Title: Replicable Online Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Online Learning

Replicability
Abstract: We investigate the concept of algorithmic replicability introduced by Impagliazzo et al.(2022) in an online setting. In our model, the input sequence received by the online learner is generated from time-varying distributions chosen by an adversary (obliviously). Our objective is to design low-regret online algorithms that, with high probability, produce the \emph{exact same sequence} of actions when run on two independently sampled input sequences generated as described above. We refer to such algorithms as adversarially replicable.

Previous works explored replicability in the online setting under inputs generated independently from a fixed distribution; we term this notion as iid-replicability. Our model generalizes to capture both adversarial and iid input sequences, as well as their mixtures, which can be modeled by setting certain distributions as point-masses.

We demonstrate adversarially replicable online learning algorithms for online linear optimization and the experts problem that achieve sub-linear regret. Additionally, we propose a general framework for converting an online learner into an adversarially replicable one within our setting, bounding the new regret in terms of the original algorithm’s regret. We also present a nearly optimal (in terms of regret) iid-replicable online algorithm for the experts problem, highlighting the distinction between the iid and adversarial notions of replicability.

Finally, we establish lower bounds on the regret (in terms of the replicability parameter and time) that any replicable online algorithm must incur.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies algorithmic replicability in the online setting. In this setting, an oblivious adversary picks a sequence of distributions over loss vectors before the game begins. The goal is to design low-regret online algorithms that, with high probability, output the exact same sequence of actions on two independently sampled sequences of loss vectors. The authors consider two online learning settings —online linear optimization and learning with expert advice —and design replicable online algorithms that achieve sublinear regret. In the special case where the adversary chooses the same loss distributions for all rounds, the authors design a replicable online learning algorithm with near-optimal regret guarantees. More generally, the authors present a generic conversion of a non-replicable online algorithm into a replicable one. Finally, the authors complement their upper bounds with lower bounds on regret that any replicable algorithm needs to suffer.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- The paper studies an important problem in machine learning and presents, to the best of my knowledge, the first formal framework for studying replicability for online learning with full-information feedback. \n- The authors provide several interesting results across various online learning settings, including both lower and upper bounds\n\nWeaknesses: My main concern with this paper is its presentation, which is the main reason behind my rating. Below, I list my concerns. \n1. There are several places where notation is not well-defined and hinders readability:\n  - In many places (Line 248, 274, ...), vectors are being added to spaces. This operation should be defined.\n  - In Line 272, $Z^n$ is not defined.\n  - In Line 274, $c_{1:t-1}$ is not defined.\n  - In Theorem 3.1, $D$ is not defined. \n  - Figure 1 is a bit unclear, it would be good if the text c_{1:t-1} was colored in red, and c'_{1:t-1} is colored blue\n  - In Algorithm 2 and Line 287, \"Geo\" and \"geometric noise\" are not defined. What does geometric noise mean? What is the distribution of this noise? This should be stated precisely in the main text.\n  - In Line 316, I think $\\Omega$ should be $O$\n  - In Line 312, $D$ is not defined. \n  - In Line 326, $K$ is not defined. Also, what am I supposed to take away from this sentence? There is always an upper bound on the regret... \n\n2. The paper currently reads like a list of results without providing much intuition. No intuition/sketches behind the proofs is provided for any of the main Theorems, as the authors repeatedly \"defer the proof to the Appendix.\" The most egregious offense is Section 4, which contains just the algorithm and the theorem statement. The authors should expand on this section and provide insight on why geometric noise is needed and why the results for online learning optimization don't just apply to the expert setting (see Question 3 in \"Additional Questions\"). Section 5 is also not illustrative as Algorithm 5 is not in the main text. The authors should move Algorithm 5 into the main text and provide a proof sketch. Overall, more exposition is needed to ensure that the main text does not just read like a list of results, and hence, I believe that this work requires a rewrite. \n\n3. A minor concern is the significant gap between the lower and upper bounds on the regret for replicable online linear optimization. It would be good if the authors could comment on whether a tighter analysis of the current approach can lead to a better upper bound than the one in Theorem 3.1, and it not, what the barrier is. \n\n4. Another minor concern is regarding novelty. It seems like the techniques used by this paper are not new: rounding has been used by existing work on replicability, and \"blocking\" (more generally, laziness) has also been studied in online learning literature. It would be great if the authors could comment on what makes this work not just a combination of these two techniques.\n\nQUESTIONS:\nIt would be great if the authors could address all my concerns in Weaknesses 1-4. \n\nAdditional questions:\n1. The technique of batching (and more generally, lazy algorithms) has been studied extensively in online learning (see [1]). What implications do these results have on replicable online learning, if any?\n\n2. Lazy online algorithms have been especially useful for designing differentially private online learning algorithms (see [2]). How does your notion of replicable online learning relate to differentially private online learning? Does one imply the other? \n\n3. Learning with expert advice is a special case of online learning optimization when the decision space is taken to be the n-dimensional simplex. Can your regret bound for replicable online linear optimization be converted into a regret bound for learning with expert advice? That is, do you really need to have a separate algorithm for the problem of replicable learning with expert advice?\n\n[1] Arora, Raman, Ofer Dekel, and Ambuj Tewari. \"Online bandit learning against an adaptive adversary: from regret to policy regret.\" arXiv preprint arXiv:1206.6400 (2012).\n\n[2] Asi, Hilal, et al. \"Private online learning via lazy algorithms.\" Advances in Neural Information Processing Systems 37 (2024): 112158-112183.\n\nLIMITATIONS:\nThere is a significant gap between the regret upper and lower bounds. It would be great if the authors could provide some intuition on how this gap can be reduced.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the recently-introduced notion of replicability in online learning setups. An adversary decides on a sequence of $T$ (possibly different) distributions, from which the instances are drawn independently. The online learning algorithm is called *replicable* if, fixing the randomness in the algorithm, the outputs match with high probability on two sequences of instances sampled independently. The goal is to design low-regret and replicable algorithms for various online learning settings.\n\nIn comparison, prior work has studied replicable online learning in more restricted settings and/or with weaker guarantees:\n- Some focused on the special case that all $T$ distributions are identical (termed iid-replicability in this work).\n- Some only requires the algorithm to be low-regret on observations drawn from the distributions (rather than on all sequences).\n\nThe main results of this work are summarized below, where polynomial dependence on other parameters (including the replicability parameter) are suppressed:\n- Theorem 3.1: A $T^{5/6}n^{1/6}$ regret upper bound for online linear optimization in $n$ dimensions.\n- Theorem 4.1: A $T^{5/6}\\log^{5/6} n$ regret upper bound for the experts problem with $n$ experts.\n- Theorem 5.1: A general transformation that translates a low-regret (but not necessarily repliable) algorithm into a replicable one.\n- Theorem 6.1: A $\\sqrt{T}\\log(n) / \\rho$ regret upper bound (modulo a $\\mathrm{polylog}(\\log T, 1 / \\rho)$ factor) for the experts problem in the iid replicable setting.\n- Theorems 7.1 and 7.2: Regret lower bounds against replicable online learners in the experts problem.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The formulation of replicable online learning is natural and clean from a theoretical perspective. It is stronger (and arguably more natural) than those in prior work. Yet, a sublinear regret bound can still be guaranteed for several fundamental online learning settings.\n- The writing is generally clear and easy to follow. Especially, the authors did a good job in comparing their model to prior ones and highlighting the differences.\n\nWeaknesses:\n- I was a bit confused about the statement of Theorem 1.3 (and Theorem 5.1): What families of online learning problems does this result apply to? As stated, there isn't any restriction other than that each instance $c_t$ is a $n$-dimensional vector.\n- However, as I followed the proof of Theorem 5.1 in Appendix F, it appeared to me that more assumptions are needed. For example, in the step from Equation (13) to (14), isn't it assumed that the loss function $\\mathsf{Cost}(\\cdot, \\cdot)$ is linear in the second argument?\n- I didn't follow the step from (14) to (15) either: even assuming the aforementioned linearity, $g_{uB}$ and $g_{(u-1)B}$ are $\\sum_{t=1}^{uB}c_t$ and $\\sum_{t=1}^{(u-1)B}c_t$ rounded to two different grids---why is the difference between them equal to $c_{(u-1)B:uB}$?\n- Except for the iid-replicable experts setting, the upper bounds do not come with lower bounds with matching dependences on $T$.\n- From a more practical perspective, I didn't find that the notion of replicability makes as much sense in the online setting as in the batch/offline/supervised setting. In supervised learning, it makes sense to collect fresh data and replicate the learning. In the online setting, why is it reasonable to assume that one can \"repeat the history\" and observe a sequence of identically distributed data points?\n\nOther comments / writing suggestions:\n- Line 63: There is an extra space before the period.\n- Line 81: The term \"adversarially $\\rho$-replicable\" was a bit misleading to me: compared to being \"iid $\\rho$-replicable\", the only change seems to be that the observations can be differently distributed across timesteps. In other words, they are still independently sampled from distributions, and I didn't quite see what is \"adversarial\" in this formulation.\n- Appendix F, Line 27 of Algorithm 5: Should the condition be \"$t - 1$ is a multiple of $B$ and $t > 1$\"? This is because we shouldn't call the internal algorithm at time $t = 1$.\n\nOverall, while I liked the formulation of the problem and the suite of results that the authors obtained, I have strong concerns over the validity (or at least the scope of applicability) of one of the main results. Therefore, I don't think this submission should be accepted without adequately addressing this point.\n\nQUESTIONS:\nRepeating the \"Weaknesses\" section above:\n- In the online setting, why is it reasonable to assume that one can \"repeat the history\" and observe a sequence of identically distributed data points?\n- For the proof of Theorem 5.1, could you clarify whether the proof relies on some unstated assumptions on the loss, as well as why the step from (14) to (15) is valid?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n\\paragraph{Summary} The paper studies online learning problems, like learning with experts and online linear regression under the requirement of replicability. Loosely speaking, a randomized  algorithm on input sampled from some distribution, is replicable if for two independent set of samples, it is very likely over the internal randomness of the algorithm (probability $\\geq \\rho$ for some replicability parameter $\\rho$) that the algorithm returns the same output on the two sets of samples. The notion was introduced motivated by the replicability crisis in science, but the property is interesting as a stability-notion in its own right, and has connections to concepts like differential privacy and generalization.\n\nThe main results of the paper are replicable algorithms for online learning with experts and online linear optimization (plus a more general framework for online learning problems) with sublinear regret. The input in each round $i$ is assumed to be sampled from some distribution $\\mathcal{D}_i$ chosen by an adversary in an oblivious fashion which significantly generalizes a framework by Esfandiari et al. [2022] where the input is assumed to be iid. The paper also proves almost matching lower bounds for the iid setting through a reduction from replicable coin testing.\n\nThe main algorithmic idea is to partition the sequence of updates into blocks (for the  iid setting the blocks have increasing lengths) where the algorithm within each block plays the same action. Long blocks of course incurs worse regret bounds but they are good for replicability where we only have to worry about the ends of blocks. In order to enforce replicability by the end of the block, their algorithm rounds the regularized cost vector to a random grid point and uses the minimum of that grid points as the action for the next block. In order to not make the grid too big, this requires some concentration bounds for the total cost incurred during a block. The final algorithm chooses the block size appropriately to ensure replicability and low regret.\n\nSTRENGTHS AND WEAKNESSES:\n\\textbf{Strengths}: \n(1) I found the problem well-motivated and their model very interesting and natural opening up for several new directions for future work.\n(2) The technique of partitioning into blocks and using random grid shifting after the blocks to enforce both replicability and low regret is very nice. \n(3) It is great that the paper provides a near matching lower bound in the iid setting\n(4) The paper is well-written, and I had little problems following the proofs. I read the first 9 pages and the proof of theorem 3.1 in the appendix, but stopped after that.\n(5) The paper states many interesting problems for future work.\n\n\\textbf{Weaknesses} (1) The paper might benefit by a discussion of how the techniques differ from past work, e.g., Esfandiari et al. [2022]\n\nQUESTIONS:\nl74: It would be nice with a discussion of whether obliviousness is necessary. I had a vague intuition that since the algorithms are replicable, an adaptive adversary wouldn't be much harder to handle since over the two independent runs, replicability ensures that the outputs are likely the same and then it seems like the adversary adaptively chosing the distribution shouldn't help much. But it's very possible I'm missing something.\n\nl117-118: I'm confused. Doesn't your algorithms have no switching within the blocks?\n\nl209-214: Don't you need some assumption on the norm of the cost vectors? Is it related to the $D$ on l279 (which I think hasn't been defined).\n\nExample 3.1. I had a hard time following the writing in this example and I don't know the precise meaning of \"gives a bonus\" or \"pretends that $a_2$ has better performance initially\". The final sentence could also be phrased stronger I think. \n\nl784: In the def of $a_t$ there is a missing $+$.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies replicable learning for online learning optimization and the experts setting under the online setting. Previous works studied a similar problem in the online setting with data being generated in an iid fashion from a single distribution whereas this work generalizes the setting to allow data points at different timesteps to be sampled from different distributions (i.e. adversarially chosen). The primary objective of this work is to produce algorithms that fit certain replicability properties along with achieving low regret in the online setting.\n\nThe authors explain that an online learning algorithm is adversarially $\\rho$-replicable to indicate that with probability $1 - \\rho$, and for two sequences $S$ and $S'$ generated iid from an adversarially chosen sequence of distributions, the algorithm generates the same set of actions at all timesteps for both sequences (fixing the algorithm's internal randomness).\n\nThe paper introduces many results for online replicable learning with the property of adversarially $\\rho$-replicability being satisfied. In both online linear optimization and the experts problem, the authors construct two algorithms, one for each setting, that satisfy adversarial replicability along with achieving sublinear regret. More specifically, in online linear optimization, the authors develop an algorithm called the FLLB which is the FLL algorithm executed on the time horizon divided into blocks where the updates happen at the end of each block. Similarly, in the experts problem, the authors take the FTPL and apply the same blocking procedure and use geometric noise instead of the standard exponential noise used by Kalai and Vempala in their FTPL algorithm. They also study regret lower bounds and show a lower bound of $\\Omega(\\frac{\\sqrt{T}}{\\rho})$ and extend their results for iid replicability as well. They additionally introduce a general framework to convert any online regret minimization algorithm to one that has a slightly worse regret guarantee but with adversarially $\\rho$-replicability satisfied.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper tackles a unique setting of a more generalized version of replicability under the online setting, namely adversarial $\\rho$-replicability.\n- The paper provides a pretty comprehensive treatment of this setting with many different results across online linear optimization, the experts problem, an online-to-replicable algorithm reduction, and derivation of lower bounds under each scenario.\n - They also show a nearly optimal iid replicable algorithm in the expert setting with a bound that matches the lower bound of Impagliazzo et al. (2022) up to logarithmic factors.\n- The paper is also quite well-written. The authors introduce the setting in an effective manner using tangible examples, give easy-to-follow overviews of their algorithmic techniques, and do a good job of explaining technical details.\n\nWeaknesses:\n- Not necessarily a major weakness but there exists a rather noticeable gap between the upper and lower regret bounds, $T^{5/6}$ vs $\\sqrt{T}$, under adversarially replicable algorithms. A small discussion, if available, discussing the challenges in closing the gap or why the gap exists would be interesting.\n\nQUESTIONS:\n- Regarding the lower bounds for adversarially replicable algorithms, could it be the case that the cost of replicability must exceed the classical $\\sqrt{T}$ barrier as seen in traditional online learning? Or is it possible that one could further refine the upper bounds to make them sharper?\n- While the authors show that one can preemptively select the block length $B$ beforehand to get the optimal guarantees, is there an adaptive, data-dependent way to choose the block length that would still preserve replicability?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}