Replicable Bandits
Authors: Hossein Esfandiari
Alkis Kalavasis
Amin Karbasi
Andreas Krause
Vahab Mirrokni
Grigoris Velegkas
Venue: ICLR 2023 poster (2023)
Keywords: Interactive Learning
Reproducible Learning
Bandit Algorithms

Abstract: In this paper, we introduce the notion of replicable policies in the context of stochastic bandits, one of the canonical problems in interactive learning. A policy in the bandit environment is called replicable if it pulls, with high probability, the exact same sequence of arms in two different and independent executions (i.e., under independent reward realizations). We show that not only do replicable policies exist, but also they achieve almost the same optimal (non-replicable) regret bounds in terms of the time horizon. More specifically, in the stochastic multi-armed bandits setting, we develop a policy with an optimal problem-dependent regret bound whose dependence on the replicability parameter is also optimal. Similarly, for stochastic linear bandits (with finitely and infinitely many arms) we develop replicable policies that achieve the best-known problem-independent regret bounds with an optimal dependency on the replicability parameter. Our results show that even though randomization is crucial for the exploration-exploitation trade-off, an optimal balance can still be achieved while pulling the exact same arms in two different rounds of executions. 

Published as a conference paper at ICLR 2023
REPLICABLE BANDITS
Hossein Esfandiari
Alkis Kalavasis
Google Research
National Technical University of Athens
es ar assFandaar@@gogge..ccoo @google.com
kalavasisal kis@maall ntua. .gr
Amin Karbasi
Andreas Krause ETH Zurich krauseaßethz.co ch
Vahab Mirrokni
Yale University, Google Research amin. karbasi@yale. edu
Google Research mir mirzokni@gooogee.oo
Grigoris Velegkas
Yale University grigoris wel..rsseeeeeeeeylll...e...
ABSTRACT
In this paper, we introduce the notion of replicable policies in the context of stochastic bandits, one of the canonical problems in interactive learning. A policy in the bandit environment is called replicable if it pulls, with high probability, the exact same sequence of arms in two different and independent executions (i.e., un- der independent reward realizations). We show that not only do replicable policies exist, but also they achieve almost the same optimal (non-replicable) regret bounds in terms of the time horizon. More specifically, in the stochastic multi-armed ban- dits setting, we develop a policy with an optimal problem-dependent regret bound whose dependence stochastic linear bandits (with finitely and infinitely many arms) we develop repli- cable policies that achieve the best-known coblem-indeeennent regret bounds with an optimal dependency on the replicability parameter. Our results show that even though randomization is crucial for the apploration-exploiaaiiii trade-off, an optimal balance can still be achieved while pulling the exact same arms in two different rounds of executions.
the replicability parameter is also optimal. Similarly, for
INTRODUCTION
In order for scientific findings to be valid and reliable, the experimental process must be repeat- able, and must provide coherent results and conclusions across these repetitions. In fact, lack of reproducibility has been a major issue in many scientific areas; a 2016 survey that appeared in Na- ture (Baker) (2166a) revealed that more than 70% of researchers failed in their attempt reproduce another researcher's experiments. What is even more concerning is that over 50% of them failed reproduce their own findings. Similar concerns have been raised by the machine learning com- munity, e.g., the ICLR 2019 Reproducibility Challenge (Pincau et al., 2019) and NeurIPS 2019 Reproducibility Program (Pineau al. 2021). due to the the exponential increase in the number of publications and the reliability of the findings.
The aforementioned empirical evidence has recently led to theoretical studies and rigorous defini- tions of replicability. In particular, the works of [Imaaliazzo et al. (2022) and Ahn et al. (2022) considered replicability as an algorithmic property through the lens of (offfine) learning and convex optimization, respectively. In similar vein, in the current work, introduce the notion of replica- bility in the context of interactive learning and decision making. In particular, we study replicable policy design for the fundamental setting of stochastie bandits.
aamti--rmmed bandit (MAB) a one one-player game that is played over T rounds where there is a set of different arms/actions A of size [A] K (in the more general case of linear bandits, can consider even an infinite number of arms). In each round 1, 2...... ,...,, T, the player pulls an arm a A and receives corresponding reward rı- In the stochastic setting, the rewards of each

---

Published as a conference paper at ICLR 2023
arm sampled in each round independently, from some fixed but unknown, distribution supported on [0, 1]. Crucially, each arm has potentially different reward distribution, but the distribution of each arm is fixed over time. A bandit algorithm & every round takes input the sequence of arm-reward pairs that it has seen so far, i.e., (#1, ........ ( 1t- then uses (potentially) some internal randomness ç to pull arm dr E A and, finally, observes the associated reward r Day-
replicable bandit algorithm, which is inspired by the definition of Impagliazzo al.) (2022). Intuitively, bandit algorithm is replicable if distinct executions of the algorithm, with internal randomness fixed between both runs, but with independent reward realizations, give the exact same sequence of played arms, with high probability. More formally, we have the following definition.
We propose the following natural notion
Definition 1 (Replicable Bandit Algorithm). Let p E [0. We call a bandit algorithm A p- replicable in the stochastic setting if for any distribution Da; over [0. 1] of the rewards of the j-th arm 0j E A, and for any two executions of A, where the internal randomness & is shared across the executions, holds that
[aaf'........
(2) (
Pr (r(),,())))
.......))
( Here, a, A(a 11 ... (-i is the t-th action taken by the algorithm A in execution i E {1,2}.
(1)
The reason why we allow for some fixed internal randomness is that the algorithm designer has control over it, e.g., they can use the same seed for their (pseudo)random generator between two executions. Clearly, naively designing replicable bandit algorithm not quite challenging. For instance, algorithm that always pulls the same arm particular random sequence determined by the shared random seed are both replicable. The caveat is that the performance of these algorithms in terms of expected regret will be quite poor. In this work, we aim to design bandit algorithms which are replicable and enjoy small expected regret. In the stochastic setting, the (expected) regret after 7 rounds is defined as
algorithm that plays the arms in
E [(RT]
max Pa aEA
E
Σ
Ma
where Pa EY,DD, [r] is the reward for arm A. In similar manner, define the regret in the more general setting of linear bandits (see. Seccton 5) Hence, the overarching question in this work the following:
Is it possible to design replicable bandit algorithms with small expected regret?
At a first glance, one might think that this is not possible, since it looks like replicability contradicts the exploratory behavior that bandit algorithm should possess. However, our main results answer this question in the affirmative and can be summarized in Table
Summary of Results
Setting
Algorit hhmm
Regret K² (T)
Theor em
Stochastic MAB
Algorithm
Theorem 3
Stochastic MAB
Algorithm
R² HH
Theorem 4
Stochastic Linear Bandits
Algorithm
K² Ver
Theorem 6
Stochastic Linear Bandits Infinite Action Space
Algorithm 4
polyid) vT
Theorem 10
p²
Our results for replicable stochastic general multi-armed and linear bandits. In the expected regret column, (-) subsumes logarithmic factors. HA is equal to Σj 1/Aj. A, is the differ- ence between the mean of action and the optimal action, K is the number of arms, d is the ambient dimension in the linear bandit setting.
Table 1:
2

---

Published as a conference paper at ICLR 2023
1.1 RELATED WORK
In this work, we introduce the notion of replicability in the con- text of interactive learning and, in particular. in the fundamental setting of stochastic bandits. Close to our work, the notion of replicable algorithm in the context of learning was proposed by Impagli-] azzo et { (202)), where it is shown how any statistical query algorithm can be made replicable with moderate increase in its sample complexity. Using this result, they provide replicable algorithms for finding approximate heavy-hitters, medians, and the learning of half-spaces. Reproducibility has been also considered in the context of optimization by [hn et al. (2022). We mention that in /Ahn et al. (2022) the notion of a replicable algorithm different from our work and that of Impaglazzo et al. 2022), in the sense that the outputs of two different executions of the algorithm do not need to be exactly the same. From a more application-oriented perspective, |Shamir & Lin (2022) study irreproducibility in recommendation systems and propose the use of smooth activations (instead of ReLUs) to improve recommendation reproducibility. In general, the reproducibility crisis is re- ported in various scientific disciplines Joannidis (2005): McNutt] (2014); [Baker (2016b); Goodman et al. (2016); Lucic et al. (2018) : Henderson et al. (2018). For more details we refer to the report of the NeurIPS 2019 Reproducibility Program Pineau et al. (02T) and the ICLR 2019 Reproducibility
Reproduccbliity/Rppiiabbb
Challenge Pineau et al. (2019).
Bandit Algorithms. Stochastic multi armed bandits for the general setting without structure have
been studied extensively Slivkins/ (2019); Lattimore & Szepesvári (2020) Bubeck et al. (201 2b);
Auer et al. (2002): ; Cesa-Bianchi & Fischer 1998); Kaufmann et al. (2012a) Audibert et al. (2010);
Agrawal & Goyal 2 Kaufmann et al. (20126). In this setting, the optimum regret achievable is ): this is achieved, e.g., by the upper confidence bound (UCB) algorithm of
0 [log(
A
Auer et al. (2002) The setting of dddimensional linear stochastic bandits is also well-explored Dani
et al. (2008) y Abbasi- Yadkori al. (201T) under the well-specified linear reward model, achiev-
ing (near) optimal problem-independent regret of O(dy T (olo)(())) Lattimore & Szepesvári 2020))
Note that the best-known lower bound is ((dv/T) Dani et al. (2008) and that the number of arms can, in principle. be unbounded. For finite number of arms K, the best known upper bound is Bubeck et al. (2012a). Our work focuses on the design of replicable bandit algo- rithms and we hence consider only stochastic environments. In general, there is also extensive work in adversarial bandits and we refer the interested reader to Lattimore & Szepesvári (2020).
O(vdT dT log(K)) (VVTlog(K))
Batched Bandits. While sequential bandit problems have been studied for almost century, there is much interest in the batched setting too. In many settings, like medical trials, one has to take lot of actions in parallel and observe their rewards later. The works of Auer & Ortner (2010) and Cesa- Bianchi et al. (2013) provided sequential bandit algorithms which can casily work in the batched setting. The works of Gao al. (2019) and Esfandiari et al.) (202T) are focusing exclusively on the batched setting. Our work on replicable bandits builds upon some of the techniques from these two lines of work.
2 STOCHASTIC BANDITS AND REPLICABILITY
In this section, we first highlight the main challenges in order to guarantee replicability and then discuss how the results of Impagliazzo et al. (022) can be applied in our setting.
2.1 WARM-UP I: NAIVE REPLICABILITY AND CHALLENGES
Let us consider the stochastic two-arm setting (K = 2) and bandit algorithm A with two in- dependent exccutions, As and Az. The algorithm As plays the sequence 1,2, until some, potentially random, round T; N after which one of the two arms is eliminated and, from that point, the algorithm picks the winning arm j: E 21, The algorithm A is p-replicable and only j2 with probability p.
if T T2 and
Assume that |يا - μ2م = A where μ4 is the mean of the distribution of the i-th arm. we assume that A is known, then we can run the algorithm for Ti = T2 = E. log(1 1/p) for some universal
constant C > 0 and obtain that, with probability P. it will hold that 72(3) in and 7(6)) p1
3

---

Published as a conference paper at ICLR 2023
j E [1, where n is the estimation of arm's i mean during execution j. Hence, knowing A implies that the stopping criterion of the algorithm A is deterministic and that, with high probability, the winning arm will be detected at time Ti T2. This will make the algorithm p-replicable.
Observe that when K = 2, the only obstacle to replicability is that the algorithm should decide at the same time select the winning arm and the selection must be the same in the two execution threads. In the presence of multiple arms, there exists the additional constraint that the above conditions must be satisfied during, potentially, multiple arm eliminations. Hence, the two questions arising from the above discussion are (i) how to modify the above approach when A is unknown and (ii) how to deal with K > arms.
A potential solution to the second question (on handling K > 2 arms) is the Execute-Then--Cmeee (ETC) strategy. Consider the stochastic K-arm bandit setting. For any p E (0, 1), the ETC algorithm with known A min; A; and horizon T' that uses phases before commitment is p-replicable. The intuition is exactly the same as in the K 2 case. The caveats of this approach are that it assumes that A is known and that the obtained regret is quite unsatisfying. In particular, it achieves regret bounded by m ExE[K] A + P (T mK) ExE(M] A,
I7 log(1/p) deterministic exploration
Next, we discuss how to improve the regret bound without knowing the gaps A, Before designing new algorithms, we will inspect the guarantees that can be obtained by combining ideas from previ- ous results in the bandits literature and the recent work in replicable learning of Impagliazzo et al. (022).
2.2 WARM-UP II: BANDIT ALGORITIMMS AND REPLICABLE MEAN ESTIMATION
First, we remark that we work in the stochastic setting and the distributions of the rewards of the two arms are subgaussian. Thus, the problem of estimating their mean is an instance of a statistical query for which we can use the algorithm of Impagliazzo et al. (0222 to get replicable mean estimator for the distributions of the rewards of the arms.
Proposition 2 (Replicable Mean Estimation (Impagliazzo al. 2022)). Let T, T, p E [0, 1]. There log(1/5) (p-0)-)2 8)2
samples from distribution with mean and computes an estimate ji that satisfies j pa| S with probability at least 8.
exists a p-replicable algorithm ReprMeanEstimatic on that draws 52
Notice that we are working in the regime where 5 < p. so the sample complexity is Ω [log(1/8) p2pp
The straightforward approach is to try to use an optimal multi-armed algorithm for the stochastic setting, such UCB or arm-elimination (Even-Dar et al. 2006), combined with the replicable mean estimator. However, is not hard to see that this approach does not give meaningful results: if we want to achieve replicability P we need to call the replicable mean estimator routine with parameter P/(T)), due to the union bound that we need to take. This means that we need to pull every arm at least 22 times, so the regret guarantee becomes vacuous. This gives us the first key insight to lackle the problem: we need to reduce the number of calls to the mean estimator. Hence, we will draw inspiration from the line of work in stochastic batched bandits (Gao al. 2019 Esfandiari et al. 2021) to derive replicable bandit algorithms.
3 REPLICABLE MEAN ESTIMATION FOR BATCHED BANDITS
As a first step, we would like to show how one could combine the existing replicable algorithms of Impagliazzo al. (2022) with the batched bandits approach Estandiari et al.) (202)) get some preliminary non-trivia invial results. We build an algorithm for the K -arm setting, where the gaps Aj are unknown to the learner. Let 5 be the confidence parameter of the arm elimination algorithm and p be the replicability guarantee we want to achieve. Our approach is the following: let us, deterministically, split the time interval into sub-intervals of increasing length. We treat each sub- interval as a batch of samples where we pull each active arm the same number of times and use the replicable mean estimation algorithm to, empirically, compute the true mean. At the end of each batch, we decide to eliminate some arm j using the standard UCB estimate. Crucially, if we condition on the event that all the calls to the replicable mean estimator return the same number, then the algorithm we propose is replicable.
4

---

Published as a conference paper at ICLR 2023
Algorithm Mean-Estimation Based Replicable Algorithm for Stochastic MAB (Theorem 3)
Input: time horizon T, number of arms K, replicability 2: Initialization: B + log(()), t T1/B co + 0, 0, [
r T, Me + va A
3: for 1
do
4: if [ [ then
5:
break
6:
4= G-1 + L4']
7:
Pull every arm a E A for [q'] times
8:
for a do
9:
Tha f ReprMeenEstiimationn = 1/(2KBB), = 1, Vogg / (K)B))
Proposition
10:
- r A 19'1
11:
for E A do
12:
if P.g < maX Pa 2T then Remove from A
13:
14:
In the last batch play the arm from A with the smallest index
Theorem 3. Ler E N. E (0, 1). There exists a p-replicable algorithm (presented in Algorithm ly for the stochastic bandit problem with K arms and gaps (Aj)jE(K] whose expected regret is
K2 lol (T)
log KT log(T) Σ A,,
[(,++
E[RT] SC-
A,
where C > O is an absolute numerical constant, and its running time is polynomial in K, T and 1/p.
The above result, whose proof can be found in Appendix A states that, by combining the tools from Impagliazzo et al.] (2022) and Esfandiari et al. (202) we can design replicable bandit algorithm with sestaacc-dependeen
expected regret O(K" log (T)/p²). Notice that the regret guarantee has (T) (T) factor compared to its non-replicable counterpart in Esfandiari et al. (2021) (Theorem 5.1). This because, due to union bound over the rounds and the arms, need to call the replicable mean estimator with parameter p/ p/(K log(?))) In the next section, we show how to get rid of the log² (T) by designing a new algorithm.
an extra K2 log
4 IMPROVED ALGORITHMS FOR REPLICABLE STOCHASTIC BANDITS
While the previous result provides a non-trivial regret bound, is not optimal with respect to the time horizon T. In this section, show how to improve by designing Algorithm 2, which satisfies the guarantees Theorem 4jand, essentially, decreases the dependence on the time horizon T from log (T) to lT). Our main result for replicable stochastic multi-armed bandits with K arms follows.
algorithm, presented in
Theorem 4. Let T E N, E (0, 1]. There exists p-replicable algorithm (presented in Algorithm 2 for the stochastic bandit problem with arms and gaps (Aj) whose expected regret is
E R <
C
K2
Σ
(A,
log( (gg(K) KTlogg(7 (T)(
A
J:A,.0
where is an absolute numerical constant, and its running time is polynomial in K, T and 1/p.
Note that, compared to the non-replicable setting, we incur an extra factor K2/ 2 in the regret.
The proof can be found in Appendix B Let us now describe how Algorithm workks We decompose the time horizon into B log(T") batches. Without the replicability constraint, one could draw q' samples in batch from each arm and estimate the mean reward. With the replicability constraint, we have to boost this: in each batch i, we pull each active arm O(3q') times, for some q to be determined, where OKK2 /p2) is the replicability blow-up. Using these samples, compute
((K2/p2
5

---

Published as a conference paper at ICLR 2023
Algorithm 2 Replicable Algorithm for Stochastic Multi-Armed Bandits (Theorem 4)
Input: time horize zon T, number of arms K, replicability p 2: Initialization: B + log(T"), + T1/B, , co + 0, Ao + [K], + T, Po + 0, Va E
AA
3: ++ max (K22/////30000
4: for to 2-11
5: if [ [A, >
then
6:
break
7: A + AA-11
8:
for E A, do
9:
Pull arm for B[q'] times
Compute the empirical mean 7
10:
11:
G - G-1 + lg"]
12:
G - Se;
13:
U. - V2In(2KTB) c
14:
U, + /2ln(2KTB B)
15:
U. + Uni[U,/2, U.]
16:
T+ --r-B. |Al Lg!!
17:
for a A; do
18:
if ((t)
pe U4 A Pa Remove a from A,
-(1) U, then
19:
20:
In the last batch play the arm from AB-1 with the smallest index
the empirical mean (1) for any active arm a. Note that Ü, in Algorithm 2 corresponds to the size of the actual confidence interval of the estimation and U, corresponds to the confidence interval of an algorithm that does not use the 3-blow-up in the number of samples. The novelty of our approach comes from the choice the interval around the mean the maximum arm: we pick a threshold uniformly random from interval of size U:/2 around the maximum mean. Then, the algorithm
checks whether Ab + U < max ((( U. where max runs over the active arms in batch i, and eliminates arms accordingly. To prove the result show that there three regions that some arm j can be in relative to the confidence interval of the best arm in batch i (cf. [Appendix B). lies in two of these regions, then the decision of whether to keep it discard it is the same in both executions of the algorithm. However, if it is in the third region, the decision could be different between parallel executions, and since relies on some external and unknown randomness, is not clear how to reason about it. To overcome this issue, we use the random threshold to argue about the probability that the decision between two executions differs. The crucial observation that allows us get rid of the extra log" (I') factor that there are correlations between consecutive batches: prove that if some arm j lies in this "bad" region in some batch i, then it will be outside this region after a constant number of batches.
5 REPLICABLE STOCHASTIC LINEAR BANDITS
We now investigate replicability in the more general setting of stochastic linear bandits. In this setting, each arm is vector E Rd belonging to some action set A C IRd, and there is pa- rameter 8' Rd unknown the player.
round f, the player chooses some action a; E A and receives reward тt = (0*, az) + Th, where is zero-mean I-subgaussian random vari- able independent any other source of randomness. This means that E[7,] = 0 and satisfies E(exp(An)) < exp(X²/2 for any X E R. For normalization purposes, it is standard to assume that ||6*|/2 < and sup.EA ||a||2 I 1. In the linear setting, the expected regret after T pulls 01,..., .. can be written
T az)
E = T s'sup(0" (0)
E
Σ
10
aEAA

---

Published as a conference paper at ICLR 2023
In Section 5.1 we provide results for the finite action space case, ie.. when |A| K. Next, in Section 5.2] we study replicable linear bandit algorithms when dealing with infinite action spaces. work in the regime where T > d. We underline that approach leverages
In the tollowing, connections of stochastic linear bandits with G-optimal experiment design, core sets constructions, and least-squares estimators. Roughly speaking, the goal of G-optimal design is find (small) subset of arms A', which is called the core set, and define a distribution over them with the following property: for any ε > 0, pulling only these arms for an appropriate number of times and computing the least-squares estimate 0 guarantees that supae. A 0" 0) e, with probability 1- 8. For an extensive discussion, refer to Chapters 21 and 22 of Lattimore & Szepesvári 2020
5.1 FINITE ACTION SET
We first introduce a lemma that allows us to reduce the size of the action set that our algorithm has to search over.
Lemma 5 (See Chapters 21 and 22 in Lattimore & Szepesvári (202)).. For any finite action set A that spans Rdd and any 8, > 0, there exists an algorithm that, in time polynomial in d, computes multi-set set of (dllog(1))) log(1/5)/e² + log log d) actions (possibly with repetitions) such that (i) they span I and (ū) if we perform these actions a batched stochastic d-dimensional linear bandits setting with true parameter l* E Rd and let @ be the least-squares estimate for 0", then, for any E A,
with probability at least 1 we have <a, 0) S.
Essentially, the multi-set in Lemma 5] is obtained using an approximate G-optimal design algo- rithm. Thus, it is crucial to check whether this can be done in a replicable manner. Recall that the above set of distinct actions called the core set and is the solution of an (approximate) G- optimal design problem. To be more specific, consider a distribution æ A - [0, 1] and define Ta (a))aa" E IRdxd and g(T)) = sup,e. A | The distribution т is called a design and the goal of G-optimal design is to find a design that minimizes g. Since the number of actions is finite, this problem reduces to an optimization problem which be solved efficiently using standard optimization methods (e.g., the Frank- Wolfe method). Since the initialization the same, the algorithm that finds the optimal (or an approximately optimal) design replicable under the assumption that the gradients and the projections do not have numerical errors. This perspective orthogonal to the work of Ahn al. (2022). that defines reproducibility from a different viewpoint.
(xx = Ene
Algorithm Replicable Algorithm for Stochastic Linear Bandits (Theorem 6
1: Input: number of arms K, time horizon T, replicability B < log(T), + (T/c) A + [K],
2: Initialization
3: + |max[K² /p" [3004]]
4: for i 1 to - do
5:
dd /dlog(KT))/00 (KT))/(3q) &
6:
E /d ldlog( /dllog(KT)///// //²)/q"
7:
n, 10d (KT))// /E?
8:
a1 an, <- multi-set given by Lemma 5 with parameters 5 = 1/(KT²) and E = E
9:
if n, then
10:
break
Pull every arm d1,..., an. and receive rewards .1.... Tia
Compute the LSE 0, + (E=n (E arr,)
12:
13:
E + Eni -+r ni
14:
15:
for a E do
16:
if (n,0.) E max (a, 0.) E, then Remove a from A In the last batch play arg max (a, 0B-1)
17:
In our batched bandit algorithm (Algorithm 3), the multi-set of arms aj,..., .......,,, an, computed in each batch is obtained via deterministic algorithm with runtime poly(K, (K d), where [A] = K. Hence, the
7

---

Published as a conference paper at ICLR 2023
multi-set will be the same in two different executions of the algorithm. On the other hand, the LSE will not be since it depends on the stochastic rewards. We apply the techniques that we developed in the replicable stochastic MAB setting in order to design our algorithm. Our main result for replicable d-dimensional stochastic linear bandits with K arms follows. For the proof, we refer to colpppddiix C
Theorem 6. Let T p,p E (0, 11. There exists a p-replicable algorithm for the stochastic d- dimensional linear bandit problem with K arms whose expected regret
K²
E[Rr] SC p² V log(KT),
where C > O is an absolute numerical constant, and its running time is polynomial in d, K,T and 1/p.
Note that the best known non-replicable algorithm achieves an upper bound of Ö(vdI dT log( ()) and, hence, our algorithm incurs a replicability overhead of order K2, p²²p The intuition behind the proof is similar to the multi-armed bandit setting in Section 4
5.2 INFINTEE ACTION SET
Let us proceed the setting where the action set A is unbounded. Unfortunately, even when d = 1, we cannot directly get an algorithm that has satisfactory regret guarantees by discretizing the space and using Algorithm 3 The approach of Essandiari et al. (2021) is discretize the action space and use an 1 -net to cover it, i.e. set A'' C A such that for all E A there exists some a' A'' with ||a - a' a21122 < 1/T. It is known that there exists such a net of size at most (3T)² (Vershynin 2018 Corollary 4.2.13). Then, they apply the algorithm for the finite arms setting, increasing their regret guarantee by factor of Vd. However, our replicable algorithm for this setting contains an additional factor of x2 in the regret bound. Thus, even when so the bound is vacuous. One way to fix this issue and get a sublinear regret guarantee is to use smaller net. We use a 1 1TT11(11+2)_ net that has size at most (3T) ममंर and this yields an expected regret of order ((Tdd+1/(ad2)) v /dloog())/p)) For further details, we refer to Appendix
1, our regret guarantee is greater than T,
Even though the regret guarantee we managed to get using the smaller net Appenni D is sublinear in T. it is not a satisfactory bound. The next step is to provide an algorithm for the infinite action setting using replicable LSE subroutine combined with the batching approach of Esfandiari et al. 2021) We will make use of the next lemma.
Lemma 7 (Section 21.2 Note 3 of Lattimore & Szepesvári (0220). There exists deterministic algorithm that, given an action space A C IK. computes 2-approximate G-optimal design n with core set of size (d log log(d)).
We additionally prove the next useful lemma, which, essentially, states that we can assume without loss of generality that every arm in the support of has mass at least ((1/(dlg(dd)) We refer to Appendix F. F. the proof.
Lemma 8 (Effective Support). Let π be the distribution that corresponds to the 2-approximate optimal G-design of Lemma with input A. Assume that π (a) S (og(dd))) where O is some absolute numerical constant, for some arm a in the core set. Then, we can construct a distribution π such that, for any arm in the core set, 7(a)= 2 Colgd))) where C 0 is an absolute constant, so that it holds
sup I|a' 12 S 4d. d'EA
The upcoming lemma is a replicable algorithm for the least- least-squares estimator and, essentially, builds upon Lemma and Lemma 8 Its proof can be found at Appendix F.2
Lemma (Replicable LSE). Let p. E E (0, 1] and < S min 1/d of d-dimensional stochastic linear bandits with infinite action space A.
Consider an environment
Assume that π is 4-
approximate optimal design with associated core set as computed by
Lemma with input A.
There exists p-replicable algorithm that pulls each arm E C a total of d' log(d/5) 5) log" log(d) log log log(d)
Ω
22p²
We can handle the case of < 8 by paying an extra log d factor in the sample complexity.
8

---

Published as a conference paper at ICLR 2023
times and outputs 83Q that satisfies supaeA (a, 0sq 8M) - with probability least 1 - 6.
Algorithm 4 Replicable LSE Algorithm for Stochastic Infinite Action Set (Theorem 10)
1: Input: time horizon T, action set A RR, replicability
2: A' f 1/T of
3: unitialization
7, log(T). (T/6)
4: for ior B 1 do
5: q' denotes the number of pulls of all arms before the replicability blow-up
6: E; = c. -cc./llo((T//
7:
The blow-up is M; = of dd log(d) log" log(d) log log log(d) (T)
8:
a1..... at,, +- core set C, of the design given by Lemma 7 with parameter AA'
9:
if [M]] then
10:
break
Pull every arm aj for N; [M,]/[C rounds and receive rewards r ( for j E [[C:]]
S. =(aa,r!!)) [N [[C|]]]
13:
0. - Reppicableeseeeeee.ee..
p = ( min 13)
14:
r r [M,]
15:
for a E A' do
if (a, O.) < max.a (a, 0t) 2e; then Remove a from A'
17:
In the last batch play arg M (a, la,0B. B-1))
19:
Repl Deplicabletss(S,pp
21: for C do
22: v(a) + Reeelecalelelfeeeeeee : (,
Impagliazzo et al. (2022)
23:
return EEes a-a)))- ( a na u(a))
The main result for the infinite actions' case, obtained by Algorithm 4 follows. Its proof can be found at Atppeddix H Theorem 10. Let T N. (0, There exists p-replicable algorithm (Algorithm 4) for the stochastic d-dimensional linear bandit problem with infinite action set whose expected regret is
d' log(d) log* log(d) log log log(d) VTlog 3/2 (T2)
E Rrl & C.
p
where C is absolute numerical constant, and its running time is polynomial in T and 1/p.
Our algorithm for the infinite arm linear bandit case enjoys an expected regret of order (poly(d)vTT), We underline that the dependence of the regret on the time horizon (almost) optimal, and we incur extra d factor in the regret guarantee compared to the non-replicable algorithm of Estandiari et al. (2021). We now comment on the time complexity of our algorithm.
Remark 11. The current implementation of our algorithm requires time exponential in d. However, for a general convex set A, given access to separation oracle for and an oracle that computes an (approximate) G-optimal design, we can execute it in polynomial time and with polynomially many calls to the oracle. Notably, when is a polytope such oracles exist. We underline that computational complexity issues also arise in the traditional setting of linear bandits with an infinite number of arms and the computational overhead that the replicability requirement adds is minimal. For further details, refer to Appendix G
CONCLUSION AND FUTURE DIRECTIONS
In this paper, we have provided formal notion of reprodubbbiityyrepllabbttt for stochastic ban- dits and we have developed algorithms for the multi-armed bandit and the linear bandit settings that satisfy this notion and enjoy a small regret decay compared to their non-replicable counterparts. We hope and believe that paper will inspire future works in replicable algorithms for more com- plicated interactive learning settings such as reinforcement learning. We also provide experimental evaluation in Appendix H.

---

Published as a conference paper at ICLR 2023
7 ACKNOWLEDGEMENTS
Alkis Kalavasis was supported by the Hellenic Foundation for Research and Innovation (H.F.R..)) under the "First Call for H.F.R.I. Research Projects support Faculty members and Researchers and the procurement of high-cost research equipment grant", project BALSAM, HFRIFM11-1424. Amin Karbasi acknowledges funding direct support of this work from NSF (ISS-1845032,, ONR (N00014- 19-1-2406), and the AI Institute for Learning-Enable Andreas Krause was supported by the European Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation program grant agreement no. 8 815943 and the Swiss National Science Foundation under NCCR Automation, grant agreement 5INF40 180545, Grig- oris Velegkas was supported by NSF (IIS- Onassis Foundation PhD) Fellowship and Bodossaki Foundation PhD Fellowship.
Optimization at Scale (TILOS).
REFERENCES
Yasin Abbasi- Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011. 1.
Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi- bandit prob- lem. In Conference on learning theory, pp. 39-1. JMLR Workshop and Conference Proceedings, 2012. 3
Kwa Kwangjun Ahn, Prateek Jain, Ziwei Ji, Satyen Kale, Praneeth Netrapalli, and Gil Shamir. Repro- ducibility in optimization: Theoretical framework and limits. arXiv preprint arXiv:2202.04598, 2022.
Jean- Yves Audibert, Sébastien Bubeck, and Rémi Munos. Best identification in multi-armed bandits. In COLT, pp. 41-53. Citeseer, 2010.
Peter Auer and Ronald Ortner. Ucb revisited: Improved regret bounds for the stochastic bandit problem. Periodica Mathematica Hungarica, 61(1-2):55-66 65, 2010. [3]
Peter Auer, Nicolo Cesa-Biancha and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235 256, 2002.
Monya Baker. 1,500 scientists lift the lid on reproducibility. Nature, 533(7604), 2016a. !
Monya Baker. Reproducibility crisis. Nature, (33(26):353-66) 2016b.
Mihir Bellare and Phillip Rogaway. The complexity of approximating nonlinear program. In Complexity in numerical optimization, pp. 16-32. World Scientific, 1993. 22
Sébastien Bubeck, Nicolo Cesa- -asa-Bianchi, and Sham M Kakade. Towards minimax policies for online linear optimization with bandit feedback. In Conference Learning Theory, pp. 41-1. JMLR Workshop and Conference Proceedings, 2012a.
Sébastien Bubeck, Nicolo Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi- armed bandit problems. Foundations and Trends@ in Muchine Learning, 1-1) 201 2b. 3
Nicolo Cesa-Bianchi and Paul Fischer. Finite-time regret bounds for the multiarmed bandit problem. In ICML, volume 98, pp. 100-108. Citeseer, 1998.
Nicolo Cesa-Bianchi Ofer Dekel, and Ohad Shamir. Online learning with switching costs and other adaptive adversaries. Advances in Neural Information Processing Systems, 26, 2013.
Varsha Dani, Thomas P Hayes, and Sham M Kakade. Stochastic linear optimization under bandit feedback. 2008.
Hossein Esfandiari, Amin Karbasi, Abbas Mehrabian, and Vahab Mirrokni. Regret bounds for batched bandits. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7340-7348, 2021. 3 g
10

---

Published as a conference paper at ICLR 2023
Eyal Even-Dar, Shie Mannor, Yishay Mansour, and Sridhar Mahadevan. Action elimination and stopping conditions for the multi armed bandit and reinforcement learning problems. Journal of machine learning research, 7(6), 2006.
Valerii Vadimovich Fedorov. Theory of optimal experiments. Elsevier, 2013.
22
Robert M Freund and James B Orlin. On the complexity of four polyhedral set containment prob- lems. Mathematical programming, 33(2):339-455, 1985. _2
Zijun Gao, Yanjun Han, Zhimei Ren, and Zhengqing Zhou. Batched multi-armed bandits problem.
Advances in Neural Information Processing Systems, 32, 2019. 4
Steven N Goodman, Daniele Fanelli, and John PA Ioannidis. What does research reproducibility mean? Science translational medicine, 16541):34 2016.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. 3
Russell Impagliazzo, Rex Lei, Toniann Pitassi, and Jessica Sorrell. Reproducibility in learning. arXiv preprint arXhv:2201.04433 2022.
20
John PA Ioannidis.
Why most published research findings are false. PLoS medicine, 2(8):e124,
2005. 3
Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On bayesian upper confidence bounds for
bandit problems. In Artificial intelligence and statistics, pp. 592-600. PMLR, 2012a. 3
Emilie Kaufmann, Nathaniel Korda, and Rémi Munos. Thompson sampling: An asymptotically optimal finite-time analysis. In International conference algorithmic learning theory, pp. 199- 213. Springer, 2012b.
Piyush Kumar and E Alper Yildirim. Minimum-volume enclosing ellipsoids and core sets. Journal of Optimization Theory and applications, 126(1):1-21, 2005. 22
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
22
Tor Lattimore, Csaba Szepesvari, and Gellert Weisz. Learning with good feature representations in bandits and in with generative model. In International Conference Machine Learning, pp. 5662-5670. PMLR, 2020. 22
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. Advances in neural information processing systems, 31, 2018.
Olvi L Mangasarian and T-H Shiau. A variable-complextty norm maximization problem. SIAM Journal on Algebraic Discrete Methods, 7(3):455-61,, 1986. 22
Marcia McNutt. Reproducibility, 2014.
Joelle Pineau, Koustuv Sinha, Genevieve Fried, Rosemary Nan Ke, and Hugo Larochelle. Ielr reproducibility challenge 2019. ReScience C, 5(2), May 2019. doi: 10.5281/zemode 3158244.
D3
Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research: a report from the neurips 2019 reproducibility program. Journal of Machine Learning Research, 22, 2021.
Gil I Shamir and Dong Lin. Real world large scale recommendation systems reproducibility and smooth activations. arXiv preprint arXiv:2202.06499 2022. 3
Aleksandrs Slivkins. Introduction to multi-armed bandits. Foundations and Trends in Machine Learning, 12(1-2) 2019.

---

Published as a conference paper at ICLR 2023
Michael Todd. Minimum-volume ellipsoids: Theory and algorithms. SIAM, 2016.
22
Stephen A Vavasis. Polynomial time weak approximation algorithms for quadratic programming. In Complexity in numerical optimization, pp. 490-500. World Scientific, 1993. 22
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Yinyu Ye. On affine scaling algorithms for nonconvex quadratic programming. Muthematical Pro-
gramming, 56(1):285-300 1992. 22
12

---

Published as a conference paper at ICLR 2023
A THE PROOF OF THEOREM 3
Theorem. Let T & N. 1) There exists p-replicable algorithm (presented in Algorithm I for the stochastic bandit problem with K arms and gaps (A))jE(K) whose expected regret is
K² log" (T)
log(2KTlog(T))
E RT SC-
Σ
(A,+
p
A
1:,,20
where an absolute numerical constant, and its running time is polynomial in K, T and 1/p.
Proof. First, we claim that the algorithm is p-replicable: since the elimination decisions are taken in the same iterates and are based solely the mean estimations, the replicability of the algorithm of Proposition iimplies the replicability of the whole algorithm. In particular,
Pr[(ai...,,a + (d),..., dy-) Pr [B], 3j [K] p)) was not replicable] S p.
During each batch i, we draw for any active arm | fresh samples for total o samples and use the replicable mean estimation algorithm to estimate its mean. For active arm, at the end of some batch & [)), we say that its estimation is "correct" if the estimation of mean is within /log(2 (KKTB)/; from the true mean. Using Proposition 2) the estimation any active arm at the end of any batch (except possibly the last batch) correct with probability at least 1/(2KTB) B) and so, by the union bound, the probability that the estimation is incorrect for some arm at the end of some batch is bounded by 1/T. We remark that when ô p, the sample complexity of Proposition 2 reduces to Ologg( (log((/o)/(²² (/0)/(72pp)) p?))) Let & denote the event that our estimates are correct. The total expected regret can be bounded
E E S 1/T E[R
suffices to bound the second term the RHS and hence we can assume that each gap is correctly estimated within an additive factor of / (oog(2KBB) after batch i. First, due to the elimination condition, we get that the best arm is never eliminated. Next, we have that
E E[
Σ A, E(T,(E). J:A,>0
where T3 is the total number of pulls of arm j. Fix a sub-optimal arm and assume that + 1 was the last batch it was active. Since this arm eliminated at the end of batch i, and the estimations are correct, we have that
A s 1og(2KTB)/c;
and so c; < (ogg(KKBB//A} Hence, the number of pulls to get the desired bound due to Propo- sition 2 is (since we need to pull arm c:/p times in order to get an estimate at distance with probability 8 in p1-replicable manner when 5 < P1)
Ty S c41/ of = q/ + ci)= S q/p; (1+ log (2g(2KK))
This implies that the total regret is bounded by
log log(2KTB)
E E < 1 1
Σ
A
A
j:Ay00
We finally set = T/B and = -T)g(T). Moreover, we have that Pi = (p((/(() These yield
K2 log" (T)
log(2KTlog())
E[RT]
Σ
A
2A,20
This completes the proof.
13

---

Published as a conference paper at ICLR 2023
B THE PROOF OF THEOREM 4
Theorem. Let & p (0, 1.. There exists a p-replicable algorithm (presented in Algorithm 2 for the stochastic bandit problem with K arms and gaps (Az)jE(k) whose expected regret is
K2
E
S
E
Ø²
2:,,20
for some absolute numerical constant C'> 0, and its running time is polynomial in K,T and 1/p.
To give some intuition, we begin with non tight analysis which, however, provides the main ideas behind the actual proof.
Non Tight Analysis Assume that the environment has K arms with unknown means μ4 and let T be the number of rounds. Consider B to the total number of batches and / > 1. We set g = T1/B In each batch i E [B], pull each arm 31 q') times. Hence, after the i-th batch, will have drawn ē. = > -isis Blod]] independent and identically distributed samples from each arm. Let us also set C Ligjsilr'-
Let us fix i [B]. Using Hoeffding's bound for subgaussian concentration, the length of the con- fidence bound for arm j E [K] that guarantees 1 5 probability of success (in the sense that the empirical estimate ūi will be close the true μj) is equal to
U = 2 log(1/
when the estimator uses & samples, Also, let
U.= 2 2loogl//))/
Assume that the active arms the batch iteration lie in the set A: Consider the estimates {P Ne(B).je. where n. is the empirical mean of arm j using ès samples. We will eliminate an arm at the end of the batch iteration i if
F ( + max TEA,
pi U.,
where U. Uni(Us/2,U,]. For the remaining of the proof, condition on the event & that for every arm j [K] and every batch E [B] the true mean is within U, from the empirical one.
We first argue about the replicability of our algorithm. Consider fixed round i (end of i-th batch) and fixed arm j. Let i* the optimal empirical arm after the i-th batch.
(0) -00) Let Pi ..g. the empirical estimates of arms j, i after the i-th batch, under some other execution of the algorithm. We condition on the event E' for the other execution well. Notice that [a(0" p S 20, In(2' p < 20, Notice that, since the randomness of U, is shared, if p.))) + U, p U, + 4U, then the arm will not be eliminated after the i-th batch in some other execution
=0) of the algorithm as well. Similarly, if (() + +Ü; < Pi:- - U, 40; the the arm will get eliminated after the i-th batch in some other execution of the algorithm as well. particular, this means that if 5 20,
(i) + U, U then the arm will not get eliminated in some other execution of
the algorithm and if fry + 5Ū, < p(() U, then the arm will also get eliminated in some other execution of the algorithm with probability 1 under the event & E E'. We call the above two cases good since they preserve replicability. Thus, suffices to bound the probability that the decision about arm will be different between the two executions when Then, the worst case bound due to the mass of the uniform probability measure is
in neither of these cases.
16 2 (1g(1/)
/ log(1/))/c
This implies that the probability mass of the bad event is at most 16/c// 16 1651/3. A union bound over all arms and batches yields that the probability that two distinct executions differ in at least one pull is
Pr((a1......e ar) (0/, -.. a4)) < 16K BGKBV1 1/3 25,
14

---

Published as a conference paper at ICLR 2023
and since 6 S p suffices to pick 768K2
We now focus on the regret of our algorithm. Let us set = 1))KTB). Fix a sub-optimal arm j and assume that batch i + was the last batch that is was active. We obtain that the total number of pulls of this arm is
T S G+1 S 3q(1 + 4) S Sq(1 + 8 (log(//))/3)
From the replicability analysis, it suffices to take 3 of order X² (g=log" logg((T) (T)/ and
C- log (T)
log (KTlog(T))
E[ S T-1/T+E|RT = 1+ Σ A E[Ty|E] ド:A,>0
Σ ;A,Y00
D +
A,
for some absolute constant C' > O.
Notice that the above analysis, which uses naive union bound, does not yield the desired regret bound. We next provide a more tight analysis of the same algorithm that achieves the regret bound Therrem 4
Improved Analysis (The Proof of Theorem 4) In the previous analysis, we used union bound over all arms and all batches in order to control the probability of the bad event. However, we can obtain an improved regret bound as follows. Fix sub-optimal arm i E [K] and let & be the first round that it appears in the bad event. We claim that after a constant number of rounds, this arm will be eliminated. This will shave the O(log((T) factor from the regret bound. Essentially, as indicated in the previous proof, the bad event corresponds to the case where the randomness of the cut-off threshold can influence the decision of whether the algorithm climinates an arm or not, The intuition is that during the rounds and t + 1, given that the two intervals intersected at round t, we know that the probability that they intersect again is quite small since the interval of the optimal mean is moving upwards, the interval of the sub-optimal mean is concentrating around the guess and the two estimations have been moved by at most a constant times the interval's length.
Since the bad event occurs round L, know that
(1)
(t) - U, 5Ū. 140
3U.
In the above fit- the estimate of the optimal mean at round t whose index is denoted by t*, Now assume that the bad event for arm j also occurs at round k. Then, have that
(t+R)
(e+R) P((tk)- ULL 5Ü44k k P (((4))- U+++/2 U
First, notice that since the concentration inequality under event holds for rounds 1, + I we have
that p.j (t-kk) s pb) + Ü. + Üttk- Thus, combining it with the above inequalities gives us
(1) + Ü. + Utt S T/ - U./2 40, + Ü ((++)) < P. (((tk)- U 50.4 S 10
((+R)
We now compare (1)) -P(tt)) Let denote the optimal arm. We have that
(t+h) P(ttay 2 ((+R)) 2 to Ü.tk 2 pr Ü44 2 1/0 - Ü. Üetk-
This gives us that
1l - ULtA -6 6ū - U, S (((tt)-
- Uptx - 5024k.
Thus, we have established that
p
U2tk GÜ2*K - -ū, S (1!
(1)) - U,/2 2 + 48, + Ü,4k +
U44 2 U2/2 7Ü -50 2 U./ 2- 12U,
Since > 2304, we get that 12U, < U./4. Thus, we get that
U+tk 2 U./4.
15

---

Published as a conference paper at ICLR 2023
Notice that
Uttk Ut
V cx+b
thus immediately follows that
Ct Ct+*
1
q
16 (1
2
2
2
16
-1 gltk+1
16
gi+i
g
1
/ºh S 16 + g S
= klogg log 17
S 5,
when we pick B IT'( batches. Thus, for every arm the bad event can happen at most times, by taking a union bound over the K arms we see that the probability that our algorithm is not replicable is most O(K /1/B), so picking 3 = @(K /p²) suffices to get the result.
C THE PROOF OF THEOREM 6
Theorem. Let T E P, (0, There exists p-replicable algorithm (presented in Algorithm 3
for the stochastic d-dimensional linear bandit problem with arms whose expected regrel is
K2
E(RT] S C-
VdT log(KT)
p2
for some absolute numerical constant C > 0, and its running time is polynomial in d, K, T and 1/p.
Proof. Let c, C be the numerical constants hidden in Lemma i.e., the size of the multi-set is in the interval [ed log(1/00 (1/5)/e,, Cd (see Lemma 5). by the end of the B batch we will have less than n pulls left. Hence, the number of batches is at most B.
We know that the size of each batch n, E [og', Cq"]]
We first define the event that the estimates of all arms after the end of each batch are accurate, i.e., for every active arm at the beginning of the i-th batch, at the end of the batch we have that
(a. 0*) S E. Since 1/(KT²) and there are at most T batches and K active arms each batch, simple union bound shows that happens with probability least 1 - 1/T.. We condition on the event & throughout the rest of the proof.
We now argue about the regret bound of our algorithm. We first show that any optimal arm a" will not get eliminated. Indeed, consider any sub-optimal arm E [K] and any batch i [B]. Under the event & we have that
(a,0,) - (,) 0.) S ((a, 0*) E.) ((a", 8-) 2.) 22, E +
Next, we need bound the number of times we pull some fixed suboptimal arm a E [K]. We let A = (a* a, 0*) denote the gap and we let be the smallest integer such that e; A/4. We claim that this arm will get eliminated by the end of batch Indeed,
(a 0.) (a, 0.) ((a". 0.) E.)
((a, 0.) + + A
23, > 42, 23, E +
This shows that during any batch i, all the active arms have gap at most 46;-1 Thus, the regret of the algorithm conditioned on the event E is at most
B
B
B-1 KT)
4n:5i-1 S 4BC
Σ
9
s /log(kK
9 112
1=1
1=0
o Bi2t1 V (log(KT)) -
K2 1
g / /dlog(KT) 0
B/2+1
K2
QV dT log log(KT)
02
Thus, the overall regret is bounded by 8 T + (1 8) .
122
=
/ d' log(KT))
16

---

Published as a conference paper at ICLR 2023
We now argue about the replicability of our algorithm. The analysis follows in similar fashion as in Theorem 4 Let 0., 0² be the LSE after the i-th batch, under two different executions of the algorithm and assume that the set of active arms. We condition on the event E' for the other execution as well. Assume that the set of active arms is the same under both executions at the beginning of batch i. Notice that since the set that is guaranteed by Lemma 5 is computed by a deterministic algorithm, both executions will pull the same arms in batch i. Consider suboptimal arm a and
let a.. = arg max (0,.a), di. = arg max (f), a). Under the event & E E' we have that (a, 0, 0'1 S 25,, (ar-, O. 0,) S 25,, and (a!.,
(a=-; 0.)1 S 25, Notice that, since the randomness of E; is shared, if (a, 0.) + + 2 (a. 0.) E, + 42,, then the arm a will not be eliminated after the i-th batch in some other execution of the algorithm as well. Similarly, if (a, 0.) + < (,) 0,) - E 42, the the arm will get eliminated after the i-th batch in some other execution of the algorithm as well. particular, this means that if (a, 0.) 26; > (a.. (04-,,,)++3 0,)+E; then the arm a will not get climinated in some other execution of the algorithm and if (a, 0,)++ +5; < (a.- 02) - E, then the arm j will also get eliminated in some other execution of the algorithm with probability 1 under the event E Thus, it suffices to bound the probability that the decision about arm j will be different between the two executions when we are in neither of these cases. Then, the worst case bound due to the mass of the uniform probability measure is
16 dl log (1/8//2
v Vllog(//0)/c
This implies that the probability mass of the bad event is at most 16/0/t/ = 16/1/3. naive union bound would require us to pick B = O(K log" T/p²). We next show to avoid the log T factor. Fix sub-optimal arm [K] and let t be the first round that it appears in the bad event.
Since the bad event occurs at round I, we know that
(a, 02)
(a--,0)) ..) Ex 55,, (ar-, 0.) -2e/2+ +5
In the above, ar is the optimal arm at round t w.r.t. the LSE. Now assume that the bad event for arm also occurs at round t k. Then, we have that
(a, 0r+a) E ( 0r++) Ec+a 5ē (u(++R),, - - 22//2+33++++
First, notice that since the concentration inequality under event & holds for rounds 1, t + 1 we have
that (a, ((tb) I (a, 0.) + Eo + ēltA- Thus, combining it with the above inequalities gives us (a(++)-- 0e4x) - E2th 50 S (a, Te4n) S (a, 0z) + + E S (ar,, (.) s./2 42, + Eetk- We now compare ( 0e), (a(e+k)) 0e+x). Let denote the optimal arm. We have that (a(+z)), Oeta) 2 (a",0 00+4)) 2 00) - Eetk 2 (ar-, 0²) E 2 (ar-, 0.) Ecta Ec. This gives us that
(ar,, ..) Eetk 6ē -,, < (a((4))) Drtk) Ec+k - 55
Thus, we have established that
(or, 0t) Ettk 65L+h - E, S (ar 02) - -0//2 + 4E, + ēzth
EL+b 2 e Tēi+k 55, 2 e1/2 12E,
Since 3 2 2304, we get that 128, < e./4. Thus, we get that
c 2 e/4.
Notice that
Et+k E1
qt
1/
git
thus it immediately follows that
g'
< 16
- k logg q S log 16
k < 4,
16
when we pick B = log( batches. Thus, for every arm the bad event can happen at most 5 times, by taking a union bound over the K arms is most O(K v1/3), so picking 3 = OK?2 /p²) suffices to get the result.
that the probability that our algorithm is not not replicable
17

---

Published as a conference paper at ICLR 2023
D NAIVE APPLICATION OF ALGORITHM 3 WITH INFINITE ACTION SPACE
We use a 1 1/T1/(10+2) nne that has size at most ( Let A' be the new set arms. We then run Algorithm Jusing A'. This gives the following result, that is proved right after.
Corollary 12. Let T p. E (0, There is p-replicable algorithm for the stochastic d- dimensional linear bandit problem with infinite arms whose expected regret most
46+1
T4a42
E[RT]
S C.
P /dlog(T),
where C is an absolute numerical constant.
Proof. Since K = (3T) adiz, we have that
T sup (a, ) -E E Σ (a.. 07) so
T
24 (3T) 2d12
dT log ((3T))sf)
= =0 (T
Tiuts
4d
v Vdlog(T)
Ø²
p²
aEA'
Comparing to the best arm in A, we have that:
T
(ae, 0-) 1)
T I'sup (a,0') 0*) aEA
(ae 8*)
T sup 0*) sup aEA
sup (a. aEd'
E T (a, aEA'
0*)
Σ
Our choice the 1/TT4(ad++)-nee implies that for every a E A there exists some a' A' such that ||a - d'll2 I 1/T1/(14+2) Thus, supaE (a, 0*) supwewrd'' 0*) I ||a - 0'112|10~12 S
1(44/(, Thus, the total regret is at most
(T(r" Vdlog(T)
T (Top
T Twupz
4d+1 Vdlog(T)
T 1-1TT +0
p²
E THE PROOF OF THEOREM 10
Theorem. Let T E p.pE p E (0, There exists a p-replicable algorithm (presented in Algorritm 4 for the stochastic d-dimensional linear bandit problem with infinite action set whose expected regret is
E C. d' log(d) log² log(d) log log log(d) (d) Vo
(T)
p²
for some absolute numerical constant C > 0, and its running time is polynomial in T and 1/p.
Proof. First, the algorithm is p-replicable since in each batch we use a replicable LSE sub-routine with parameter p' P/B. This implies that
Pr((ar.... ar) # (a' d/r) Pr[Bi [B] 0 was not replicable] S p.
Let us fix a batch iteration E [B - 1]. Set C, be the core set computed by Lemma 7 The algorithm
first pulls n; Caa log(d/)) log" log(d) log log log(d) times each one of the arms of the i-th core set C, as <fp""
indicated by Lemma 9 and computes the LSE 0, in replicable way using the algorithm Lemma 9 Let be the event that over all batches the estimations are correct. We pick = A/(2[A'TT) so that this good event does hold with probability at least 1/T. Our goal is to control the expected regret which can be written as
T
RT] = T sup (a, 0*)
E
ag 8*)
asA
[=1
We have that
sup
sup (a',0) 0') < 1,
aEA
d'EA'
18

---

Published as a conference paper at ICLR 2023
since A' is a deterministic 1/T- of A. Also, let us set the expected regret of the bounded action sub-problent as
E [
T sup (a',0%
Σ a,, 0-)
a'EA'
We employ the analysis of the finite arm During batch i, any active arm has gap most 41 so the instantaneous regret in any round is not more than 45;-1- The expected regret conditional the good event is upper bounded by
E B[ S Σ 4M,,11
where Mj is the total number of pulls in batch (using the replicability blow-up) and 6;-1 is the would achieve by drawing q² samples (ignoring the blow-up). Then, for some absolute constant C > O, we have that
q" log(d) log" log( (d) log log log(d) 1
(
B 8
4
/ log(T)/a'"
which yields that
E S C dt log(d) log" log(d) log log log(d) (d) log(?") log
S
where we
B
= i=1
q 1/2
Σ
1/2
= 102=9114 (1/B222
11121/ 1)/2
i=1
We pick B = log(T)) and get that, if B then S = O(VT). We remark that this choice of valid since
B
B+1
Tp²
Σ
=0(g - 12
d log () log" log( (d) log log log(d)
mm1
Hence, we have that
dt log(d) log² log(d) log log log(d) VT log 2/
BI
(/2((1/))))
3/2 (T)
Note that when & does not hold, we can bound the expected regret by 1/T . T = 1. This implies that the overall regret E R[ < E[R [8] and so it satisfies the desired bound and the proof is complete.
F DEFERRED LEMMATA
F. THE PROOF OF LEMMA g
Proof. Consider the distribution that is a 2-approximation to the optimal G-design and has support CC| = (d log d). Let C" be the set of arms in the support such that π (aa) S c/ log d. We consider = (1 π)x + ra, where a E C' and x will be specified later. Consider now the matrix V(T).. formula, we have that
Using the
V( V(T)
xV -
-- aa V(T)
((T)-1
xV(=) aa 'aa T (
(1 r) 1
1 1-r | 1
1-x
1 x. + |1a ||al
Consider any arm a'. Then,
1 2 (m)
(a ( aa
1
|| [
<
[all
|a
T-
1- x
a (
1-x
19

---

Published as a conference paper at ICLR 2023
Note that apply this transformation most Od log d) times. Let IT be the distribution end up with. We see that
log log d
odlog log d'
| Ia'll S x
|||| [a)]v(n)= $2
d.
1-x
Notice that there is constant d' such that when x d' //gl d we have that cd log log
S 2.
Moreover, notice that the mass of every arm least x(1 - 2))(| z - |c|22 c' (d c" log log (d² logo (d)) 2 s/(logg()) for some absolute numerical constant 0. This con- cludes the claim.
F.2 THE PROOF OF LEMMA 9
(/) i.c., as long (a) 12(=² (dfe//doo/// log(1/))) In the regime are this point is handled thanks to Lemma 8 Combining the following proof with Lemma 8) we can obtain the desired result.
Proof. The proof works when we can treat ( (doo((//))((()//"")) as 2id
deterministically cho- sen independently of the rewards rt. Assume that the core set of Lemma 7 is the set C. Fix the multi-set S {(rr r.) na = (o(a)) log([C1/6)/e time Hence, we have that
We underline that work in the fixed design setting: the arms
[M]] where each
lies in the core set and is pulled
M
Σ na (oggg())) aEC
Let also V =Eie[M] aza. The least-squares estimator can be written as
((e)
LSE =
=V
Σ orr, ar.=v= MEIM
Σ aEC iE(na)
Σ rı(a),
where each a lies in the core set (deterministically) and ra(a) is the i-th reward generated indepen- dently by the linear regression process (0*,a)+$ +5 where & is a fresh zero mean sub-gaussian random variable. Our goal is to reproducibly estimate the value (r(()) for any a. This sufficient since two independent executions of the algorithm share the set C and na for any a. Note that the above sum is random variable. the following, we condition on the high-probabillty event that the average reward of the arm is e-close to the expected one, i.e., the value (8".a). a). This happens with probability least 1 - (2(2))) given ((m(a) d log(d) log([CI/5)/e" samples from arm a E C. In order to guarantee replicability, we will apply result from Impagliazzo et al. (2022). Since we will union bound over all arms in the core set and |C| Od log ldg( (via Lemma 7), we will make use of /a/pp//rrelllbbllle algorithm that gives estimate v(a) E R such that
(a(
(88" a) - v(a)) <T.
with probability at least 5/(2/C]). For d < p, the algorithm uses
S. Ω (d log(d/5) log" log(d) log log log(d)
many samples from the linear regression with fixed arm E C. Since we have conditioned on the randomness of (a) for any we get
1 Σ r:(a) (0", a) +100" v(a)) S
ri(a) v(a)
The
Na ve(m..
:E(ma]]
with probability at least - 8/(2(C]). Hence, by repeating this approach for all arms in the core set, we set 0sq = V-1 Enec a na v(a). Let us condition on the randomness of the estimate ((8) We have that
(aa, (a',0sq 8*) S sup (a', Asq - 0 I+ + sup (a', ( 8*)
sup a'EA
d'EA
d'EAA
Recall that (())= 2 (d))))))) for constant the previous expression is ((log(6/(C))/
20

---

Published as a conference paper at ICLR 2023
Note that the second term is with probability least 1 5 via Lemma 5 Our next goal is to tune the accuracy т E (0, 1) so that the first term yields another E error. For the first term, we have that
sup a'EA
(a', Usq
oisEN
I sup a' a'EA
(a', V
Σ aEc
a na (e T))
Note that V log(od)) Eace (a)aa" T and so V-1
Calog(d)ooa([IA ((+) for
some absolute constant > O. This implies that
( d'Colog(d) (oglogg))ogg((///]))) V(x)=
E²
Cd log log log((C1/6)7(a)
sup a'EA
(a', Bsq LSE S (E+T) sup a'EA
2²
aEC
Hence, get that
(a. ['. Σ T(a)a) )/
sup (a 0s0
((8) 1.S
S (e + r) sup
a.,
a'EA
e'EA
aEC
Consider fixed arm a' E A. Then,
( (d. Σ T(a)a
S (d) (xx)-aa))
aEC
aCC
T(a) (1 + (a', ((z)
aEC
1 + Σ (a') ((T)
aEC
+ || (
S 4d 1,
where the last inequality follows from the fact that is 4-approximation of the G-optimal design. Hence, in total, by picking T = e, we get that
sup e'EA
((a ,sq 0*) < 11ds.
Thus, for any
0, the total number of pulls of each arm is
Ω (d' log(d/8) log" log(d) log log (og(d)/(o()))
to get
sup e'EAA
(a OsQ - 0*) se.
G COMPUTATIONAL PERFORMANCE OF ALGORITHH
In this appendix, we discuss the barriers towards computational efficiency regarding Algorithm 4 The reasons why Algorithm i computationally inefficient are the following: (a) we have to com- pute the arm in the set of active arms that has maximum correlation with the estimate 0., (b) we have eliminate arms based on this value and (c) we have run each batch the Frank- Wolfe algo- rithm (or some other optimization method needed for Lemma 5) in order to obtain an approximate G-optimal design. As minimal assumption what follows, we focus on the case where the action set A is convex and we have access to a separation oracle for it.
Note that executing both (a) and (b) naively requires time exponential in d. However, on the one side arm elimination (issue (b)) reduces to finding the intersection of the current active set with halfspace H. whose normal vector is Ø' and the threshold is, roughly speaking, the maximum correlation. This maximum correlation can also be computed efficiently. Finding an arm with (almost) maximum correlation relates to the problem of finding point that maximizes linear
21

---

Published as a conference paper at ICLR 2023
objective under the constraint that the point lies in the intersection of the active arm set with some linear constraints. Thus, we can use the ellipsoid algorithm to implement this step.
The above discussion deals with issues (a) and (b) and, essentially, states that even with infinitely many actions, one could implement these steps efficiently. We now focus on issue (c). The Frank- Wolfe method first requires a proper initialization. As mentioned Lat & Szepesvári (2020) the starting point is chosen to be the uniform distribution over A', then the number of iterations before getting 2-approximate optimal design is roughly (dd) The issue is that since A'' is ex- ponential in d. it is not clear how to work with such an initialization efficiently. Notably there is different initialization Fedorov (2013); Lattimore et al. (2020) with support O(d) for which the method runs in (d log (d) rounds (see Note 3 at Section 21.2 of Lattimore & Szepesvári (2020) and Lattimore et al. (2020). There are two issues: first, one requires an oracle to provide this good initialization. Second, each iteration of the Frank-Wolft method (with current design guess r) requires computing point in the current active set with maximum Todd (2016), a good initialization for finding a G-optimal design, i.e., a minimum volume enclosing ellipsoid (MVEE) should be sufficiently sparse (compared the number of active arms) and assign positive mass to arms that correspond to extreme points, i.e., points that are close to the border of MVEE. The work of Kumar& & Yildirim (2005) provides an initial core set that depends only on but not on the number of points. The algorithm works as follows: it runs for d iterations and, in each round, adds 2 arms into the core set. Initially, we set the core set C. = 0 and let v/ {0}. In each iteration i E [d]. the algorithm draws a random direction v; in the orthogonal complement of Ve (this step is replicable thanks to the shared randomness) and computes the vectors in the ac- tive arms' set with the maximum and the minimum correlation with vi say ag , It then extends Co < Co U {a a. } and sets V + span(V, faat 0, }). Hence, the runtime of this algorithm corresponds to the runtime of the tasks max,e.A' (a, 04) and min (a, 21). One can efficiently ap- proximate these values using the ellipsoid algorithm and hence efficiently initialize the Frank-Wolfe algorithm as in Todd 2016 (e.g., set the weights uniformly 1/(2d))
-nomm. As noted in
Our second challenge deals with finding point in the active arm set with maximum V(T)- norm for some current guess x. Even the current active set is a polytope. finding an exact norm maximizer is NP-hard Freund & Orlin 1985) y Mangasarian & Shiau (9866 Hence, one should focus on efficient approximation algorithms. We note that even a poly (d)ppppoxiii maximizer is sufficient to get (poly(d))VT) regret. Such an algorithm for polytopes, which gets an 1/d
approximation, is provided in Ye [992]; Vavasis (1993).
As general note, if we assume that we have access to an oracle O that computes a 2-approximate G-optimal design in time To, then our our/Algorithm 4 rrnns in time polynomial in To.
H EXPERIMENTAL EVALUATION
In this section, we provide some experimental evaluation of our proposed algorithms in the setting of multi-an aumti- stochastic bandits. In particular, we will compare the performance of our algorithm with the standard UCB approach. We first analyze the experimental setup.
Experimental Setup. We consider a multi-armed bandit setting with K = 6 arms that have Bemoulli rewards with bias (0.44, 0.47, 0.5, 0.53, 0.56, 0.59), we run the algorithms for T 45000 iterations and we execute both algorithms 20 different times. For our algorithm, we set its repli- cability parameter p = 0.3. We are interested in comparing the replicability and the regret of the two algorithms. We underline that our theoretical bound in this setting shows that the regret of our algorithm could be, roughly, at most 360 times worse than the regret of UCB.
Observations. We first observe that our proposed bandit algorithm is indeed replicable, i.e., it satisfies our proposed reproduciilittyeepliabbtty definition. In particular, for the 10 pairs of con- secutive executions we consider, we observe that our algorithm pulls the exact same sequence of arms in all of them. On the other hand, the standard UCB algorithm is far from being replicable. To be more precise, between consecutive executions we see that the algorithm makes a different choice
In fact, even finding constant factor approximation, for some appropriate constant, is NP-hard Bellare & Rogaway (1933..
22

---

Published as a conference paper at ICLR 2023
2500
2000
1500
1000
500
Average Regret UCB
Average Regret Repr. Batched Elimin.
10000
20000
30000
40000
Figure 1: Cumulative Regret UCB vs. Replicable Batched Algorithm
between 26208 28002 rounds out of the 45000 rounds, which is that half of the rounds! However, the regret of our approach is quite competitive compared to the standard UCB algorithm. In particular, the regret of the standard UCB algorithm has the expected polylog(2)) shape, while the regret of our replicable variant incurs multiplicative overhead that is, roughly, 2.5 times worse than UCB. This is depicted in Figure I executions) of the two algorithms.
where we plot the average cumulative regret (across the 20
23


Reviews:
[{"review_id": "docMBNG6H7", "reviewer": "Reviewer_QnVR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the stochastic multi-armed bandit problem (K-armed as well as the linear version) under \"reproducibility constraints,\" i.e., the policy should play the same sequence of arms in any two i.i.d. instances of the problem (using the same algorithmic random seed) with probability at least $1-\\rho$. The authors propose $\\rho$-reproducible policies with rate-optimal regret (w.r.t. $T$). ", "summary_of_the_review": "Based on my assessment of this paper's contributions, I vote for an acceptance. ", "strength_and_weaknesses": "The paper is technically sound. It is insightful to see that while standard bandit algorithms are not reproducible in general, one can with only a slight multiplicative increase in sample complexity ensure reproducibility.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "UT0a32-hkvY", "reviewer": "Reviewer_171s", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The purpose of this paper is to investigate reproducible bandit algorithms while keeping a reasonable regret guarantee. The paper starts with a notion of reproducibility for a bandit algorithm following the definition of Impagliazzo et al. (2022). It states that a bandit algorithm is reproducible if two different executions of the algorithm would lead to the same sequence of played arms with high probability.\n\nThe authors first studied the problem for standard stochastic bandits in Section 3 and proposed a first version of algorithm based on reproducible mean estimation (Impagliazzo et al., 2022) and a batched bandits algorithm (Esfandiari et al., 2021) (using traditional optimal algorithms like UCB is unlikely to be feasible under current definition). This first version incurs an extra factor of $O(K^2\\log^2(T)/\\rho^2)$ compared to its non-reproducible counterpart. The authors then managed to get rid of a $\\log^2(T)$ factor in Section 4, which reduces the extra factor to $O(K^2/\\rho^2)$.\n\nThe authors also managed to propose an algorithm with the same extra $O(K^2/\\rho^2)$ factor for linear bandits (finitely-armed) proposed in Section 5.1. A study in the infinitely-armed case is also provided (with an extra $d^2$ factor against its non-reproducible counterpart).", "summary_of_the_review": "In summary, I appreciate the authors' effort and I think the topic of reproducibility is important to the community. The algorithms and regret bounds provided are rather solid under the current definition of reproducibility. But I do have some doubt on whether the definition really makes sense given the dependence of $K$ in the proposed algorithms and the lack of experimental illustration.\n\nMisc:\n- Section 1, Paragraph 1: \"due to the to the\" -> \"due to the\"\n\n__________________________________________________________________\n\nAfter rebuttal, paper solid in theoretical aspects. Some experimental illustration added, which is appreciated, though still moderate for a paper on reproducibility. I decide to increase my score.\n\n", "strength_and_weaknesses": "Strength:\n- The topic is novel and could have an impact on the community.\n- A first definition of reproducible bandit algorithms is provided and is supported by some non-trivial algorithms.\n- A very clear thinking process on how the algorithms (and improvements) are proposed.\n- Correct theoretical contributions: regret bounds provided are non-trivial.\n\nWeakness:\n- One of my major concerns is that the paper didn't provide any experimental illustration while the paper is talking about reproducibility. I do understand that the authors want to provide a fundamental study of the problem, but reproducibility itself is a very practical-oriented notion for which lack of experiments seems not quite reasonable to me.\n- Another related point, not necessarily a weak point but rather something arguable, is the extra dependence of $K$ in the regret bound compared to non-reproducible algorithms. It seems that the extra $K^2/\\rho^2$ factor is rather difficult to be get rid of under current context. This could particularly cause problems when we have a large arm space in practice. And somehow, reproducibility makes more sense in a large scale environment. This is a question rather about the validity of the definition itself in my opinion: does it really make sense in practice? I think the authors could probably provide more discussion on that rather than (alongside some experiments ideally) stacking theoretical results for different problem settings one by one. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8tyWhpB2aGC", "reviewer": "Reviewer_p4Aa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The submission studies the intersection of reproducibility and bandits. The submission proposes and analyses four algorithms for the stochastic MABs and stochastic linear bandits, with optimal dependency on the reproducibility parameter. The submission reveals a valuable insight between reproducible interpretability and exploration-exploitation tradeoff.", "summary_of_the_review": "Reproducibility is becoming crucial in comparing, interpreting, and understanding different methods. Introducing reproducible bandit algorithms gives us a perspective to make better sense of bandits. The reproducible bandit algorithms also link themself to the field of reproducible learning. Given these merits and the strengths listed above, I would like to recommend an acceptance of the submission.", "strength_and_weaknesses": "- (a) Introducing reproducibility to bandits provides a new perspective to contemplate the bandit problem.\n- (b) The submission presents its algorithms step-by-step, providing a clear picture of which parts are the novelties and how a novel design is combined with existing techniques to achieve the desired regret bound. The critical components of leveraging batch bandits, uniformly sampled thresholds (step 13 of Algorithm2 and step 13 of Algorithm 3), G-optimal design, LSE, and ReproducibleLSE become intuitive and easy to follow.\n- (c) The submission clearly explains the challenges and the intuitions of algorithm design. Regarding similar approaches, the submission carefully compares differences between the proposed method and published ones.\n- (d) The regret bounds offer insights and the impact of reproducible learning. The submission explains how to spend additional rounds to guarantee reproducibility and the extra factors compared to the conventional bounds.\n\nGiven the above, the results and contribution of this submission are substantial. The proofs are rigorous.\n\n- (e) The minor question is, in step 9 of Algorithm 1, is a \"min\" missing in the assignment of \\tau?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "2mj6C8tPYkf", "reviewer": "Reviewer_4C9J", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper designs reproducible policies for classic multi-armed bandit model and linear bandit model. In the classic multi-armed bandit setting, the authors first design an algorithm that directly applies existing reproducible algorithms for estimating the means of random distributions, and propose an regret upper bound that is  $O({K^2\\log^2 T \\over \\rho^2})$ larger than non-reproducible algorithms. Here $K$ is the number of arms, $T$ is the time horizon, and $\\rho$ is the reproducible rate (i.e., with probability at least $1-\\rho$, the algorithm will output the same sequence). Then they improve their algorithm by some specific properties of bandits setting, and reduce the $O(\\log^2 T)$ factor in the regret upper bound. As for the linear bandit case, the authors first consider the finite arm set case, and propose an algorithm that achieves regret upper bound of $\\tilde{O}({K^2 \\over \\rho^2}\\sqrt{dT})$, which is also $\\tilde{O}({K^2 \\over \\rho^2})$ higher than existing non-reproducible algorithms. When the arm set is infinite, the authors also provide a reproducible algorithm that achieves $\\tilde{O}({d^3 \\over \\rho^2})$ higher regret than non-reproducible ones. ", "summary_of_the_review": "My main concern here is about the motivation of this paper, and therefore I give the score of \"marginally below the threshold\".\n\n=======After the rebuttal===========\n\nAfter reading the explanations about the motivation and the lower bound, I am willing to change my score to 6.", "strength_and_weaknesses": "Strength\n\n1. Applying reproducity to bandits problems in an interesting and novel idea.\n\n2. I check some of the proofs in appendix, and they seem to be correct. \n\nWeaknesses\n\n1. About the motivation.\n\nThough the reproducity of scientific findings are very important, I do not really understand why we need to design reproducible algorithms for bandit problems. Maybe one can claim that we can reproduce others' experimental results easily when the designed algorithms are reproducible, but in this case, I think i) the definition of reproducible (Definition 1) is too strict; and ii) the cost is too large. And  I guess that ii) maybe a consequence of i). In fact, to reproduce others' experimental results, we do not really need the sequence to be exact the same.\n\n2. About the regret upper bounds.\n\nThe regret upper bound seems to be not very tight. For example, when we choose $\\rho \\to 1$, the algorithms does not have any reproducible properties, but we still suffer an extra $K^2$ factor in the regret upper bound. What is the reason of this?\n\n3. Lack of regret lower bounds.\n\nI guess that the regret lower bound analysis could be difficult, especially for the linear bandit case. However, I think some discussions would be very helpful here. After I read the paper, I cannot really understand the difficulty of designing reproducible algorithms, and also do not know which kinds of instances could be very hard to solve in this case. Besides, it is mentioned in the comclusion that the factor of ${1\\over \\rho^2}$ is tight (according to (Impagliazzo et al., 2022)). However, after I read that paper, I do not think its proof can be directly used for the bandit case. Maybe I miss some important things, but I think there should be more detailed explanations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster