Title: An $\varepsilon$-Best-Arm Identification Algorithm for Fixed-Confidence and Beyond
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi-armed bandits

pure-exploration

epsilon best arm identification

Top Two algorithm

anytime
Abstract: We propose EB-TC$\varepsilon$, a novel sampling rule for $\varepsilon$-best arm identification in stochastic bandits.
	It is the first instance of Top Two algorithm analyzed for approximate best arm identification. EB-TC$\varepsilon$ is an *anytime* sampling rule that can therefore be employed without modification for fixed confidence or fixed budget identification (without prior knowledge of the budget).
	We provide three types of theoretical guarantees for EB-TC$\varepsilon$.
	First, we prove bounds on its expected sample complexity in the fixed confidence setting, notably showing its asymptotic optimality in combination with an adaptive tuning of its exploration parameter.
	We complement these findings with upper bounds on its probability of error at any time and for any slack parameter, which further yield upper bounds on its simple regret at any time.
	Finally, we show through numerical simulations that EB-TC$\varepsilon$ performs favorably compared to existing algorithms for different approximate best arm identification tasks.

An ε -Best-Arm Identification Algorithm for Fixed-Confidence and Beyond

Abstract

We propose EB-TC ε , a novel sampling rule for ε -best arm identification in stochastic bandits. It is the first instance of Top Two algorithm analyzed for approximate best arm identification. EB-TC ε is an anytime sampling rule that can therefore be employed without modification for fixed confidence or fixed budget identification (without prior knowledge of the budget). We provide three types of theoretical guarantees for EB-TC ε . First, we prove bounds on its expected sample complexity in the fixed confidence setting, notably showing its asymptotic optimality in combination with an adaptive tuning of its exploration parameter. We complement these findings with upper bounds on its probability of error at any time and for any error parameter, which further yield upper bounds on its simple regret at any time. Finally, we show through numerical simulations that EB-TC ε performs favorably compared to existing algorithms, in different settings.

1 Introduction

In pure exploration problems, the goal is to answer a question about a set of unknown distributions (modelling for example the efficacy of a treatment) from which we can collect samples (measure its effect), and to provide guarantees on the candidate answer. Practitioners might have different pre-defined constraints, e.g. the maximal budget might be fixed in advance or the error made should be smaller than a fixed admissible error. However, in many cases, fixing such constraints in advance can be challenging since a 'good' choice typically depends on unknown quantities. Moreover, while the budget is limited in clinical trials, it is often not fixed beforehand. The physicians can decide to stop earlier or might obtain additional fundings for their experiments. In light of those real-world constraints, regardless of its primal objective any strategy for choosing the next treatment should ideally come with guarantees on its current candidate answer that hold at any time.

We formalize our investigations in the well-studied stochastic bandit model [4, 29], in which a learner interacts sequentially with an environment composed of K ∈ N arms, which are unknown distributions ( ν i ) i ∈ [ K ] with finite means ( µ i ) i ∈ [ K ] . At each stage n ∈ N , the learner chooses an arm I n ∈ [ K ] based on the samples previously observed and receives a sample X n,I n , random variable with conditional distribution ν I n given I n . It then proceeds to the next stage. An algorithm for the learner in this interaction is specified by a sampling rule , a procedure that determines I n based on previously observed samples. Formally, the sampling rule defines for all n ∈ N a function from ([ K ] × R ) n -1 to the probability distribution on [ K ] , which is measurable with respect to the σ -algebra F n := σ ( { I t , X t,I t } t ∈ [ n -1] ) . We call that σ -algebra history before n .

Identification tasks Wefocus on best arm identification (BAI). In that task, the goal of the algorithm is to find which of the arms has the largest mean, and to do so with a small probability of error, as

quickly as possible. If several arms have means very close to the maximum, finding the one with the highest mean might be difficult. However in practice we are often satisfied by any good enough arm, in the sense that its mean is greater than µ ⋆ -ε , where µ ⋆ = max i ∈ [ K ] µ i . This is the ε -BAI task. Our results can also be adapted to the multiplicative ε -BAI task, in which all means are non-negative and we want to find an arm with mean µ i ≥ (1 -ε ) µ ⋆ [19] (see Appendix I for details).

Now that we have a (for now informal) goal, we need to complement the sampling rule with a recommendation rule that specifies which arm is the candidate returned by the algorithm for the best arm. We follow [5] and define that rule for all stages: for all n ∈ N , we denote by ˆ ı n this F n -measurable function from ([ K ] × R ) n -1 to [ K ] . We call algorithm the combination of a sampling and a recommendation rule.

Performance criteria There are several ways to evaluate the performance of an algorithm for ε -BAI. Let I ε ( µ ) = { i ∈ [ K ] | µ i ≥ µ ⋆ -ε } be the set of ε -good arms. The probability of ε -error or the recommendation at n is defined as P ν (ˆ ı n / ∈ I ε ( µ )) . Introduced in [1], the expected simple regret is defined as E ν [ µ ⋆ -µ ˆ ı n ] , and is independent of any parameter ε . Based on those notions, several setting are studied in the bandit identification literature.

Fixed confidence: we augment the algorithm with a stopping rule , a stopping time τ ε,δ with respect to the history of samples and we impose that the algorithm should be ( ε, δ ) -PAC. That is, its probability of ε -error at τ ε,δ must satisfy P ν ( τ ε,δ < + ∞ , ˆ ı τ ε,δ / ∈ I ε ( µ )) ≤ δ . The parameter δ is known to the algorithm. An algorithm is judged based on its expected sample complexity E [ τ ε,δ ] , the expected number of samples it needs to collect before it can stop and return a good arm with the required confidence.

Fixed budget: we run the algorithm until a predefined time T and we evaluate it based on the probability of error at T . This setting has been mostly studied for ε = 0 [1, 25], but [41] present the first bounds for ε > 0 for an algorithm that is actually agnostic to this value.

Simple regret minimization: we evaluate the expected simple regret at T [5, 41].

̸

Simple regret is typically studied in an anytime setting: [5] contains upper bounds on the simple regret at time n for any n ∈ N ∗ . Similarly, [23] propose the anytime exploration setting, in which they control the error probability P (ˆ ı n = i ⋆ ) for exact best arm identification. Interestingly, the authors build on an algorithm for the fixed-confidence setting, LUCB [24], whose sampling rule depends on the risk parameter δ , which they replace by a sequence δ n . The algorithm that we study in this paper, motivated by the fixed-confidence ε -BAI problem, will already be anytime , which means that it does not depend on a given final time T or a confidence level δ . We shall analyze its sample complexity in the fixed confidence setting but thanks to the anytime property we will also be able to prove guarantees on its probability of ε -error for every ε ≥ 0 and its simple regret at any time.

Additional notation and assumption We denote by D a set to which the distributions of the arms are known to belong. We suppose that all distributions in D are 1-sub-Gaussian. A distribution ν 0 is 1-sub-Gaussian if it satisfies E X ∼ ν 0 [ e λ ( X -E X ∼ ν 0 [ X ]) ] ≤ e λ 2 / 2 for all λ ∈ R . For example, all distributions bounded in [ -1 , 1] are 1-sub-Gaussian. Let us denote by i ⋆ ( µ ) := arg max i ∈ [ K ] µ i the set of arms with largest mean (i.e. i ⋆ ( µ ) = I 0 ( µ ) ). Let ∆ i := µ ⋆ -µ i denote the sub-optimality gap of arm i . We denote by △ K ⊂ R K the simplex of dimension K -1 .

Fixed-confidence ε -best-arm identification Let ε ≥ 0 and δ ∈ (0 , 1) be fixed error and confidence parameters. In the fixed-confidence ε -BAI setting [31, 13, 35, 16], the probability of error of an algorithm is required to be less than δ for all instances ν ∈ D K . That requirement leads to an asymptotic lower bound on the expected sample complexity on any instance.

Lemma 1 ([10]) . For all ( ε, δ ) -PAC algorithms and all instances ν i = N ( µ i , 1) with µ ∈ R K , lim inf δ → 0 E ν [ τ ε,δ ] log(1 /δ ) ≥ T ε ( µ ) where T ε ( µ ) = min i ∈I ε ( µ ) min β ∈ (0 , 1) T ε,β ( µ, i ) with

̸

We say that an algorithm is asymptotically (resp. β -)optimal if its sample complexity matches that lower bound, that is if lim sup δ → 0 E ν [ τ ε,δ ] log(1 /δ ) ≤ T ε ( µ ) (resp. T ε,β ( µ ) = min i ∈I ε ( µ ) T ε,β ( µ, i ) ). Note

that the expected sample complexity of an asymptotically 1 / 2 -optimal algorithm is at worst twice higher than that of any asymptotically optimal algorithm since T ε, 1 / 2 ( µ ) ≤ 2 T ε ( µ ) [34].

The asymptotic characteristic time T ε ( µ ) is of order ∑ K i =1 min { ε -2 , ∆ -2 i } . It is computed as a minimum over all ε -good arms i ∈ I ε ( µ ) of an arm-specific characteristic time, which can be interpreted as the time required to verify that i is ε -good. Each of the times min β ∈ (0 , 1) T ε,β ( µ, i ) correspond to the complexity of a BAI instance (i.e. ε -BAI with ε = 0 ) in which the mean of arm i is increased by ε (Lemma 9). Let w ε,β ( µ, i ) be the maximizer of (1). In [16], they show that T ε ( µ ) = T ε,β ⋆ ( i ⋆ ) ( µ, i ⋆ ) and T ε,β ( µ ) = T ε,β ( µ, i ⋆ ) , where i ⋆ ∈ i ⋆ ( µ ) and β ⋆ ( i ⋆ ) = arg min β ∈ (0 , 1) T ε,β ( µ, i ⋆ ) . For ε = 0 , a similar lower bound to Lemma 1 holds for all δ [15]. Lower bounds of order ∑ K i =1 ∆ -2 i log log ∆ -2 i (independent of δ , but with a stronger dependence in the gaps) were also shown [17, 6, 38, 7]. Note that the characteristic time for σ -sub-Gaussian distributions (which does not have a form as 'explicit' as (1)) is always smaller than the ones for Gaussian having the same means and variance σ 2 .

A good algorithm should have an expected sample complexity as close as possible to these lower bounds. Several algorithms for ( ε -)BAI are based on modifications of the UCB algorithm [24, 17, 14]. Others compute approximate solutions to the lower bound maximization problem and sample arms in order to approach the solution [15, 11, 39]. Our method belongs to the family of Top Two algorithms [34, 36, 21], which select at each time two arms called leader and challenger, and sample among them. It is the first Top Two algorithm for the ε -BAI problem (for ε > 0 ).

Any time and uniform ε -error bound In addition to the fixed-confidence guarantees, we will prove a bound on the probability of error for any time n and any error ε , similarly to the results of [41]. That is, we bound P ν (ˆ ı n / ∈ I ε ( µ )) for all n and all ε . This gives a bound on the probability of error in ε -BAI, and a bound on the simple regret of the sampling rule by integrating: E ν [ µ ⋆ -µ ˆ ı n ] = ∫ P ν (ˆ ı n / ∈ I ε ( µ ))d ε .

The literature mostly focuses on the fixed budget setting, where the time T at which we evaluate the error probability is known and can be used as a parameter of the algorithm. Notable algorithms are successive rejects (SR, [1]) and sequential halving (SH, [25]). These algorithms can be extended to not depend on T by using a doubling trick [23, 41]. That trick considers a sequence of algorithms that are run with budgets ( T k ) k , with T k +1 = 2 T k and T 1 = 2 K ⌈ log 2 K ⌉ . Past observations are dropped when reaching T k , and the obtained recommendation is used until the budget T k +1 is reached.

1.1 Contributions

We propose the EB-TC ε 0 algorithm for identification in bandits, with a slack parameter ε 0 > 0 , originally motivated by ε 0 -BAI. We study its combination with a stopping rule for fixed confidence ε -BAI (possibly with ε 0 = ε ) and also its probability of error and simple regret at any time.

̸

EB-TC ε 0 performs well empirically compared to existing methods, both for the expected sample complexity criterion in fixed confidence ε -BAI and for the anytime simple regret criterion. It is in addition easy to implement and computationally inexpensive in our regime.

We prove upper bounds on the sample complexity of EB-TC ε 0 in fixed confidence ε -BAI with sub-Gaussian distributions, both asymptotically (Theorem 1) as δ → 0 and for any δ (Theorem 2). In particular, EB-TC ε with ε > 0 is asymptotically optimal for ε -BAI with Gaussian distributions.

We prove a uniform ε -error bound valid for any time for EB-TC ε 0 . This gives in particular a fixed budget error bound and a control of the expected simple regret of the algorithm (Theorem 3 and Corollary 1).

2 Anytime Top Two sampling rule

We propose an anytime Top Two algorithm, named EB-TC ε 0 and summarized in Figure 1.

Recommendation rule Let N n,i := ∑ t ∈ [ n -1] 1 ( I t = i ) be the number of pulls of arm i before time n , and µ n,i := 1 N n,i ∑ t ∈ [ n -1] X t,I t 1 ( I t = i ) be its empirical mean. At time n > K , we recommend the Empirical Best (EB) arm ˆ ı n ∈ arg max i ∈ [ K ] µ n,i (where ties are broken arbitrarily).

1: Input: slack ε 0 > 0 , proportion β ∈ (0 , 1) (only for fixed proportions).

̸

3: Set ¯ β n +1 ( B n , C n ) = ( T n ( B n , C n ) ¯ β n ( B n , C n ) + β n ( B n , C n )) /T n +1 ( B n , C n ) with [ fixed ] β n ( i, j ) = β or [ IDS ] β n ( i, j ) = N n,j / ( N n,i + N n,j ) and T n +1 ( B n , C n ) = T n ( B n , C n ) + 1 .

2: Set ˆ ı n ∈ arg max i ∈ [ K ] µ n,i , B n =ˆ ı n and C n ∈ arg min i = B n µ n,Bn -µ n,i + ε 0 √ 1 /N n,Bn +1 /N n,i .

4: Set I n = C n if N B n n,C n ≤ (1 -¯ β n +1 ( B n , C n )) T n +1 ( B n , C n ) , otherwise set I n = B n .

5: Output : next arm to sample I n and next recommendation ˆ ı n .

2.1 Anytime Top Two sampling rule

̸

We start by sampling each arm once. At time n > K , a Top Two sampling rule defines a leader B n ∈ [ K ] and a challenger C n = B n . It then chooses the arm to pull among them. For the leader/challenger pair, we consider the Empirical Best (EB) leader B EB n = ˆ ı n and, given a slack ε 0 > 0 , the Transportation Cost (TC ε 0 ) challenger

̸

The algorithm then needs to choose between B n and C n . In order to do so, we use a so-called tracking procedure [15]. We define one tracking procedure per pair of leader/challenger ( i, j ) ∈ [ K ] 2 such that i = j , hence we have K ( K -1) independent tracking procedures. For each pair ( i, j ) of leader and challenger, the associated tracking procedure will ensure that the proportion of times the algorithm pulled the leader i remains close to a target average proportion ¯ β n ( i, j ) ∈ (0 , 1) . At each round n , only one tracking rule is considered, i.e. the one of the pair ( i, j ) = ( B n , C n ) .

̸

We define two variants of the algorithm that differ in the way they set the proportions ¯ β n ( i, j ) . Fixed proportions set ¯ β n ( i, j ) = β for all ( n, i, j ) ∈ N × [ K ] 2 , where β ∈ (0 , 1) is fixed beforehand. Information-Directed Selection ( IDS ) [40] defines β n ( i, j ) = N n,j / ( N n,i + N n,j ) and sets ¯ β n ( i, j ) := T n ( i, j ) -1 ∑ t ∈ [ n -1] 1 (( B t , C t ) = ( i, j )) β t ( i, j ) where T n ( i, j ) := ∑ t ∈ [ n -1] 1 (( B t , C t ) = ( i, j )) is the selection count of arms ( i, j ) as leader/challenger.

Let N i n,j := ∑ t ∈ [ n -1] 1 (( B t , C t ) = ( i, j ) , I t = j ) be the number of pulls of arm j at rounds in which i was the leader. We set I n = C n if N B n n,C n ≤ (1 -¯ β n +1 ( B n , C n )) T n +1 ( B n , C n ) and I n = B n otherwise. Using Theorem 6 in [12] for each tracking procedure (i.e. each pair ( i, j ) ) yields Lemma 2 (proved in Appendix H).

̸

The TC ε 0 challenger seeks to minimize an empirical version of a quantity that appears in the lower bound for ε 0 -BAI (Lemma 1). As such, it is a natural extension of the TC challenger used in the T3C algorithm [36] for ε 0 = 0 . In earlier works on Top Two methods [34, 32, 36], the choice between leader and challenger is randomized: given a fixed proportion β ∈ (0 , 1) , set I n = B n with probability β , otherwise I n = C n . [20] replaced randomization by tracking, and [40] proposed IDS to define adaptive proportions β n ( B n , C n ) ∈ (0 , 1) . In this work we study both fixed proportions with β = 1 / 2 and adaptive proportions with IDS. Empirically, we observe slightly better performances when using IDS (e.g. Figure 7 in Appendix J.2.1). While [20] tracked the leader with K procedures, we consider K ( K -1) independent tracking procedures depending on the current leader/challenger pair.

Choosing ε 0 [21] shows that EB-TC (i.e. EB-TC ε 0 with slack ε 0 = 0 ) suffers from poor empirical performance for moderate δ in BAI (see Appendix D.3 in [21] for a detailed discussion). Therefore, the choice of the slack ε 0 > 0 is critical since it acts as a regularizer which naturally induces sufficient exploration. By setting ε 0 too small, the EB-TC ε 0 algorithm will become as greedy as EB-TC and perform poorly. Having ε 0 too large will flatten differences between sub-optimal arms, hence it will behave more uniformly. We observe from the theoretical guarantees and from our experiments that it is best to take ε 0 = ε for ε -BAI, but the empirical performance is only degrading slowly for ε 0 > ε . Taking ε 0 < ε leads to very poor performance. We discuss this trade-off in more details in our experiments (e.g. Figures 5, 6 and 7 in Appendix J.2.1). When tackling BAI, the limitation of EB-TC can be solved by adding an implicit exploration mechanism in the choice of the leader/challenger

pair. For the choice of leader, we can use randomization (TS leader [34, 32, 36]) or optimism (UCB leader [20]). For the choice of the challenger, we can use randomization (RS challenger [34]) or penalization (TCI challenger [21], KKT challenger [40] or EI challenger [32]).

Anytime sampling rule EB-TC ε 0 is independent of a budget of samples T or a confidence parameter δ . This anytime sampling rule can be regarded as a stream of empirical means/counts ( µ n , N n ) n>K (which could trigger stopping) and a stream of recommendations ˆ ı n = i ⋆ ( µ n ) . These streams can be used by agents with different kinds of objectives. The fixed-confidence setting couples it with a stopping rule to be ( ε, δ ) -PAC. It can also be used to get an ε -good recommendation with large probability at any given time n .

2.2 Stopping rule for fixed-confidence ε -best-arm identification

In addition to the sampling and recommendation rules, the fixed-confidence setting requires a stopping rule. Given an error/confidence pair, the GLR ε stopping rule [15] prescribes to stop at the time

̸

where c : N × (0 , 1) → R + is a threshold function. Lemma 3 gives a threshold ensuring that the GLR ε stopping rule is ( ε, δ ) -PAC for all ε ≥ 0 and δ ∈ (0 , 1) , independently of the sampling rule.

Lemma 3 ([27]) . Let ε ≥ 0 and δ ∈ (0 , 1) . Given any sampling rule, using the threshold

with the stopping rule (3) with error/confidence pair ( ε, δ ) yields a ( ε, δ ) -PAC algorithm for subGaussian distributions. The function C G is defined in (23) . It satisfies C G ( x ) ≈ x +log( x ) .

3 Fixed-confidence theoretical guarantees

To study ε -BAI in the fixed-confidence setting, we couple EB-TC ε 0 with the GLR ε stopping rule (3) using error ε ≥ 0 , confidence δ ∈ (0 , 1) and threshold (4). The resulting algorithm is ( ε, δ ) -PAC by Lemma 3. We derive upper bounds on the expected sample complexity E ν [ τ ε,δ ] both in the asymptotic regime of δ → 0 (Theorem 1) and for finite confidence when ε = ε 0 (Theorem 2).

Theorem 1. Let ε ≥ 0 , ε 0 > 0 and ( β, δ ) ∈ (0 , 1) 2 . Combined with GLR ε stopping (3) , the EB-TC ε 0 algorithm is ( ε, δ ) -PAC and it satisfies that, for all ν ∈ D K with mean µ such that | i ⋆ ( µ ) | = 1 ,

̸

While Theorem 1 holds for all sub-Gaussian distributions, it is particularly interesting for Gaussian ones, in light of Lemma 1. When choosing ε = ε 0 (i.e. D ε 0 ,ε 0 ( µ ) = 1 ), Theorem 1 shows that EBTC ε 0 is asymptotically optimal for Gaussian bandits when using IDS proportions and asymptotically β -optimal when using fixed proportions β . We also note that Theorem 1 is not conflicting with the lower bound of Lemma 1, as shown in Lemma 11 in Appendix C. Empirically we observe that the empirical stopping time can be drastically worse when taking ε 0 < ε , and close to the optimal one when ε 0 > ε (Figures 5, 6 and 7 in Appendix J.2.1).

Until recently [40], proving asymptotic optimality of Top Two algorithms with adaptive choice β was an open problem in BAI. In this work, we prove that their choice of IDS proportions also yield asymptotically optimal algorithms for ε -BAI. While the proof of Theorem 1 assumes the existence of a unique best arm, it holds for instances having sub-optimal arms with the same mean. This is an improvement compared to existing asymptotic guarantees on Top Two algorithms which rely on the assumption that the means of all arms are different [32, 36, 21]. The improvement is possible thanks to the regularization induced by the slack ε 0 > 0 .

While asymptotic optimality in the ε -BAI setting was already achieved for various algorithms (e.g. ε -Track-and-Stop (TaS) [16], Sticky TaS [10] or L ε BAI [19]), none of them obtained nonasymptotic guarantees. Despite their theoretical interest, asymptotic results provide no guarantee on the performance for moderate δ . Furthermore, asymptotic results on Top Two algorithms require a unique best arm regardless of the considered error ε : reaching asymptotic ( β -)optimality implicitly

means that the algorithm eventually allocates samples in an optimal way that depends on the identity of the unique best arm, and that requires the unique best arm to be identified. As our focus is ε -BAI, our guarantees should only require that one of the ε -good arms is identified and should hold for instances having multiple best arms. The upper bound should scale with ε -2 0 instead of ∆ -2 min when ∆ min is small. Theorem 2 satisfies these requirements.

Theorem 2. Let δ ∈ (0 , 1) and ε 0 > 0 . Combined with GLR ε 0 stopping (3) , the EB-TC ε 0 algorithm with fixed β = 1 / 2 is ( ε 0 , δ ) -PAC and satisfies that, for all ν ∈ D K with mean µ ,

where T µ,ε 0 ( δ, ˜ ε ) and S µ,ε 0 (˜ ε ) are defined by

̸

where γ ∈ (0 , 1 / 2] is an analysis parameter and h 1 ( y, z ) ≈ z + y log( z + y log( y )) as in Lemma 51. T ε 0 , 1 / 2 ( µ, i ) and w ε 0 , 1 / 2 ( µ, i ) are defined in (1) and

The upper bound on E ν [ τ ε 0 ,δ ] involves a δ -dependent term T µ,ε 0 ( δ, ˜ ε ) and a δ -independent term S µ,ε 0 (˜ ε ) . The choice of ˜ ε influences the compromise between those, and the infimum over ˜ ε means that our algorithm benefits from the best possible trade-off. In the asymptotic regime, we take ˜ ε = 0 and γ → 0 and we obtain lim δ → 0 E ν [ τ ε 0 ,δ ] / log(1 /δ ) ≤ 2 | i ⋆ ( µ ) | T ε 0 , 1 / 2 ( µ ) . When | i ⋆ ( µ ) | = 1 , we recover the asymptotic result of Theorem 1 up to a multiplicative factor 2 . For multiple best arms, the asymptotic sample complexity is at most a factor 2 | i ⋆ ( µ ) | from the β -optimal one.

Given a finite confidence, the dominant term will be S µ,ε 0 (˜ ε ) . For ˜ ε = 0 , we show that H µ,ε 0 (0) = O ( K min { ∆ min , ε 0 } -2 ) , hence we should consider ˜ ε > 0 to avoid the dependency in ∆ min . For ˜ ε = ε 0 , there exists instances such that max i ∈I ε 0 ( µ ) T ε 0 , 1 / 2 ( µ, i ) is arbitrarily large, hence S µ,ε 0 ( ε 0 ) will be very large as well. The best trade-off is attained in the interior of the interval (0 , ε 0 ) . For ˜ ε = ε 0 / 2 , Lemma 10 shows that T ε 0 , 1 / 2 ( µ, i ) = O ( K/ε 2 0 ) for all i ∈ I ε 0 / 2 ( µ ) and H µ,ε 0 ( ε 0 / 2) = O ( K/ε 2 0 ) . Therefore, we obtain an upper bound O ( |I ε 0 / 2 ( µ ) | Kε -2 0 log ε -1 0 ) .

̸

Likewise, Lemma 10 shows that min j = i w ε 0 , 1 / 2 ( µ, i ) j ≥ (16( K -2) + 2) -1 for all i ∈ I ε 0 / 2 ( µ ) . While the dependency in a µ,ε 0 ( ε 0 / 2) is milder in ε -BAI than in BAI (as it is bounded away from 0 ), we can improve it by using a refined analysis (see Appendix E). Introduced in [20], this method allows to clip min j = i w ε 0 , 1 / 2 ( µ, i ) j by a fixed value x ∈ [0 , ( K -1) -1 ] for all i ∈ I ˜ ε ( µ ) .

̸

Comparison with existing upper bounds The LUCB algorithm [24] has a structure similar to a Top Two algorithm, with the differences that LUCB samples both the leader and the challenger and that it stops when the gap between the UCB and LCB indices is smaller than ε 0 . As LUCB satisfies E µ [ τ ε 0 ,δ ] ≤ 292 H ε 0 ( µ ) log( H ε 0 ( µ ) /δ ) + 16 where H ε 0 ( µ ) = ∑ i (max { ∆ i , ε 0 / 2 } ) -2 , it enjoys better scaling than EB-TC ε 0 for finite confidence. However, since the empirical allocation of LUCB is not converging towards w ε 0 , 1 / 2 ( µ ) , it is not asymptotically 1 / 2 -optimal. While LUCB has better moderate confidence guarantees, there is no hope to prove anytime performance bounds since LUCB indices depends on δ . In contrast, EB-TC ε 0 enjoys such guarantees (see Section 4).

Key technical tool for the non-asymptotic analysis We want to ensure that EB-TC ε 0 eventually selects only ε -good arms as leader, for any error ε ≥ 0 . Our proof strategy is to show that if the leader is not an ε -good arm and empirical means do not deviate too much from the true means, then either the current leader or the current challenger was selected as leader or challenger less than a given quantity. We obtain a bound on the total number of times that can happen.

̸

Lemma 4. Let δ ∈ (0 , 1] and n > K . Let T n ( i ) := ∑ j = i ( T n ( i, j ) + T n ( j, i )) be the number of times arm i was selected in the leader/challenger pair. Assume there exists a sequence of events ( A t ( n, δ )) n ≥ t>K and positive reals ( D i ( n, δ )) i ∈ [ K ] such that, for all t ∈ { K +1 , . . . , n } , under the event A t ( n, δ ) ,

To control the deviation of the empirical means and empirical gaps to their true value, we use a sequence of concentration events ( E n,δ ) n>T defined in Lemma 45 (Appendix G.2) such that P ν ( E ∁ n,δ ) ≤ K 2 δn -s where s ≥ 0 and δ ∈ (0 , 1] . For the EB-TC ε 0 algorithm with fixed β = 1 / 2 , we prove that, under E n,δ , { B EB t / ∈ I ε ( µ ) } is a 'bad' event satisfying the assumption of Lemma 4. This yields Lemma 5, which essentially says that the leader is an ε -good arm except for a logarithmic number of rounds.

Lemma 5. Let δ ∈ (0 , 1] , n > K and ε ≥ 0 . Under the event E n,δ , we have

where f 2 ( n, δ ) = log(1 /δ ) + (2 + s ) log n and H µ,ε 0 ( ε ) is defined in (5) .

Noticeably, Lemma 5 holds for any ε ≥ 0 even when there are multiple best arms. As expected the number of times the leader is not among the ε 0 -good arms depends on H µ,ε 0 ( ε 0 ) = O ( K/ε 2 0 ) . The number of times the leader is not among the best arms depends on H µ,ε 0 (0) = O ( K (min { ∆ min , ε 0 } ) -2 ) .

Time-varying slack Theorem 1 shows the asymptotic optimality of the EB-TC ε 0 algorithm with IDS for ε 0 -BAI (where ε 0 > 0 ). To obtain optimality for BAI, we consider time-varying slacks ( ε n ) n , where ( ε n ) n is decreasing, ε n > 0 and ε n → + ∞ 0 . A direct adaptation of our asymptotic analysis on E ν [ τ 0 ,δ ] (see Appendix D), regardless of the choice of ( ε n ) n , one can show that using GLR 0 stopping, the EB-TC ( ε n ) n algorithm with IDS is (0 , δ ) -PAC and is asymptotically optimal in BAI. Its empirical performance is however very sensitive to the choice of ( ε n ) n (Appendix J.2.3).

4 Beyond fixed-confidence guarantees

Could an algorithm analyzed in the fixed-confidence setting be used for the fixed-budget or even anytime setting? This question is especially natural for EB-TC ε 0 , which does not depend on the confidence parameter δ . Yet its answer is not obvious, as it is known that algorithms that have optimal asymptotic guarantees in the fixed-confidence setting can be sub-optimal in terms of error probability. Indeed [28] prove in their Appendix C that for any asymptotically optimal (exact) BAI algorithm, there exists instances in which the error probability cannot decay exponentially with the horizon, which makes them worse than the (minimax optimal) uniform sampling strategy [5].

Their argument also applies to β -optimal algorithms, hence to EB-TC 0 with β = 1 / 2 . However, whenever ε 0 is positive, Theorem 3 reveals that the error probability of EB-TC ε 0 always decays exponentially, which redeems the use of optimal fixed-confidence algorithms for a relaxed BAI problem in the anytime setting. Going further, this result provides an anytime bound on the probability to recommend an arm that is not ε -optimal, for any error ε ≥ 0 . This bound involves instancedependent complexities depending solely on the gaps in µ . To state it, we define C µ := |{ µ i | i ∈ [ K ] }| as the number of distinct arm means in µ and let C µ ( i ) := { k ∈ [ K ] | µ ⋆ -µ k = ∆ i } be the set of arms having mean gap ∆ i where the gaps are sorted by increasing order 0 = ∆ 1 < ∆ 2 < · · · < ∆ C µ . For all ε ≥ 0 , let i µ ( ε ) = i if ε ∈ [∆ i , ∆ i +1 ) (with the convention ∆ C µ +1 = + ∞ ). Theorem 3 shows that the exponential decrease of P ν (ˆ ı n / ∈ I ε ( µ )) is linear.

Theorem 3. (see Theorem 6 in Appendix F) Let ε 0 > 0 . The EB-TC ε 0 algorithm with fixed proportions β = 1 / 2 satisfies that, for all ν ∈ D K with mean µ , for all ε ≥ 0 , for all n > 5 K 2 / 2 ,

where p ( x ) = x -log x and ( H i ( µ, ε 0 )) i ∈ [ C µ -1] are such that H 1 ( µ, ε 0 ) = K (2∆ -1 min +3 ε -1 0 ) 2 and K/ ∆ -2 i +1 ≤ H i ( µ, ε 0 ) ≤ K min j ∈ [ i ] max { 2∆ -1 j +1 , 2 ∆ j /ε 0 +1 ∆ i +1 -∆ j +3 ε -1 0 } 2 for all i > 1 .

This bound can be compared with the following uniform ε -error bound of the strategy using uniform sampling and recommending the empirical best arm:

Recalling that the quantity H i ( µ, ε 0 ) in Theorem 3 is always bounded from below by 2 K ∆ -1 i +1 , we get that our upper bound is larger than the probability of error of the uniform strategy, but the two should be quite close. For example for ε = 0 , we have

Even if they provide a nice sanity-check for the use of a sampling rule with optimal fixed-confidence guarantees for ε 0 -BAI in the anytime regime, we acknowledge that these guarantees are far from optimal. Indeed, the work of [41] provides tighter anytime uniform ε -error probability bounds for two algorithms: an anytime version of Sequential Halving [25] using a doubling trick (called DSH), and an algorithm called Bracketting Sequential Halving, that is designed to tackle a very large number of arms. Their upper bounds are of the form P ν (ˆ ı n / ∈ I ε ( µ )) ≤ exp( -Θ( n/H ( ε ))) with H ( ε ) = 1 g ( ε/ 2) max i ≥ g ( ε )+1 i ∆ 2 i where g ( ε ) = |{ i ∈ [ K ] | µ i ≥ µ ⋆ -ε }| . Therefore, they can be much smaller than K ∆ -2 i µ ( ε )+1 .

The BUCB algorithm of [26] is also analyzed for any level of error ε , but in a different fashion. The authors provide bounds on its ( ε, δ ) -unverifiable sample complexity , defined as the expectation of the smallest stopping time ˜ τ satisfying P ( ∀ t ≥ ˜ τ, ˆ ı n ∈ I ε ( µ )) ≥ 1 -δ . This notion is different from the sample complexity we use in this paper, which is sometimes called verifiable since it is the time at which the algorithm can guarantee that its error probability is less than δ . Interestingly, to prove Theorem 3 we first prove a bound on the unverifiable sample complexity of EB-TC ε 0 which is valid for all ( ε, δ ) , neither of which are parameters of the algorithm. More precisely, we prove that P ν ( ∀ n > U i µ ( ε ) ,δ ( µ, ε 0 ) , ˆ ı n ∈ I ε ( µ ) ) ≥ 1 -δ for U i,δ ( µ, ε 0 ) = δ → 0 8 H i ( µ, ε 0 ) log(1 /δ ) + O (log log(1 /δ )) . As this statement is valid for all δ ∈ (0 , 1) , applying it for each n to δ n such that U i µ ( ε ) ,δ n ( µ, ε 0 ) = n , we obtain Theorem 3. We remark that such a trick cannot be applied to BUCB to get uniform ε -error bounds for any time, as the algorithm does depend on δ .

Simple regret As already noted by [41], uniform ε -error bounds easily yield simple regret bounds. We state in Corollary 1 the one obtained for EB-TC ε 0 . As a motivation to derive simple regret bounds, we observe that they readily translate to bounds on the cumulative regret for an agent observing the stream of recommendations (ˆ ı n ) and playing arm ˆ ı n . An exponentially decaying simple regret leads to a constant cumulative regret in this decoupled exploration/exploitation setting [2, 33].

Corollary 1. Let ε 0 > 0 . Let p ( x ) and ( H i ( µ, ε 0 )) i ∈ [ C µ -1] be defined as in Theorem 3. The EB-TC ε 0 algorithm with fixed β = 1 / 2 satisfies that, for all ν ∈ D K with mean µ , for all n > 5 K 2 / 2 ,

Following the discussion above, this bound is not expected to be state-of-the-art, it rather justifies that EB-TC ε 0 with ε 0 > 0 is not too much worse than the uniform sampling strategy. Yet, as we shall see in our experiments, the practical story is different. In Section 5, we compare the simple regret of EB-TC ε 0 to that of DSH in synthetic experiments with a moderate number of arms, revealing the superior performance of EB-TC ε 0 .

5 Experiments

We assess the performance of the EB-TC ε 0 algorithm on Gaussian instances both in terms of its empirical stopping time and its empirical simple regret, and we show that it perform favorably compared to existing algorithms in both settings. For the sake of space, we only show the results for large sets of arms and for a specific instance with | i ⋆ ( µ ) | = 2 .

Empirical stopping time We study the impact of large sets of arms (up to K = 1000 ) in ε -BAI for ( ε, δ ) = (0 . 1 , 0 . 01) on the ' α = 0 . 3 ' scenario of [18] which sets µ i = 1 -(( i -1) / ( K -1)) α for all i ∈ [ K ] . EB-TC ε 0 with IDS and slack ε 0 = ε is compared to existing ε -BAI algorithms having low computational cost. This precludes algorithms such as ε -Track-and-Stop (TaS) [16], Sticky TaS [10] or ε -BAI adaptation of FWS [39] and DKM [11]. In Appendix J.2.2, we compare EB-TC ε to those algorithms on benchmarks with smaller number of arms. We show that EB-TC ε performs on par with ε -TaS and ε -FWS, but outperforms ε -DKM. As Top Two benchmarks with fixed β = 1 / 2 , we consider T3C [36], EB-TCI [21] and TTUCB [20]. To provide a fair comparison, we adapt them to tackle ε -BAI by using the stopping rule (3) and by adapting their sampling rule to use the TC ε challenger from (2) (with a penalization log N n,i for EB-TCI). We use the heuristic threshold c ( n, δ ) = log((1 + log n ) /δ ) . While it is too small to ensure the ( ε, δ ) -PAC property, it still yields an empirical error which is several orders of magnitude lower than δ . Finally, we compare with LUCB [24] and uniform sampling. For a fair comparison, LUCB uses √ 2 c ( n -1 , δ ) /N n,i as bonus, which is also too small to yield valid confidence intervals. Our results are averaged over 100 runs, and the standard deviations are displayed. In Figure 2(a), we see that EB-TC ε performs on par with the ε -T3C heuristic, and significantly outperforms the other algorithms. While the scaling in K of ε -EB-TCI and LUCB appears to be close to the one of EB-TC ε , ε -TTUCB and uniform sampling obtain a worse one. Figure 2(a) also reveals that the regularization ensured by the TC ε challenger is sufficient to ensure enough exploration, hence other exploration mechanisms are superfluous (TS/UCB leader or TCI challenger).

Anytime empirical simple regret The EB-TC ε 0 algorithm with fixed β = 1 / 2 and ε 0 = 0 . 1 is compared to existing algorithms on the instance µ = (0 . 6 , 0 . 6 , 0 . 55 , 0 . 45 , 0 . 3 , 0 . 2) from [16], which has two best arms. As benchmark, we consider Doubling Successive Reject (DSR) and Doubling Sequential Halving (DSH), which are adaptations of the elimination based algorithms SR [1] and SH [25]. SR eliminates one arm with the worst empirical mean at the end of each phase, and SH eliminated half of them but drops past observations between each phase. These doubling-based algorithms have empirical error decreasing by steps: they change their recommendation only before they restart. In Figure 2(b), we plot the average of the simple regret over 10000 runs and the standard deviation of that average (which is too small to see clearly). We observe that EB-TC ε 0 outperforms uniform sampling, as well as DSR and DSH, which both perform worse due to the dropped observations. The favorable performance of EB-TC ε 0 is confirmed on other instances from [16], and for 'two-groups' instances with varying | i ⋆ ( µ ) | (see Figures 10 and 12).

Supplementary experiments Extensive experiments and implementation details are available in Appendix J. In Appendix J.2.1, we compare the performance of EB-TC ε 0 with different slacks ε 0 for IDS and fixed β = 1 / 2 . In Appendix J.2.2, we demonstrate the good empirical performance of EB-TC ε 0 compared to state-of-the art methods in the fixed-confidence ε -BAI setting, compared to DSR and DSH for the empirical simple regret, and compared to SR and SH for the probability of 0 -error in the fixed-budget setting (Figure 13). We consider a wide range of instances: random ones, benchmarks from the literature [18, 16] and 'two-groups' instances with varying | i ⋆ ( µ ) | .

6 Perspectives

We have proposed the EB-TC ε 0 algorithm, which is easy to understand and implement. EB-TC ε 0 is the first algorithm to be simultaneously asymptotically optimal in the fixed-confidence ε 0 -BAI setting (Theorem 1), have finite-confidence guarantees (Theorem 2), and have also anytime guarantees on the probability of error at any level ε (Theorem 3), hence on the simple regret (Corollary 1). Furthermore, we have demonstrated that the EB-TC ε 0 algorithm achieves superior performance compared to other algorithms, in benchmarks where the number of arms is moderate to large. In future work, we will investigate its possible adaptation to the data-poor regime of [41] in which the number of arms is so large that any algorithm sampling each arm once is doomed to failure.

While our results hold for general sub-Gaussian distributions, the EB-TC ε 0 algorithm with IDS and slack ε 0 > 0 only achieves asymptotic optimality for ε 0 -BAI with Gaussian bandits. Even though IDS has been introduced by [40] for general single-parameter exponential families, it is still an open problem to show asymptotic optimality for distributions other than Gaussians. While our non-asymptotic guarantees on E ν [ τ ε 0 ,δ ] and E ν [ µ ⋆ -µ ˆ ı n ] were obtained for the EB-TC ε 0 algorithm with fixed β = 1 / 2 , we observed empirically better performance when using IDS. Deriving similar (or better) non-asymptotic guarantees for IDS is an interesting avenue for future work.

Finally, the EB-TC ε 0 algorithm is a promising method to tackle structured bandits. Despite the existence of heuristics for settings such as Top-k identification [40], it is still an open problem to efficiently adapt Top Two approaches to cope for other structures such as ε -BAI in linear bandits.

Acknowledgments and Disclosure of Funding

Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr). This work has been partially supported by the THIA ANR program 'AI_PhD@Lille'. The authors acknowledge the funding of the French National Research Agency under the project BOLD (ANR-19-CE23-0026-04) and the project FATE (ANR22-CE23-0016-01).

References

J.-Y. Audibert, S. Bubeck, and R. Munos. Best Arm Identification in Multi-armed Bandits. In Conference on Learning Theory , 2010.

O. Avner, S. Mannor, and O. Shamir. Decoupling exploration and exploitation in multi-armed bandits. In Proceedings of the 29th International Coference on International Conference on Machine Learning , pages 1107-1114, 2012.

A. Barrier, A. Garivier, and T. Kocák. A non-asymptotic approach to best-arm identification for gaussian bandits. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , 2022.

S. Bubeck and N. Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Fondations and Trends in Machine Learning , 5(1):1-122, 2012.

S. Bubeck, R. Munos, and G. Stoltz. Pure Exploration in Finitely Armed and Continuous Armed Bandits. Theoretical Computer Science 412, 1832-1852 , 412:1832-1852, 2011.

L. Chen, J. Li, and M. Qiao. Towards instance optimal bounds for best arm identification. Conference on Learning Theory , 2017.

L. Chen, J. Li, and M. Qiao. Nearly instance optimal sample complexity bounds for top-k arm selection. In Artificial Intelligence and Statistics , 2017.

Y. Chen and I. O. Ryzhov. Balancing optimal large deviations in sequential selection. Management Science , 69(6):3457-3473, 2023.

R. Degenne. Impact of structure on the design and analysis of bandit algorithms . PhD thesis, Université de Paris, 2019.

R. Degenne and W. M. Koolen. Pure exploration with multiple correct answers. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

R. Degenne, W. M. Koolen, and P. Ménard. Non-asymptotic pure exploration by solving games. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

R. Degenne, H. Shao, and W. M. Koolen. Structure adaptive algorithms for stochastic bandits. In International Conference on Machine Learning (ICML) , 2020.

E. Even-Dar, S. Mannor, and Y. Mansour. Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems. Journal of Machine Learning Research , 7:1079-1105, 2006.

V. Gabillon, M. Ghavamzadeh, and A. Lazaric. Best Arm Identification: A Unified Approach to Fixed Budget and Fixed Confidence. In Advances in Neural Information Processing Systems , 2012.

A. Garivier and E. Kaufmann. Optimal best arm identification with fixed confidence. In Proceedings of the 29th Conference On Learning Theory , 2016.

A. Garivier and E. Kaufmann. Non-asymptotic sequential tests for overlapping hypotheses and application to near optimal arm identification in bandit models. Sequential Analysis , 40(1): 61-96, 2021.

K. Jamieson, M. Malloy, R. Nowak, and S. Bubeck. lil'UCB: an Optimal Exploration Algorithm for Multi-Armed Bandits. In Proceedings of the 27th Conference on Learning Theory , 2014.

K. G. Jamieson and R. D. Nowak. Best-arm identification algorithms for multi-armed bandits in the fixed confidence setting. In Conference on Information Sciences and Systems (CISS) , 2014.

M. Jourdan and R. Degenne. Choosing answers in ε -best-answer identification for linear bandits. In International Conference on Machine Learning (ICML) , 2022.

M. Jourdan and R. Degenne. Non-asymptotic analysis of a ucb-based top two algorithm. arXiv preprint arXiv:2210.05431 , 2022.

M. Jourdan, R. Degenne, D. Baudry, R. De Heide, and E. Kaufmann. Top two algorithms revisited. Advances in Neural Information Processing Systems , 2022.

M. Jourdan, R. Degenne, and E. Kaufmann. Dealing with unknown variances in best-arm identification. International Conference on Algorithmic Learning Theory , 2023.

K.-W. Jun and R. Nowak. Anytime exploration for multi-armed bandits using confidence information. In International Conference on Machine Learning (ICML) , 2016.

S. Kalyanakrishnan, A. Tewari, P. Auer, and P. Stone. PAC subset selection in stochastic multi-armed bandits. In International Conference on Machine Learning (ICML) , 2012.

Z. Karnin, T. Koren, and O. Somekh. Almost optimal Exploration in multi-armed bandits. In International Conference on Machine Learning (ICML) , 2013.

J. Katz-Samuels and K. Jamieson. The true sample complexity of identifying good arms. In International Conference on Artificial Intelligence and Statistics , 2020.

E. Kaufmann and W. M. Koolen. Mixture martingales revisited with applications to sequential tests and confidence intervals. Journal of Machine Learning Research , 22(246):1-44, 2021.

J. Komiyama, T. Tsuchiya, and J. Honda. Minimax optimal algorithms for fixed-budget best arm identification. In Advances in Neural Information Processing Systems , 2022.

T. Lattimore and C. Szepesvari. Bandit Algorithms . Cambridge University Press, 2019.

A. Locatelli, M. Gutzeit, and A. Carpentier. An optimal algorithm for the thresholding bandit problem. In International Conference on Machine Learning (ICML) , 2016.

S. Mannor and J. Tsitsiklis. The Sample Complexity of Exploration in the Multi-Armed Bandit Problem. Journal of Machine Learning Research , pages 623-648, 2004.

C. Qin, D. Klabjan, and D. Russo. Improving the expected improvement algorithm. In Advances in Neural Information Processing Systems 30 (NIPS) , 2017.

C. Rouyer and Y. Seldin. Tsallis-inf for decoupled exploration and exploitation in multi-armed bandits. In Conference on Learning Theory , pages 3227-3249. PMLR, 2020.

D. Russo. Simple Bayesian algorithms for best arm identification. In Proceedings of the 29th Conference on Learning Theory (COLT) , 2016.

S. Sabato. Epsilon-best-arm identification in pay-per-reward multi-armed bandits. Advances in Neural Information Processing Systems , 32, 2019.

X. Shang, R. de Heide, E. Kaufmann, P. Ménard, and M. Valko. Fixed-confidence guarantees for bayesian best-arm identification. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2020.

D. Shin, M. Broadie, and A. Zeevi. Tractable sampling strategies for ordinal optimization. Operations Research , 66(6):1693-1712, 2018.

M. Simchowitz, K. Jamieson, and B. Recht. The simulator: Understanding adaptive sampling in the moderate-confidence regime. In Conference on Learning Theory , 2017.

P.-A. Wang, R.-C. Tzeng, and A. Proutiere. Fast pure exploration via frank-wolfe. Advances in Neural Information Processing Systems , 2021.

W. You, C. Qin, Z. Wang, and S. Yang. Information-directed selection for top-two algorithms. In The Thirty Sixth Annual Conference on Learning Theory , pages 2850-2851. PMLR, 2023.

Y. Zhao, C. J. Stephens, C. Szepesvári, and K.-S. Jun. Revisiting simple regret: Fast rates for returning a good arm. In 40th International Conference on Machine Learning (ICML) , 2023.

A Outline

The appendices are organized as follows:

Notation are summarized in Appendix B.

In Appendix C, we show how the characteristic times and optimal allocations for ε -BAI can be reduced to the ones of a BAI problem with modified instances (Lemma 9).

The asymptotic upper bound on the expected sample complexity of the EB-TC ε 0 algorithm when combined with the stopping rule (4) is proven in Appendix D (Theorem 1).

The non-asymptotic upper bound on the expected sample complexity of the EB-TC ε 0 algorithm when combined with the stopping rule (4) is proven in Appendix E (Theorem 5).

The upper bound on the probability of error and on the simple regret of the EB-TC ε 0 algorithm is proven in Appendix F (Theorem 6), as well as the upper bound on its unverifiable expected sample complexity (Theorem 7).

Appendix G gathers concentration results used for the calibration of the stopping rule (Lemma 3) and to control the empirical means and gaps.

Appendix H gathers existing and new technical results which are used for our proofs.

In Appendix I, we discuss the multiplicative notion of ε -goodness, propose the EB-TC m ε 0 algorithm and show asymptotic guarantees on the expected sample complexity when combined with the appropriate GLR stopping rule (Theorem 8).

Implementation details and additional experiments are presented in Appendix J.

B Notation

We recall some commonly used notation: the set of integers [ n ] := { 1 , · · · , n } , the complement X ∁ and interior ˚ X of a set X , the indicator function 1 ( X ) of an event, the probability P ν and the expectation E ν under distribution ν , Landau's notation o , O , Ω and Θ , the ( K -1) -dimensional probability simplex △ K := { w ∈ R K + | w ≥ 0 , ∑ i ∈ [ K ] w i = 1 } . The functions C G and g G are defined in (23), W -1 in Lemma 49, h 1 in Lemma 51, ζ is the Riemann ζ function. While Table 1 gathers problem-specific notation, Table 2 groups notation for the algorithms.

C Characteristic Times

In Appendix C, we first recall properties satisfied by the ( β -)characteristic times and ( β -)optimal allocation for the BAI setting with Gaussian bandits. Then, we show how a ε -BAI problem can be reduced to a BAI one on a modified (and easier) instance.

BAI setting Let ν ∈ D K with mean µ ∈ R K . The ( β -)characteristic times for the fixed-confidence BAI setting with Gaussian bandits N ( µ, 1) are defined as

̸

see [15, 34] for example. Let us denote by w ⋆ β ( µ ) and w ⋆ ( µ ) their maximizer, which we will refer to as ( β -)optimal allocations. Note that ( T ⋆ ( µ ) , T ⋆ β ( µ )) = ( T 0 ( µ ) , T 0 ,β ( µ )) and ( w ⋆ ( µ ) , w ⋆ β ( µ )) = ( w 0 ( µ ) , w 0 ,β ( µ )) .

We recall in the following some fundamental results on those quantities ( β -)characteristic times and ( β -)optimal allocations. Lemma 6 proves that the ( β -)optimal allocations are unique, have strictly positive elements, and shows a worst-case inequality T ⋆ 1 / 2 ( µ ) ≤ 2 T ⋆ ( µ ) . Lemma 6 also holds for any single-parameter exponential family of distributions and for the non-parametric family of bounded distributions [21].

Lemma 6 ([15, 34, 21]) . If i ⋆ ( µ ) is a singleton and β ∈ (0 , 1) , then w ⋆ ( µ ) and w ⋆ β ( µ ) are singletons, i.e. the optimal allocations are unique, and w ⋆ ( µ ) i > 0 and w ⋆ β ( µ ) i > 0 for all i ∈ [ K ] . Moreover, we have T ⋆ 1 / 2 ( µ ) ≤ 2 T ⋆ ( µ ) and with β ⋆ = w ⋆ i ⋆ ( µ ) . Additionally, we have H ( µ ) ≤ T ⋆ ( µ ) ≤ 2 H ( µ ) where H ( µ ) = 2 ∑ i ∈ [ K ] ∆ -2 i where ∆ i ⋆ = ∆ min .

For Gaussian bandits, Lemma 7 shows that the ( β -)characteristic times and ( β -)optimal allocations can be obtained by solving a simpler optimization problem, which is amenable to faster implementation (e.g. Newton's iterates). It also provides upper and lower bound on the optimal allocation of the best arm, and gives hints that a tighter worst-case inequality T ⋆ 1 / 2 ( µ ) ≤ r K T ⋆ ( µ ) might hold.

̸

Lemma 7 ([3, 20]) . Assume that i ⋆ ( µ ) = { i ⋆ } . Let r ( µ ) and r β ( µ ) be the solution of ψ µ ( r ) = 0 and φ µ,β ( r ) = 0 with, for all r ∈ (1 / min i = i ⋆ ( µ i ⋆ -µ i ) 2 , + ∞ ) ,

̸

̸

Then, ψ µ and φ µ,β are convex and decreasing. Moreover,

̸

̸

Additionally, for all i = i ⋆ , we have βw ⋆ β ( µ ) -1 i = βT ⋆ β ( µ )( µ i ⋆ -µ i ) 2 / 2 -1 . For K = 2 , w ⋆ ( µ ) = (0 . 5 , 0 . 5) and T ⋆ ( µ ) = T ⋆ 1 / 2 ( µ ) = 8( µ 1 -µ 2 ) 2 . For K ≥ 3 , we have

̸

Let r K = 2 K/ (1 + √ K -1) 2 , j ⋆ ∈ arg min j = i ⋆ µ i ⋆ -µ j and x ∈ R K -2 such that x j = µ i ⋆ -µ j µ i ⋆ -µ j ⋆ . Then, we have T ⋆ 1 / 2 ( µ ) /T ⋆ ( µ ) = Ω( x ) (i.e. independent of µ i ⋆ and µ i ⋆ -µ j ⋆ ) and

Lemma 8 gathers necessary conditions on the ( β -)optimal allocations at the equilibrium such as equality of the transportation costs, and a link between the squared allocation which was refered to as overall balance in [40].

̸

Lemma 8 ([3, 40]) . Assume that i ⋆ ( µ ) = { i ⋆ } and β ∈ (0 , 1) . Then, for all i = i ⋆ ,

Moreover, we have

̸

Reduction of an ε -BAI problem to a BAI problem Lemma 9 gives a reduction of a ε -BAI problem to a BAI one on a modified instance, which is easier. Thanks to Lemma 9, it is possible to leverage existing results on T ⋆ ( µ ) , T ⋆ β ( µ ) , w ⋆ ( µ ) , w ⋆ β ( µ ) (such as Lemmas 6, 7 and 8) in order to study T ε ( µ ) , T ε,β ( µ ) , T ε,β ( µ, i ) , w ε ( µ ) , w ε,β ( µ ) and w ε,β ( µ, i ) .

̸

Lemma 9. Let µ ∈ R K , ε ≥ 0 and ˜ ε ∈ [0 , ε ] and β ∈ (0 , 1) . For all i ∈ I ˜ ε ( µ ) , we define µ ε ( i ) as µ ε ( i ) j = µ j -ε for all j = i and µ ε ( i ) i = µ i . Then, for all i ∈ I ˜ ε ( µ ) , T ε,β ( µ, i ) = T ⋆ β ( µ ε ( i )) and w ε,β ( µ, i ) = w ⋆ β ( µ ε ( i )) . Moreover, for all i ⋆ ∈ i ⋆ ( µ ) ,

Moreover, we have w ε ( µ ) = ⋃ i ⋆ ∈ i ⋆ ( µ ) w ⋆ ( µ ε ( i ⋆ )) and w ε,β ( µ ) = ⋃ i ⋆ ∈ i ⋆ ( µ ) w ⋆ β ( µ ε ( i ⋆ )) .

̸

Proof. For ε = 0 , the result is direct by definition. Let ε > 0 and ˜ ε ∈ [0 , ε ] . The first part is obtained by definition of T ⋆ ε,β ( µ, i ) , w ⋆ ε,β ( µ, i ) . Let i ⋆ ∈ i ⋆ ( µ ) and µ ε ( i ⋆ ) defined as above, hence i ⋆ ( µ ε ( i ⋆ )) = { i ⋆ } . For all i ∈ I ε ( µ ) \ i ⋆ ( µ ) , let us denote by λ ( i ⋆ ,i ) ε the instance such that λ ( i ⋆ ,i ) ε,i = µ i and λ ( i ⋆ ,i ) ε,j = µ j -ε for all j = i . If µ i ⋆ -µ i = ε then i ⋆ ( λ ( i ⋆ ,i ) ε ) = { i } ∪ i ⋆ ( µ ) \ { i ⋆ } , otherwise i ⋆ ( λ ( i ⋆ ,i ) ε ) = { i } . We consider the permutation σ that swaps arm i with arm i ⋆ . By symmetry, we have T ⋆ ( λ ( i ⋆ ,i ) ε ) = T ⋆ ( σ ( λ ( i ⋆ ,i ) ε )) . Moreover, we have that the gaps of σ ( λ ( i ⋆ ,i ) ε ) are all strictly smaller than the gaps of µ ε since ε ≥ ∆ i > 0 . Therefore, Lemma 11 of Barrier et al (2022) yields that T ⋆ ( σ ( λ ( i ⋆ ,i ) ε )) > T ⋆ ( µ ε ) . We have proved that

By symmetry T ⋆ ( µ ε ( i ⋆ )) is constant for all i ⋆ ∈ i ⋆ ( µ ) , hence we have shown that

It also shows that w ε ( µ ) = ⋃ i ⋆ ∈ i ⋆ ( µ ) w ⋆ ( µ ε ( i ⋆ )) . The same reasoning yields the result for T ε,β ( µ ) and w ε,β ( µ ) .

0

ε -BAI problems cannot be arbitrarily difficult Lemma 10 shows that, for all i ∈ I ε/ 2 ( µ ) , T ε, 1 / 2 ( µ, i ) cannot be arbitrarily large and min j = i w ε 0 , 1 / 2 ( µ, i ) j cannot be arbitrarily small. The proof relies on Lemma 9 and existing results for BAI.

̸

̸

Lemma 10. Let ε > 0 and µ ∈ R K . Then, for all i ∈ I ε/ 2 ( µ ) , we have T ε, 1 / 2 ( µ, i ) ≤ 32 K/ε 2 and min j = i w ε, 1 / 2 ( µ, i ) j ≥ (16( K -2) + 2) -1 .

̸

Proof. Let i ∈ I ε/ 2 ( µ ) , hence µ i ≥ µ ⋆ -ε/ 2 . Let µ ε ( i ) as in Lemma 9 which satisfies that i ⋆ ( µ ε ( i )) = { i } . Let ∆ i,j = µ i -µ j and ∆ i,i = µ i -max j = i µ j . Then, we have

where we used Lemma 6 for the two first inequality and i ∈ I ε/ 2 ( µ ) for the last one. Since ∆ j ≥ 0 , we conclude that T ε,β ( µ, i ) ≤ 32 K/ε 2 . Moreover, using Lemma 6, we have

̸

Likewise, using Lemma 9 and Lemma 7, we obtain that, for all j = i ,

where the last inequality uses what we proved above. When µ k ≤ µ j , the ratio is smaller than one. When µ k > µ j , we have µ ⋆ ≥ µ k > µ j ≥ µ ⋆ -ε/ 2 , hence µ k -µ j ≤ ε/ 2 and

-1 , which concludes the result.

Therefore, we obtain w ε, 1 / 2 ( µ ) j / 2 ≤ 8( K -2) + 1

Lemma 11 links the characteristic times for ε -BAI where ε ∈ { ε 0 , ε 1 } .

Lemma 11. Let µ ∈ R K such that | i ⋆ ( µ ) | = 1 . Let ε 0 > ε 1 . Then, for all β ∈ (0 , 1) , we have T ε 0 ( µ )(∆ min + ε 0 ) 2 ≥ T ε 1 ( µ )(∆ min + ε 1 ) 2 and T ε 0 ,β ( µ )(∆ min + ε 0 ) 2 ≥ T ε 1 ,β ( µ )(∆ min + ε 1 ) 2 . Let ε 0 < ε 1 . Then,

T

ε

(

µ

)(∆

+

ε

0

)

T

ε

1

(

µ

)(∆

+

ε

1

)

and

T

ε

0

,β

(

µ

)(∆

Proof. Let i ⋆ ( µ ) = { i ⋆ } . Let ε 0 > ε 1 . Using Lemma 9, we have

̸

̸

To conclude the first part of the first result, a sufficient condition is to show that ˜ ∆ j ( ε 1 ) ≥ ˜ ∆ j ( ε 0 ) for all j = i ⋆ . Direct manipulations show that, for all j = i ⋆ ,

̸

hence the result holds. The same proof can be used to obtain the second part of the first result. Let ε 0 < ε 1 . Using Lemma 9, we have

̸

̸

To conclude the first part of the second result, a sufficient condition is to show that ¯ ∆ j ( ε 1 ) ≥ ¯ ∆ j ( ε 0 ) for all j = i ⋆ . Direct manipulations show that, for all j = i ⋆ ,

̸

hence the result holds. The same proof can be used to obtain the second part of the second result.

2

2

+

ε

0

)

2

T

ε

,β

(

µ

)(∆

+

ε

1

)

2

1

.

D Asymptotic analysis

Let ε 0 > 0 , β ∈ (0 , 1) and δ ∈ (0 , 1) . In this section, we provide an asymptotic analysis of EB-TC ε 0 slack ε 0 > 0 combined with the stopping rule (3) with parameters ( ε 0 , δ ) . First, we detail IDS proportions in Appendix D.1. Then, we sketch the proof for fixed proportions β in Appendix D.2.

In the following, we consider a sub-Gaussian bandit with distribution ν ∈ D K having mean parameter µ ∈ R K with a unique best arm, i.e. i ⋆ ( µ ) = { i ⋆ } . We restate Theorem 1 below.

Theorem 4 (Theorem 1) . Let ε 0 > 0 , ε 1 ≥ 0 and δ ∈ (0 , 1) . Using the threshold (4) in the stopping rule (3) with slack ε 1 , the EB-TC ε 0 algorithm is ( ε 1 , δ ) -PAC. For IDS proportions, it satisfies that, for all ν ∈ D K such that | i ⋆ ( µ ) | = 1 ,

̸

Let β ∈ (0 , 1) . For fixed proportions β , it satisfies that, for all ν ∈ D K such that | i ⋆ ( µ ) | = 1 ,

̸

.

Theorem 1 provides asymptotic upper bound on the expected sample complexity of the EB-TC ε 0 algorithm when combined with the stopping rule (3). When ε 1 = ε 0 and for Gaussian distributions, combining Lemma 1 and Theorem 1 shows that the EB-TC ε 0 algorithm is asymptotically optimal for ε 0 -BAI when using IDS proportions and asymptotically β -optimal for ε 0 -BAI when using fixed proportions β . Until recently [40], proving asymptotic optimality of Top Two algorithms with adaptive choice β was an open problem in the BAI literature. Our work extends their analysis to tackle the ε -BAI setting. Compared to previous work, we removed the assumption that sub-optimal arms should have distinct means.

̸

Considering ε 1 = ε 0 can be interesting when the practitioner decides to tackle ε -BAI with an error ε which is different from the slack ε 0 used by the EB-TC ε 0 algorithm. Those situations occur when the practitioner is observing the data collected by the EB-TC ε 0 algorithm without having control over it. Theorem 1 show that the resulting sequential hypothesis testing has still asymptotic guarantees on the expected sample complexity. Quite naturally, those guarantees are not asymptotically optimal even for Gaussian distributions. The sub-optimality multiplicative gap of IDS proportions (that is, the ratio of the upper bound in Theorem 1 and the characteristic time for ε 1 -BAI, T ε 1 ( µ ) ) is

where the inequalities comes from Lemma 11 and justify that there is no contradiction with the asymptotic lower bound on the expected sample complexity (Lemma 1). The same reasoning applies to fixed proportions β by replacing T ε ( µ ) by T ε,β ( µ ) .

Concentration result For both proofs, we use the following concentration result of the empirical mean for sub-Gaussian observations (Lemma 12). This is a standard tool for the asymptotic analysis of Top Two algorithms (e.g. Lemma 3 in [32], Lemma 5 in [36] or Lemma 14 in [21]), hence we omit the proof.

Lemma 12. There exists a sub-Gaussian random variable W µ such that almost surely, for all i ∈ [ K ] and all n such that N n,i ≥ 1 ,

In particular, any random variable which is polynomial in W µ has a finite expectation.

D.1 IDS proportions

Using Lemma 9, the optimal allocation for ε 0 -BAI are defined as

̸

Since | i ⋆ ( µ ) | = 1 , let µ ε 0 = µ ε 0 ( i ⋆ ) where µ ε ( i ⋆ ) as in Lemma 9. Since we have T ε 0 ( µ ) = T ⋆ ( µ ε 0 ) and w ε 0 ( µ ) = w ⋆ ( µ ε 0 ) , Lemma 6 and 8 yield that w ε 0 ( µ ) = { w ⋆ } is a singleton with unique element denotes by w ⋆ which satisfies that min i ∈ [ K ] w ⋆ i > 0 ,

̸

and

[40] refers to (8) as the overall balance equation. The key novelty of their analysis is to show that asymptotically the empirical proportions satisfy the same equation, referred to as the empirical overall balance, when using IDS proportions with sampling to choose among the leader and the challenger. The key novelty in our work lies in proving that the empirical overall balance equation is also satisfied when using IDS proportions with K ( K -1) tracking procedures to select between the leader and the challenger (see Appendix D.1.2).

̸

Convergence time We follow an asymptotic analysis similar to the ones used in [32, 36, 21] with the improvement from [40]. Let γ > 0 . Let us define the convergence time T ε 0 ,γ , which is a random variable quantifies the number of samples required for the empirical allocations ( N n,i /N n,i ⋆ ) i = i ⋆ to be γ -close to ( w ⋆ i /w ⋆ i ⋆ ) i = i ⋆ :

̸

̸

Lemma 13 gives a sufficient condition to prove Theorem 1 for IDS proportions. The case ε 0 = ε 1 is a direct consequence of combining Theorem 13 and Proposition 14 in [40]. The proof is inspired by existing methods, e.g. Theorem 2 in [21] or Theorem 3 in [32].

Lemma 13. Let ε 0 > 0 , ε 1 ≥ 0 and δ ∈ (0 , 1) . Assume that there exists γ 1 ( µ ) > 0 such that for all γ ∈ (0 , γ 1 ( µ )] , E ν [ T ε 0 ,γ ] < + ∞ . Using the threshold (4) in the stopping rule (3) with slack ε 1 yields an algorithm such that, for all ν ∈ D K such that | i ⋆ ( µ ) | = 1 ,

̸

Proof. Let γ > 0 . Let us define by

Using that

̸

it is direct to see that for all n ≥ T γ

Therefore, for all γ ∈ (0 , γ 1 ( µ ) /K ] , E ν [ ˜ T γ,ε 0 ] < + ∞ .

Let µ ε 0 as in Lemma 9. Since we have T ε 0 ( µ ) = T ⋆ ( µ ε 0 ) and w ε 0 ( µ ) = w ⋆ ( µ ε 0 ) , we can leverage existing results from the BAI literature, e.g. Theorem 2 in [21] or Theorem 3 in [32]. The sole criterion on the stopping threshold is to be asymptotically tight (Definition 1).

̸

Definition 1. A threshold c : N × (0 , 1] → R + is said to be asymptotically tight if there exists α ∈ [0 , 1) , δ 0 ∈ (0 , 1] , functions f, ¯ T : (0 , 1] → R + and C independent of δ satisfying: (1) for all δ ∈ (0 , δ 0 ] and n ≥ ¯ T ( δ ) , then c ( n, δ ) ≤ f ( δ ) + Cn α , (2) lim sup δ → 0 f ( δ ) / log(1 /δ ) ≤ 1 and (3) lim sup δ → 0 ¯ T ( δ ) / log(1 /δ ) = 0 .

Since C G defined in (23) satisfies C G ≈ x +log( x ) , it is direct to see that

is asymptotically tight, e.g. by taking ( α, δ 0 , C ) = (1 / 2 , 1 , 4) , f ( δ ) = 2 C G ( 1 2 log ( K -1 δ )) and ¯ T ( δ ) = 1 .

Based on continuity arguments and using that E ν [ T ε 0 ,γ ] < + ∞ , the proof of Theorem 2 in [21] yields that

̸

̸

where the equality uses the condition at the equilibrium (9). This concludes the proof.

Using Lemma 13, the proof of Theorem 1 for IDS proportions boils down to showing that E ν [ T ε 0 ,γ ] < + ∞ . As in [32, 36, 21, 40], the proof is divided in several steps. First, we show that all the arms are sufficient explored for n large enough (Appendix D.1.1). Second, we prove that the empirical overall balance equation approximately holds for n large enough (Appendix D.1.2). Finally, we show convergence of the empirical ratio of proportions towards the ratio of optimal allocation, i.e. E ν [ T ε 0 ,γ ] < + ∞ (Appendix D.1.3).

D.1.1 Sufficient exploration

To upper bound the expected convergence time, as prior work we first establish sufficient exploration. Given an arbitrary threshold L ∈ R ∗ + , we define the sampled enough set and its arms with highest mean (when not empty) as

In all generality I ⋆ n is a set, yet we obtain I ⋆ n = { i ⋆ } as soon as i ⋆ ∈ S L n . We define the highly and the mildly under-sampled sets

[21] identifies the properties that the leader and the challenger should satisfy to ensure sufficient exploration. We improve on their analysis by removing the need for the distinct means assumption, and supporting IDS instead of a fixed allocation β .

Lemma 14 shows that the transportation cost is strictly positive and increases linearly. Compared to previous results, the key improvement lies in the fact that the lower bound holds for ( i, j ) ∈ I ⋆ n × S L n instead of ( i, j ) ∈ I ⋆ n × ( S L n \ I ⋆ n ) . This is due to the ε 0 regularization, which removes the need to assume that the means are distinct.

̸

Lemma 14. Let S L n and I ⋆ n as in (11). There exists L 0 with E µ [( L 0 ) α ] < + ∞ for all α > 0 such that if L ≥ L 0 , for all n such that S L n = ∅ , for all ( i, j ) ∈ I ⋆ n × S L n such that i = j , we have

Proof. Using Lemma 12 and min { N n,i , N n,j } ≥ L , we obtain

̸

where the last inequality is obtained for L ≥ L 0 = L 1 -e which is defined as

The last inequality is obtained by using Lemma 51, and we recall that h 1 is defined in Lemma 51. Since h 1 ( x, e ) ∼ x → + ∞ x log x , we have E µ [( L 0 ) α ] < + ∞ for all α > 0 by using Lemma 12 (polynomial in W µ ). Then, we have

Lemma 15 gives an upper bound on the transportation costs between a sampled enough arm and an under-sampled one.

Lemma 15. Let S L n as in (11) . There exists L 1 with E µ [( L 1 ) α ] < + ∞ for all α > 0 such that for all L ≥ L 1 and all n ∈ N ,

where D 1 > 0 is a problem dependent constant and W µ is the random variables defined in Lemma 12.

Proof. Using Lemma 12 and N n,i ≥ L ≥ N n,j ≥ 1 , we obtain

̸

Lemma 16 shows the desired property for the EB leader. Since it corresponds to Lemma 17 in [21], we omit the proof.

̸

Lemma 16 (Lemma 17 in [21]) . There exists L 2 with E ν [( L 2 ) α ] < + ∞ for all α > 0 such that if L ≥ L 2 , for all n (at most polynomial in L ) such that S L n = ∅ , B EB n ∈ S L n implies B EB n ∈ I ⋆ n .

Lemma 17 show that the desired property for the TC challenger. Compared to existing result, we improve by remove the assumption that all arms have distinct means thanks to the regularization ε 0 > 0 .

̸

Lemma 17. There exists L 3 with E ν [ L 3 ] < + ∞ such that if L ≥ L 3 , for all n (at most polynomial in L ) such that U L n = ∅ , B EB n / ∈ V L n implies C TC ε 0 n ∈ V L n .

̸

Proof. In the following, we consider U L n = ∅ (hence V L n = ∅ ) and B EB n / ∈ V L n . Let J ⋆ n = arg max i ∈ V L n µ i and L 2 as in Lemma 16. Then, for L ≥ L 4 / 3 2 , we have B EB n ∈ J ⋆ n .

̸

Let L 0 and ( L 1 , D 1 ) as in Lemmas 14 and 15. Then, for all L ≥ max { L 4 / 3 0 , L 4 / 3 2 , L 2 1 } ,

̸

̸

As B EB n ∈ J ⋆ n , the definition C TC ε 0 n yields that C TC ε 0 n ∈ V L n . Otherwise the above strict inequality would wield a contradiction. Using Lemma 12, we show

hence this concludes the proof.

Lemma 18 shows that all arms are sufficiently explored and that the leader is the unique best arm for n large enough.

Lemma 18. There exist N 1 with E ν [ N 1 ] < + ∞ such that for all n ≥ N 1 and all i ∈ [ K ] , N n,i ≥ √ n/K and B EB n = i ⋆ .

̸

Proof. Let L 2 and L 3 as in Lemmas 16 and 17. Therefore, for L ≥ L 4 := max { L 3 , L 4 / 3 2 } , for all n such that U L n = ∅ , B EB n ∈ V L n or C TC ε 0 n ∈ V L n . We have E ν [ L 4 ] < + ∞ . There exists a deterministic L 5 such that for all L ≥ L 5 , ⌊ L ⌋ ≥ KL 3 / 4 . Let L ≥ max { L 5 , L 4 } .

Suppose towards contradiction that U L ⌊ KL ⌋ is not empty. Then, for any 1 ≤ t ≤ ⌊ KL ⌋ , U L t and V L t are non empty as well. Using the pigeonhole principle, there exists some i ∈ [ K ] such that N ⌊ L ⌋ ,i ≥ L 3 / 4 . Thus, we have ∣ ∣ ∣ V L ⌊ L ⌋ ∣ ∣ ∣ ≤ K -1 . Our goal is to show that ∣ ∣ ∣ V L ⌊ 2 L ⌋ ∣ ∣ ∣ ≤ K -2 . A sufficient condition is that one arm in V L ⌊ L ⌋ is pulled at least L 3 / 4 times between ⌊ L ⌋ and ⌊ 2 L ⌋ -1 . If we have

then, using that V L t ⊆ V L ⌊ L ⌋ , we obtain

Therefore, there exists i ∈ V L ⌊ L ⌋ which is sampled L 3 / 4 times between ⌊ L ⌋ and ⌊ 2 L ⌋ -1 . In the following, we assume that

Since B EB t ∈ V L t or C TC ε 0 t ∈ V L t , we have

Therefore, we have

Since β t ( i, j ) = N t,j / ( N t,j + N t,i ) , we obtain

This lower bound will be crucial to argue that the challenger is sampled enough in case 2 and the leader is sampled enough in case 1.

Case 1. Using Lemma 46 and the above, we obtain

̸

̸

where the last inequality is obtained for L ≥ L 6 +1 with

Therefore, there exists i ∈ V L ⌊ L ⌋ which is sampled L 3 / 4 times between ⌊ L ⌋ and ⌊ 2 L ⌋ -1 .

Case 2. Using Lemma 46 and the above, we obtain

̸

̸

where the last inequality is obtained for L ≥ L 6 + 1 . Therefore, there exists i ∈ V L ⌊ L ⌋ which is sampled L 3 / 4 times between ⌊ L ⌋ and ⌊ 2 L ⌋ -1 .

Summary. We have shown ∣ ∣ ∣ V L ⌊ 2 L ⌋ ∣ ∣ ∣ ≤ K -2 . By induction, for any 1 ≤ k ≤ K , we have ∣ ∣ ∣ V L ⌊ kL ⌋ ∣ ∣ ∣ ≤ K -k , and finally U L ⌊ KL ⌋ = ∅ for all L ≥ L 7 := max { L 5 , L 4 , L 6 +1 } . Defining N 0 = KL 7 , we have E ν [ N 0 ] < + ∞ . For all n ≥ N 0 , we let L = n/K , hence we have U n/K n = U L ⌊ KL ⌋ = ∅ . This concludes the proof of sufficient exploration.

Leader is best arm. Using Lemma 12, we obtain that for all n ≥ N 0 ,

̸

Therefore, we have B EB n = i ⋆ for all n ≥ N 1 := max { N 0 , K ( X 0 -e ) 2 +1 } where

̸

̸

where the last inequality is obtained by using Lemma 51 with h 1 defined therein. Since h 1 ( x, e ) ∼ x → + ∞ x log x , we have

This concludes the proof.

D.1.2 Empirical overall balance

As in [40], the key to obtain asymptotic optimality is to show that the empirical proportion satisfy the empirical overall balance equation. Compared to them, the novelty of Lemma 19 is that we use IDS proportions with K ( K -1) tracking procedures to select between the leader and the challenger instead of sampling.

Lemma 19. Let γ > 0 . There exists N 2 with E ν [ N 2 ] < + ∞ such that for all n ≥ N 2

̸

Proof. Let N 1 as in Lemma 18. Then, we have for all n > N 1

̸

̸

̸

Chaining the inequalities with Lemma 46, we obtain that

Using that a 2 -b 2 = ( a -b )( a + b ) ≤ 2( a -b ) for ( a, b ) ∈ (0 , 1) 2 , we obtain

For all n > N 1 , let us denote by

Therefore, we have

̸

̸

̸

Direct manipulations yield that

Therefore, we obtain by using the above inequalities that

where we used that β n ( i ⋆ , C n ) N n,i ⋆ = (1 -β n ( i ⋆ , C n )) N n,C n since β n ( i ⋆ , C n ) = N n,Cn N n,i ⋆ + N n,Cn . Summing those inequalities, we obtain

where we used that G N 1 +1 = 2 β N 1 ( i ⋆ , C N 1 ) -1 ∈ [ -1 , 1] . Combining everything together, we have shown that

̸

Therefore, we have shown the desired results for n ≥ N 2 defined as

which satisfies E ν [ N 2 ] < + ∞ since it is a linear function of N 1 .

As in [40], using Lemma 19 allows to bound the empirical proportion allocated to the unique best arm N n,i ⋆ / ( n -1) away from 0 (Lemma 20).

Lemma 20. There exists N 3 with E ν [ N 3 ] < + ∞ such that for all n ≥ N 3

Proof. Let N 2 as in Lemma 19 for γ = 1 / 2 . Using Lemma 19, we obtain N n,i ⋆ n -1 ≤ 3 4 since

̸

Let ˜ N 2 as in Lemma 19 for γ = 1 32( K -1) . Similarly by using Lemma 19, we obtain N n,i ⋆ n -1 ≥ 1 4 √ 2( K -1) since

̸

Taking N 3 = max { N 2 , ˜ N 2 } yields the result.

Lemma 21 is a rescaled version of the empirical overall balance equation which is proven by simply combining Lemma 19 and Lemma 20.

Lemma 21. Let γ > 0 . There exists N 4 with E ν [ N 4 ] < + ∞ such that for all n ≥ N 4

̸

Proof. Let N 2 as in Lemma 19 for γ 32( K -1) . Let N 3 as in Lemma 20. Direct manipulation shows that, for all n ≥ N 4 = max { N 2 , N 3 } ,

̸

This concludes the result.

D.1.3 Convergence towards optimal ratio of allocation

As in [40], we show that a challenger will not be sampled next when the ratio of its empirical proportion compared to the one of i ⋆ overshoots the ratio of the optimal allocations (Lemma 22).

̸

Lemma 22. Let γ > 0 . There exists N 5 with E ν [ N 5 ] < + ∞ such that for all n ≥ N 5 and all i = i ⋆ ,

̸

Proof. Let γ > 0 . Let i = i ⋆ such that

Suppose towards contradiction that

̸

Let ˜ γ > 0 . Let N 4 as in Lemma 21 for ˜ γ . then using Lemma 21 and 8 yields that, for all n ≥ N 4 ,

̸

Therefore, we have a contradiction if we take ˜ γ small enough (e.g. ˜ γ < γ 2 ), hence we have shown that, for all n ≥ N 4 ,

Let N 1 as in Lemma 18. Using that B EB n = i ⋆ for all n ≥ N 1 and the definition of C TC ε 0 n , we known that

̸

To conclude the proof, it is sufficient to show that the ratio of the two quantities is strictly larger than one. For all n ≥ max { N 1 , N 4 } , we obtain

̸

̸

̸

Let ˜ γ > 0 . Using Lemmas 12 and 18, we have, for all n ≥ N 1 and all k = i ⋆ ,

̸

̸

for all n ≥ N 6 = K ( X 0 -e ) 2 +1 which is defined as

̸

where the last inequality is obtained by using Lemma 51 with h 1 defined therein. Since h 1 ( x, y ) ∼ x → + ∞ x log x , we have E ν [ N 6 ] < + ∞ since it polynomial in W µ (by using Lemma 12). Let κ > 0 . We have shown that, for all n ≥ max { N 1 , N 4 , N 6 } ,

where the equality uses that the transportation costs are equal at the equilibrium, i.e. (9). Taking ˜ γ small enough, we have that shown that, for all n ≥ max { N 1 , N 4 , N 6 } ,

̸

This concludes the proof.

Lemma 23 shows no sub-optimal arm is overshooting the ratio of its optimal allocation for n large enough.

̸

Lemma 23. Let γ > 0 . There exists N 6 with E ν [ N 6 ] < + ∞ such that for all n ≥ N 6 and all i = i ⋆ ,

Proof. Let γ > 0 . Let N 1 as in Lemma 18. Let N 3 and N 5 as in Lemmas 20 and 22. Let M ≥ max { N 3 , N 5 , N 1 } and n > M . Let t n,i ( γ ) = max { M, max { t ∈ { M, · · · , n -1 } | N t,i N t,i ⋆ < w ⋆ i w ⋆ i ⋆ + γ/ 2 }} for all i = i ⋆ . In particular, using Lemma 22, for all n > M

̸

̸

and

Then, we have for all n ≥ M ,

Using that N t n,i ( γ ) ,i ⋆ ≤ N n,i ⋆ , N n,i ⋆ ≥ n -1 4 √ 2( K -1) (Lemma 20) and max { a, b } ≤ a + b , we obtain that

where the last inequality holds for n ≥ N 7 = 8( M + 1) √ 2( K -1) /γ + 1 . Taking N 6 = max { N 3 , N 5 , N 1 , N 7 } yields the result since E ν [ N 6 ] < + ∞ .

Finally, Lemma 24 shows that the sufficient condition (see Lemma 13) to obtain the asymptotic upper bound on the expected sample complexity is satisfied.

Lemma 24. Let ε 0 > 0 , γ > 0 and T ε 0 ,γ as in (10) . Under the EB-TC ε 0 sampling rule with IDS proportions, we have E ν [ T ε 0 ,γ ] < + ∞ .

Proof. Let γ > 0 and ˜ γ . Let N 4 and N 6 as in Lemmas 21 and 23 for ˜ γ . Then, using Lemmas 21 and 23, we have for all n ≥ max { N 4 , N 6 } ,

where the equality uses (9). Therefore, we have shown that for all n ≥ max { N 4 , N 6 } ,

where the last inequality holds for ˜ γ small enough as a function of γ and w ⋆ . Therefore, we obtained

̸

which concludes the proof, i.e. E ν [ T γ ] < + ∞

Combining Lemmas 13 and 24 concludes the first part of Theorem 1, i.e.

̸

D.2 Fixed proportions

Using Lemma 9, the β -optimal allocation for ε 0 -BAI are defined as

̸

Since | i ⋆ ( µ ) | = 1 , let µ ε 0 = µ ε 0 ( i ⋆ ) where µ ε ( i ⋆ ) as in Lemma 9. Since we have T ε 0 ,β ( µ ) = T ⋆ β ( µ ε 0 ) and w ε 0 ,β ( µ ) = w ⋆ β ( µ ε 0 ) , Lemma 6 and 8 yield that w ε 0 ,β ( µ ) = { w ⋆ β } is a singleton with unique element denotes by w ⋆ β which satisfies that min i ∈ [ K ] w ⋆ β,i > 0 and

When considering fixed proportions β , the proof strategy is the same as in Appendix D.1. The sole difference lies in the fact that the empirical allocation of the unique best arm is converging towards a fixed β .

Convergence time Let γ > 0 . Let us define the convergence time T ε 0 ,β,γ , which is a random variable quantifies the number of samples required for the empirical allocations N n to be γ -close to w ⋆ β :

Lemma 25 gives a sufficient condition for asymptotic β -optimality. The case ε 0 = ε 1 is a direct consequence of existing methods, e.g. Theorem 2 in [21] or Theorem 3 in [32].

Lemma 25. Let ε 0 > 0 , ε 1 ≥ 0 and δ ∈ (0 , 1) . Assume that there exists γ 1 ( µ ) > 0 such that for all γ ∈ (0 , γ 1 ( µ )] , E ν [ T ε 0 ,β,γ ] < + ∞ . Using the threshold (4) in the stopping rule (3) with slack ε 1 yields an algorithm such that, for all ν ∈ D K such that | i ⋆ ( µ ) | = 1 ,

̸

Proof. The proof is the same of the ones of Lemma 13. The sole difference is that we have w ⋆ β,i ⋆ = β by definition.

Sufficient exploration Lemma 26 shows that all arms are sufficiently explored and that the leader is the unique best arm for n large enough.

Lemma 26. There exist N 1 with E ν [ N 1 ] < + ∞ such that for all n ≥ N 1 and all i ∈ [ K ] , N n,i ≥ √ n/K and B EB n = i ⋆ .

̸

Proof. Since Lemmas 16 and 17 holds, we can use the intermediate results of Lemma 18. It is direct to see that the proof can be conducted similarly by noting that, for all i = j ,

Convergence towards optimal allocation Combining Lemma 26 with the proof of Lemma F.8 in [20] yields Lemma 27.

Lemma 27. Let γ > 0 . There exists N 2 with E µ [ N 2 ] < + ∞ such that for all n ≥ N 2 ,

Combining Lemmas 26 and 27 with the proof of Lemma F.9 in [20] yields Lemma 28.

̸

Lemma 28. Let γ > 0 . There exists N 5 with E ν [ N 5 ] < + ∞ such that for all n ≥ N 5 and all i = i ⋆ ,

̸

Combining Lemmas 27 and 28 with the proof of Lemma F.10 in [20] yields Lemma 29.

Lemma 29. Let γ > 0 . There exists N 6 with E ν [ N 6 ] < + ∞ such that for all n ≥ N 6 ,

Lemma 30 is a direct corollary of Lemma 29.

Lemma 30. Let ε 0 > 0 , β ∈ (0 , 1) , γ > 0 and T ε 0 ,β,γ as in (14) . Under the EB-TC ε 0 sampling rule with fixed proportions β , we have E ν [ T ε 0 ,β,γ ] < + ∞ .

Combining Lemmas 25 and 30 concludes the second part of Theorem 1, i.e.

̸

E Non-asymptotic analysis

Let ε 0 > 0 , β ∈ (0 , 1) and δ ∈ (0 , 1) . In this section, we provide a non-asymptotic analysis of EB-TC ε 0 with fixed proportions β and slack ε 0 > 0 when combined with the stopping rule (3) with parameters ( ε 0 , δ ) . In practice, we will mostly be interested in the case β = 1 / 2 . First, we prove Lemma 5 in Appendix E.1. Then, we detail the proof of Theorem 2 in Appendix E.2.

In the following, we consider a sub-Gaussian bandit with distribution ν ∈ D K having mean parameter µ ∈ R K . In particular, our analysis holds when several arms have the largest mean µ ⋆ .

E.1 Proof of Lemma 5

Let s ≥ 0 . For all n > K and δ ∈ (0 , 1] , let E n,δ = E 1 n,δ ∩ E 2 n,δ with ( E 1 n,δ ) n>K and ( E 2 n,δ ) n>K as in (24) and (25), i.e.

̸

where f 2 ( x, δ ) = log(1 /δ ) + (1 + s ) log( x ) and f 2 ( x, δ ) = log(1 /δ ) + (2 + s ) log( x ) .

Lemma 31 shows that when there are arms with strictly higher true mean than the one of the leader, then at least one of those arms is undersampled.

Lemma 31. Under E 1 n,δ , for all t ∈ [ n ] \ [ K ] let B EB t = k . Then,

̸

̸

Proof. Under E 1 n,δ , for all t ∈ [ n ] \ [ K ] , let B EB t = k . Then, for all i = k , we have

Re-ordering the above equations for i such that µ i > µ k yields the result.

Lemma 32. Let ε ≥ 0 , ∆ µ ( ε ) = min k/ ∈I ε ( µ ) ∆ k and C µ,ε 0 ( ε ) = max { 2∆ µ ( ε ) -1 -ε -1 0 , ε -1 0 } 2 . Let A ε 0 ,ε,i = 2 / ∆ µ ( ε ) 2 for all i ∈ i ⋆ ( µ ) , A ε 0 ,ε,i = C µ,ε 0 ( ε ) for all i ∈ I ε ( µ ) \ i ⋆ ( µ ) , otherwise A ε 0 ,ε,i = max { C µ,ε 0 ( ε ) , 2 / ∆ 2 i } . For all n > K , under event E n,δ , for all t ∈ [ n ] \ [ K ] such that B EB t / ∈ I ε ( µ ) , there exists i t ∈ [ K ] such that

̸

Proof. Let ∆ i = µ ⋆ -µ i and ∆ max = max i ∈ [ K ] ∆ i . When ε ≥ ∆ max , we have I ε ( µ ) ∁ = ∅ , hence the above result holds trivially since the event B EB t / ∈ I ε ( µ ) cannot happen. Let ε ∈ [0 , ∆ max ) , i.e. I ε ( µ ) ∁ = ∅ . We will consider in two distinct cases since

Case 1. Let t ∈ [ n ] \ [ K ] such that ( B EB t , C TC ε 0 t ) = ( i, j ) with i / ∈ I ε ( µ ) and j ∈ i ⋆ ( µ ) . Using Lemmas 46 and 31, we obtain

which can be rewritten as

Let us define ∆ µ ( ε ) = min k/ ∈I ε ( µ ) ∆ k , and

The above shows that there exists k t ∈ I ε ( µ ) ∁ ∪ i ⋆ ( µ ) such that

Case 2. Let t ∈ [ n ] \ [ K ] such that ( B EB t , C TC ε 0 t ) = ( i, j ) with i / ∈ I ε ( µ ) and j / ∈ i ⋆ ( µ ) . Let i 0 ∈ i ⋆ ( µ ) . Using the TC challenger, we obtain

Using Lemma 31, we obtain

By distinguishing between ε 0 > ∆ i and ε 0 ≤ ∆ i and using that ∆ i > 0 , we have

Using Lemma 46 to lower bound min { N t,i , N t,j } and reordering, we have shown that

Let us define C µ,ε 0 ( ε ) = max { 2∆ µ ( ε ) -1 -ε -1 0 , ε -1 0 } 2 . The above shows that, there exists k t / ∈ i ⋆ ( µ ) such that

Summary. Let us define ( A ε 0 ,ε,i ) i ∈ [ K ] as in the statement of Lemma 32. Under E n,δ , we have show that, when B t / ∈ I ε ( µ ) , there exists i t ∈ [ K ] such that

For all n > K , under event E n,δ , combining Lemma 4 and 32 for A t ( n, δ ) = { B EB t / ∈ I ε ( µ ) } and D i ( n, δ ) = 4 f 2 ( n,δ ) min { β, 1 -β } A ε 0 ,ε,i +3( K -1) / 2 yields that

where we used that ∑ i ∈ [ K ] A ε 0 ,ε,i = H µ,ε 0 ( ε ) where H µ,ε 0 ( ε ) is defined in (5). To conclude the proof of Lemma 5, we use that

Let ˜ f 1 ( n, δ ) and ˜ f 2 ( n, δ ) defined as in (26) and (28). Using Lemma 50, we have ˜ f 1 ( n, δ ) ≤ ˜ f 2 ( n, δ ) . Therefore, it is direct to see that Lemma 33 can be proven with the same proof as for Lemma 5 based on the concentration event ˜ E n,δ = ˜ E 1 n,δ ∩ ˜ E 2 n,δ with ( ˜ E 1 n,δ ) n>K and ( ˜ E 2 n,δ ) n>K as in (27) and (29).

Lemma 33. Let δ ∈ (0 , 1] , ε ≥ 0 and H µ,ε 0 ( ε ) as in (5) . For all n > K , under the event ˜ E n,δ ,

E.2 Proof of Theorem 2

Let s > 1 . For all n > K , let E n = E 1 n, 1 ∩ E 2 n, 1 with ( E 1 n,δ ) n>K and ( E 2 n,δ ) n>K as in (24) and (25). Using Lemma 45, we know that ∑ n>K P ν ( E ∁ n ) ≤ K ( K +1) 2 ζ ( s ) .

Let ε 0 > 0 , ε 1 ≥ ε 0 and δ ∈ (0 , 1) . Since ε 1 ≥ ε 0 , and using the definition of the stopping rule (3), it is direct to see that E ν [ τ ε 1 ,δ ] ≤ E ν [ τ ε 0 ,δ ] .

Suppose that we have constructed a time T ( δ ) > K be such that for n ≥ T ( δ ) , E n ⊂ { τ ε 0 ,δ ≤ n } Then, Lemma 47 yields

We will construct an infinite number of times { T ( δ, u ) } u ∈U such that the above property holds, hence taking the infimum yields

This proof strategy is the one used to prove Theorem 5. Theorem 2 is a corollary of Theorem 5. Note that the notation of both theorems differ slightly since we provide a shorter statement in the main content. The key difference lies in the refined analysis used to clip min j = i w ε 0 , 1 / 2 ( µ, i ) j by a fixed value x ∈ [0 , ( K -1) -1 ] for all i ∈ I ε ( µ ) .

̸

Theorem 5. Let ε 0 > 0 , ε 1 ≥ ε 0 and δ ∈ (0 , 1) . Using the threshold (4) in the stopping rule (3) with error ε 1 , the EB-TC ε 0 algorithm with fixed proportions β = 1 / 2 is ( ε 1 , δ ) -PAC and satisfies that, for all ν ∈ D K , E ν [ τ ε 1 ,δ ] ≤ E ν [ τ ε 0 ,δ ] and

where

̸

where γ ∈ (0 , 1 / 2] and s > 1 are analysis parameters. T ε 0 , 1 / 2 ( µ, i ) and w ε 0 , 1 / 2 ( µ, i ) are defined in (1) , H µ,ε 0 ( ε ) in (5) , ζ is the Riemann ζ function and h 1 ( z, y ) = zW -1 (log( z ) + y/z ) is defined in Lemma 51 and satisfies that h 1 ( z, y ) ≈ x log( z ) + y +log(log( z ) + y/z ) .

Proof. Let ε ∈ [0 , ε 0 ] be an analysis parameter, and I ε = |I ε ( µ ) | . Let v ∈ ˚ △ I ε defined as

The main technical result on which our proof relies on is Lemma 34, which we will prove afterwards.

̸

Lemma 34. There exist D 0 > 0 and T µ > 0 such that for all n ≥ T µ such that E n ∩ { n < τ ε 1 ,δ } holds true, there exists i 0 ∈ I ε ( µ ) and t ≤ n with B EB t = i 0 , which satisfies

Before proving Lemma 34, we show how it can be used to conclude. Based on Lemma 34, let D 0 > 0 , T µ > 0 , n > T µ such that E n ∩ { n < τ ε 1 ,δ } holds true, i 0 and t ≤ n such that as B EB t = i 0 and

Using the stopping rule (3) with error ε 0 , t ≤ n < τ ε 1 ,δ , B EB t = ˆ ı t = i 0 and the definition of the TC ε 0 challenger in (2) with slack ε 0 , we obtain

̸

The third inequality is obtained since E n holds. The fourth inequality is obtained since N i 0 t,i ≥ N i 0 t,i for all i ∈ [ K ] and by using Lemmas 8 and 9. The last inequality is obtained by Lemma 34 and using the definition of v i 0 .

Let's define T ( δ, D 0 ) := sup { n > K | n -1 < D 0 ∑ i ∈I ε ( µ ) T ε 0 ,β ( µ, i )( √ c ( n -1 , δ ) + √ (2 + s ) log n ) 2 } . Therefore, we have E n ∩ { n < τ ε 0 ,δ } = ∅ (i.e. E n ⊂ { τ ε 0 ,δ ≤ n } ) for all n ≥ max { T ( δ, D 0 ) , T µ +1 } . This concludes the construction.

To conclude the proof, we will show that Lemma 34 for many choices of D 0 and T µ .

Vanilla Proof of Lemma 34. Let g ε ( n ) = 4 H µ,ε 0 ( ε ) min { β, 1 -β } (2 + s ) log n + 3 K ( K -1) / 2 . Using Lemma 5, under the event E n ,

Therefore, the pigeonhole principle yields that there exists i 0 ∈ I ε ( µ ) such that

The crux of the problem is to relate N i 0 t,C t / ( t -1) and w ⋆ β ( i 0 ) C TC ε 0 t . To do so, we use the approach of [20] that builds on the idea behind the proof for APT from [30]: consider an arm being over-sampled and study the last time this arm was pulled.

̸

By the pigeonhole principle, at time n , there is an index k 1 = i 0 such that

̸

and we take such k 1 . Let t 1 := sup { t < n | ( B t , C t ) = ( i 0 , k 1 ) } be the last time at which i 0 was the leader and k 1 was the challenger. If I t 1 = B t 1 then N i 0 t 1 ,k 1 = N i 0 n,k 1 , otherwise N i 0 t 1 ,k 1 = N i 0 n,k 1 -1 .

In both cases, we have N i 0 t 1 ,k 1 ≥ N i 0 n,k 1 -1 . Combined with the above and using that w ⋆ β ( i 0 ) k 1 ≤ 1 -β and v i 0 ≤ 1 , we obtain

̸

̸

̸

Let w i 0 , -> 0 be a lower bound on w ⋆ β ( i 0 ) k 1 , e.g. w i 0 , -= min i = i 0 w ⋆ β ( i 0 ) i . For instances µ such that w ⋆ β ( i 0 ) k 1 is small, the equation (16) can be satisfied at the very beginning, hence t 1 might be sublinear in n . Due to the missing link between t 1 and n , we use the following inequality

which is a suboptimal step which artificially introduces 1 /β .

Let γ ∈ (0 , 1 / 2] . It remains to control both terms. First, we obtain

for all n > C 1 ( w i 0 , -, v µ,ε 0 ( ε )) . The last inequality is obtained by definition of

which ensures that, for all n > C 1 ( w i 0 , -, v µ,ε 0 ( ε )) , the last condition of the equivalence

Second, using Lemma 46, we can show that

Then, re-ordering the above and using that N i 0 t 1 ,k 1 ≥ w ⋆ β ( i 0 ) k 1 v i 0 ( n -1) -h ε ( n ) , we obtain

for all n > C 2 ( w i 0 , -, v µ,ε 0 ( ε )) . The last inequality is obtained by definition of

which ensures that, for all n > C 2 ( w i 0 , -, v µ,ε 0 ( ε )) , the last condition of the equivalence

is satisfied since w ⋆ β ( i 0 ) k 1 ≥ w i 0 , -, v i 0 ≥ v µ,ε 0 ( ε ) and n > C 2 ( w i 0 , -, v µ,ε 0 ( ε )) .

Let T µ ≥ max k ∈ [2] C k (min i ∈I ε ( µ ) w i, -, v µ,ε 0 ( ε )) and D 0 = (1 + γ ) 2 /β . In summary, we have shown that for all n > T µ , there exists i 0 ∈ I ε ( µ ) and t 1 ≤ n with B t 1 = i 0 and such that (15) holds.

For β = 1 / 2 , using that K ≥ 2 , 1 < 1 + γ -1 and 5 / 2 + γ -1 ≤ 3(1 + γ -1 ) / 2 (since γ ∈ (0 , 1 / 2] ), we obtain that

where the last inequality uses Lemma 51 and h 1 ( x, y ) defined therein.

̸

Using w i 0 , -= min i = i 0 w ⋆ β ( i 0 ) i and β = 1 / 2 , the first part of Theorem 5 (i.e. x = 0 ) is obtained by taking the infimum over ( ε, x ) ∈ [0 , ε 0 ] ×{ 0 } .

̸

Refined Proof of Lemma 34. When considering large K or instances with unbalanced β -optimal allocation, min i 0 ∈I ε ( µ ) min i = i 0 w ⋆ β ( i 0 ) i can become arbitrarily small, hence a dependence in its inverse is undesired. Thanks to the refined analysis from [20], we can clip it with a value of our choosing which is away from zero.

̸

Let x ∈ (0 , 1 / ( K -1)] be an allocation threshold and d µ ( x, i ) := |{ j ∈ [ K ] \ { i } | w ⋆ β ( i ) j < (1 -β ) x }| for all i ∈ I ε ( µ ) . The equation (16) is only informative when (1) w ⋆ β ( i 0 ) k 1 ≥ (1 -β ) x or (2) w ⋆ β ( i 0 ) k 1 < (1 -β ) x and N i 0 n,k 1 ≥ x ∑ j = i 0 N i 0 n,j . The problematic case occurs when (3) w ⋆ β ( i 0 ) k 1 < (1 -β ) x and N i 0 n,k 1 < x ∑ j = i 0 N i 0 n,j .

̸

̸

When w ⋆ β ( i 0 ) k 1 ≥ (1 -β ) x , we can use the above computations by considering w i 0 , -= max { (1 -β ) x, min i = i 0 w ⋆ β ( i 0 ) i } .

̸

When w ⋆ β ( i 0 ) k 1 < (1 -β ) x and N i 0 n,k 1 ≥ x ∑ j = i 0 N i 0 n,j , the above proof can be conducted by using (1 -β ) x instead of w ⋆ β ( i 0 ) k 1 since we will obtain that

̸

Then, we can likewise consider w i 0 , -= max { (1 -β ) x, min i = i 0 w ⋆ β ( i 0 ) i } to conclude.

̸

When w ⋆ β ( i 0 ) k 1 < (1 -β ) x and N i 0 n,k 1 < x ∑ j = i 0 N i 0 n,j , as in (16), the pigeonhole principle yields that there exists k 2 ∈ [ K ] \ { i 0 , k 1 } such that

̸

Based on k 2 the same dichotomy as for k 1 happens: either we can conclude the proof when we are in case 1 or 2 or we cannot since we are in case 3. If case 3 occurs also for k 2 , i.e. w ⋆ β ( i 0 ) k 2 < (1 -β ) x and N i 0 n,k 2 < x ∑ j / ∈{ i 0 ,k 1 } N i 0 n,j , we should also ignore it since it is non informative and construct a third arm k 3 .

The main idea is then to peel off arms that are not informative, till we find an informative one. By induction, we construct a sequence ( k a ) a ∈ [ d ] of such arms, where k d is the first arm for which either (1) w ⋆ β ( i 0 ) k d ≥ (1 -β ) x or (2) w ⋆ β ( i 0 ) k d < (1 -β ) x and N i 0 n,k d ≥ x ∑ j / ∈{ i 0 }∪{ k a } a ∈ [ d -1] N i 0 n,j holds. This means that for all a ∈ [ d -1] , we have w ⋆ β ( i 0 ) k a < (1 -β ) x and

Using the pigeonhole principle for i ∈ [ K ] \ ( { i 0 } ∪ { k a } a ∈ [ d -1] ) yields that k d satisfies

Using a simple recurrence on the arms { k a } a ∈ [ d -1] , we obtain that

̸

Let t d := sup { t < n | ( B t , C t ) = ( i 0 , k d ) } . Since the arm k d satisfies case 1 or case 2, we can conclude similarly as above with an extra multiplicative factor (1 -x ) d -1 .

To remove the dependency in the random variable d , we consider the worst case scenario where { k a } a ∈ [ d -1] = { i ∈ [ K ] \ { i 0 } | w ⋆ β ( i 0 ) i < (1 -β ) x } , i.e. d -1 ≤ d µ ( x, i 0 ) . In words, it means that we had to enumerate over all arms with small allocation, such that case 2 didn't hold, before finding an arm with large allocation, i.e. satisfying case 1. Let d µ,ε ( x ) = max i ∈I ε ( µ ) d µ ( x, i ) . Using that

̸

we can conclude the proof by taking T µ ≥ max k ∈ [2] C k (min i ∈I ε ( µ ) w i, -, v µ,ε 0 ( ε )) and D 0 = (1+ γ ) 2 β (1 -x ) dµ,ε ( x ) . In other words, we have shown that n > T µ , there exists i 0 ∈ I ε ( µ ) and t d ≤ n with B t d = i 0 and such that (15) holds. Using β = 1 / 2 , the second part of Theorem 5 is obtained by taking the infimum over ( ε, x ) ∈ [0 , ε 0 ] × (0 , 1 / ( K -1)] .

E.2.1 Consequence of tighter concentration

It is direct to see that the proof of Theorem 5 will also work when using the concentration event ˜ E n = ˜ E 1 n, 1 ∩ ˜ E 2 n, 1 with ( ˜ E 1 n,δ ) n>K and ( ˜ E 2 n,δ ) n>K as in (27) and (29). Let us define

Using Lemma 50, we know that ˜ f 1 ( n ) ≤ ˜ f 2 ( n ) . Then, we will be able to show

where

with ˜ h 1 ( z, y ) > sup { n ≥ 1 | n < z ˜ f 2 ( n ) + y } . Using Lemma 49, we obtain that

Therefore, we can consider ˜ h 1 ( z, y ) ≥ z (1 + s ) h 2 ( z, y ) with

where C ( z, y ) = y (1+ s ) z + s s +1 log z +log(1 + s ) + 2 1+ s . Therefore, we will obtain a mulitplicative factor (1 + s ) instead of (2 + s ) in front of H µ,ε 0 ( ε ) . This improvement is rather mild, hence we don't elaborate on it further.

F Probability of error and simple regret

Let ε 0 > 0 and β ∈ (0 , 1) . In Appendix F, we provide an analysis on the probability of error and the simple regret of EB-TC ε 0 with fixed proportions β and slack ε 0 > 0 (Theorem 6 proved in Appendix F.1). In Appendix F.2, we provide an upper bound on the unverifiable expected sample complexity of EB-TC ε 0 (Theorem 7). Then, by using Theorem 6, we prove that the policy pulling arm ˆ ı n at time n has a constant induced regret (Corollary 2 proved in Appendix F.3). In practice, we will mostly be interested in the case β = 1 / 2 .

Theorem 3 and Corollary 1 is a corollary of Theorem 6 up to some additional computations on the complexities ( H i ( µ, ε 0 )) i ∈ [ C µ -1] , which we will detail below. Theorem 6 provides upper bound on P ν (ˆ ı n / ∈ I ε ( µ )) holding for any time n and any error ε ≥ 0 , and upper bound on E ν [ µ ⋆ -µ ˆ ı n ] holding for any time n .

Theorem 6. Let ε 0 > 0 . For all ε ≥ 0 , let i µ ( ε ) = i if ε ∈ [∆ i , ∆ i +1 ) and i ∈ [ C µ -1] , otherwise i µ ( ε ) = C µ . For all i ∈ [ C µ -1] , let C i ( ε 0 ) = 2∆ -1 i -ε -1 0 and C i,j ( ε 0 ) = 2 ∆ j /ε 0 +1 ∆ i -∆ j +3 ε -1 0 . For all i ∈ [ C µ -1] , let H i ( µ, ε 0 ) := min j ∈ [ i ] max { ¯ H i,j ( µ, ε 0 ) , ˜ H i,j ( µ, ε 0 ) } where

The EB-TC ε 0 algorithm with fixed proportions β = 1 / 2 satisfies that, for all ν ∈ D K and all n ≥ D µ , if the algorithm has not stopped yet, then

where p ( x ) = x exp( -x ) and D µ = 8 H 1 ( µ, ε 0 ) h 2 ( 8 H 1 ( µ, ε 0 ) , 5 K 2 / 2 , 2 + log ( K ( K +1) / 2) ) + 5 K 2 / 2 with h 2 defined in Lemma 51.

Hardness of BAI We have C 2 ( ε 0 ) = 2∆ -1 min -ε -1 0 and C 2 , 1 ( ε 0 ) = 2∆ -1 min + 3 ε -1 0 . Then, we obtain H 1 ( µ, ε 0 ) = ¯ H 1 , 1 ( µ, ε 0 ) = K (2∆ -1 min +3 ε -1 0 ) 2 since

This corresponds to the complexity of the two-groups instances which are defined as D ∆ := { ν ∈ D K | C µ = 2 , ∆ 2 = ∆ } where ∆ > 0 . More importantly, it corresponds to the complexity of identifying one of the best arms, i.e. taking ε = 0 yields

Hardness of ε -BAI Let i > 1 and j ∈ [ i ] . We have C i +1 ,j ( ε 0 ) = 2 ∆ j /ε 0 +1 ∆ i +1 -∆ j +3 ε -1 0 , C j +1 ( ε 0 ) = 2∆ -1 j +1 -ε -1 0 ≤ 2∆ -1 j +1 and √ 2∆ -1 k ≤ 2∆ -1 j +1 for all k ≥ j +1 . Direct manipulations show that, for all i > 1 and all j ∈ [ i ] , we have

Therefore, we have

Taking j = 1 and then j = 2 , we see that, for all i > 1 ,

where the last inequality uses that ∆ 3 ≤ ∆ i +1 for all i > 1 .

Similarly, we can derive a lower bound on H i ( µ, ε 0 ) by noting that H i ( µ, ε 0 ) ≥ min j ∈ [ i ] ¯ H i,j ( µ, ε 0 ) . For all j ∈ [ i ] , we have that √ 2∆ -1 j +1 ≥ √ 2∆ -1 i +1 , C j +1 ( ε 0 ) ≥ 2∆ -1 i +1 -ε -1 0 and C i +1 ,j ( ε 0 ) ≥ 2∆ -1 i +1 +3 ε -1 0 , hence we obtain

Three-groups instances The three-groups instances are defined as D ∆ , Λ := { ν ∈ D K | C µ = 3 , ∆ 2 = ∆ , ∆ 3 = Λ } where ∆ > 0 and Λ > ∆ . Above results yield

F.1 Proof of Theorem 6

In the following, we consider a sub-Gaussian bandit with distribution ν ∈ R K having mean parameter µ ∈ R K . In particular, our analysis holds when several arms have the largest mean µ ⋆ .

Let ∆ max = max i ∈ [ K ] µ ⋆ -µ i . Let ε 1 be an analysis parameter (i.e. not the error parameter from the stopping rule (3)). When ε 1 ≥ ∆ max , we have I ε 1 ( µ ) ∁ = ∅ , hence P ν (ˆ ı n / ∈ I ε 1 ( µ )) = 0 . In the following, we consider ε 1 ∈ [0 , ∆ max ) .

For all n > K and δ ∈ (0 , 1) , let ˜ E n,δ = ˜ E 1 n,δ ∩ ˜ E 2 n,δ with ( ˜ E 1 n,δ ) n>K and ( ˜ E 2 n,δ ) n>K as in (27) and (29) for s = 0 , i.e.

̸

where

Using Lemma 50, we know that ˜ f 1 ( n, δ ) ≤ ˜ f 2 ( n, δ ) . Using Lemmas 42 and 44, we obtain that P ν ( ˜ E ∁ n,δ ) ≤ K ( K +1) 2 δ .

Suppose that we have constructed a time T ε 1 ( δ ) > K such that ˜ E n,δ ⊆ { ˆ ı n ∈ I ε 1 ( µ ) } for n > T ε 1 ( δ ) . Then, using Lemma 48, we obtain

We will construct an infinite number of times { T ε 1 ( δ, ε ) } ε ∈ [0 ,ε 1 ] such that the above property holds, hence taking the infimum yields

This proof strategy is the one used to prove Theorem 6.

Let ε ∈ [0 , ε 1 ] . Since ˆ ı n = B EB n , our goal is to construct a time T ε 1 ( δ, ε ) such that

Error due to undersampled arms Let us denote by U ε,ε 1 ,t ( n, δ ) the set of undersampled arms which are not ε 1 -good, i.e.

Lemma 35 shows that, for n large enough, a necessary condition for an error to occur is to have an undersampled leader.

Lemma 35. Let ε 1 > 0 and ε ∈ [0 , ε 1 ] . Let H µ,ε 0 ( ε ) as in Lemma 5 and define

Let us define

For all n > S ε,ε 1 ,ε 0 ,µ ( δ ) , under the event ˜ E n,δ , we have B EB n / ∈ I ε 1 ( µ ) implies that B EB n ∈ U ε,ε 1 ,n ( n, δ ) .

Proof. Let |I ε ( µ ) | = I ε and g ε,ε 0 ( n, δ ) = 4 H µ,ε 0 ( ε ) min { β, 1 -β } ˜ f 2 ( n, δ ) + 3 K ( K -1) / 2 . Using Lemma 5 and the pigeonhole principle, there exists i 0 ∈ I ε ( µ ) such that

Therefore, we have

Let S ε,ε 1 ,ε 0 ,µ ( δ ) defined as in the statement of Lemma 35. Direct manipulations shows that

Therefore, we have N n,i 0 > 4 ˜ f 2 ( n, δ ) max i/ ∈I ε 1 ( µ ) C ε,i for all n > S ε,ε 1 ,ε 0 ,µ ( δ ) .

Let n > S ε,ε 1 ,ε 0 ,µ ( δ ) . Suppose that B EB n = i / ∈ I ε 1 ( µ ) . Using Lemma 31, under the event ˜ E n,δ , we obtain that

Suppose towards contradiction that min { N n,i , N n,i 0 } = N n,i 0 , then we have N n,i 0 ≤ 4 C ε,i ˜ f 2 ( n, δ ) . This is a direct contradiction with N n,i 0 > 4 ˜ f 2 ( n, δ ) max i/ ∈I ε 1 ( µ ) C ε,i since n > S ε,ε 1 ,ε 0 ,µ ( δ ) . Therefore, we have shown that min { N n,i , N n,i 0 } = N n,i , hence i ∈ U ε,ε 1 ,n ( n, δ ) . This concludes the proof.

No remaining undersampled arms Lemma 36 shows that, if there are still undersampled arms which are not ε 1 -good, then either the leader or the challenger was not often selected as leader or challenger.

̸

Lemma 36. Let ε 1 ≥ 0 and ε ∈ [0 , ε 1 ] . Let ∆ µ ( ε ) = min k/ ∈I ε ( µ ) ∆ k , C µ,ε 0 ( ε, ε 1 ) = 2 max j / ∈I ε 1 ( µ ) ∆ j /ε 0 +1 min k ∈I ε ( µ ) (∆ j -∆ k ) + ε -1 0 and C µ,ε 0 ( ε ) = 2 / ∆ µ ( ε ) -ε -1 0 . Let A ε,ε 1 ,ε 0 ,i = max { 2 / ∆ µ ( ε ) 2 , C µ,ε 0 ( ε, ε 1 ) 2 } for all i ∈ i ⋆ ( µ ) , A ε,ε 1 ,ε 0 ,i = max { C µ,ε 0 ( ε ) 2 , C µ,ε 0 ( ε, ε 1 ) 2 } for all i ∈ ( I ε ( µ ) \ i ⋆ ( µ )) ∪ ([ K ] \ I ε 1 ( µ )) , and otherwise A ε,ε 1 ,ε 0 ,i = max { C µ,ε 0 ( ε ) 2 , C µ,ε 0 ( ε, ε 1 ) 2 , 2 / ∆ 2 i } for all i ∈ I ε 1 ( µ ) \ I ε ( µ ) . For all n > K , under event ˜ E n,δ , for all t ∈ [ n ] \ [ K ] such that U ε,ε 1 ,t ( n, δ ) = ∅ , there exists i t ∈ [ K ] such that

Proof. We will be interested in three distinct cases since

̸

̸

̸

Case 1. Let t ∈ [ n ] \ [ K ] such that { B EB t , C TC t } ∩ U ε,ε 1 ,t ( n, δ ) = ∅ . Let k t ∈ { B EB t , C TC t } ∩ U ε,ε 1 ,t ( n, δ ) . For this k t / ∈ I ε 1 ( µ ) , we have T t +1 ( k t ) = T t ( k t ) + 1 and, by combining Lemma 31 and N t,k t ≤ 4 C ε,k t ˜ f 2 ( n, δ ) , we obtain that

̸

Case 2. Let t ∈ [ n ] \ [ K ] such that U ε,ε 1 ,t ( n, δ ) = ∅ , U ε,ε 1 ,t ( n, δ ) ∩ { B EB t , C TC t } = ∅ and B EB t / ∈ I ε ( µ ) . Let ∆ µ ( ε ) and C µ,ε 0 ( ε ) defined as in the statement of Lemma 36. Let D ε 0 ,ε,i = 2 / ∆ µ ( ε ) 2 for all i ∈ i ⋆ ( µ ) , D ε 0 ,ε,i = C µ,ε 0 ( ε ) for all i ∈ I ε ( µ ) \ i ⋆ ( µ ) , otherwise D ε 0 ,ε,i = max { C µ,ε 0 ( ε ) , 2 / ∆ 2 i } . Using Lemma 32, there exists k t ∈ [ K ] such that

̸

Case 3. Let t ∈ [ n ] \ [ K ] such that U ε,ε 1 ,t ( n, δ ) = ∅ , U ε,ε 1 ,t ( n, δ ) ∩ { B EB t , C TC t } = ∅ and B EB t ∈ I ε ( µ ) . Let j 0 ∈ U ε,ε 1 ,t ( n, δ ) \ { B EB t , C TC t } , which is possible since U ε,ε 1 ,t ( n, δ ) ∩{ B EB t , C TC t } = ∅ and U ε,ε 1 ,t ( n, δ ) = ∅ . Let us denote by ( B EB t , C TC t ) = ( i, j ) with i ∈ I ε ( µ ) and j = j 0 . Using the TC challenger, under the event E n,δ , we obtain

̸

̸

Since µ i -µ j 0 + ε 0 ≥ ε 0 > 0 , we have

Using Lemma 31 to lower bound min { N t,i , N t,j } and µ i -µ j 0 ≤ ∆ j 0 for all i ∈ I ε ( µ ) , direct manipulation yields that

̸

where C µ,ε 0 ( ε, ε 1 ) is defined as in the statement of Lemma 36. It is direct to see that C µ,ε 0 ( ε, ε 1 ) 2 ≥ max { 1 /ε 2 0 , max i/ ∈I ε 1 ( µ ) C ε,i } and C ε,i ≥ 2 / ∆ 2 i for all i / ∈ I ε 1 ( µ ) . The above shows that there exists k t ∈ [ K ] such that

̸

Summary. Let us define ( A ε,ε 1 ,ε 0 ,i ) i ∈ [ K ] as in the statement of Lemma 36. Under ˜ E n,δ , we have show that, when U ε,ε 1 ,t ( n, δ ) = ∅ , there exists i t ∈ [ K ] such that

Lemma 37 shows that, for n large enough, there is no undersampled arms which are not ε 1 -good.

Let us define

For all n > T ε,ε 1 ,ε 0 ,µ ( δ ) , under the event ˜ E n,δ , we have U ε,ε 1 ,n ( n, δ ) = ∅ .

̸

Proof. For all n > K , under event ˜ E n,δ , combining Lemma 4 and 36 for A t ( n, δ ) = { U ε,ε 1 ,t ( n, δ ) = ∅} and D i ( n, δ ) = 4 ˜ f 2 ( n,δ ) min { β, 1 -β } A ε,ε 1 ,ε 0 ,i +3( K -1) / 2 yields that

̸

where we used that ∑ i ∈ [ K ] A ε,ε 1 ,ε 0 ,i = ¯ H ε,ε 1 ( µ, ε 0 ) where ¯ H ε,ε 1 ( µ, ε 0 ) is defined in (20).

For all i / ∈ I ε 1 ( µ ) , let us define

By definition, for all i / ∈ I ε 1 ( µ ) , we have i ∈ U ε,ε 1 ,t ( n, δ ) for all t ∈ ( K,t i ( n, δ )] and i / ∈ U ε,ε 1 ,t ( n, δ ) for all t ∈ ( t i ( n, δ ) , n ] . Therefore, for all t ∈ ( K, max i/ ∈I ε 1 ( µ ) t i ( n, δ )] , we have U ε,ε 1 ,t ( n, δ ) = ∅ and U ε,ε 1 ,t ( n, δ ) = ∅ for all t > max i/ ∈I ε 1 ( µ ) t i ( n, δ ) , hence

̸

̸

Let T ε,ε 1 ,ε 0 ,µ ( δ ) defined as in the statement of Lemma 37. Direct manipulations show that

Let n > T ε,ε 1 ,ε 0 ,µ ( δ ) . Then, we have

Upper bound probability of error Let S ε,ε 1 ,ε 0 ,µ ( δ ) and T ε,ε 1 ,ε 0 ,µ ( δ ) as in Lemmas 35 and 37. For β = 1 / 2 , direct manipulations show that max { S ε,ε 1 ,ε 0 ,µ ( δ ) , T ε,ε 1 ,ε 0 ,µ ( δ ) } ≤ E ε,ε 1 ,ε 0 ,µ ( δ ) with

Combining Lemmas 35 and 37, for all n > E ε,ε 1 ,ε 0 ,µ ( δ ) , we proved that

̸

hence { ˆ ı n / ∈ I ε 1 ( µ ) } ⊆ ˜ E ∁ n,δ .

Using Lemma 48, P ν ( ˜ E ∁ n,δ ) ≤ K ( K +1) δ/ 2 and taking the infimum over ε ∈ [0 , ε 1 ] , yields

where we used Lemma 52 and inf ε ∈ [0 ,ε 1 ] E ε,ε 1 ,ε 0 ,µ ( δ ) = E ε 1 ,ε 0 ,µ ( δ ) with

Using Lemma 52, we obtain that

where p ( x ) = x exp( -x ) .

More explicit complexity In order to obtain a more interpretable result, we notice that the dependence in ( ε, ε 1 ) of ¯ H ε,ε 1 ( µ, ε 0 ) and ˜ H ε,ε 1 ( µ, ε 0 ) is only with respect to its position in the ordered means. In the following, we will replace the dependency in ( ε, ε 1 ) by a dependency in such indices.

Let µ ∈ R K . Let us denote by C µ := |{ µ i | i ∈ [ K ] }| the number of different arm means in µ . For all i ∈ [ C µ ] , we denote the set of arms having mean gap ∆ i by C µ ( i ) := { k ∈ [ K ] | µ ⋆ -µ k = ∆ i } where the gaps are sorted by increasing order

In particular, we have C µ (1) = i ⋆ ( µ ) and ∆ C µ = ∆ max . Let us denote by

It is direct to see that i µ ( ε ) = max { i ∈ [ C µ ] , C µ ( i ) ⊂ I ε ( µ ) } for all ε > 0 . For all i ∈ [ C µ -1] , let us define H i ( µ, ε 0 ) = min j ∈ [ i ] max { ¯ H i,j ( µ, ε 0 ) , ˜ H i,j ( µ, ε 0 ) } with ¯ H i,j ( µ, ε 0 ) and ˜ H i,j ( µ, ε 0 ) as in Theorem 6. Let ε 1 ≥ 0 and ε ∈ [0 , ε 1 ] . Then, we have by direct manipulations that

since ¯ H ε,ε 1 ( µ, ε 0 ) = ¯ H i µ ( ε 1 ) ,i µ ( ε ) ( µ, ε 0 ) and ˜ H ε,ε 1 ( µ, ε 0 ) = ˜ H i µ ( ε 1 ) ,i µ ( ε ) ( µ, ε 0 ) . This concludes the first part of Theorem 6, i.e.

Simple regret Since µ ⋆ -µ ˆ ı n is a positive random variable, we obtain that

As a function of ε , our upper bound on P ν (ˆ ı n / ∈ I ε 1 ( µ )) is piecewise constant on [∆ i , ∆ i +1 ) for all i ∈ [ C µ -1] and has null value for ε 1 ≥ ∆ max . Our upper is informative, i.e. smaller than 1 , for all n ≥ D µ where

where the inequality is obtained by using Lemma 51, and h 2 is defined therein. Therefore, we obtain that, for all n ≥ D µ ,

which concludes the second part of Theorem 6.

F.2 Unverifiable sample complexity

The ( ε, δ ) -unverifiable sample complexity is defined as the expectation of the smallest stopping time ˜ τ satisfying P ( ∀ t ≥ ˜ τ, ˆ ı n ∈ I ε ( µ )) ≥ 1 -δ . Theorem 7 gives an upper bound on the unverifiable expected sample complexity of the EB-TC ε 0 algorithm with fixed proportions β = 1 / 2 .

Theorem 7. Let ε 0 > 0 and δ ∈ (0 , 1) . The EB-TC ε 0 algorithm with fixed proportions β = 1 / 2 satisfies that, for all ν ∈ D K with mean µ , for all ε ≥ 0 ,

The times U i µ ( ε ) ,δ ( µ, ε 0 ) are the ( ε, δ ) -unverifiable sample complexity of EB-TC ε 0 defined as

with ( H i ( µ, ε 0 )) i ∈ [ C µ -1] defined in Theorem 6. The function h 2 (log(1 /δ ) , A, B ) = 2 AW -1 ( 1 2 log(1 /δ ) + B 2 A + log(2 A )) satisfies that h 2 (log(1 /δ ) , A, B ) = δ → 0 A log(1 /δ ) + O (log log(1 /δ )) .

Proof. In Appendix F.1, we consider the concentration event ˜ E n,δ that involved tighter concentration results with thresholds ˜ f 1 ( n, δ ) and ˜ f 2 ( n, δ ) . Let n > K and δ ∈ (0 , 1) . It is direct to see that the same argument holds for the concentration events E n,δ = E 1 n,δ ∩E 2 n,δ with ( E 1 n,δ ) n>K and ( E 2 n,δ ) n>K as in (24) and (25) for s = 0 , i.e.

̸

where f 1 ( n, δ ) = log(1 /δ ) + log n + log ( K ( K +1) / 2) and f 2 ( n, δ ) = log(1 /δ ) + 2 log n + log ( K ( K +1) / 2) , which satisfy f 1 ( n, δ ) ≤ f 2 ( n, δ ) . Using Lemmas 40 and 41, we obtain that P ν ( E ∁ n,δ ) ≤ δ .

In the following, we use the notation on the gaps introduced in Appendix F.1, the complexities ( H i ( µ, ε 0 )) i ∈ [ C µ -1] defined in Theorem 6 and the function i µ ( ε ) defined in (21). To prove Theorem 6, we obtain as an intermediary step that: for all n > E ε 1 ,ε 0 ,µ ( δ ) , { ˆ ı n / ∈ I ε 1 ( µ ) } ⊆ E ∁ n,δ , where

where the equality holds by using the rewriting of the complexities done in the end of Appendix F.1. Let A = 8 H i µ ( ε 1 ) and B = 8 H i µ ( ε 1 ) log ( K ( K +1) / 2) + 5 K 2 / 2 . Using a proof similar to Lemma 51, applying Lemma 49 yields that

Let us define h 2 (log(1 /δ ) , A, B ) = 2 AW -1 ( 1 2 log(1 /δ ) + B 2 A +log(2 A ) ) and, for all i ∈ [ C µ -1] , we define

Therefore, we have shown that, for all ε 1 ≥ 0 ,

Using that P ν ( E ∁ n,δ ) ≤ δ , we can conclude that, for all ε 1 ≥ 0 ,

F.3 Cumulative regret of induced policy

Given a stream of recommendations (ˆ ı n ) n>K , an external agent aiming at minimizing its cumulative regret can simply pull the arm ˆ ı n at time n . Since the EB-TCa algorithm has anytime guarantees on the simple regret of this recommendation, the regret of the induced policy is constant (Corollary 2).

Corollary 2. Let ε 0 > 0 . Let (ˆ ı n ) n>K be the recommendations of the EB-TC ε 0 algorithm with fixed proportions β = 1 / 2 . An agent pulling ˆ ı n at time n has constant induced regret, i.e.

Proof. Using that H 1 ( µ, ε 0 ) = max i ∈ [ C µ -1] H i ( µ, ε 0 ) , a direct upper bound based on Corollary 1 yields that, for all n ≥ D µ ,

Let c > 0 . Let f ( c, x ) = ( c +log( x )) 2 x exp( -x ) for all x ≥ 1 . Let A > 0 , B > 0 . By integration, for all C ≥ A + B , we obtain

where C α = ∫ + ∞ 1 log( x ) α exp( -x )d x < + ∞ for all α ≥ 0 . For n ≥ 5 K 2 , we have n ≤ 2( n -5 K 2 / 2) . Recall that D µ = 8 H 1 ( µ, ε 0 ) h 2 ( 8 H 1 ( µ, ε 0 ) , 5 K 2 / 2 , 2 + log ( K ( K +1) / 2) ) +5 K 2 / 2 with h 2 defined in Lemma 51. For all n < 5 K 2 / 2+ D µ , we have trivially that E ν [ µ ⋆ -µ ˆ ı n ] ≤ ∆ max . Therefore, we obtain

Studying the function h 2 ( x, y, z ) , it is direct to see that h 2 ( x, y, z ) = x → + ∞ o (log( x ) α ) for all α > 0 , hence ∆ max D µ is dominated by the other term. Therefore, by studying the dominating term when min { ∆ min , ε 0 } → 0 , our upper bound scales as

The idea of decoupling exploration and exploitation when minimizing the regret in the multi-armed bandits literature was introduced by [2]. At each round, the agent is allowed to choose one arm to explore and one arm to exploit at every round. While the agent suffers the loss from the exploited arm, it observes the one of the explored arm without cost. In the stochastic regime for instances having a unique best arm, the Decoupled-Tsallis-INF algorithm [33] achieves a constant regret O ( K/ ∆ min ) . Therefore, in this setting, our constant regret in Corollary 2 does have the best achievable scaling in K and ∆ min .

Beyond synchronized policy When the external agent plays ˆ ı n at time n , it is said to be synchronized with the recommendation rule. In all generality, one could pull arm ˆ ı L ( n ) at time n , where L : N → N is a non-decreasing function, and obtain similar guarantees as Corollary 2. The agent can be slower than the recommendation rule (e.g. offline hyper-parameter optimization) hence L can be rapidly increasing, e.g. L ( n ) = Bn + K where B > 1 . The recommendation rule can be slower than the agent (e.g. pulling the same arm multiple times) hence L can be piecewise constant with small steps, e.g. L ( n ) = ⌈ n/B ⌉ + K . The communication between the agent and the recommendation rule can happen infrequently (e.g. paying to access the recommendation) hence L can be piecewise constant with large steps, e.g. L ( n ) = ( ⌈ n/B ⌉ ) α + K where α > 1 .

G Concentration results

The proof of Lemma 3 is given in Appendix G.1. Appendix G.2 gathers concentration results to control the empirical means and gaps.

G.1 Proof of Lemma 3

Proving that a GLR stopping rule ensures the algorithm to be ( ε 1 , δ ) -PAC is done by leveraging concentration results. In particular, we build upon Theorem 9 of [27] which is restated below. While Theorem 9 was only stated for Gaussian distributions, it is direct to notice that the result also holds for sub-Gaussian distributions with variance proxy σ 2 = 1 (as the authors mentioned).

Lemma 38 (Theorem 9 of [27]) . Consider a sub-Gaussian bandit ν with means µ ∈ R K . Let S ⊆ [ K ] and x > 0 .

where C G is defined in [27] by C G ( x ) = min λ ∈ ]1 / 2 , 1] g G ( λ )+ x and

̸

λ g G ( λ ) = 2 λ -2 λ log(4 λ ) + log ζ (2 λ ) -1 2 log(1 -λ ) , (23) where ζ is the Riemann ζ function and C G ( x ) ≈ x +log( x ) . Since ˆ ı n = i ⋆ ( µ n ) , standard manipulations yield that for all i =ˆ ı n )

⋆ ⋆

̸

where the second inequality is obtained with ( k, x, y ) = ( i ⋆ , µ i , µ i ⋆ ) since i / ∈ I ε 1 ( µ ) , and the third by union bound. By concavity of x ↦→ log(4 + log( x )) and N n,i ⋆ + N n,i ≤ ∑ k ∈ [ K ] N n,k = n -1 , we obtain

Combining the above with Lemma 38 for all i / ∈ I ε 1 ( µ ) and using that |I ε 1 ( µ ) | ≤ K -1 , we obtain

G.2 Sequence of concentration events

Lemma 39 is a standard concentration result for sub-Gaussian distribution, hence we omit the proof.

Lemma 39. Let X be an observation from a sub-Gaussian distribution with mean 0 and variance proxy σ 2 . Then, for all δ ∈ (0 , 1] ,

Lemma 40 gives a sequence of concentration events under which the empirical means are close to their true values.

Lemma 40. Let δ ∈ (0 , 1] and s ≥ 0 . For all n > K , let f 1 ( x, δ ) = log(1 /δ ) + (1 + s ) log x and

Then, for all n > K , P ν (( E 1 n,δ ) ∁ ) ≤ Kδ n s .

Proof. Let ( X s ) s ∈ [ n ] be i.i.d. observations from one sub-Gaussian distribution with mean 0 and variance proxy σ 2 = 1 . Then, 1 m ∑ m i =1 X i is sub-Gaussian with mean 0 and variance proxy σ 2 = 1 /m . By union bound over [ K ] and over m ∈ [ n ] , we obtain

where we used that µ t,k -µ k = 1 N t,k ∑ t s =1 1 ( I s = k ) X s,k and concentration results for subGaussian observations (Lemma 39).

Lemma 41 gives a sequence of concentration events under which the empirical gaps are close to their true values.

Lemma 41. Let δ ∈ (0 , 1] and s ≥ 0 . For all n > K , let f 2 ( x, δ ) = log(1 /δ ) + (2 + s ) log( x ) and

̸

Then, for all n > K , P ν (( E 2 n,δ ) ∁ ) ≤ K ( K -1) δ 2 n s .

Proof. Let ( X s ) s ∈ [ n ] and ( Y s ) s ∈ [ n ] be two streams of i.i.d. observations from two sub-Gaussian distributions with mean 0 and variance proxy σ 2 = 1 . Then, 1 m 1 ∑ m 1 i =1 X i -1 m 2 ∑ m 2 i =1 Y i is subGaussian with mean 0 and variance proxy σ 2 = 1 m 1 + 1 m 2 . By union bound , we obtain

̸

̸

where we used that ( µ t,i ⋆ -µ i ⋆ ) -( µ t,k -µ k ) = 1 N t,i ⋆ ∑ t s =1 1 ( I s = i ⋆ ) X s,i ⋆ -1 N t,k ∑ t s =1 1 ( I s = k ) X s,k and concentration results for sub-Gaussian observations (Lemma 39).

Tighter concentration Lemma 42 provides concentration results on the empirical means, which are tighter than the one obtained in Lemma 40.

Lemma 42. Let δ ∈ (0 , 1] and s ≥ 0 . Let W -1 defined in Lemma 49. For all n > K , let

and

Then, for all n > K , P ν (( ˜ E 1 n,δ ) ∁ ) ≤ Kδ n s .

Proof. Let ( X s ) s ∈ [ n ] be i.i.d. observations from one sub-Gaussian distribution with mean 0 and variance proxy σ 2 = 1 . Let S t = ∑ s ∈ [ t ] X s . To derived concentration result, we use peeling.

Let η > 0 , γ > 0 and D = ⌈ log( n ) log(1+ η ) ⌉ . For all i ∈ [ D ] , let N i = (1 + η ) i -1 . For all i ∈ [ D ] , we define the family of priors f N i ,γ ( x ) = √ γN i 2 π exp ( -x 2 γN i 2 ) with weights w i = 1 D and process

which satisfies M (0) = 1 . It is direct to see that M ( t ) = exp ( xS t -1 2 x 2 t ) is a non-negative supermartingale since sub-Gaussian distributions with mean 0 and variance proxy σ 2 = 1 satisfy

By Tonelli's theorem, then M ( t ) is also a non-negative supermartingale of unit initial value.

̸

Let i ∈ [ D ] and consider t ∈ [ N i , N i +1 ) . For all x ,

Direct computations shows that

Minoring M ( t ) by one of the positive term of its sum, we obtain

Using Ville's maximal inequality for non-negative supermartingale, we have that with probability greater than 1 -δ , log M ( t ) ≤ log (1 /δ ) . Therefore, with probability greater than 1 -δ , for all i ∈ [ D ] and t ∈ [ N i , N i +1 ) ,

Since this upper bound is independent of t , we can optimize it and choose γ as in Lemma 43.

Lemma 43 (Lemma A.3 in [9]) . For a, b ≥ 1 , the minimal value of f ( η ) = (1 + η )( a +log( b + 1 η )) is attained at η ⋆ such that f ( η ⋆ ) ≤ 1 -b + W -1 ( a + b ) . If b = 1 , then there is equality.

Therefore, with probability greater than 1 -δ , for all i ∈ [ D ] and t ∈ [ N i , N i +1 ) ,

The second inequality is obtained since D ≤ 1+ log n log(1+ η ) . The last equality is obtained for the choice η ⋆ = e 2 -1 , which minimizes η ↦→ log(1 + η ) -2 log(log(1 + η )) . Since [ n ] ⊆ ⋃ i ∈ [ D ] [ N i , N i +1 ) and N t,k ( µ t,k -µ k ) = ∑ s ∈ [ N t,k ] X s,k (unit-variance), this yields

Since 3 -2 log 2 ≤ 2 and W -1 is increasing, taking δn -s yields

Doing a union bound over arms yields the result.

Lemma 44 provides concentration results on the empirical gaps, which are tighter than the ones obtained in Lemma 41.

Lemma 44. Let δ ∈ (0 , 1] and s ≥ 0 . Let W -1 defined in Lemma 49. For all n > K , let

and

̸

Then, for all n > K , P ν (( ˜ E 2 n,δ ) ∁ ) ≤ K ( K -1) 2 δ n s .

Proof. Without loss of generality, we show the result for arms ( i, k ) = (1 , 2) . Let ( X s ) s ∈ [ n ] and ( Y s ) s ∈ [ n ] be i.i.d. observations from two sub-Gaussian distribution with mean 0 and variance proxy σ 2 = 1 . Let S t,k = ∑ s ∈ [ t ] 1 ( I s = k ) X s for all k ∈ { 1 , 2 } . To derived concentration result, we use peeling as in Lemma 42.

Let η > 0 , γ > 0 and D = ⌈ log( n ) log(1+ η ) ⌉ . For all i ∈ [ D ] , let N i = (1 + η ) i -1 . For all i ∈ [ D ] , we define the family of priors f N i ,γ ( x ) = √ γN i 2 π exp ( -x 2 γN i 2 ) with weights w i = 1 D and process

which satisfies M k (0) = 1 for all k ∈ { 1 , 2 } . As in the proof of Lemma 42, we obtain that M 1 ( t ) and M 2 ( t ) are non-negative supermartingale such that

where we used ( i 1 , i 2 ) ∈ [ D ] 2 and consider N t,k ∈ [ N i k , N i k +1 ) for all k ∈ { 1 , 2 } . Let us define M ( t ) = M 1 ( t 1 ) M 2 ( t 2 ) . Then, we have that M ( t ) is a non-negative supermartingale such that

Using Ville's maximal inequality for non-negative supermartingale, we have that with probability greater than 1 -δ , log M ( t ) ≤ log (1 /δ ) . Therefore, with probability greater than 1 -δ , for all ( i 1 , i 2 ) ∈ [ D ] 2 and N t,k ∈ [ N i k , N i k +1 ) for all k ∈ { 1 , 2 } ,

Since this upper bound is independent of t , we can optimize it and choose γ as in Lemma 43. Therefore, with probability greater than 1 -δ , for all ( i 1 , i 2 ) ∈ [ D ] 2 and N t,k ∈ [ N i k , N i k +1 ) for all k ∈ { 1 , 2 } ,

The second inequality is obtained as in Lemma 42 by using that D ≤ 1 + log n log(1+ η ) , W -1 increasing, taking η ⋆ = e 2 -1 and using 3 -2 log 2 ≤ 2 .

Since [ n ] ⊆ ⋃ i ∈ [ D ] [ N i , N i +1 ) , N t,k ≤ n and N t,k ( µ t,k -µ k ) = S t,k (unit-variance), this yields

Let C ( n, δ ) := W -1 (log (1 /δ ) + 2 log (2 + log n ) + 2) . In the following, we consider that we are under the event,

which has probability at least 1 -δ . Since it satisfies the above constraint, the quantity µ t, 1 -µ 1 -( µ t, 2 -µ 2 ) is upper bounded by

Introducing a Lagrange multiplier α , the above optimization problem is equivalent to

Solving for x by cancelling the derivative yields that x 1 = µ t, 1 -1 αN t, 1 and x 2 = µ t, 2 + 1 αN t, 2 . Therefore, we obtain as solution

where the last equality is obtained by solving the optimization since the derivative is null at α ⋆ = √∑ k ∈{ 1 , 2 } 1 2 C ( n,δ ) N t,k . By symmetry, we obtain the same result to upper bound µ t, 2 -µ 2 -( µ t, 1 -µ 1 ) . Therefore, under E n,δ (1 , 2) , we have shown that

̸

The same argument as above can be applied for all ( i, k ) ∈ [ K ] 2 such that i = k . A direct union bound yields

̸

where we used the inclusion proven above for ( n, δ/n s ) . This concludes the proof.

Global events Lemma 45 combines the above sequences of concentration events.

Lemma 45. Let s > 1 . Let ( E 1 n,δ ) n>K and ( E 2 n,δ ) n>K as in (24) and (25) . Let ( ˜ E 1 n,δ ) n>K and ( ˜ E 2 n,δ ) n>K as in (27) and (29) . Let us define E n,δ := E 1 n,δ ∩ E 2 n,δ and ˜ E n,δ := ˜ E 1 n,δ ∩ ˜ E 2 n,δ for all n > K . Then,

Proof. It is direct to see that

The same proof can be applied to upper bound ∑ n>K P ν ( ˜ E ∁ n,δ ) .

H Technicalities

Appendix H gathers existing and new technical results which are used for our proofs.

Key technical result Lemma 4 is the key technical ingredient on which our proofs rely on. It builds on a sequence of 'bad' events such that, under each 'bad' event, either the leader or the challenger was not often selected as leader or challenger, and shows that the number of times those 'bad' events occur is small.

Lemma (Lemma 4) . Let δ ∈ (0 , 1] and n > K . Let ( A t,δ ( n, δ )) n ≥ t>K be a sequence of events and ( D i ( n, δ )) i ∈ [ K ] be positive thresholds satisfying that, for all t ∈ [ n ] \ [ K ] , under the event A t,δ ( n, δ ) ,

Then, we have ∑ n t = K +1 1 ( A t,δ ( n, δ )) ≤ ∑ i ∈ [ K ] D i ( n, δ ) .

Proof. Using the inclusion of events given by the assumption on ( A t,δ ( n, δ )) n ≥ t>K , we obtain

The second inequality is obtained by union bound. The third inequality is direct since the number of times one can increase by one a quantity that is positive and bounded by D i ( n, δ ) is at most D i ( n, δ ) .

Tracking Lemma 46 provide general results for the K ( K -1) tracking procedures both for IDS and fixed proportions, which includes the ones of Lemma 2. The main theoretical argument behind those results is based on applying Theorem 6 in [12].

̸

Lemma 46. For all n > K , i ∈ [ K ] , j = i , we have

̸

̸

Let β ∈ (0 , 1) . For fixed proportions β , we have

Proof. We have K ( K -1) independent two-arms C-Tracking between the challenger and the leader. Theorem 6 in [12] yields the result. The second inequality is a simple re-ordering.

To obtain the second part, direct manipulations yield that

̸

̸

̸

̸

which allows to conclude since N i n,i = ∑ j = i ( T n ( i, j ) -N i n,j ) . The lower bound is obtained similarly.

The third is a direct consequence of the first part since

̸

Having shown

it is direct to conclude.

Methodology Lemma 47 is a standard result to upper bound the expected sample complexity of an algorithm, e.g. see Lemma 1 in [11]. This is a key method extensively used in the literature.

Lemma 47. Let ( E n ) n>K be a sequence of events and T ( δ ) > K be such that for n ≥ T ( δ ) , E n ⊆ { τ δ ≤ n } . Then, E ν [ τ δ ] ≤ T ( δ ) + ∑ n>K P ν ( E ∁ n ) .

̸

̸

̸

̸

̸

̸

̸

̸

̸

Proof. Since the random variable τ δ is positive and { τ δ > n } ⊆ E ∁ n for all n ≥ T ( δ ) , we have

which concludes the proof by adding positive terms.

Lemma 48 is a key method to upper bound the probability of error of an algorithm.

Lemma 48. Let ε ≥ 0 and ( E n,δ ) n>K,δ ∈ (0 , 1) be a sequence of events such that P ν ( E ∁ n,δ ) ≤ Cδ with C > 0 . Suppose that T ε ( δ ) > K is such that for n > T ε ( δ ) , E n,δ ⊆ { ˆ ı n ∈ I ε ( µ ) } . Then,

Proof. Using the assumption, we have P ν (ˆ ı n / ∈ I ε ( µ )) ≤ P ν ( E ∁ n,δ ) ≤ Cδ for all n > T ε ( δ ) . Taking the infimum yields the result.

Inversion results Lemma 49 gathers properties on the function W -1 , which is used in the literature to obtain concentration results.

Lemma 49 ([22]) . Let W -1 ( x ) = -W -1 ( -e -x ) for all x ≥ 1 , where W -1 is the negative branch of the Lambert W function. The function W -1 is increasing on (1 , + ∞ ) and strictly concave on (1 , + ∞ ) . In particular, W ′ -1 ( x ) = ( 1 -1 W -1 ( x ) ) -1 for all x > 1 . Then, for all y ≥ 1 and x ≥ 1 ,

Moreover, for all x > 1 ,

Lemma 50 provides an ordering on the thresholds for tighter concentration results. It leverages properties of W -1 .

Lemma 50. Let s ≥ 0 . Let W -1 defined in Lemma 49. For all δ ∈ (0 , 1] and n ≥ 1 , let us denote by

Then, we have ˜ f 1 ( n, δ ) ≤ ˜ f 2 ( n, δ ) .

Proof. Let a > 0 and b ≥ 1 . Let us define f ( x ) = 1 x W -1 ( ax + b ) . Using Lemma 49, we have

Therefore, we obtain f ′ ( x ) ≤ 0 if and only if W -1 ( ax + b ) ≥ ax +1 . Since W -1 ( ax + b ) ≥ ax + b and b ≥ 1 , the function f ( x ) is decreasing for all x ≥ 1 . Therefore, 1 2 W -1 (2 x + b ) ≤ W -1 ( a + b ) . Applying this result with a = log (1 /δ ) + s log n and b = 2log(2 + log n ) + 2 ≥ 1 for all n ≥ 1 , we obtain ˜ f 1 ( n, δ ) ≤ ˜ f 2 ( n, δ ) .

Lemma 51 is an inversion result to upper bound a time which is implicitly defined. It is a direct consequence of Lemma 49.

Lemma 51. Let W -1 defined in Lemma 49. Let A > 0 , B > 0 such that B/A +log A > 1 and

Then, C ( A,B ) < h 1 ( A,B ) with h 1 ( z, y ) = zW -1 ( y/z +log z ) and C ( A,B ) < Ah 2 ( A,B,D ) + B where

Proof. Since B/A +log A > 1 , we have C ( A,B ) ≥ A , hence

Using Lemma 49 yields that

By changing the variable, we obtain

Lemma 52 is an inversion result to upper bound a probability which is implicitly defined based on times that are implicitly defined.

Lemma 52. Let W -1 defined in Lemma 49. Let A > 0 , B > 0 , C > 0 , E > 0 , α > 0 , β > 0 and D A,B,C ( δ ) = sup { x | x ≤ A (log(1 /δ ) + C log x ) + B } ,

Then,

Proof. Direct manipulations yield that

Likewise, using Lemma 49, we obtain

I Multiplicative setting

Let ν ∈ D K with mean parameters µ ∈ ( R ⋆ + ) K . Compared to the additive setting, we only consider strictly positive mean parameters. In many applications, this assumption is natural. In other applications, there is a known lower bound on the value of the true mean parameters, hence we can simply translate the problem by adding this lower bound. We also note that the problem of identifying which arms are above the threshold 0 is the thresholding bandit problem, which has been extensively studied in the literature. Overall, while this assumption might seem restrictive, we think it is rather mild. Since the means are strictly positive, the multiplicative error ε has also to be positive and strictly smaller than 1 . Given a multiplicative error ε ∈ [0 , 1) , the set of multiplicative ε -good arms are denoted by I mul ε ( µ ) := { i ∈ [ K ] | ∆ i ≤ µ ⋆ ε } . We will refer to this problem as multiplicative ε -BAI.

First, we discuss the characteristic times involved in the fixed-confidence multiplicative ε -BAI (Appendix I.1). Second, for ε 0 ∈ (0 , 1) , we present the EB-TC m ε 0 with IDS or fixed β proportions (Appendix I.2). Finally, we prove that EB-TC m ε 0 is asymptotically (resp. β -)optimal when using IDS (resp. fixed β ) when combined with the appropriate GLR stopping rule (Appendix I.3).

I.1 Characteristic times

Let ν ∈ D K with mean parameters µ ∈ ( R ⋆ + ) K . We assume in the following that there exists a unique best arm, i.e. i ⋆ ( µ ) = { i ⋆ } . Let ε ∈ [0 , 1) . The ( β -)characteristic times for the fixed confidence multiplicative ε -BAI setting with Gaussian bandits N ( µ, 1) are defined as T mul ε ( µ ) = min i ∈I mul ε ( µ ) ,β ∈ (0 , 1) T mul ε,β ( µ, i ) and T mul ε,β ( µ ) = min i ∈I mul ε ( µ ) T mul ε,β ( µ, i ) where

̸

We denote by w mul ε ( µ, i ) and w mul ε,β ( µ, i ) their maximizer, which we refer to as the ( β -)optimal allocation for multiplicative ε -BAI.

Even though the multiplicative ε -BAI is less studied, it is direct to see that it has similar properties as T 0 ( µ ) . Due to the factor (1 -ε ) 2 in the denominator, it is not possible to simply use a reduction to a BAI problem as we did for the additive ε -BAI problem (see Lemma 9). Since the properties listed in Appendix C are obtained by studying the optimization problem defining T 0 ( µ ) , we will have the same properties for T mul ε ( µ ) by accounting for the extra multiplicative factor (1 -ε ) 2 . We list here the properties on which our proof relies:

T mul ε ( µ ) = min β ∈ (0 , 1) T mul ε,β ( µ, i ⋆ ) and T mul ε,β ( µ ) = T mul ε,β ( µ, i ⋆ ) , meaning the unique best arm corresponds to the arm which is the easiest to verify multiplicative ε -BAI.

w mul ε ( µ ) = { w mul ε } and w mul ε,β ( µ ) = { w mul ε,β } , meaning there is a unique ( β -)optimal allocation which satisfies min i ∈ [ K ] ( w mul ε ) i > 0 and min i ∈ [ K ] ( w mul ε,β ) i > 0 .

̸

There is equality at the equilibrium of the transportation costs, meaning for all i = i ⋆

and the multiplicative overall balance equation is satisfied

̸

For the sake of space, we omit the proofs and defer the reader to Appendix C for ideas on how to prove them.

I.2 Anytime Top Two algorithm

Let ε 0 ∈ (0 , 1) . The EB-TC m ε 0 algorithm, detailed in Figure 3, has the same structure as EB-TC ε 0 . There are two differences. First, the IDS proportions are β n ( i, j ) = N n,j / ((1 -ε 0 ) 2 N n,i + N n,j ) when considering multiplicative ε -BAI. Second, the challenger will be based on transportation costs for multiplicative ε -BAI, namely

̸

Stopping rule for fixed-confidence multiplicative ε -BAI Similarly, the fixed-confidence setting requires a stopping rule, which will be different since we tackle multiplicative ε -BAI. Given an error/confidence pair ( ε, δ ) ∈ [0 , 1) × (0 , 1) , the GLR m ε stopping rule [15] prescribes to stop at the time

̸

where c : N × (0 , 1) → R + is a threshold function. Lemma 53 gives a threshold ensuring that the GLR m ε stopping rule is ( ε, δ ) -PAC for all ε ∈ [0 , 1) and δ ∈ (0 , 1) , independently of the sampling rule.

̸

1: Input: slack ε 0 ∈ (0 , 1) , proportion β ∈ (0 , 1) (only for fixed proportions).

̸

2: Set ˆ ı n ∈ arg max i ∈ [ K ] µ n,i , B EB n =ˆ ı n and C TC m ε 0 n ∈ arg max i = B EB n µ n,B EB n -(1 -ε 0 ) µ n,i √ 1 /N n,B EB n +(1 -ε 0 ) 2 /N n,i .

3: Set [ fixed ] ¯ β n +1 ( i, j ) = β or [ IDS ] β n ( i, j ) = N n,j / ((1 -ε 0 ) 2 N n,i + N n,j ) and update ¯ β n +1 ( i, j ) .

4: Set I n = C TC m ε 0 n if N B EB n n,C TC m ε 0 n ≤ (1 -¯ β n +1 ( B EB n , C TC m ε 0 n )) T n +1 ( B EB n , C TC m ε 0 n ) , otherwise set I n = B EB n .

5: Output : next arm to sample I n and next recommendation ˆ ı n .

Lemma 53. Let ε ∈ [0 , 1) and δ ∈ (0 , 1) . Given any sampling rule, using the threshold (4) with the stopping rule (34) with error/confidence pair ( ε, δ ) yields a ( ε, δ ) -PAC algorithm for sub-Gaussian distributions.

̸

Proof. The proof of Lemma 53 is actually almost identical to the one of Lemma 3. The only change is at the beginning of the proof. Since ˆ ı n = i ⋆ ( µ n ) , standard manipulations yield that for all i =ˆ ı n

⋆ ⋆

where the second inequality is obtained with ( k, x, y ) = ( i ⋆ , µ i , µ i ⋆ ) since i / ∈ I mul ε ( µ ) , and the third by union bound. Then, we can conclude as in Appendix G.1 that

Asymptotic expected sample complexity While Theorem 8 holds for all sub-Gaussian distributions, it is particularly interesting for Gaussian ones, in light of Lemma 1. It shows that, for multiplicative ε -BAI, EB-TC m ε 0 is asymptotically optimal for Gaussian bandits when using IDS proportions and asymptotically β -optimal when using fixed proportions β .

Theorem 8. Let ε 0 ∈ (0 , 1) , β ∈ (0 , 1) and δ ∈ (0 , 1) . Using the threshold (4) in the stopping rule (34) with error/confidence ( ε 0 , δ ) , the EB-TC m ε 0 algorithm is ( ε 0 , δ ) -PAC and it satisfies that, for all ν ∈ D K such that | i ⋆ ( µ ) | = 1 ,

I.3 Proof of Theorem 8

The proof is very similar to the one of Theorem 1, hence we will only highlight key differences. We defer the reader to Appendix D for a detailed discussion on the proof method and intuition on the different steps of the proof, see also [36, 21, 40].

In the following, we consider a slack ε 0 for EB-TC m ε 0 which matches the error ε considered by the GLR m ε stopping rule (34), i.e. ε 0 = ε . For conciseness, we denote w ⋆ = w mul ε and w ⋆ β = w mul ε,β .

Let γ > 0 . Let us define the convergence times T ε 0 ,γ and T ε 0 ,β,γ as in (10) and (14), i.e.

̸

Lemma 54 gives a sufficient condition for asymptotic optimality and asymptotic β -optimality. The proof is the same as for Lemmas 13 and 25, hence we omit it.

Lemma 54. Let ε 0 ∈ (0 , 1) and δ ∈ (0 , 1) . Assume that there exists γ 1 ( µ ) > 0 such that for all γ ∈ (0 , γ 1 ( µ )] , E ν [ T ε 0 ,γ ] < + ∞ . Using the threshold (4) in the stopping rule (3) with error ε 0 yields an algorithm such that, for all ν ∈ D K such that | i ⋆ ( µ ) | = 1 ,

Assume that there exists γ 1 ( µ ) > 0 such that for all γ ∈ (0 , γ 1 ( µ )] , E ν [ T ε 0 ,β,γ ] < + ∞ . Using the threshold (4) in the stopping rule (3) with error ε 0 yields an algorithm such that, for all ν ∈ D K such that | i ⋆ ( µ ) | = 1 ,

I.3.1 Sufficient exploration

Lemma 55 shows that the transportation cost is strictly positive and increases linearly, it bares similarity with Lemma 14.

̸

Lemma 55. Let S L n and I ⋆ n as in (11). There exists L 0 with E µ [( L 0 ) α ] < + ∞ for all α > 0 such that if L ≥ L 0 , for all n such that S L n = ∅ , for all ( i, j ) ∈ I ⋆ n × S L n such that i = j , we have

where D µ > 0 is a problem dependent constant.

Proof. Using Lemma 12 and min { N n,i , N n,j } ≥ L , we obtain

where the last inequality is obtained for L ≥ L 0 = L 1 -e which is defined as

The last inequality is obtained by using Lemma 51, and we recall that h 1 is defined in Lemma 51. Since h 1 ( x, e ) ∼ x → + ∞ x log x , we have E µ [( L 0 ) α ] < + ∞ for all α > 0 by using Lemma 12 (polynomial in W µ ). Then, we have

̸

Lemma 56 gives an upper bound on the transportation costs between a sampled enough arm and an under-sampled one it bares similarity with Lemma 15.

Lemma 56. Let S L n as in (11) . There exists L 1 with E µ [( L 1 ) α ] < + ∞ for all α > 0 such that for all L ≥ L 1 and all n ∈ N ,

where D 1 > 0 is a problem dependent constant and W µ is the random variables defined in Lemma 12.

Proof. Using Lemma 12 and N n,i ≥ L ≥ N n,j ≥ 1 , we obtain for

̸

Lemma 57 show that the desired property for the TC challenger, we omit the proof since it is the same as for Lemma 17.

̸

Lemma 57. There exists L 1 with E ν [ L 3 ] < + ∞ such that if L ≥ L 3 , for all n (at most polynomial in L ) such that U L n = ∅ , B EB n / ∈ V L n implies C TC m ε 0 n ∈ V L n .

Combining Lemma 16 and Lemma 57 yields Lemma 58. Importantly, Lemma 58 holds for the EB-TC ε 0 algorithm with fixed proportions β ∈ (0 , 1) and IDS proportions. Since the proof is the same as for Lemmas 18 and 26, we omit it.

Lemma 58. There exist N 1 with E ν [ N 1 ] < + ∞ such that for all n ≥ N 1 and all i ∈ [ K ] , N n,i ≥ √ n/K and B EB n = i ⋆ .

I.3.2 Empirical overall balance

As in [40], the key to obtain asymptotic optimality is to show that the empirical proportion satisfy the empirical overall balance equation. Compared to them, the novelty of Lemma 59 is that we use IDS proportions with K ( K -1) tracking procedures to select between the leader and the challenger instead of sampling. The proof of Lemma 59 is highly similar to the one of Lemma 19.

Lemma 59. Let γ > 0 . There exists N 2 with E ν [ N 2 ] < + ∞ such that for all n ≥ N 2

̸

Proof. Let N 1 as in Lemma 58. We proceed as in Lemma 19. Let us define

̸

Direct manipulations yield that

Using that β n ( i ⋆ , C n ) N n,i ⋆ = (1 -β n ( i ⋆ , C n )) N n,C n / (1 -ε 0 ) 2 since β n ( i ⋆ , C n ) = N n,C n / ((1 -ε 0 ) 2 N n,i ⋆ + N n,C n ) and the inequalities proven in Lemma 19, we can exhibit deterministic constants A,B > 0 (depending on K and ε 0 ) such that

| G n +1 -G n | ≤ AN 1 + B hence | G n | ≤ ( n -1 -N 1 )( AN 1 + B ) + (1 -ε 0 ) 2 . where the second inequality is obtained by telescopic sum (as in Lemma 19) and using that G N 1 +1 = β N 1 ( i ⋆ , C N 1 ) 2 -(1 -β N 1 ( i ⋆ , C N 1 )) 2 / (1 -ε 0 ) 2 .

As in Lemma 19, it is possible to exhibit deterministic constants C, D > 0 (depending on K and ε 0 ) such that

̸

Combining both results, we can show

̸

Therefore, we have shown the desired result for n ≥ N 2 defined as

which satisfies E ν [ N 2 ] < + ∞ since it is a linear function of N 1 .

As in [40], using Lemma 59 allows to bound the empirical proportion allocated to the unique best arm N n,i ⋆ / ( n -1) away from 0 (Lemma 60).

Lemma 60. There exists N 3 with E ν [ N 3 ] < + ∞ such that for all n ≥ N 3

Proof. Let N 2 as in Lemma 59 for γ (1 -ε 0 ) 4 . Using Lemma 59, we obtain for all n ≥ N 2

̸

where we used that N n,i ⋆ n -1 ≤ 1 . Taking γ = (1 -ε 0 ) 2 / 2 yields the upper bound.

Let ˜ N 2 as in Lemma 59 for γ ( K -1)(1 -ε 0 ) 2 . Using Lemma 59, we obtain for all n ≥ max { N 2 , ˜ N 2 }

̸

Taking γ = (1 -c U ) 2 / 2 yields the lower bound. Note that

˜

where the last condition is trivially true. Taking N 3 = max { N 2 , N 2 } yields the result.

Lemma 61 is a rescaled version of the empirical overall balance equation which is proven by simply combining Lemma 59 and Lemma 60.

Lemma 61. Let γ > 0 . There exists N 4 with E ν [ N 4 ] < + ∞ such that for all n ≥ N 4

̸

Proof. Let N 3 and c L as in Lemma 60. Let N 2 as in Lemma 59 for γc 2 L / (1 -ε 0 ) 2 . Direct manipulation shows that, for all n ≥ N 4 = max { N 2 , N 3 } ,

̸

This concludes the result.

I.3.3 Convergence towards optimal allocation

As in [40], we show that a challenger will not be sampled next when the ratio of its empirical proportion compared to the one of i ⋆ overshoots the ratio of the optimal allocations (Lemma 62).

̸

Lemma 62. Let γ > 0 . There exists N 5 with E ν [ N 5 ] < + ∞ such that for all n ≥ N 5 and all i = i ⋆ ,

̸

Proof. Let γ > 0 . Let i = i ⋆ such that

̸

̸

Let N 4 as in Lemma 61. As in Lemma 22, we can show for all n ≥ N 4

Let N 1 as in Lemma 18. Using that B EB n = i ⋆ for all n ≥ N 1 and the definition of C TC m ε 0 n , we known that

̸

To conclude the proof, it is sufficient to show that the ratio of the two quantities is strictly larger than one. For all n ≥ max { N 1 , N 4 } , we obtain

̸

Let ˜ γ > 0 . Using Lemmas 12 and 18, we have, for all n ≥ N 1 and all k = i ⋆ ,

̸

for all n ≥ N 6 = K ( X 0 -e ) 2 +1 which is defined as

̸

̸

where the last inequality is obtained by using Lemma 51 with h 1 defined therein. Since h 1 ( x, y ) ∼ x → + ∞ x log x , we have E ν [ N 6 ] < + ∞ since it polynomial in W µ (by using Lemma 12). Let κ > 0 . We have shown that, for all n ≥ max { N 1 , N 4 , N 6 } ,

where the equality uses that the transportation costs are equal at the equilibrium, i.e. (31). Taking ˜ γ small enough, we have that shown that, for all n ≥ max { N 1 , N 4 , N 6 } ,

̸

N n,i N n,i ⋆ ≥ w ⋆ i w ⋆ i ⋆ + γ = ⇒ µ n,i ⋆ -(1 -ε 0 ) µ n,i µ n,i ⋆ -(1 -ε 0 ) µ n,j √ 1 + (1 -ε 0 ) 2 N n,i ⋆ /N n,j 1 + (1 -ε 0 ) 2 N n,i ⋆ /N n,i > 1 , hence C TC m ε 0 n = i . This concludes the proof.

Lemma 63 is obtained with the same proof as Lemma 23.

Lemma 63. Let γ > 0 . There exists N 6 with E ν [ N 6 ] < + ∞ such that for all n ≥ N 6 and all i = i ⋆ ,

Lemma 64 is obtained with the same proof as Lemma 24.

Lemma 64. Let ε 0 ∈ (0 , 1) , γ > 0 and T ε 0 ,γ introduced before (adaptation of (10) ). Under the EB-TC m ε 0 sampling rule with IDS proportions, we have E ν [ T ε 0 ,γ ] < + ∞ .

Combining Lemmas 54 and 64 yields the first of Theorem 8.

I.3.4 Convergence towards β -optimal allocation

Combining Lemmas 58 and 27 with the proof of Lemma F.9 in [20] yields Lemma 65.

̸

Lemma 65. Let γ > 0 . There exists N 5 with E ν [ N 5 ] < + ∞ such that for all n ≥ N 5 and all i = i ⋆ ,

̸

Lemma 66 is obtained with the same proof as Lemmas 29 and 30.

Lemma 66. Let ε 0 ∈ (0 , 1) , β ∈ (0 , 1) , γ > 0 and T ε 0 ,β,γ introduced before (adaptation of (14) ). Under the EB-TC m ε 0 sampling rule with fixed proportions β , we have E ν [ T ε 0 ,β,γ ] < + ∞ .

Combining Lemmas 54 and 66 yields the second part of Theorem 8.

J Implementation details and additional experiments

After presenting the implementations details in Appendix J.1, we display supplementary experiments in Appendix J.2.

̸

J.1 Implementation details

Top Two sampling rules Given a fixed β , most Top Two algorithms use randomization to select I n ∈ { B n , C n } , namely they sample B n with probability β , otherwise sample C n . Instead of randomization, TTUCB [20] uses K tracking procedures to select I n ∈ { B n , C n } . Namely, they choose I n = B n if N B n n,B n ≤ β ∑ i = B n T n +1 ( B n , i ) , and I n = C n otherwise. In comparison, EB-TC ε 0 relies on K ( K -1) tracking procedures both for fixed proportion β and IDS proportions [40]. In [40], they consider IDS proportions to define β n adaptively. For Gaussian with known homoscedastic variance, their update mechanism is

̸

where the second equality is obtained by direct computations which uses that

[8] proposed a new methodology called Balancing Optimal Large Deviations (BOLD) to select I n ∈ { B n , C n } adaptively. While their approach aims at minimizing the probability of incorrect selection (i.e. fixed-budget setting), it is possible to adapt their idea to minimize the expected sample complexity (i.e. fixed-confidence setting) by 'swapping' the arguments of the KL divergence. Namely, the BOLD procedure can be rewritten in our setting as selecting I n = B n if

̸

and I n = C n otherwise. For Gaussian with known homoscedastic variance, their approach recovers the heuristic 'adaptive Welch divergence' algorithm of proposed in [37], i.e. I n = B n if N 2 n,B n < ∑ i = B n N 2 n,i , and I n = C n otherwise. The analysis of [8] is focused on asymptotic guarantees in probability. It is an interesting direction for future research to show that the BOLD procedure also yields asymptotically optimal algorithms.

̸

TTTS [34] uses a TS (Thompson Sampling) leader and a RS (Re-Sampling) challenger based on a sampler Π n . For Gaussian bandits, the sampler Π n is the posterior distribution × i ∈ [ K ] N ( µ n,i , 1 /N n,i ) given the improper prior Π 1 = ( N (0 , + ∞ )) K . The TS leader is B TS n ∈ arg max i ∈ [ K ] θ i where θ ∼ Π n . The RS challenger samples vector of realizations θ ∈ Π n until B n / ∈ arg max i ∈ [ K ] θ i , then it is defined as C RS n arg max i ∈ [ K ] θ i for this specific vector of realization. When the posterior Π n and the leader B n have almost converged towards the Dirac distribution on µ and the best arm i ⋆ ( µ ) respectively, the event B n / ∈ arg max i ∈ [ K ] θ i becomes very rare. The experiments in [21] reveals that computing the RS challenger can require more than millions of re-sampling steps. Therefore, the RS challenger can become computationally intractable even for Gaussian distribution where sampling from Π n can be done more efficiently.

T3C [36] combines the TS leader and the TC challenger. EB-TCI [21] combines the EB leader with the TCI challenger defined as

̸

TTUCB [20] combined the TC challenger with a UCB leader which is defined as

where √ g ( n ) /N n,i is a bonus coping for uncertainty. TS-KKT [40] uses the TS leader and a challenger based on transportation costs with a different penalization than the one used in [21], namely

̸

where ρ is a parameter that needs to be selected beforehand. The authors highlight that the choice of ρ has an important impact on the empirical performance.

While the above Top Two algorithms are BAI algorithms, we can adapt them straightforwardly to tackle the ε -BAI setting by (1) using the stopping rule as in (3) and (2) using transportation costs for ε -BAI, i.e. the ones in (2). The modification (1) is inspired by [19] and the modification (2) by [22]. As an example, the modified TCI challenger can be written as

̸

Fixed-confidence BAI algorithms At each time n , Track-and-Stop (TaS) [15] computes the optimal allocation for the current empirical mean, w n = w ⋆ ( µ n ) . Given w n ∈ △ K , it uses a tracking procedure to obtain an arm I n to sample. On top of this tracking a forced exploration is used to enforce convergence towards the optimal allocation for the true unknown parameters. The optimization problem defining w ⋆ ( µ ) can be rewritten as solving an equation ψ µ ( r ) = 0 , where

̸

̸

̸

The function ψ µ is decreasing, and satisfies lim r → + ∞ ψ µ ( r ) = -1 and lim y → 1 / min i = i ⋆ ( µ i ⋆ -µ i ) 2 F µ ( y ) = + ∞ . For the practical implementation of the optimal allocation, we use the approach of [15] and perform binary searches to compute the unique solution of ψ µ ( r ) = 0 . A faster implementation based on Newton's iterates was proposed by [3] after proving that ψ µ is convex. While this improvement holds only for Gaussian distributions, the binary searches can be used for more general distributions. Similarly, one can adapt TaS to tackle ε -BAI as done in [16], i.e. by tracking w n = w ε ( µ n ) . Using Lemma 9, it is direct to see that one can use the same implementation to compute it efficiently.

DKM[11] view T ⋆ ( µ ) -1 as a min-max game between the learner and the nature, and design saddlepoint algorithms to solve it sequentially. At each time n , a learner outputs an allocation w n , which is used by the nature to compute the worst alternative mean parameter λ n . Then, the learner is updated based on optimistic gains based on λ n . Similarly, one can adapt DKM to tackle ε -BAI by using a modified alternative mean parameter. This coincides with the L ε BAI algorithm [19] when used on unstructured multi-armed bandits.

FWS [39] alternates between forced exploration and Frank-Wolfe (FW) updates. Similarly, FWS can be adapted to tackle ε -BAI

LUCB [24] samples and stops based on upper/lower confidence indices for a bonus function g . For Gaussian distributions, it rewrites for all i ∈ [ K ] as

̸

At each time n , it samples ˆ ı n and arg max i =ˆ ı n U n,i . For ε -BAI, LUCB stops when L n, ˆ ı n + ε ≥ max i =ˆ ı n U n,i .

̸

Fixed-budget BAI algorithms We consider Successive Reject (SR) [1] and Sequential Halving (SH) [25]. SR eliminates one arm with the worst empirical mean at the end of each phase, and SH eliminated half of them but drops past observations between each phase. Within each phase, both algorithms use a round-robin uniform sampling rule on the remaining active arms. It is possible to convert the fixed-budget SH algorithm into an anytime algorithm by using the doubling trick. It considers a sequences of algorithms that are run with increasing budgets ( T k ) k ≥ 1 , with T k +1 = 2 T k and T 1 = 2 K ⌈ log 2 K ⌉ , and recommend the answer outputted by the last instance that has finished to run. It is well know that the 'cost' of doubling is to have a multiplicative factor 4 in front of the hardness constant. The first two-factor is due to the fact that we forget half the observations. The second two-factor is due to the fact that we use the recommendation from the last instance of SH that has finished. The doubling version of SR and SH are named Doubling SR (DSR) and Doubling SH (DSH).

Reproducibility Our code is implemented in Julia 1.9.0 , and the plots are generated with the StatsPlots.jl package. Other dependencies are listed in the Readme.md . The Readme.md file also provides detailed julia instructions to reproduce our experiments, as well as a script.sh to run them all at once. The general structure of the code (and some functions) is taken from the

tidnabbil library. This library was created by [11], see https://bitbucket.org/wmkoolen/ tidnabbil . No license were available on the repository, but we obtained the authorization from the authors. Our experiments are conducted on an institutional cluster with 4 Intel Xeon Gold 5218R CPU with 20 cores per CPU and an x86_64 architecture.

J.2 Supplementary experiments

For the sake of space, we only presented a small subset of our experiments in Section 5, hence we provide additional empirical evidence below. In Appendix J.2.1, we assess the impact of the choice of the slack ε 0 on the empirical performance of EB-TC ε 0 with IDS or with fixed proportion β = 1 / 2 . Appendix J.2.2 complements Section 5 by providing more comparison between EB-TC ε 0 and existing algorithms. In Appendix J.2.3, we study the BAI problem and show the empirical performance of the EB-TC ( ε n ) n>K with fixed proportion β = 1 / 2 and using varying slack parameters ( ε n ) n>K (as described in Sections 3 and 4). As in Section 5, we consider δ = 0 . 01 and use the heuristic threshold c ( n, δ ) = log((1 + log n ) /δ ) , which yields an empirical error which is several orders of magnitude lower than δ .

J.2.1 Numerical simulations on EB-TC ε 0

Appendix J.2.1 is meant to be a sensitivity analysis of the EB-TC ε 0 algorithm. It will allow us to numerically substantiate our recommendation to the practitioner on how to set the slack ε 0 , and why using IDS yields better empirical performance. For slacks ε 0 ∈ { 0 . 15 , 0 . 1 , 0 . 05 } , we will consider the EB-TC ε 0 algorithms with IDS proportions and fixed proportions β = 1 / 2 .

Large sets of arms First, we evaluate the impact of larger number of arms (up to K = 1000 ). The ' α = 0 . 6 ' scenario of [18] sets µ i = 1 -(( i -1) / ( K -1)) α for all i ∈ [ K ] . The 'sparse' scenario of [18] sets µ 1 = 1 / 4 and µ i = 0 otherwise. We average on 100 runs, and the standard deviations are displayed.

In Figure 4, we see that all instances of EB-TC ε 0 with IDs or fixed β = 1 / 2 have the same scaling with the dimension K . Figure 4 also reveals that using IDS instead of fixed β = 1 / 2 consistently yields smaller empirical stopping time. Therefore, IDS should be preferred to choosing fixed β = 1 / 2 . Finally, Figure 4 shows that better empirical performance are obtained when using a slack ε 0 which matches the error ε , i.e. choosing ε 0 = ε to tackle ε -BAI. While choosing ε 0 > ε only slightly damage the empirical performance, taking ε 0 < ε is clearly detrimental to the empirical performance.

Random instances We further investigate the phenomenon described above on random instances. For ε ∈ { 0 . 05 , 0 . 1 } , we assess the performance on 1000 random Gaussian instances with K = 5 (resp. K = 10 and K = 10 ) such that µ 1 = 1 , µ i ∼ U ([1 -ε, 1]) for i ∈ { 2 } (resp. i ∈ { 2 , 3 } and i ∈ { 2 , 3 , 4 } ) and µ i ∼ U ([0 , 1 -ε )) for i ≥ 3 (resp. i ≥ 4 and i ≥ 6 ), hence I ε ( µ ) = [2] (resp. I ε ( µ ) = [3] and I ε ( µ ) = [5] ). We display the boxplots of the empirical stopping time on 1000 runs.

In Figure 5, the tendencies glimpsed by inspecting Figure 4 are more apparent. Figure 5 shows that better empirical performance are obtained when using a slack ε 0 which matches the error ε , i.e. choosing ε 0 = ε to tackle ε -BAI. We also see that the empirical gain of taking ε 0 = ε is increasing with the dimension K . Figure 5 shows that ε 0 > ε slightly damage the empirical performance: the larger the missmatch ε 0 -ε is, the worse the performances are. Moreover, Figure 5 highlights how choosing ε 0 < ε is highly detrimental to the empirical performance. Finally, we see that there is a mild advantage in using IDS over fixed β = 1 / 2 . Note that the mild empirical gain of adaptive proportions is a known phenomenon which was studied empirically in [20], and larger gains can be observed for 'two-groups' instances. Intuitively, the characteristic times T ε ( µ ) and T ε, 1 / 2 ( µ ) are very close to each other on average, and the difference is the largest when all the sub-optimal arms have the same mean. Therefore, on average, using IDS has mild gain compared to using fixed β = 1 / 2 . This worst-case scenario of the 'two-groups' instances is studied in Figure 7.

Specific instances We continue the experimental validation of our the phenomenon described above by considering the three specific instances from the experimental section of [16] which are described in Table 3. Instance µ 3 corresponds to the specific instances studied in Section 5. It is particularly interesting since it has two best arms, a third ε -good arm which is equally distant from ε than the bad arm with largest mean, and has two additional bad arms. Instance µ 2 has also three ε -good arms, which are equally spaced, and three bad arms. Instance µ 1 has only one ε -good arm, and four bad arms. We display the boxplots of the empirical stopping time on 1000 runs, and the empirical simple regret on 10000 runs (with associated standard deviation).

The left column of Figure 6 confirms our previous observations: taking ε = ε 0 yields the best performance, taking ε 0 > ε slightly damage performance and taking ε 0 < ε highly damage performance.

Figure 6(middle left) shows that the the larger the missmatch ε -ε 0 is, the worse the performances are.

The right column of Figure 6 reveals that the choice of ε 0 has few impact on the empirical simple regret, at least for reasonably chosen ε 0 . Likewise, there is few differences between IDS proportions and fixed β = 1 / 2 . The good empirical performance of IDS proportions as regards the simple regret hints that EB-TC ε 0 with IDS has (most likely) similar theoretical guarantees as the ones obtained in Section 4 for EB-TC ε 0 with fixed β = 1 / 2 (Theorem 3 and Corollary 1). This is an interesting open problem that we leave for future work.

Two-groups instances We conclude our sensitivity analysis on EB-TC ε 0 by considering the 'twogroups' instances µ ∈ { 0 . 6 , 0 . 4 } 10 for varying number of best arms, i.e. | i ⋆ ( µ ) | ∈ [8] . Our aim is to assess the impact of having multiple best arms on the empirical stopping time. We average on 1000 runs, and the standard deviations are displayed.

On this specific instances where the number of arms is fixed, it would be intuitive to think that the problem is easier when there are more best arms. However, this is actually the opposite since there less bad arms (which were easy to detect) and more good arms which need to be estimated well enough. While Figure 7 shows how poor the performances are when taking ε 0 < ε , it also shows that the impact of having multiple best arms is rather mild for slacks ε 0 ≥ ε . We will see in Figure 12 that EB-TC ε 0 is the most resilient algorithm with respect to multiple arms, meaning that its slope is the smallest.

'Two-groups' instances are interesting since they are the instances for which the ratio T ε, 1 / 2 ( µ ) /T ε ( µ ) is the largest. Using Lemma 9 and the theoretical results of [20] (see their Lemma C.6), we conjecture that T ε, 1 / 2 ( µ ) ≤ r K T ε ( µ ) with r K = 2 K/ (1 + √ K -1) 2 and that the equality occurs for 'twogroups' instances. Since r 10 = 5 / 4 , using fixed β = 1 / 2 should be roughly 25% slower than using IDS. While Figure 7 confirms that using IDS yields better empirical performance than using fixed β = 1 / 2 , the empirical ratio between their performance is lower than the one suggested by the theory.

J.2.2 Comparison with modified BAI algorithms

Appendix J.2.2 is meant to provide further empirical evidence that EB-TC ε 0 outperforms its competitors on different tasks and for different types of instances. Overall, we will use the same instances as used in Appendix J.2.1 since they provide a wide range of interesting problems. Throughout this section, we will be using EB-TC ε 0 with slack ε 0 = 0 . 1 . While IDS proportions are used for the experiments on the empirical stopping time, we use fixed β = 1 / 2 for the empirical simple regret.

As benchmarks we consider modified BAI which are tailored to tackle ε -BAI, and we combine them with the GLR ε stopping rule. One can still wonder whether the unmodified sampling rule would perform well for ε -BAI when combined with the GLR ε stopping rule. While we could hope that modifying the stopping rule is enough, our experiments reveal that it is not the case. If we don't adapt the sampling rule as well, Table 4 shows that the empirical stopping time can be more than ten times larger than their modified version.

Large sets of arms To complement Figure 2(a), we consider the ' α = 0 . 6 ' instances and 'sparse' instances for varying K . We average on 100 runs, and the standard deviations are displayed.

Figure 8 confirms the observations made in Figure 2(a). We see that EB-TC ε performs on par with the ε -T3C, and outperforms the other algorithms. It also highlights that the regularization ensured by the TC ε challenger is sufficient to ensure enough exploration. As a consequence, other exploration mechanism are superfluous when using Top Two algorithms for ε -BAI (e.g. using TS/UCB leader or TCI challenger).

Random instances For ε ∈ { 0 . 05 , 0 . 1 } , we assess the performance on 1000 random Gaussian instances with K ∈ { 5 , 10 , 20 } as described above. We display the boxplots of the empirical stopping time on 1000 runs.

Figure 9 confirms that EB-TC ε 0 performs on par with the state-of-the-art ε -BAI algorithms and greatly outperform ε -DKM, LUCB and uniform sampling. Interestingly, for larger sets of arms ( K = 20 ), EB-TC ε 0 appears to be more robust than its competitors, closely followed by ε -T3C and ε -EB-TCI. The performance ε -TTUCB, ε -TaS and ε -FWS is slighlty worse, while the one of ε -DKM is greatly impacted. LUCB seems relatively robust to larger sets of arms, but it is still significantly worse than EB-TC ε 0 . It is even the best when considering slack ε 0 = ε in the bottom row of Figure 9. The slightly worse (but still competitive) empirical performance of EB-TC ε 0 in the top row of Figure 9 is explained by the slack ε 0 = 0 . 1 which differs from the error ε = 0 . 05 considered in the GLR ε stopping rule.

Specific instances To complement Figure 2(b), we consider the two other instances from Table 3. We display the boxplots of the empirical stopping time on 1000 runs, and the empirical simple regret on 10000 runs (with associated standard deviation).

The left column of Figure 10 validates the previous conclusions as regards the good empirical performance of EB-TC ε 0 compare to existing ε -BAI algorithms. We also see that it seems to perform even better when there are multiple ε -good arms and multiple best arms, i.e. | i ⋆ ( µ ) | > 1 . The right column of Figure 10 corroborates the observations made in Figure 2(b): EB-TC ε 0 outperforms uniform sampling, as well as DSR and DSH.

Two groups We consider the 'two-groups' instances µ ∈ { 0 . 6 , 0 . 4 } 10 for varying number of best arms, i.e. | i ⋆ ( µ ) | ∈ [8] . We average on 1000 runs, and the standard deviations are displayed.

Figure 11 reveals that EB-TC ε 0 is the most robust to increased number of best arms, i.e. its slope is the smallest. The good performance of EB-TC ε 0 is closely followed by other Top Two algorithms. This confirms that additional regularization mechanisms (e.g. TS/UCB leader or TCI challenger) are superfluous since the TC ε challenger is enough to ensure sufficient exploration. ε -TaS and ε -FWS have slightly worse slope, and the one of ε -DKM seems to become less steep after a large initial increase. The highest slope is achieved by LUCB. Quite surprisingly, uniform sampling seems to reach a plateau after which it is scaling better than EB-TC ε 0 (in terms of slope), while still having worse empirical performance. It would be interesting to quantify theoretically the good scaling of EB-TC ε 0 with respect to increased number of best arms.

Figure 12 provides additional empirical validation that EB-TC ε 0 outperforms uniform sampling as well as DSR and DSH in terms of empirical simple regret. While the empirical stopping time increased with the number of best arms, Figure 12 confirms the intuition that the problem gets easier with respect to the simple regret. First, the range of empirical simple regret becomes smaller with increased number of best arms. This was expected as we have more chances to recommend one of the best arms at early stage even though this recommendation is close to random. Second, we observe a steeper decrease of the empirical simple regret with increased number of best arms. Likewise, this is a natural phenomenon since the collected data will reveal that there multiple good options very fast.

Quite surprisingly, EB-TC ε 0 is still better than DSH when the number of best arms increases. To understand why it is surprising, we recall that the exponential decrease of DSH's probability of error is linear with a rate proportional to a hardness constant H DSH ( µ ) . By definition, H DSH ( µ ) gets small when there are multiple best arms, hence we would expect to observe lower empirical simple regret (i.e. a 'speed-up'). In contrast, our hardness constant H 1 ( µ, ε 0 ) = K (2∆ -1 min +3 ε -1 0 ) 2 does not enjoy such property. Therefore, we would expect that DSH will outperform EB-TC ε 0 when the number of best arms increases. Theoretically proving that EB-TC ε 0 has a better scaling than the one given in Theorem 3 is still an open problem. Solving it would allow to better understand its good empirical performance on this task.

Fixed-budget performance In addition, we compare the performance of EB-TC ε 0 with ε 0 and β = 1 / 2 to the one of SR and SH. Since SR and SH are fixed-budget algorithms, we ran a different instance for each budget T . Therefore, this comparison gives an unfair advantage to the fixed-budget

̸

algorithms which fully leverage the knowledge of T but have no theoretical guarantees at time n = T (even at time T ± 1 ). As a result, the empirical performances of SR and SH are better than the ones of DSR and DSH. We consider the instances from Table 3, as well as 'two-groups' instances.

Overall, according to Figure 13, EB-TC ε 0 seems to perform on par with SR and SH in all instances considered, and to outperform SH for some instances. In our experiments, we consider SH in which the samples are dropped between each phase, as analyzed theoretically in [1]. This loss of information explains why SR has better empirical performance than SH. To the best of our knowledge, there is no theoretical analysis for the heuristic SH where all the samples are preserved, even though it has better empirical performance.

J.2.3 Varying slack parameter

In Appendix J.2.3, we study the BAI problem. Those experiments have two goals: (1) sensitivity analysis of EB-TC ( ε n ) n described in Section 3 and (2) performance assessment of EB-TC ε 0 on BAI problems. We consider fixed proportions β = 1 / 2 , and use ε 0 = 0 . 1 for EB-TC ε 0 . For the rate of decrease of ( ε n ) n , we will be considering two archetypal choices: (a) polynomial by taking ε n = n -α/ 2 and (b) polylogarithmic by taking ε n = log( n ) -α/ 2 . We compare empirically those two rates of decrease for different choices of α , namely α ∈ { 0 . 05 , 0 . 1 , 0 . 5 } . To tackle BAI, we consider the GLR 0 stopping rule (3) with ( ε, δ ) = (0 , 0 . 01) and the heuristic threshold c ( n, δ ) = log((1 + log n ) /δ ) . Even though this choice is not sufficient to prove (0 , δ ) -PAC, it yields an empirical error which is several orders of magnitude lower than δ . As benchmark, we use the unmodified BAI algorithms.

̸

Random instances We assess the performance on 1000 random Gaussian instances with K ∈ { 5 , 10 , 20 } such that µ 1 = 1 and µ i ∼ U ([0 . 5 , 08]) for i = 1 . We display the boxplots of the empirical stopping time on 1000 runs.

The most stricking feature of Figure 14 is that it reveals how poor the performance of EB-TC ( ε n ) n can be for many choices of ( ε n ) n . For the considered α , we observe that polynomial decrease is always bad since it yields roughly the same empirical performance as uniform sampling. This can be explained by the fact that this decrease is too fast, hence EB-TC ( ε n ) n will ressemble EB-TC 0 which is know to have poor empirical performance [21]. For the considered α , we see that polylogarithmic decrease is often bad, except for α = 0 . 5 . In that case, it performs on par with TaS, slightly worse than standard Top Two algorithms for BAI and better than LUCB and uniform sampling. This can be explained by the fact that EB-TC ( ε n ) n is similar to uniform sampling when the decrease is too slow. The above sensitivity analysis truly shows how difficult it is to choose ( ε n ) n beforehand. The

best 'trade-off' between a slow decrease (yet not too slow) highly depends on the complexity of the unknown instance. While EB-TC ( ε n ) n reaches asymptotic optimality for BAI, we recommend the practitioner to use other Top Two algorithms which enjoy the same theoretical guarantees and better empirical performance. When one needs to have a deterministic algorithms, EB-TCI seems to be the best. Without the deterministic constraint, T3C has great performance.

In Figure 14, we also see that, when combined with the GLR 0 stopping rule, EB-TC ε 0 has good empirical performance for BAI. It outperforms EB-TC ( ε n ) n , performs on par with TaS and is only slightly worse than standard Top Two algorithms for BAI. Experiments conducted in Appendix J.2.1 already provide experimentally confirmation that EB-TC ε 0 has still good empirical performance on ε -BAI problems when ε 0 ≥ ε . Theoretically, Theorem 1 provide some intuition on why this is true. It shows that the gap between the asymptotic lower bound T 0 ( µ ) and the asymptotic upper bound of EB-TC ε 0 is bounded by

̸

where we used Lemmas 6, 9 and 7.

Specific instances We consider the two instances µ 1 and µ 2 from Table 3 with ε = 0 . Since most theoretical guarantees on BAI algorithms assume that there is a unique best arm, we don't study µ 3 . We display the boxplots of the empirical stopping time on 1000 runs.

Figure 15 confirms the empirical observations from Figure 14. Overall, EB-TC ( ε n ) n performs poorly and EB-TC ε 0 has good empirical performance for BAI.

̸

Peer reviews:
comments: SUMMARY:
The paper gives a new epsilon-best arm identification algorithm which is simple and anytime, just amounting to balancing between the empirical best arm and roughly speaking, the arm with the best chance to be superior to it by epsilon. Its performance is analyzed mainly for fixed confidence, and it also attains decent (not particularly optimal) bounds for fixed budget. The results assume a unique best arm but allow repeated mean values among the other arms.

STRENGTHS:
The fixed confidence bounds are very good (sharp for Gaussians) and the algorithm is nice and simple. The epsilon parameter can be mismatched, though this degrades the bounds a bit.

WEAKNESSES:
No new minimax regret bounds are obtained. As I understand it the paper is basically giving a simpler algorithm with some nice guarantees. Which is fine but not the most revolutionary.

QUESTIONS:
Maybe ``\beta-optimal'' could be defined more carefully, and the ratio bound of T_{\eps} vs T_{\eps,\beta} could be stated near the definitions. 

Just before line 200, it would be nice to cite a few of these other works on Top Two algorithms.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
This work proposed the EB-TC algorithm for $\varepsilon$-best arm identification in stochastic bandits. 
EB-TC is a BAI algorithm that does not require the knowledge of confidence or horizon, and hence it can be applied in both fixed-confidence and fixed-budget settings. Beyond theoretical guarantees, this work also provided numerical experiments to evaluate the performance of EB-TC.



==========================

Thanks for the response. I would like to keep the score.

STRENGTHS:
1. The paper is generally easy to follow.
2. EB-TC seems to be a nice BAI algorithm that does not need to know confidence or horizon as priori knowledge, which can be applied in both fixed-confidence and fixed-budget settings.
3. EB-TC can be viewed as a generalization of the Top Two Sampling rule which was designed in an earlier work.
4. The paper provides a clear comparison to existing works.

WEAKNESSES:
As discussed in Section 2.1, the EB-TC algorithm may not perform well to identify the optimal arm. Can this issue be solved?

QUESTIONS:
As mentioned in the Weakness section: what is the key obstacle such that EB-TC cannot work well for the identification of the optimal arm? I appreciate some explanation of this point. A (possibly) suboptimal theoretical bound is even better.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an anytime $\\epsilon$-best-arm identification algorithm for multi-armed bandits that can be employed in both fixed-confidence and fixed-budget setting. The algorithm is analyzed in asymptotic fixed-confidence, non-asymptotic fixed-confidence and fixed-budget settings. Meanwhile, the paper also provides empirical results in many different settings.\n\nSTRENGTHS:\n- The proposed algorithm is anytime in nature and thus widely applicable.\n- The analysis for fixed $\\beta$ is thorough and covers all asymptotic fixed-confidence, non-asymptotic fixed-confidence and fixed-budget settings.\n- The empirical experiments are extensive.\n\nWEAKNESSES:\n- The analysis for the IDS variant is limited.\n- This may be just due to my misunderstanding, but it seems the expected stopping time of the proposed algorithm with finite confidence $\\delta>0$ will explode for BAI ($\\epsilon=0$) problem.\n\n### Suggestions on Writing\n- It may be better to also give the formula of $\\bar{\\beta}_{n+1}(i, j)$ under the IDS variant in the algorithm summary.\n- Do we simply have $T_{\\epsilon_0, \\beta}(\\mu)=\\min_{i\\in\\mathcal{I}\\_{\\epsilon_0}(\\mu)}T_{\\epsilon_0, \\beta}(\\mu, i)$ in the statement of Theorem 1? It seems this quantity is not formally defined in the main paper.\n- It seems the statement of Theorem 1 does not mention that the result for IDS variant only holds for Gaussian distribution.\n- If possible, it may be better to present results in the main paper more in terms of big-$\\mathcal{O}$ instead of exact formula.\n\nQUESTIONS:\n- How does the proposed algorithm perform, compared with uniform sampling, SH and SR, in fixed-budget BAI ($\\epsilon=0$) problem?\n- Based on Theorem 1, the proposed algorithm is asymptotically optimal for 0-BAI when taking $\\epsilon_0=0$. However, based on Theorem 2, it seems the expected stopping time just explode when taking $\\epsilon_0=0$ with positive $\\delta>0$. Is this a contradiction? Did I misunderstand anything? How does the algorithm perform in 0-BAI with finite $\\delta>0$?\n\nLIMITATIONS:\nThe limitations are addressed well in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the epsilon-BAI problem in the multi-armed bandit setting with sub-gaussian arm distributions. It proposes a top2 algorithm for the problem that works across different performace criteria: fixed-confidence, fixed-budget, as well as simple-regret. The proposed algorithm is asymptotically optimal (as delta-> 0) for the fixed confidence setting for gaussian bandits, where delta is the bound on probability of errors. The paper also provides finite delta analysis of the upper bound on the sample complexity in the fixed confidence setting. Another novel feature of the proposed algorithm is that it can be used in the fixed-budget setting, without a prior knowledge of the budget. The theoretical results are accompanied with extensive simulations and numerical studies.\n\nSTRENGTHS:\nI believe that the results of this work are novel. The proposed sampling rule, when coupled with either stopping rule or recommendation rule (or both) works across different BAI settings: fixed-confidence, fixed-budget, simple regret. The paper also discusses well and brings out the comparison with existing works and techniques for each of these problems. I especially like the discussions that are included after each result. I believe that the literature review is also thorough. The extensive numerical studies undertaken definitely add value to the theoretical work.\n\nWEAKNESSES:\nI believe that the paper is dense and too long for a conference submission with many details in the appendix. But this may be because of the space constraints. \n\nPlots in Figures 6, 10, and 12 are very light. Also the legends are too small to read for most of the plots. In general, it will help to spread out the plots for readability. Since the plots are already in appendix, space isn't an issue.\n\nQUESTIONS:\n1. Could you clarify the definition of simple regret in the epsilon-BAI framework? In line 47, should there also be an indicator on chosen arm not being epsilon-best? In line 97-98, is Delta_{\\hat{i}_n} missing in the integration? \n\n2. The lower bound in Lemma 1 is specifically for Gaussian bandits. How does it compare with sub-Gaussian bandits? Is there some monotonicity? It will be good to add a discussion around this. If not, including an example (if known) could help.\n\n3. In Figure 1 (EP-TC_eps algorithm with fixed or IDS) line 3, it will be good to refer to the equation for updating \\bar{beta}_{n+1}(i,j) in the main text. \n\n4. It will be helpful if the discussion in lines 150-151 could be elaborated. What are the K(K-1) different tracking rules?  Are these the different choices of beta that are maintained for each leader/challenger pair? \n\n5. How should one choose epsilon_0 in practice? \n\n6.  In lines 201-203, is that a requirement for asymp. optimality, or is it a requirement for proofs to work? Can one show that if number of optimal arms is greater than 1, then the algorithm doesn't converge (may be b/c of some discontinuity)?\n\n7. Lines 235-237: Would using any-delta methods to construct UCB and LCB indexes in LUCB help in this regard?\n\n8. In Lemma 9, Line 530, could you clarify what j is in mu_eps(i)_j ?\n\n9. Is Lemma 12 a standard result? Adding a citation or proof would help.\n\n10. What are the challenges in generalizing the proofs to beyond gaussian arms to exponential families? What specific properties of Gaussians are used in analysis that may not hold for exponential families? Can one pin down to the level that if we can prove this, this, and this about KL (for example) or other functions in exponential families, then we should get the bound? \n\nMinor:\n1. Line 207 satisfies these ...\n2. Line 587: eeror --> error\n\nLIMITATIONS:\nNa", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors proposed EB-TC_epsilon, a novel sampling rule for epsilon-best arm identification. While the proposed algorithm is analyzed for approximate best arm identification, it is also an anytime sampling rule that can be used for fixed confidence, fixed budget, or anytime settings. The algorithm is analyzed for those settings and is empirically evaluated with numerical simulations.\n\nSTRENGTHS:\n1. The proposed algorithm is simple and easy to implement. Without modification, it works for approximate best arm identification, fixed confidence, fixed budget, or anytime settings.\n\n2. The proposed algorithm is theoretically analyzed and empirically evaluated with numerical simulations. The results show that the proposed algorithm can achieve competitive performance compared to other methods.\n\nWEAKNESSES:\n1. The presentation of this paper can be improved. I found it hard to follow this paper in its current form, and I can not discuss its contribution with confidence. \n   (a) The main results and theoretical discussions are mixed with related works. While it is nice to relate the proposed algorithms with previous works, it is hard to see the difference/improvement without clearly presenting previous results -- many of the previous algorithms are only cited by name and without further explanation. \n   (b) Many of the high-level discussions and insights are also mixed with mathematical derivations, which makes it hard to follow. It would be nice to focus more on the main results and insights and present the details in the appendix.\n\n2. The algorithm presentation is not well organized. The algorithm is introduced in line 120 and a crucial quantity $\\bar \\beta_{n+1}$ is not defined until line 140. It would be nice to define all the quantities before or when introducing the algorithm.\n\nQUESTIONS:\nWhat is the motivation for proposing one algorithm that works for approximate best arm identification and works, without modification, for fixed confidence, fixed budget, or anytime settings? There are other algorithms proposed specifically for different settings, and one might choose an algorithm based on the setting. It would be nice to see some discussion on this.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-14"}