Title: Non-Asymptotic Analysis of a UCB-based Top Two Algorithm
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi-armed bandits

best-arm identification

Gaussian bandits

Top Two algorithm

fixed confidence

finite confidence
Abstract: A Top Two sampling rule for bandit identification is a method which selects the next arm to sample from among two candidate arms, a *leader* and a *challenger*. Due to their simplicity and good empirical performance, they have received increased attention in recent years. However, for fixed-confidence best arm identification, theoretical guarantees for Top Two methods have only been obtained in the asymptotic regime, when the error level vanishes. In this paper, we derive the first non-asymptotic upper bound on the expected sample complexity of a Top Two algorithm, which holds for any error level. Our analysis highlights sufficient properties for a regret minimization algorithm to be used as leader. These properties are satisfied by the UCB algorithm, and our proposed UCB-based Top Two algorithm simultaneously enjoys non-asymptotic guarantees and competitive empirical performance.

Non-Asymptotic Analysis of a UCB-based Top Two Algorithm

Abstract

A Top Two sampling rule for bandit identification is a method which selects the next arm to sample from among two candidate arms, a leader and a challenger . Due to their simplicity and good empirical performance, they have received increased attention in recent years. However, for fixed-confidence best arm identification, theoretical guarantees for Top Two methods have only been obtained in the asymptotic regime, when the error level vanishes. In this paper, we derive the first non-asymptotic upper bound on the expected sample complexity of a Top Two algorithm, which holds for any error level. Our analysis highlights sufficient properties for a regret minimization algorithm to be used as leader. These properties are satisfied by the UCB algorithm, and our proposed UCB-based Top Two algorithm simultaneously enjoys non-asymptotic guarantees and competitive empirical performance.

1 Introduction

Faced with a collection of items ('arms') with unknown probability distributions, a question that arises in many applications is to find the distribution with the largest mean, which is referred to as the best arm. Different approaches have been considered depending on the data collection process. Sequential hypothesis testing [10, 37] encompasses situations where there is no control on the collected samples. Experimental design [7, 35] aims at choosing the data collection scheme a priori. In the multi-armed bandit [4, 20] and the ranking and selection [18] literature, an algorithm chooses sequentially the distribution from which it will collect an additional sample based on past data.

In order to have theoretical guarantees for this identification problem, one should adopt a statistical model on the underlying distributions. While parametric models are reasonable for applications such as A/B testing [28], they are unrealistic in other fields such as agriculture [21]. Despite the restricted scope of its applications, studying the identification task for Gaussian distributions is a natural first step. Hopefully the insights gained will then be generalized to wider classes of distributions.

In the fixed confidence identification problem, an algorithm aims at identifying the best arm with an error of at most δ ∈ (0 , 1) while using as few samples as possible. Since each sample has a cost, those algorithms should provide an upper bound on the expected number of samples used by the algorithm before stopping. For those guarantees to be useful in practice, they should hold for any δ , which is referred to as the non-asymptotic (or moderate) regime. In contrast, the asymptotic regime considers vanishing error level, i.e. δ → 0 . For Gaussian distributions, Top Two algorithms [36, 39]

have only been studied in the asymptotic regime. We show the first non-asymptotic guaranty for a Top Two algorithm holding for any instance with a unique best arm.

1.1 Setting and related work

A Gaussian bandit problem is described by K arms whose probability distributions belongs to the set D of Gaussian distributions with known variance σ 2 . By rescaling, we assume σ 2 i = 1 for all i ∈ [ K ] . Since an element of D is uniquely characterized by its mean, the vector µ ∈ R K refers to a K -arms Gaussian bandit. Let △ K ⊆ R K be the ( K -1) -dimensional simplex.

A best arm identification (BAI) algorithm aims at identifying an arm with highest mean parameter, i.e. an arm belonging to the set i ⋆ ( µ ) = arg max i ∈ [ K ] µ i . At each time n ∈ N , the algorithm (1) chooses an arm I n based on previous observations, (2) observes a sample X n,I n ∼ N ( µ I n , 1) , and (3) decides whether it should stop and return an arm ˆ ı n or continue sampling. We consider the fixed confidence identification setting, in which the probability of error of an algorithm is required to be less than a given δ ∈ (0 , 1) on all instances µ . The sample complexity of an algorithm corresponds to its stopping time τ δ , which counts the number of rounds before termination. An algorithm is said to be δ -correct on D K if P µ ( τ δ < + ∞ , ˆ ı τ δ / ∈ i ⋆ ( µ )) ≤ δ for all µ ∈ D K [14]. We aim at designing δ -correct algorithms minimizing E [ τ δ ] .

As done in all the literature on fixed-confidence BAI, we assume that there is a unique best arm and we denote it by i ⋆ ( µ ) or i ⋆ when µ is clear from the context. To ensure δ -correctness on D K , an algorithm has to be able to distinguish the unknown µ from any instance having a different best arm, hence it needs to estimates the gaps between arms. Lemma 1.1 gives a lower bound on the expected sample complexity which is known to be tight in the asymptotic regime, i.e. when δ goes to zero.

Lemma 1.1 ([17]) . An algorithm which is δ -correct on all problems in D K satisfies that for all µ ∈ R K , E µ [ τ δ ] ≥ T ⋆ ( µ ) log(1 / (2 . 4 δ )) where T ⋆ ( µ ) = min β ∈ (0 , 1) T ⋆ β ( µ ) and, for all β ∈ (0 , 1) ,

̸

When considering the sub-class of algorithms allocating a fraction β of their sample to the best arm, we obtain a lower bound as in Lemma 1.1 with T ⋆ β ( µ ) instead of T ⋆ ( µ ) . An algorithm is said to be asymptotically optimal (resp. β -optimal) if its sample complexity matches that lower bound asymptotically, that is if lim sup δ → 0 E µ [ τ δ ] / log(1 /δ ) ≤ T ⋆ ( µ ) (resp. T ⋆ β ( µ ) ). [38] showed the worst-case inequality T ⋆ 1 / 2 ( µ ) ≤ 2 T ⋆ ( µ ) for any single-parameter exponential families. Therefore, the expected sample complexity of an asymptotically β -optimal algorithm with β = 1 / 2 is at worst twice higher than that of any asymptotically optimal algorithm. Leveraging the symmetry of Gaussian distributions, a tighter worst-case inequality can be derived (Lemma C.6). The allocations w ⋆ ( µ ) and w ⋆ β ( µ ) realizing T ⋆ ( µ ) and T ⋆ β ( µ ) are known to be unique, and satisfy min i ∈ [ K ] min { w ⋆ ( µ ) i , w ⋆ β ( µ ) i } > 0 . [6] showed that 2 ≤ w ⋆ ( µ ) -1 i ⋆ ≤ √ K -1 + 1 for Gaussian distributions (Lemma C.4).

̸

Related work The first BAI algorithms were introduced and studied under the assumption that the observation have bounded support, with a known upper bound [15, 24, 16, 19]. The sample complexity bounds proved for these algorithms scale as the sum of squared inverse gap, i.e. H ( µ ) := 2∆ -2 min + ∑ i = i ⋆ 2( µ i ⋆ -µ i ) -2 where ∆ min := min i = i ⋆ ( µ i ⋆ -µ i ) , which satisfies H ( µ ) ≤ T ⋆ ( µ ) ≤ 2 H ( µ ) [17]. Following their work, a rich literature designed asymptotically optimal algorithms in the fixed-confidence setting for parametric distributions, such as single-parameter exponential families, and non-parametric distributions such as bounded ones. Those algorithms build on two main ideas. The Tracking approach computes at each round the optimal allocation for the empirical estimator, and then tracks it [17]. To achieve lower computational cost, Game-based algorithms [12] view T ⋆ ( µ ) -1 as a min-max game between the learner and the nature, and design saddle-point algorithms to solve it sequentially.

̸

Top Two algorithms arose as an identification strategy based on the Thompson Sampling algorithm for regret minimization [41]: [38] introduced Top Two Probability Sampling (TTPS) and Top Two Thompson Sampling (TTTS). Adopting a Bayesian viewpoint, Russo studied the convergence rate of the posterior probability that i ⋆ is not the best arm, under some conditions on the prior. For Gaussian bandits, other Bayesian Top Two algorithms with frequentist components have been shown to be

asymptotically β -optimal: Top Two Expected Improvement (TTEI, [36]) and Top Two Transportation Cost (T3C, [39]). [21] introduces fully frequentist Top Two algorithms. Their analysis proves asymptotic β -optimality for several Top Two algorithms and distribution classes, beyond Gaussian. [34] provides guarantees for single-parameter exponential families, at the price of adding forced exploration. [44] proposes an algorithm to tackle the topk identification problem and introduces information-directed selection (IDS) to choose β in an adaptive manner, which differs from the one proposed in [33]. In addition to their success in the fixed-confidence setting, Top Two algorithms have also been studied for fixed-budget problems [2], in which guarantees on the error probability should be given after T samples. While existing Top Two sampling rules differ by how they choose the leader and the challenger, they all sample the leader with probability β . By design, Top Two algorithms with a fixed β can reach β -optimality at best, and cannot be optimal on all instances µ .

Shortcomings of the asymptotic regime While the literature provides a detailed understanding of the asymptotic regime, many interesting questions are unanswered in the non-asymptotic regime. Recent works [9, 40, 32, 31] have shown that the sample complexity is affected by strong moderate confidence terms (independent of δ ). The analysis of [21] applies to their β -EB-TC algorithm whose empirical stopping times is order of magnitude larger than its competitors for δ = 0 . 01 . Since the proof of asymptotic β -optimality hides design flaws, non-asymptotic guarantees should be derived to understand which Top Two algorithms will perform well in practice for any reasonable choice of δ .

1.2 Contributions

Our main contribution is to propose the first non-asymptotic analysis of Top Two algorithms. We identify sufficient properties of the leader (seen as a regret-minimization algorithm) for it to hold. This solves two open problems: obtaining an upper bound which (1) is non-asymptotic (Theorem 2.4 holds for any δ ) and (2) holds for all instances having a unique best arm (i.e. sub-optimal arms can have the same mean, which was not allowed in the analyzes of existing Top Two algorithms). As a consequence, we propose the TTUCB (Top Two UCB) algorithm which builds on the UCB algorithm.

By using tracking instead of sampling to choose between the leader and the challenger, TTUCB is the first Top Two algorithm which is asymptotically β -optimal (Theorem 2.3) and has non-asymptotic guarantees (Theorem 2.4). Our experiments reveal that TTUCB performs on par with existing Top Two algorithms, which are only proven to be asymptotically β -optimal, even for large sets of arms. Numerically, we show that considering adaptive proportions compared to a fixed β = 1 / 2 yields a significant speed-up on hard instances, and to a moderate improvement on random instances.

2 UCB-based Top Two algorithm

We propose a fully deterministic Top Two algorithm based on UCB [5], named TTUCB and detailed in Algorithm 1. We prove a non-asymptotic upper bound on the expected sample complexity holding for any instance having a unique best arm.

Stopping and recommendation rules The σ -algebra F n := σ ( { I t , X t,I t } t ∈ [ n -1] ) encompasses all the information available to the agent before time n . Let N n,i := ∑ t ∈ [ n -1] 1 ( I t = i ) be the number of pulls of arm i before time n , and its empirical mean by µ n,i := 1 N n,i ∑ t ∈ [ n -1] X t,I t 1 ( I t = i ) .

The algorithm stops as soon as the generalized likelihood ratio exceeds a threshold c ( n -1 , δ ) , i.e.

̸

where we recommend ˆ ı n = arg max i ∈ [ K ] µ n,i at time n . Lemma 2.1 provides an explicit threshold ensuring δ -correctness, which relies on concentration inequalities derived in [27].

Lemma 2.1. Let C G defined in (16) s.t. C G ( x ) ≈ x +log( x ) . Given any sampling rule, taking

in the stopping rule (1) ensures δ -correct for Gaussian distributions.

Algorithm 1 TTUCB

̸

̸

̸

Sampling rule We initialize by sampling each arms once. At time n > K , a Top Two sampling rule defines a leader B n ∈ [ K ] and a challenger C n = B n , and chooses I n = B n or I n = C n based on a fixed allocation β . In prior work this choice was done at random, which means that the leader was sampled with probability β . We replace randomization by tracking, and show similar theoretical and numerical results (see Figure 4 in Appendix G.2). For fixed β , we recommend to use β = 1 / 2 without prior knowledge on the unknown mean parameters (see Section 3.4 for adaptive proportions). This recommendation is supported theoretically by the fact that w ⋆ ( µ ) i ⋆ ≤ 1 / 2 (Lemma C.4) and that T ⋆ 1 / 2 ( µ ) /T ⋆ ( µ ) is significantly smaller than 2 for most instances (Lemma C.6 and Figure 2).

Let L n,i := ∑ t ∈ [ n -1] 1 ( B t = i ) be the number of time arm i was the leader, and N i n,j := ∑ t ∈ [ n -1] 1 (( B t , I t ) = ( i, j )) be the number of pulls of arm j at rounds in which i was the leader. We use K independent tracking procedures. A tracking procedure is a deterministic method to convert a sequence of allocations over arms into a sequence of arms, which ensures that the empirical proportions are close to the averaged allocation over arms. For each leader, we track the allocation ( β, 1 -β ) between the leader and the challenger. Formally, we set I n = B n if N B n n,B n ≤ βL n +1 ,B n , else I n = C n . Using Theorem 6 in [13] for each tracking procedure yields Lemma 2.2.

Using tracking over randomization is motivated by practical and theoretical reasons. First, in some specific applications, the practitioner might be only willing to use a deterministic algorithm. Second, in the analysis, it is easier to control deterministic counts since it removes the need for martingales arguments to bound the deviations of the samples. Therefore, tracking simplifies the non-asymptotic analysis. Third, Lemma 2.2 shows that the speed of convergence is at least O (1 /n ) for tracking, while we would obtain a speed of O (1 / √ n ) for randomization.

At time n , the UCB leader is defined as

where √ g ( n ) /N n,i is a bonus coping for uncertainty. Let α > 1 and s > 1 be two concentration parameters. The choice of g ( n ) should ensure that we have an upper confidence bound on µ i holding with high probability: with probability 1 -Kn -s , for all t ∈ [ n 1 /α , n ] and all arms i ∈ [ K ] , µ i ∈ [ µ t,i ± √ g ( t ) /N t,i ] . For Gaussian observations, a function g which is sufficient for the purpose of our proof can be obtained by a union bound over time, giving g u ( n ) = 2 α (1 + s ) log n . We can improve on g u with mixtures of martingales, yielding g m ( n ) = W -1 (2 sα log( n ) + 2 log(2 + α log n ) + 2) with W -1 ( x ) = -W -1 ( -e -x ) for all x ≥ 1 , where W -1 is the negative branch of the Lambert W function, and W -1 ( x ) ≈ x +log( x ) . A UCB leader with g 0 ( n ) = 0 recovers the Empirical Best (EB) leader [21]. Choosing g is central for empirical performance and non-asymptotic guarantees, but not for asymptotic ones. The lowest g will yield better empirical performance since larger g means more conservative confidence bounds. In our experiments where α = s = 1 . 2 , we will consider g m since g m ( n ) ≤ g u ( n ) for n ≥ 50 .

Given a leader B n , the TC challenger is defined as

̸

where x + = max { x, 0 } . [39] introduced the TC challenger as a computationally efficient approximation of the challenger in TTTS [38], which uses re-sampling till an unlikely event occurs. Both T3C and TTTS use the TS leader which takes the best arm of a vector of realization drawn from a sampler, e.g. θ i ∼ N ( µ n,i , 1 /N n,i ) for Gaussian distributions with unit variance.

̸

Computational cost Computing the stopping rule (1) and the UCB leader (3) can be done in O ( K ) . At time n where B n coincides with ˆ ı n , computing the TC challenger (4) is done as a by-product of the computation of the stopping rule, without additional cost. When B n =ˆ ı n , we draw at random an arm with larger empirical mean. The per-round computational and memory cost of TTUCB is O ( K ) .

2.1 Sample complexity upper bound

Leveraging the unified analysis of Top Two algorithms proposed by [21], we obtain the asymptotic β -optimality of TTUCB (Theorem 2.3). After showing the required properties for the UCB leader, we proved that tracking Top Two algorithms have similar properties as their sampling-based counterparts.

̸

Theorem 2.3. Let ( δ, β ) ∈ (0 , 1) 2 , s > 1 and α > 1 . Using the threshold (2) in (1) and g u (or g m ) in (3) , the TTUCB algorithm is δ -correct and asymptotically β -optimal for all µ ∈ R K such that min i = j | µ i -µ j | > 0 , i.e. it satisfies lim sup δ → 0 E µ [ τ δ ] / log(1 /δ ) ≤ T ⋆ β ( µ ) .

Theorem 2.3 and guarantees for other Top Two algorithms hold only for arms having distinct means. Moreover, an asymptotic result provides no guarantees on the performance in moderate regime of δ . We address those two limitations.

Non-asymptotic upper bound Theorem 2.4 gives an upper bound on the expected sample complexity holding for any δ and any instance having a unique best arm. It is a direct corollary of a more general result holding for any β ∈ (0 , 1) , s > 1 and α > 1 (Theorem D.4).

Theorem 2.4. Let δ ∈ (0 , 1) . Using the threshold (2) in (1) and g u in (3) with s = α = 1 . 2 , the TTUCB algorithm with β = 1 / 2 satisfies that, for all µ ∈ R K such that | i ⋆ ( µ ) | = 1 ,

where C µ = h 1 (26 H ( µ )) , C 0 ( w 0 ) = 2 / ( εa µ ( w 0 )) + 1 with ε ∈ (0 , 1] ,

̸

with a µ ( w 0 ) = (1 -w 0 ) d µ ( w 0 ) max { min i = i ⋆ ( µ ) w ⋆ 1 / 2 ( µ ) i , w 0 / 2 } and d µ ( w 0 ) = |{ i = i ⋆ ( µ ) | w ⋆ 1 / 2 ( µ ) i < w 0 / 2 }| . The function h 1 ( x ) := xW -1 ( log( x ) + 2+2 K x ) is positive, increasing for x ≥ 2 + 2 K , and satisfies h 1 ( x ) ≈ x (log x +loglog x ) .

̸

̸

The TTUCB sampling rule using g m in (3) satisfies a similar upper bound (Corollary D.5). Since Theorem 2.4 holds for any instance having a unique best arm, we corroborate the intuition that assuming min i = j | µ i -µ j | > 0 is an artifact of the existing proof to obtain asymptotic β -optimality.

The upper bound on E µ [ τ δ ] involves several terms. The δ -dependent term is T 0 ( δ ) . In the asymptotic regime, we can show that lim sup δ → 0 T 0 ( δ ) / log(1 /δ ) ≤ 2 T ⋆ 1 / 2 ( µ ) by taking w 0 = 0 and letting ε go to zero. While there is (sub-optimal) factor 2 in T 0 ( δ ) , Theorem 2.3 shows that TTUCB is asymptotically 1 / 2 -optimal. This factor is a price we paid to obtain more explicit non-asymptotic terms, and removing it would require more sophisticated arguments in order to control the convergence of the empirical proportions N n / ( n -1) towards w ⋆ 1 / 2 ( µ ) .

In the regime where H ( µ ) → + ∞ , the upper bound is dominated by the δ -independent term C 1 . 2 µ (when α = 1 . 2 ) with satisfies C µ = O ( H ( µ ) log H ( µ )) . Compared to the best known upper and lower bounds in this regime (see discussion below), our non-asymptotic term has a sub-optimal scaling in O (( H ( µ ) log H ( µ )) α ) with α > 1 . While taking α ≈ 1 would mitigate this sub-optimality, it would yield a larger dependency in C 0 ( w 0 ) α/ ( α -1) . Empirically, Figures 1(b) and 5 (Appendix G.2) hints that the empirical performance of TTUCB has a better scaling with H ( µ ) than H ( µ ) α .

̸

For instances such that min i = i ⋆ w ⋆ 1 / 2 ( µ ) i is arbitrarily small, taking w 0 = 0 yields an arbitrarily large C 0 (0) . By clipping with w 0 / 2 , we circumvent this pitfall and ensure that C 0 ( w 0 ) = O ( K/ε ) .

̸

̸

̸

Since it yields a larger T 0 ( δ ) , we are trading-off asymptotic terms for improved non-asymptotic ones. We illustrate this with two archetypal instances. For the ' 1 -sparse' instance, in which µ 1 > 0 and µ i = 0 for all i = 1 , we have by symmetry that 2 w ⋆ 1 / 2 ( µ ) i = 1 / ( K -1) for all i = 1 . Therefore, we have C 0 ( w 0 ) = O ( K/ε ) since d µ ( w 0 ) = 0 for all w 0 ∈ [0 , 1 / ( K -1)] . The 'almost dense' instance is such that µ 1 = 1 , µ K = 0 and µ i = 1 -γ for all i / ∈ { 1 , K } . By symmetry, there exists a function h : [0 , 1) → [0 , ( K -1) -1 ) with lim γ → 0 h ( γ ) = 0 , such that 2 w ⋆ 1 / 2 ( µ ) K = h ( γ ) and 2 w ⋆ 1 / 2 ( µ ) i = (1 -h ( γ )) / ( K -2) for all i / ∈ { 1 , K } . While lim γ → 0 C 0 (0) = + ∞ , we obtain lim γ → 0 C 0 ( w 0 ) = O ( K/ε ) by taking w 0 = (1 -h ( γ )) / ( K -2) since d µ ( w 0 ) = 1 .

̸

Comparison with existing upper bounds Table 1 summarizes the asymptotic and non-asymptotic scalings of the upper bound on the sample complexity of existing BAI algorithms. Among the class of asymptotically ( β -)optimal algorithms, very few of them also enjoy non-asymptotic guarantees, e.g. the analyses of Track-and-Stop and Top Two algorithms are asymptotic. The gamification approach of [12] is the first attempt to provide both. Their non-asymptotic upper bound on E µ [ τ δ ] involves an implicit time T 1 ( δ ) which scales with KT ⋆ ( µ ) 2 and is only valid for log(1 /δ ) ≳ KT ⋆ ( µ ) (see Lemma 2, with constants in Appendix D.7). Let T ⋆ δ := T ⋆ ( µ ) log(1 /δ ) . As a first order approximation, they obtain T 1 ( δ ) ≈ T ⋆ δ + Θ (√ T ⋆ δ log T ⋆ δ ) , and we obtain T 0 ( δ ) ≈ Θ( T ⋆ δ +log T ⋆ δ ) (Lemma D.13). [42] were the first to obtain an upper bound on E µ [ τ δ ] of the form Θ( T ⋆ δ +loglog(1 /δ )) . While they improved the second-order δ -dependent term, the δ -independent term scales with e K H ( µ ) 19 / 2 (see their Theorem 2 for ε -1 ≳ T ⋆ ( µ ) , with constants given by Appendix N). The algorithm proposed by [6] has a non-asymptotic upper bound on E µ [ τ δ 1 ( E )] of the form (1+ ε ) T ⋆ δ + f ( µ, δ ) which is valid for log(1 /δ ) ≳ w -2 min K/ ∆ min , where E is such that P µ ( E ∁ ) ≤ γ . Since f ( µ, δ ) = δ → 0 o (1) , they obtain a better δ -dependency. However, f ( µ, δ ) is arbitrarily large when w min := min i ∈ [ K ] w ⋆ ( µ ) i is arbitrarily small since it scales with KH ( µ ) 4 /w 2 min . Therefore, they suffer from the pitfall which we avoided by clipping. In light of Table 1, TTUCB enjoys the best scaling when H ( µ ) → + ∞ in the class of asymptotically ( β -)optimal BAI algorithms.

The LUCB1 algorithm [24] has a structure similar to a Top Two algorithm, with the difference that LUCB samples both the leader and the challenger instead of choosing one. As LUCB1 satisfies E µ [ τ δ ] ≤ 292 H ( µ ) log( H ( µ ) /δ ) + 16 , it enjoys better scaling when H ( µ ) → + ∞ than TTUCB. Since the empirical allocation of LUCB1 is not converging towards w ⋆ 1 / 2 ( µ ) , it is not asymptotically 1 / 2 -optimal. The Peace algorithm [26] has a non-asymptotic upper bound on τ δ of the form O (( T ⋆ δ + γ ⋆ ( µ )) log( K/ ∆ min )) holding with probability 1 -δ . The term γ ⋆ ( µ ) is a Gaussian-width which originates from concentration on the suprema of Gaussian processes and satisfies γ ⋆ ( µ ) = Ω( H ( µ )) .

̸

Another class of BAI algorithms focus on the dependency in the gaps ∆ i := µ i ⋆ -µ i , and derive nonasymptotic upper bound on τ δ holding with high probability. [25, 19, 8, 9] gives δ -PAC algorithms with an upper bound of the form O ( H ( µ ) log(1 /δ ) + ∑ i = i ⋆ ∆ -2 i log log ∆ -1 i ) , and [19] shows that

for two arms the dependency ∆ -2 log log ∆ -1 is optimal when ∆ → 0 . While those algorithms obtain the best scaling when H ( µ ) → + ∞ , they are not asymptotically ( β -)optimal.

3 Non-asymptotic analysis

3.1 Proof sketch of Theorem 2.4

Existing analyses of Top Two algorithms are asymptotic in nature and requires too much control on the empirical means and proportions to yield any meaningful information in the finite-confidence regime. Therefore, we adopt a different approach which ressembles the non-asymptotic analysis of [12]. We first define concentration events to control the deviations of the random variables used in the UCB leader and the TC challenger. For all n > K , let E n := ⋂ i ∈ [ K ] ⋂ t ∈ [ n 5 / 6 ,n ] ( E 1 t,i ∩ E 2 t,i ) where

Using Lemmas D.8 and E.6, the proof boils down to constructing a time T ( δ ) after which E n ⊂ { τ δ ≤ n } for n > T ( δ ) since it would yield that E µ [ τ δ ] ≤ T ( δ ) + 12 K .

̸

Let n > K such that E n ∩ { n < τ δ } holds true, and t ∈ [ n 5 / 6 , n ] such that B UCB t = i ⋆ . Using that t ≤ n < τ δ , under ⋂ i = i ⋆ E 2 t,i , the stopping condition yields that

Let w ⋆ 1 / 2 be the unique element of w ⋆ 1 / 2 ( µ ) . Lemma 3.1 links the empirical proportions N i ⋆ t,i / ( t -1) to w ⋆ 1 / 2 ,i for i ∈ { i ⋆ , C TC t } . It is the key technical challenge of our non-asymptotic proof strategy.

Lemma 3.1. Let ε ∈ (0 , 1] . There exist T µ > 0 such that for all n > T µ such that E n ∩ { n < τ δ } holds true, there exists t ∈ [ n 5 / 6 , n ] with B UCB t = i ⋆ , which satisfies

Before proving Lemma 3.1, we conclude the proof of Theorem 2.4. Let ε, T µ and t as in Lemma 3.1 and T ( δ ) := sup { n | n -1 ≤ T ⋆ 1 / 2 ( µ )(1 + ε ) 2 ( √ c ( n -1 , δ ) + √ 4 log n ) 2 /β } . For all n > max { T µ , T (1) } , we have √ c ( n -1 , δ ) + √ 4 log n ≥ √ β ( n -1) T ⋆ 1 / 2 ( µ ) -1 (1 + ε ) -2 . Therefore, we have proved that E n ∩ { n < τ δ } = ∅ for all n > max { T µ , T ( δ ) } . This concludes the proof.

Provided that B t = i ⋆ , the above only used the stopping condition and the TC challenger, and no other properties of the leader. Lemma 3.2 shows that B UCB t = i ⋆ , except for a sublinear number of times. Section 3.3 exhibits sufficient conditions on a regret minimization algorithm to obtain a non-asymptotic upper bound.

Proof sketch of Lemma 3.1 The key technical challenge is to link N i ⋆ t,C TC t / ( n -1) with w ⋆ 1 / 2 ,C TC t . We adopt the approach used to analyze of APT [30]: consider an arm being over-sampled and study the last time this arm was pulled. By the pigeonhole principle, at time n ,

̸

Let t 1 be the last time at which B UCB t = i ⋆ and C TC t = k 1 , hence N i ⋆ t 1 ,k 1 ≥ N i ⋆ n,k 1 -1 . Using Lemmas 2.2 and 3.2, we show that N i ⋆ t 1 ,k 1 ⪆ w ⋆ 1 / 2 ,k 1 ( n -1) , hence t 1 ≥ n 5 / 6 for n large enough (see Appendix D.2). Then, we need to link N i ⋆ t 1 ,i ⋆ to ( n -1) / 2 . When w ⋆ 1 / 2 ,k 1 is small, (5) can be true at t 1 = n 5 / 6 , hence there is no hope to show that t 1 = n -o ( n ) . To circumvent this problem, we link N i ⋆ t 1 ,i ⋆ to N i ⋆ t 1 ,k 1 thanks to Lemma 2.2, and use that

̸

for n > T µ ( w -) with T µ ( w -) ≤ max { C 1 . 2 µ , (2 / ( εw -) + 1) 6 , (2 /ε ) 1 . 2 } (Lemmas D.10 and D.11), where w -= min i = i ⋆ w ⋆ 1 / 2 ,i > 0 lower bounds w ⋆ 1 / 2 ,k 1 . This concludes the proof for w 0 = 0 . The (sub-optimal) multiplicative factor 2 in T 0 ( δ ) comes from the inequality (6). To remove it, we need to control the deviation between the empirical proportion of arm i and w ⋆ 1 / 2 ,i for all i ∈ [ K ] . Nevertheless, TTUCB is asymptotically 1 / 2 -optimal (Theorem 2.3).

̸

Refined analysis For w 0 ∈ (0 , ( K -1) -1 ] , we clip min i = i ⋆ w ⋆ 1 / 2 ,i by w 0 / 2 (see Appendix D). Our method can be used to analyze other algorithms, and it improves existing results on APT.

3.2 Beyond Gaussian distributions

Theorems 2.3 and 2.4 hold for sub-Gaussian r.v. thanks to direct adaptations of concentration results (Lemmas 2.1, E.2 and E.5). The situation is akin to the regret bound of UCB: it holds for any sub-Gaussian, but it is close to optimality in a distribution-dependent sense only for Gaussians. However, if the focus is on asymptotically β -optimal algorithms, then it is challenging to express the characteristic time T ⋆ ( µ ) for the non-parametric class of sub-Gaussian distributions.

The TTUCB algorithm can also be defined for more general distributions such as single-parameter exponential families or bounded distributions. It is only a matter of adapting the definition of the UCB leader and the TC challenger. For bounded distributions, the UCB leader was studied in [1] and the TC challenger was analyzed in [21]. Leveraging their unified analysis of Top Two algorithms with our tracking-based results, we can show asymptotic β -optimality of TTUCB for bounded distributions and single-parameter exponential families with sub-exponential tails. We believe that non-asymptotic guaranties could be obtained for more general distributions, but it will come at the price of more technical arguments and less explicit non-asymptotic terms.

3.3 Generic regret minimizing leader

Our non-asymptotic analysis highlights that any regret minimization algorithm that selects the arm i ⋆ except for a sublinear number of times (Property 1) can be used as leader with the TC challenger.

Property 1 . There exists ( ˜ E n ) n with ∑ n P µ ( ˜ E ∁ n ) < + ∞ and a function h with h ( n ) = O ( n γ ) for some γ ∈ (0 , 1) such that under event ˜ E n , L n,i ⋆ ≥ n -1 -h ( n ) .

For asymptotic guarantees, the sufficient properties on the leader from [21] are weaker since they are even satisfied by the greedy choice B n =ˆ ı n . While Top Two algorithms were introduced by [38] to adapt Thompson Sampling to BAI, we have shown that other regret minimization algorithms can be used: the Top Two method is a generic wrapper to convert any regret minimization algorithm into a best arm identification strategy .

̸

The regret of an algorithm at time n , ¯ R n = ∑ i = i ⋆ ∆ i N n,i , is almost always studied through its expectation E [ ¯ R n ] . This is however not sufficient for our application. We need to prove that with high probability, N n,i is small for all arm i = i ⋆ . Such guarantees are known for UCB [3] and ETC [29], but are yet unknown for Thompson Sampling. We cannot in general obtain a good enough bound on N n,i from a bound on E [ ¯ R n ] . However, we can if we have high probability bounds on ¯ R n . Suppose that a regret minimization algorithm Alg 1 satisfies Property 2 and is independent of the horizon n .

̸

Property 2 . There exists s > 1 , γ ∈ (0 , 1) , ( E n,δ ) ( n,δ ) with ∑ n P µ [ E ∁ n,n -s ] < + ∞ and a function h with h ( n, n -s ) = O ( n γ ) such that under event E n,δ , ¯ R n ≤ h ( n, δ ) .

̸

̸

Let Alg 2 be the algorithm Alg 1 used in a Top Two procedure, but which uses only the observations obtained at times n such that I n = B n and discards the rest. Let ˜ E n = E n,n -s and ∆ min = min i = i ⋆ ∆ i . Then, under ˜ E n , Alg 2 satisfies ∑ i = i ⋆ N i n,i ≤ h ( n, n -s ) / ∆ min and Lemma 2.2 yields N i ⋆ n,i ⋆ ≥ β ( n -1) -h ( n, n -s ) / ∆ min -K/ 2 . Therefore, Property 1 holds for ˜ E n and h ( n ) = ( h ( n, n -s ) / ∆ min + K/ 2 + 1) /β . Given a specific algorithm, a finer analysis could avoid discarding information by using Alg 1 with every observations.

3.4 Adaptive proportions

Given a fixed allocation β , any Top Two algorithm can at best be asymptotically β -optimal. Since the optimal allocation β ⋆ ∈ arg min β ∈ (0 , 1) T ⋆ β ( µ ) is unknown, it should be learned from the observations by a Top Two algorithm using an adaptive proportion β n at time n . Recently, [44] proposes IDS to choose β n in an adaptive manner. For BAI with Gaussian observations, IDS yields β n = N n,C n / ( N n,B n + N n,C n ) . Let ¯ β i n := 1 L n,i ∑ t ∈ [ n -1] β t 1 ( B t = i ) be the average proportion when arm i was the leader before time n . Tracking with IDS requires to use ¯ β B n n +1 instead of β . Using the analysis of [44], it is reasonable to believe that one could obtain asymptotic optimality of TTUCB with IDS. However it is not clear how to adapt the non-asymptotic analysis since it heavily relies on β being fixed and bounded away from { 0 , 1 } . Experiments with IDS are available in Appendix G.2.1.

4 Experiments

In the moderate regime ( δ = 0 . 1 ), we assess the empirical performance of TTUCB with bonus g m and concentration parameters s = α = 1 . 2 . As benchmarks, we compare our algorithm with three sampling-based Top Two algorithms: TTTS, T3C and β -EB-TCI. In addition, we consider Track-and-Stop (TaS) [17], FWS [42], DKM [12], LUCB [24] and uniform sampling. At time n , the LUCB algorithm computes a leader and a challenger, then sample them both (see Appendix G.1). To provide a clear comparison with Top Two algorithms, we define a new β -LUCB algorithm which sample the leader with probability β , else sample the challenger. At the exception of LUCB and β -LUCB which have their own stopping rule, all algorithms uses the stopping rule (1) with the heuristic threshold c ( n, δ ) = log((1 + log n ) /δ ) . Even though this choice is not sufficient to prove δ -correctness, it yields an empirical error which is several orders of magnitude lower than δ . Top Two algorithms and β -LUCB use β = 1 / 2 . To allow for a fair numerical comparison, LUCB and β -LUCB use √ 2 c ( n -1 , δ ) /N n,i as bonus, which is too tight to yield valid confidence intervals. Supplementary experiments are available in Appendix G.

̸

Random instances We assess the performance on 5000 random Gaussian instances with K = 10 such that µ 1 = 0 . 6 and µ i ∼ U ([0 . 2 , 0 . 5]) for all i = 1 . Numerically, we observe w ⋆ ( µ ) i ⋆ ≈ 1 / 3 ± 0 . 02 (mean ± std). In Figure 1(a), we see that TTUCB performs on par with existing Top Two algorithms, and slightly outperforms TaS and FWS. Our algorithm achieves significantly better result than DKM, LUCB, 1 / 2 -LUCB and uniform sampling. The CPU running time is reported in Table 4, and the observed empirical errors before stopping is displayed in Figure 3 (Appendix G.2).

̸

Larger sets of arms We evaluate the impact of larger number of arms. The ' 1 -sparse' scenario of [20] sets µ 1 = 1 / 4 and µ i = 0 for all i = 1 , i.e. H ( µ ) = 32( K -1) (see Appendix G.2 for other instances). We consider algorithms with low computational cost. In Figure 1(b), all algorithms have the same linear scaling in K (i.e. in H ( µ ) ). Faced with an increase in the number of arms, the TS leader used in T3C appears to be more robust than the UCB leader in TTUCB. This is a common feature of UCB algorithms which have to overcome the bonus of sub-optimal arms.

5 Conclusion

In this paper, we have shown the first non-asymptotic upper bound on the expected sample complexity of a Top Two algorithm, which holds for any error level and for any instance having a unique best arm. Furthermore, we have demonstrated that the TTUCB algorithm achieves competitive empirical performance compared to other algorithms, including Top Two methods.

While our guarantees hold for a fixed proportion β allocated to the leader, [44] recently introduced IDS to define an adaptive proportion β n at time n and show asymptotic optimality for Gaussian distributions. Deriving guarantees for IDS for single-parameter exponential families is a challenging open problem. Finally, Top Two algorithms are a promising method to tackle complex settings. While heuristics exist for some structured bandits such as Topk , it would be interesting to efficiently adapt Top Two methods to deal with sophisticated structure, e.g. linear bandits.

Acknowledgments and Disclosure of Funding

Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr). This work has been partially supported by the THIA ANR program 'AI_PhD@Lille'. The authors acknowledge the funding of the French National Research Agency under the project FATE (ANR22-CE23-0016-01).

References

S. Agrawal, S. K. Juneja, and W. M. Koolen. Regret minimization in heavy-tailed bandits. In Conference on Learning Theory , 2021.

K. Ariu, M. Kato, J. Komiyama, K. McAlinn, and C. Qin. Policy choice and best arm identification: Asymptotic analysis of exploration sampling under posterior weighted policy regret. arXiv preprint arXiv:2109.08229 , 2021.

J.-Y. Audibert, R. Munos, and C. Szepesvári. Exploration-exploitation trade-off using variance estimates in multi-armed bandits. Theoretical Computer Science , 410(19), 2009.

J.-Y. Audibert, S. Bubeck, and R. Munos. Best Arm Identification in Multi-armed Bandits. In Conference on Learning Theory , 2010.

P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning , 47(2):235-256, 2002.

A. Barrier, A. Garivier, and T. Kocák. A non-asymptotic approach to best-arm identification for gaussian bandits. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics , 2022.

K. Chaloner and I. Verdinelli. Bayesian experimental design: A review. Statistical Science , 1995.

L. Chen, J. Li, and M. Qiao. Towards instance optimal bounds for best arm identification. Conference on Learning Theory , 2017.

L. Chen, J. Li, and M. Qiao. Nearly instance optimal sample complexity bounds for top-k arm selection. In Artificial Intelligence and Statistics , 2017.

H. Chernoff. Sequential design of Experiments. The Annals of Mathematical Statistics , 30(3): 755-770, 1959.

R. Degenne. Impact of structure on the design and analysis of bandit algorithms . PhD thesis, Université de Paris, 2019.

R. Degenne, W. M. Koolen, and P. Ménard. Non-asymptotic pure exploration by solving games. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

R. Degenne, H. Shao, and W. M. Koolen. Structure adaptive algorithms for stochastic bandits. In International Conference on Machine Learning (ICML) , 2020.

E. Even-Dar, S. Mannor, and Y. Mansour. Pac bounds for multi-armed bandit and markov decision processes. In International Conference on Computational Learning Theory , 2002.

E. Even-Dar, S. Mannor, and Y. Mansour. Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems. Journal of Machine Learning Research , 7:1079-1105, 2006.

V. Gabillon, M. Ghavamzadeh, and A. Lazaric. Best Arm Identification: A Unified Approach to Fixed Budget and Fixed Confidence. In Advances in Neural Information Processing Systems , 2012.

A. Garivier and E. Kaufmann. Optimal best arm identification with fixed confidence. In Proceedings of the 29th Conference On Learning Theory , 2016.

L. Hong, W. Fan, and J. Luo. Review on ranking and selection: A new perspective. Frontiers of Engineering Management , 8:321-343, 2021.

K. Jamieson, M. Malloy, R. Nowak, and S. Bubeck. lil'UCB: an Optimal Exploration Algorithm for Multi-Armed Bandits. In Proceedings of the 27th Conference on Learning Theory , 2014.

K. G. Jamieson and R. D. Nowak. Best-arm identification algorithms for multi-armed bandits in the fixed confidence setting. In Conference on Information Sciences and Systems (CISS) , 2014.

M. Jourdan, R. Degenne, D. Baudry, R. De Heide, and E. Kaufmann. Top two algorithms revisited. Advances in Neural Information Processing Systems , 2022.

M. Jourdan, R. Degenne, and E. Kaufmann. Dealing with unknown variances in best-arm identification. International Conference on Algorithmic Learning Theory , 2023.

K.-W. Jun and R. Nowak. Anytime exploration for multi-armed bandits using confidence information. In International Conference on Machine Learning (ICML) , 2016.

S. Kalyanakrishnan, A. Tewari, P. Auer, and P. Stone. PAC subset selection in stochastic multi-armed bandits. In International Conference on Machine Learning (ICML) , 2012.

Z. Karnin, T. Koren, and O. Somekh. Almost optimal Exploration in multi-armed bandits. In International Conference on Machine Learning (ICML) , 2013.

J. Katz-Samuels, L. Jain, K. G. Jamieson, et al. An empirical process approach to the union bound: Practical algorithms for combinatorial and linear bandits. Advances in Neural Information Processing Systems , 2020.

E. Kaufmann and W. M. Koolen. Mixture martingales revisited with applications to sequential tests and confidence intervals. Journal of Machine Learning Research , 22(246):1-44, 2021.

E. Kaufmann, O. Cappé, and A. Garivier. On the Complexity of A/B Testing. In Proceedings of the 27th Conference On Learning Theory , 2014.

T. Lattimore and C. Szepesvari. Bandit Algorithms . Cambridge University Press, 2019.

A. Locatelli, M. Gutzeit, and A. Carpentier. An optimal algorithm for the thresholding bandit problem. In International Conference on Machine Learning (ICML) , 2016.

A. A. Marjani, T. Kocák, and A. Garivier. On the complexity of all ε -best arms identification. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , 2022.

B. Mason, L. Jain, A. Tripathy, and R. Nowak. Finding all ε -good arms in stochastic bandits. Advances in Neural Information Processing Systems , 2020.

A. Mukherjee and A. Tajer. Best arm identification in stochastic bandits: Beyond β -optimality. arXiv preprint arXiv:2301.03785 , 2023.

A. Mukherjee and A. Tajer. Sprt-based efficient best arm identification in stochastic bandits. IEEE Journal on Selected Areas in Information Theory , 2023.

F. Pukelsheim. Optimal design of experiments . SIAM, 2006.

C. Qin, D. Klabjan, and D. Russo. Improving the expected improvement algorithm. In Advances in Neural Information Processing Systems 30 (NIPS) , 2017.

H. Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society , 58(5):527-535, 1952.

D. Russo. Simple Bayesian algorithms for best arm identification. In Proceedings of the 29th Conference on Learning Theory (COLT) , 2016.

X. Shang, R. de Heide, E. Kaufmann, P. Ménard, and M. Valko. Fixed-confidence guarantees for bayesian best-arm identification. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2020.

M. Simchowitz, K. Jamieson, and B. Recht. The simulator: Understanding adaptive sampling in the moderate-confidence regime. In Conference on Learning Theory , 2017.

W. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25:285-294, 1933.

P.-A. Wang, R.-C. Tzeng, and A. Proutiere. Fast pure exploration via frank-wolfe. Advances in Neural Information Processing Systems , 2021.

E. B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association , 22(158):209-212, 1927.

W. You, C. Qin, Z. Wang, and S. Yang. Information-directed selection for top-two algorithms. In The Thirty Sixth Annual Conference on Learning Theory , pages 2850-2851. PMLR, 2023.

A Outline

The appendices are organized as follows:

Appendix B gathers notation used in this work.

In Appendix C, we study the link between T ⋆ ( µ ) and T ⋆ β ( µ ) for Gaussian distributions.

The detailed analysis of our non-asymptotic upper bound (Theorem 2.4), sketched in Section 3, is detailed in Appendix D. We also give a non-asymptotic upper bound on the TTUCB using g m in (3) (Corollary D.5), and on uniform sampling (Theorem D.6).

We show the asymptotic optimality of our algorithm (Theorem 2.3) in Appendix F.

Appendix E gathers concentration results used by the stopping rule (Lemma 2.1) and the sampling rule.

Implementation details and supplementary experiments are detailed in Appendix G.

B Notation

We recall some commonly used notation: the set of integers [ n ] := { 1 , · · · , n } , the complement X ∁ and interior ˚ X of a set X , Landau's notation o , O , Ω and Θ , the ( K -1) -dimensional probability simplex △ K := { w ∈ R K + | w ≥ 0 , ∑ i ∈ [ K ] w i = 1 } . While Table 2 gathers problem-specific notation, Table 3 groups notation for the algorithms. We emphasize that N i n,i is the number of times where we pulled arm i as a leader before time n .

C Characteristic times

Let µ ∈ D K such that i ⋆ ( µ ) = { i ⋆ } . Let β ∈ (0 , 1) and w ⋆ β be the unique allocation β -optimal allocation satisfying w ⋆ β,i > 0 for all i ∈ [ K ] (Lemma C.2), i.e. w ⋆ β ( µ ) = { w ⋆ β } where

̸

̸

We restate without proof two fundamental results on the characteristic time and the associated allocation, which were first shown in [38]. Lemma C.1 gives an upper bound on T ⋆ β ( µ ) /T ⋆ ( µ ) and Lemma C.2 shows that the ( β -)optimal allocation is unique with strictly positive values. [38] shows that these two results hold for any single-parameter exponential families. [21] extended their proof for the non-parametric family of bounded distributions. Moreover, they argue that these results should hold for more general distributions provided some regularity assumptions are satisfied.

Lemma C.2 ([38]) . If i ⋆ ( µ ) is a singleton and β ∈ (0 , 1) , then w ⋆ ( µ ) and w ⋆ β ( µ ) are singletons, i.e. the optimal allocations are unique, and w ⋆ ( µ ) i > 0 and w ⋆ β ( µ ) i > 0 for all i ∈ [ K ] .

Gaussian distributions Since Lemma C.1 is a worst-case inequality holding for general distributions, we expect that tighter inequality can be achieved for Gaussian distributions by leveraging their symmetry. This intuition is fueled by recent results of [6]. Using a rewriting of the optimization problem underlying T ⋆ ( µ ) (Lemma C.3), they provide a better understanding of characteristic times and their optimal allocations (Lemma C.4). In particular, for Gaussian distributions, Lemma C.4 shows that the optimal allocation of arm i ⋆ is never above 1 / 2 and is larger than 1 / ( √ K -1 + 1) ≥ 1 /K .

Lemma C.3 (Proposition 8 in [6]) . Let µ ∈ R K be a K -arms Gaussian bandits and r ( µ ) be the solution of ψ µ ( r ) = 0 , where

̸

and ψ µ is convex and decreasing. Then,

̸

Lemma C.4 (Proposition 10 in [6]) . Let µ ∈ R K be a K -arms Gaussian bandits. For K = 2 ,

For K ≥ 3 , we have and

̸

̸

̸

where ∆ 2 = 1 K -1 ∑ i = i ⋆ ( µ i ⋆ -µ i ) 2 . In particular, the equalities w ⋆ ( µ ) i ⋆ = 1 / ( K -1 + 1) and T ⋆ ( µ ) = 2(1+ √ K -1) 2 min i = i ⋆ ( µ i ⋆ -µ i ) 2 are reached if and only if µ i = max i = i ⋆ µ i for all i = i ⋆ .

̸

̸

By inspecting the proof of Proposition 8 in [6], we obtain directly the following rewriting of T ⋆ β ( µ ) .

Lemma C.5. Let µ ∈ R K be a K -arms Gaussian bandits and r β ( µ ) be the solution of φ µ,β ( r ) = 0 , where

̸

and φ µ,β is convex and decreasing. Then,

̸

̸

̸

Proof. Using the proof of Proposition 8 in [6], we obtain directly that

where C β ( µ ) is the solution of Φ µ ( C ) = 0 where

̸

The idea behind the above result is that at the equilibrium, i.e. at w ⋆ , all the transportation costs are equal to C . Then, the implicit equation defining C is obtained by using the constraints that ∑ i = i ⋆ w ⋆ i = 1 -β . To conclude, we simply use r = β/C .

̸

Since it is the sum of K -1 convex and decreasing functions, φ µ,β is also convex and decreasing.

Lemma C.6 aims at improving the worst-case inequality between T ⋆ 1 / 2 ( µ ) and T ⋆ ( µ ) in the Gaussian setting. For K = 2 , those two quantities are equal. For K ≥ 3 , we showed that max µ : | i ⋆ ( µ ) | =1 T ⋆ 1 / 2 ( µ ) /T ⋆ ( µ ) is at least r K , which is achieved when all sub-optimal arms have the same mean. As the gradient of the ratio is the null vector for those instances, we conjecture this is the maximum, i.e. T ⋆ 1 / 2 ( µ ) ≤ r K T ⋆ ( µ ) . Our conjecture is supported by numerical simulations for K ≥ 3 . In Figure 2, we plot the ratio of characteristic times Ω( x ) = T ⋆ 1 / 2 ( µ ) /T ⋆ ( µ ) for K ∈ { 3 , 4 } . We observed that our conjecture is validated empirically, and that T ⋆ 1 / 2 ( µ ) /T ⋆ ( µ ) is often close to 1 .

̸

Lemma C.6. For K = 2 , we have T ⋆ 1 / 2 ( µ ) = T ⋆ ( µ ) . For K ≥ 3 , let r K = 2 K/ (1 + √ K -1) 2 . Then, for all µ such that i ⋆ ( µ ) is unique, we have T ⋆ 1 / 2 ( µ ) T ⋆ ( µ ) = Ω( x ) where x j = µ i ⋆ -µ j µ i ⋆ -µ j ⋆ ≥ 1 for all j / ∈ { i ⋆ , j ⋆ } with j ⋆ ∈ arg min j = i ⋆ µ i ⋆ -µ j . In other words, T ⋆ 1 / 2 ( µ ) T ⋆ ( µ ) is independent from µ i ⋆ and min j = i ⋆ µ i ⋆ -µ j . Moreover, we have

Proof. Using Lemma C.4, we have T ⋆ 1 / 2 ( µ ) = T ⋆ ( µ ) directly for K = 2 .

For K ≥ 3 , we want to upper bound T ⋆ 1 / 2 ( µ ) /T ⋆ ( µ ) . If we denote by ∆ min ( µ ) = min i = i ⋆ ( µ i ⋆ -µ i ) 2 , it is easy to see that T ⋆ 1 / 2 ( µ ) /T ⋆ ( µ ) = T ⋆ 1 / 2 (˜ µ ) /T ⋆ (˜ µ ) when ∆ min ( µ ) = ∆ min (˜ µ ) . Likewise, this ratio is invariant by translation of all means by a same quantity. Therefore, we consider without restriction an instance µ such that ∆ min ( µ ) = 1 and with

where x 2 = 1 and x i ≥ 1 for all i ≥ 3 .

First, we can rewrite

̸

̸

where the inequality is an equality if and only x = 1 for all i ≥ 2 . Using Lemma C.4, we know that

i √

if and only x i = 1 for all i ≥ 2 . Therefore, we have exhibited an instance such that

Applying Lemma C.5 for β = 1 / 2 , we obtain T ⋆ 1 / 2 ( x ) = 4 C ( x ) where C ( x ) is the implicit solution of the equation

Using Lemma C.3, we know that

where r ( x ) is the implicit solution of the equation

Therefore, we obtain that

and our goal is to show that the above quantity is maximum if and only x i = 1 for all i ≥ 2 . One way of doing this is by computing the gradient and showing it is negative, which would imply that the function is decreasing. Let j ≥ 3 . Using the implicit differentiation theorem, we obtain

and

Direct computations yield that

̸

and j ∂ ∂x j ( C ( x ) r ( x )( r ( x ) x 2 j -1) ) x =1 K -2 = ( 1 K -1 -1 ) 2 C (1 K -2 ) ( r (1 K -2 ) -1) 2 , and, for i ≥ 3 s.t. i = j , K -2

Then, plugging everything together, we obtained that

Therefore, we have shown that ∇ Ω(1 K -2 ) = 0 K -2 .

D Non-asymptotic analysis

In Appendix D.1, we state and prove one key result for each one of the three main components of the TTUCB sampling rule: the UCB leader (Lemma D.1), the TC challenger (Lemma D.2) and the tracking (Lemma D.3). The proof of Theorem 2.4 is detailed in Appendix D.2, which uses the stopping rule (1) and a proof method from [30]. It is a direct consequence of a more general result (Theorem D.4). In Appendix D.3, we prove a non-asymptotic upper bound for the TTUCB when using g m instead of g u (Corollary D.5). We compare our results with uniform sampling in Appendix D.4 (Theorem D.6). Other technicalities are gathered in Appendix D.5.

D.1 Key properties

Before delving in the proof of Theorem D.4 itself, we present the key properties of each component of the TTUCB sampling rule under a some concentration event.

Let α > 1 and s > 1 . Let ( E n ) n>K be the sequence of concentration events defined as E n := E 1 ,n ∩ E 2 ,n for all n > K where E 1 ,n and E 2 ,n are defined in (17) and (19) as

̸

with g u ( n ) = 2 α (1 + s ) log n . In Lemma E.6, it is shown that ∑ n>K P ( E ∁ n ) ≤ (2 K -1) ζ ( s ) .

UCBleader Lemma D.1 shows that the UCB leader is different from i ⋆ for only a sublinear number of times under a certain concentration event. It is slightly more general than Lemma 3.2 presented in Section 2, which follows from H 1 ( µ ) ≤ H 1 ( µ ) + 2∆ -2 min = H ( µ ) .

̸

Lemma D.1. Let ( E 1 ,n ) n and g u as in Lemma E.2. Let H 1 ( µ ) = ∑ i = i ⋆ ( µ ) 2 ( µ i ⋆ ( µ ) -µ i ) 2 . For all n > K , under the event E 1 ,n ,

Let ( E 3 ,n ) n and g m as in Lemma E.3. Under the event E 3 ,n , (6) holds by using g m instead of g u .

̸

Proof. Suppose that at time t ∈ [ n 1 /α , n ] , the UCB leader is different from i ⋆ , i.e. B t = k = i ⋆ . Using the event E 1 ,n and the definition of B t yields

We get that if t ≥ n 1 /α , then N t,k ≤ 4 g u ( t ) ( µ i ⋆ -µ k ) 2 . Therefore, we obtain the following upper bound on the number of times the leader is different from i ⋆ up to time t

̸

̸

̸

̸

where we used Lemma D.3 for the second inequality and N k t,k ≤ N t,k for the third. This concludes the proof for g u . The same reasoning can be applied for g m .

̸

TC challenger Lemma D.2 shows a lower bound on the 'transportation' costs used by the TC challenger provided a certain concentration holds. This lower bound depends only on the empirical counts when the best arm is the leader.

Lemma D.2. For all n > K , under the event E 2 ,n , for all t ∈ [ n 1 /α , n ] such that B t = i ⋆ ,

̸

Proof. Under E 2 ,n , using B t = i ⋆ yields

̸

t,B t t,k √ 1 /N t,B t +1 /N t,k ≥ i ⋆ k √ 1 /N t,i ⋆ +1 /N t,k -√ 2 α (2 + s ) log t ≥ i ⋆ k √ 1 /N i ⋆ t,i ⋆ +1 /N i ⋆ t,k -√ 2 α (2 + s ) log where the second inequality uses that N t,k ≥ N i ⋆ t,k for all k = i ⋆ .

Tracking Lemma D.3 shows the key property satisfied by the K independent tracking procedures used by the TTUCB sampling rule. It is slightly more general than Lemma 2.2 presented in Section 2. It is a simple corollary of Theorem 6 in [13].

̸

Lemma D.3. For all n > K and i ∈ [ K ] , we have -1 / 2 ≤ N i n,i -βL n,i ≤ 1 and for all k = i ,

Proof. We can rewrite the tracking condition N B n n,B n ≤ βL n +1 ,B n as

For all k ∈ [ K ] , this corresponds to a two-arms C-Tracking between the leader k and the challengers with w n = ( β, 1 -β ) for all n such that B n = k . The leader's pulling count is N B n n,B n and the challengers' pulling count is L n +1 ,B n -N B n n,B n . We recall that C-Tracking was defined as I n = arg min k ∈ [ K ] N n,k -∑ t ∈ [ n ] w t,k .

Theorem 6 in [13] yields for all n > K , -1 2 ≤ N B n n,B n -βL n,B n ≤ 1 . The K parallel tracking procedures are independent since they are considering counts partitioned on the considered leader. Therefore, the above results holds for all i ∈ [ K ] . This concludes the first part of the proof.

Direct manipulations yield the second part of the result, namely

̸

The choice of K independent tracking procedures was made for two reasons. First, independent procedures are simpler to analyze for theoretical purpose. Second, independent procedures yields better empirical performance since it avoids over-sampling a sub-optimal arm when it is mistakenly chosen as leader. To understand the second argument, let's look at another design with one tracking procedure. Namely we set I n = B n if N n,B n ≤ βn , else I n = C n . When B n = i ⋆ , then it will (almost always) take I n = B n since N n,B n is lower than βn . On the other hand, when B n = i ⋆ with K independent tracking procedures, both N B n n,B n and L n,B n are small, hence it is less systematic.

D.2 Proof of Theorem 2.4

Theorem 2.4 is a direct corollary of Theorem D.4, which holds for any β ∈ (0 , 1) , s > 1 and α > 1 . Theorem D.4. Let ( δ, β ) ∈ (0 , 1) 2 , s > 1 and α > 1 . Using the threshold (2) in (1) and g u in (3) , the TTUCB algorithm satisfies that, for all µ ∈ R K such that | i ⋆ ( µ ) | = 1 ,

̸

where

̸

with ε ∈ (0 , 1] , w 0 ∈ [0 , 1 / ( K -1)] and ζ is the Riemann ζ function. For all x > 0 , the function h 1 ( x ) := xW -1 ( log( x ) + 2+ K/β x ) is positive, increasing for x ≥ 2 + K/β , and satisfies h 1 ( x ) ≈ x (log x +loglog x ) .

Let n > K such that E n holds true and the algorithm has not stop yet, i.e. E n ∩ { n < τ δ } . Let t ∈ [ n 1 /α , n ] such that B t = i ⋆ . Let c ( n, δ ) as in (2), which satisfies that n ↦→ c ( n, δ ) is increasing. Using the stopping rule (1) and t ≤ n < τ δ , we obtain

̸

̸

̸

The last inequality is an equality when B t =ˆ ı t , and trivially true when B t =ˆ ı t since a positive term is higher than zero (already null when taking i = ı t ). Using Lemma D.2, we obtain

̸

̸

where the second inequality is obtained by artificially making appear 1 /β +1 /w ⋆ β,C t and taking the minimum of i = i ⋆ . The last inequality simply uses the definition of w i ⋆ β .

While combining Lemma D.1 and Lemma D.3 links N i ⋆ t,i ⋆ / ( t -1) with β , we need another argument to compare the empirical allocation N i ⋆ t,C t / ( t -1) of the sub-optimal arm C t with its β -optimal allocation w ⋆ β,C t . Before delving into this key argument, we conclude the proof under an assumption that will be shown to hold later: there exists D 0 > 0 and T µ > 0 such that for all n > T µ , there exists a well chosen t ∈ [ n 1 /α , n ] with B t = i ⋆ , which satisfies

Let's define

For all n > max { T µ , T (1 , D 0 ) } , the lower bound on µ t,B t -µ t,C t √ 1 /N t,B t +1 /N t,C t is positive, hence we can use that it is upper bounded by √ 2 c ( n -1 , δ ) . Putting everything together, we have shown that

Therefore, we have E n ∩ { n < τ δ } = ∅ (i.e. E n ⊂ { τ δ ≤ n } ) for all n ≥ max { T ( δ, D 0 ) , T (1 , D 0 ) , T µ } + 1 . Using that δ → T ( δ, D 0 ) is an decreasing function (since C G is increasing), we obtain that T ( δ, D 0 ) = max { T ( δ, D 0 ) , T (1 , D 0 ) } .

Combining Lemmas D.8 and E.6 yields

At this stage, the proof of Theorem D.4 boils down to exhibiting D 0 > 0 and T µ > 0 such that: for all n > T µ , there exists a well chosen t ∈ [ n 1 /α , n ] with B t = i ⋆ and such that (7) holds. As mentioned above, the crux of the problem is to relate N i ⋆ t,C t / ( t -1) and w ⋆ β,C t . To do so, we will build on the idea behind the proof for APT from [30]: consider an arm being over-sampled and study the last time this arm was pulled.

̸

By the pigeonhole principle, at time n , there is an index k 1 = i ⋆ such that (5) holds, i.e.

and we take such k 1 . Let t 1 := sup { t < n | ( B t , C t ) = ( i ⋆ , k 1 ) } be the last time at which i ⋆ was the leader and k 1 was the challenger. If I t 1 = B t 1 then N i ⋆ t 1 ,k 1 = N i ⋆ n,k 1 , else N i ⋆ t 1 ,k 1 = N i ⋆ n,k 1 -1 . In both cases, we have N i ⋆ t 1 ,k 1 ≥ N i ⋆ n,k 1 -1 . Let f 1 as in (14). Combined the above with Lemma D.9, we obtain

̸

Let w -> 0 be a lower bound on w ⋆ β,k 1 , for example consider w -= min i = i ⋆ w ⋆ β,i . Let

Since t 1 -1 ≥ N i ⋆ t 1 ,k 1 ≥ w ⋆ β,k 1 ( n -1) -f 1 ( n ) , we obtain that t 1 ≥ n 1 /α for all n > N 0 ( w -) . For instances µ such that w ⋆ β,k 1 is small, the equation (5) can be satisfied at the very beginning, hence t 1 might be sublinear in n . Therefore, while combining Lemma D.1 and Lemma D.3 yields N i ⋆ t 1 ,i ⋆ ⪆ β ( t 1 -1) , it is not possible to obtain N i ⋆ t 1 ,i ⋆ ⪆ β ( n -1) . Due to the missing link between t 1 and n , we use the following inequality

which is a suboptimal step which artificially introduces 1 /β , and is responsible for the multiplicative factor 1 /β in Theorem D.4. Improving on this suboptimal step is an interesting question, whose answer still eludes us. One idea would be to leverage the information of the sampled arm at time t since we have N i ⋆ t,i ⋆ ≤ βL t +1 ,i ⋆ when i ⋆ = B t = I t , else N i ⋆ t,i ⋆ ≥ βL t +1 ,i ⋆ .

Let ε ∈ (0 , 1] . It remains to control both terms. First, we obtain

for all n > C 1 ( w -) . The last inequality is obtained by definition of

which ensures that, for all n > C 1 ( w -) , the last condition of the equivalence

is satisfied since w ⋆ β,k 1 ≥ w -and n > C 1 ( w -) . Second, using Lemma D.3 with N i ⋆ t 1 ,k 1 ≥ w ⋆ β,k 1 ( n -1) -f 1 ( n ) , we obtain

for all n > C 2 ( w -) . The last inequality is obtained by definition of

which ensures that, for all n > C 2 ( w -) , the last condition of the equivalence

is satisfied since w ⋆ β,k 1 ≥ w -and n > C 2 ( w -) . By comparison between (8) and (10), we notice that

Putting everything together, we have shown that taking D 0 = (1 + ε ) 2 /β , we have for all n > max { K,C 0 ( w -) , C 1 ( w -) , C 2 ( w -) } , there exists t 1 ∈ [ n 1 /α , n ] with B t 1 = i ⋆ and such that (7) holds. Let h 1 defined in (15). Since ε ≤ 1 , using Lemmas D.10 and D.11 with the above yields

̸

Using w -= min i = i ⋆ w ⋆ β,i yields the first part of Theorem D.4, i.e. the special case of w 0 = 0 .

̸

Refined non-asymptotic upper bound When considering large K or instances with unbalanced β -optimal allocation, min i = i ⋆ w ⋆ β,i can become arbitrarily small. Therefore, the dependency in the inverse of min i = i ⋆ w ⋆ β,i is undesired, and we would like to clip it with a value of our choosing which is away from zero.

̸

Let w 0 ∈ (0 , 1 / ( K -1)] be an allocation threshold and d µ ( w 0 ) := |{ i ∈ [ K ] \ { i ⋆ } | w ⋆ β,i < (1 -β ) w 0 }| be the number of arms having a β -optimal allocation strictly smaller than (1 -β ) w 0 . As discussed above, for instances µ such that w ⋆ β,k 1 (defined above) is small, the equation (5) can be satisfied at the very beginning for a small empirical allocation N i ⋆ n,k 1 . To provide a more meaningful result, one needs to have a sub-optimal arm k 1 such that either:

Case 1: w ⋆ β,k 1 is not too small, i.e. w ⋆ β,k 1 ≥ (1 -β ) w 0 .

Case 2: w ⋆ β,k 1 is too small but N i ⋆ n,k 1 is large enough, i.e. w ⋆ β,k 1 < (1 -β ) w 0 and N i ⋆ n,k 1 ≥ w 0 ( L n,i ⋆ -N i ⋆ n,i ⋆ ) .

̸

In case 1, we can conduct the same manipulations as above simply by using w -= max { (1 -β ) w 0 , min i = i ⋆ w ⋆ β,i } instead of w -= min i = i ⋆ w ⋆ β,i , since it is a lower bound for w ⋆ β,k 1 .

̸

̸

In case 2, the above proof can also be applied by conducting the same algebraic manipulations with (1 -β ) w 0 instead of w ⋆ β,k 1 , and using w -= (1 -β ) w 0 = max { (1 -β ) w 0 , min i = i ⋆ w ⋆ β,i } . To slightly detail the argument, we can show similarly that N i ⋆ t 1 ,k 1 ≥ (1 -β ) w 0 ( n -1) -f 1 ( n ) , where f 1 as in (14) since we have (1 -β ) w 0 < 1 -β . Then, for all n > C 1 ((1 -β ) w 0 ) ,

The problematic situation happens when we are neither in case 1 nor in case 2:

Case 3: both w ⋆ β,k 1 and N i ⋆ n,k 1 are too small, i.e. w ⋆ β,k 1 < (1 -β ) w 0 and N i ⋆ n,k 1 < w 0 ( L n,i ⋆ -N i ⋆ n,i ⋆ ) .

̸

In case 3 it is not possible to conclude the proof with the arm k 1 without paying the price of an inverse of min i = i ⋆ w ⋆ β,i . To overcome this price, we need to find another arm k such that either case 1 or case 2 happens. Since N i ⋆ n,k 1 and w ⋆ β,k 1 are small, we will ignore arm k 1 and use the pigeonhole principle on all arm i ∈ [ K ] \ { i ⋆ , k 1 } . As in (5), we obtain that there exists k 2 such that

where the last inequality is obtained by using w ⋆ β,k 1 > 0 and N i ⋆ n,k 1 < w 0 ( L n,i ⋆ -N i ⋆ n,i ⋆ ) . Based on k 2 the same dichotomy happens: either we can conclude the proof when we are in case 1 or 2 or we cannot since we are in case 3. If case 3 occurs also for k 2 , i.e. w ⋆ β,k 2 < (1 -β ) w 0 and N i ⋆ n,k 2 < w 0 ( L n,i ⋆ -N i ⋆ n,i ⋆ -N i ⋆ n,k 1 ) , we should also ignore it since it is non informative.

The main idea is then to peel off arms that are not informative, till we find an informative one. By induction, we construct a sequence ( k a ) a ∈ [ d ] of such arms, where k d is the first arm for which either case 1 or case 2 holds. This means that for all a ∈ [ d -1] , we have w ⋆ β,k a < (1 -β ) w 0 and

The construction, which rely on the pigeonhole principle for i ∈ [ K ] \ ( { i ⋆ } ∪ { k a } a ∈ [ d -1] ) , yields that k d satisfies

where the last inequality is obtained since ∑ a ∈ [ d -1] w ⋆ β,k a > 0 and by a simple recurrence on the arms { k a } a ∈ [ d -1] . Since the arm k d satisfies case 1 or case 2, we can conclude similarly as above. Let t d := sup { t < n | ( B t , C t ) = ( i ⋆ , k d ) } .

̸

When w ⋆ β,k d ≥ (1 -β ) w 0 , the above proof can also be applied by conducting the same algebraic manipulations with (1 -w 0 ) d -1 w ⋆ β,k d , and using w -= (1 -w 0 ) d -1 max { (1 -β ) w 0 , min i = i ⋆ w ⋆ β,i } . In more details, we can show that N i ⋆ t d ,k d ≥ (1 -w 0 ) d -1 w ⋆ β,k d ( n -1) -f 1 ( n ) , where f 1 as in (14) since we have (1 -w 0 ) d -1 w ⋆ β,k d < 1 -β . Then, for all n > C 1 ( w -) ,

This allow to conclude the result with D 0 = (1+ ε ) 2 β (1 -w 0 ) d -1 , hence paying a multiplicative factor of 1 / (1 -w 0 ) d -1 .

̸

When w ⋆ β,k d < (1 -β ) w 0 and N i ⋆ n,k a ≥ w 0 ( L n,i ⋆ -N i ⋆ n,i ⋆ -∑ a ∈ [ d -1] N i ⋆ n,k a ) , we conclude similarly by manipulating (1 -w 0 ) d -1 (1 -β ) w 0 , and using w -= (1 -w 0 ) d -1 (1 -β ) w 0 = (1 -w 0 ) d -1 max { (1 -β ) w 0 , min i = i ⋆ w ⋆ β,i } . First, we have N i ⋆ t d ,k d ≥ (1 -w 0 ) d -1 w 0 ( n -1) -f 1 ( n ) , where f 1 as in (14) since we have (1 -β )(1 -w 0 ) d -1 w 0 < 1 -β . Then, for all n > C 1 ( w -) ,

This allow to conclude the result with D 0 = (1+ ε ) 2 β (1 -w 0 ) d -1 .

To remove the dependency in the random variable d , we consider the worst case scenario where { k a } a ∈ [ d -1] = { i ∈ [ K ] \ { i ⋆ } | w ⋆ β,i < (1 -β ) w 0 } , i.e. d -1 ≤ d µ ( w 0 ) . In words, it means that we had to enumerate over all arms with small allocation, such that case 2 didn't hold, before finding an arm with large allocation, i.e. satisfying case 1.

̸

In all the cases considered above, the parameters always satisfied w -≥ (1 -w 0 ) d µ ( w 0 ) max { (1 -β ) w 0 , min i = i ⋆ w ⋆ β,i } and D 0 ≤ (1+ ε ) 2 β (1 -w 0 ) dµ ( w 0 ) . This yields the second part of Theorem D.4, i.e. for w 0 ∈ (0 , 1 / ( K -1)] .

D.3 Tighter UCB leader

While Theorem 2.3 holds for the TTUCB sampling rule using g m and g u , Theorem D.4 is formulated solely for g u . Experiments highlight that using g u leads to worse performances than when using g m . This is not surprising since g m is smaller than g u .

It is direct to see that the proof of Theorem D.4 also holds for g m up to additional technicalities which we detail below. As the obtained non-asymptotic upper bound is less explicit, we chose not to include it in Theorem 2.3.

Let α > 1 and s > 1 . Let ( ˜ E n ) n be the sequence of concentration events defined as ˜ E n := E 3 ,n ∩E 2 ,n for all n > K where E 3 ,n is defined in (18) as

where W -1 ( x ) = -W -1 ( -e -x ) for all x ≥ 1 , with W -1 is the negative branch of the Lambert W function.

Let n > K such that ˜ E n holds true and the algorithm has not stop yet, i.e. ˜ E n ∩ { n < τ δ } . Using the second part of Lemma D.1, the vast majority of the proof is unchanged. Modifying the definition of f 1 in (14) of Lemma D.9 to account for g m , we define f 2 ( n ) := 2 H ( µ ) g m ( n ) /β + K/β +2 for all n > K .

In light of the proofs of the technical Lemmas D.10 and D.11, we can define

and obtain directly Corollary D.5 with the same proof as in Appendix D.2.

Corollary D.5. Let ( δ, β ) ∈ (0 , 1) 2 , ε ∈ (0 , 1] , s > 1 , α > 1 , w 0 ∈ [0 , 1 / ( K -1)] . Combining the stopping rule (1) with threshold (2) and the TTUCB sampling rule using g u in (3) yields a δ -correct algorithm such that, for all µ ∈ R K with | i ⋆ ( µ ) | = 1 ,

where T 0 ( δ ) , C 0 , C 1 and C 2 are defined in Theorem 2.3, and

Explicit upper bound Since g m is itself a non-standard function (like logarithm), it is not straightforward to obtain an explicit upper bound on (11). For g u , it was done in Lemma D.10 by using Lemma D.7.

Using Lemma D.7, we obtain that

where we used y = x c µ (2 sα 2 +1) , c 1 = log(2 sα 2 +1) + 1 sα 2 +1 / 2 , c µ = 2 H ( µ ) , c 0 = K/β +2 and define

For both equivalences above, we used that we are interested in larger values of x , hence we used that x -c 0 c µ ≥ 1 , x c µ (2 sα 2 +1) ≥ 1 , 2 sα 2 log( x ) + 2log(2 + α 2 log x ) + 2 ≥ 1 and

1 sα 2 +1 / 2 ( sα 2 log c µ + c 0 2 c µ +log(2 + α 2 log x ) ) + c 1 ≥ 1 . Since those conditions are implied by x ≥ h 2 ( c µ , x ) , those conditions are neither restrictive nor informative.

Therefore, we have shown that ˜ C µ defined in (11) satisfies

where h 2 ( z, x ) is defined in (12) is such that h 2 ( z, x ) ≈ x 2 z log(2 + α 2 log x ) and h 2 ( z, x ) ≈ z 2 sα 2 z log z .

D.4 Uniform sampling

While the shortcomings of uniform sampling are well known for general bandit instances, it is also clear that uniform sampling perform well for highly symmetric instances µ such that w ⋆ ( µ ) ≈ 1 K /K . Therefore, we derive a non-asymptotic upper bound for uniform sampling (Theorem D.6), which allow a clear comparison with Theorem D.4 (and Corollary D.5).

Theorem D.6. Let δ ∈ (0 , 1) . Combining the stopping rule (1) with threshold (2) and the uniform sampling rule yields a δ -correct algorithm such that, for all µ ∈ R K with | i ⋆ ( µ ) | = 1 ,

̸

where h 3 ( x ) = xW -1 (log( x )) for all x ≥ e and h 3 ( x ) = x for all x ∈ (0 , e ) and

̸

Proof. Let n > K such that E n holds true and the algorithm has not stop yet, i.e. E n ∩{ n < τ δ } . Let t := sup { t ∈ [ n 1 /α , n ] | ( t -1) /K ∈ N , ˆ ı t = i ⋆ } . Using the stopping rule (1) and t ≤ n < τ δ , we obtain

̸

̸

where k t = arg min i =ˆ ı t µ t, ˆ ı t -µ t,i √ 1 /N t, ˆ ı t +1 /N t,i . Since we sample uniformly, for all time t such that ( t -1) /K ∈ N , we have N t,i = ( t -1) /K for all i ∈ [ K ] . Combining the above with Lemma D.2, we obtain

̸

where the last inequality is obtained by taking the minimum over i ∈ [ K ] . To conclude the proof, we simply have to link t with n . More precisely, we will show that n -t ≤ K -1 and ˆ ı t = i ⋆ for n large enough. By concentration, we have

̸

Therefore, we have µ t,i ⋆ > max i = i ⋆ µ t,i , i.e. ˆ ı t = i ⋆ , for all t > N 3 where

̸

which means that we have at worse n -t ≤ K -1 . For n ≥ N 3 + K , we obtain t ≥ n -( K -1) > N 3 .

̸

Let c µ = 8 αK (1+ s ) min i = i ⋆ ( µ i ⋆ -µ i ) 2 and x = n/c µ . Assume that c µ ≥ e . Using the definition of g u and Lemma D.7, direct manipulations yields

̸

where the last inequality uses that log( c µ ) ≥ 1 and x ≥ 1 , which is not restrictive (or informative) since it is implied by x > W -1 (log( c µ )) . When c µ < e , which means that the problem is easy, we have directly that x -log( x ) > 1 > log( c µ ) for x > 1 . Therefore, the condition becomes x > 1 , i.e. n ≥ c µ . Defining h 3 ( x ) = xW -1 (log( x )) for all x ≥ e and h 3 ( x ) = x for all x ∈ (0 , e ) , we have

̸

Using a similar argument as the one in Appendix D.2 which rely on D.8 and the definition of T 1 ( δ ) , we can conclude the proof.

The structure of the non-asymptotic upper bound of uniform sampling in (13) is similar to the one for the TTUCB sampling rule in Theorem D.4. Therefore, we can compare the dominating terms for both the asymptotic and the non-asymptotic regime.

̸

First, we look at the asymptotic dominant term, namely we compare T 0 ( δ ) and T 1 ( δ ) . Taking w = 1 K /K instead of w ⋆ ( µ ) in the definition of T ⋆ ( µ ) , it is direct to see that T ⋆ ( µ ) ≤ 4 K min i = i ⋆ ( µ i ⋆ -µ i ) 2 .

̸

Using Lemma C.1, we have T ⋆ β ( µ ) ≤ T ⋆ ( µ ) max { β ⋆ β , 1 -β ⋆ 1 -β } where β ⋆ = w ⋆ i ⋆ ( µ ) . Therefore, while we can't say that T ⋆ β ( µ ) (1+ ε ) 2 β (1 -w 0 ) dµ ( w 0 ) ≤ 4 K min i = i ⋆ ( µ i ⋆ -µ i ) 2 for all instances µ , empirical evidences suggest that the gap is significant for reasonable instances. It is even possible to design instances where the gap between both notion of complexity explodes.

Second, we examine the dominating δ -independent term. Using Lemma D.7, we obtain by concavity that for all x > e

̸

̸

which yields that lim x → + ∞ h 1 ( x ) -h 3 ( x ) ≤ 2 . While those two functions diverges when x → + ∞ , their difference remains bounded by a finite quantity. Therefore, h 1 and h 3 are qualitatively similar. It is direct to see that 2 min i = i ⋆ ( µ i ⋆ -µ i ) 2 < H ( µ ) ≤ 2 K min i = i ⋆ ( µ i ⋆ -µ i ) 2 Therefore, while we can't say that αH ( µ ) ≤ 2 K min i = i ⋆ ( µ i ⋆ -µ i ) 2 for all instances µ , the gap can become significant for some instances.

̸

D.5 Technicalities

Lemma D.7 gathers properties on the function W -1 which we used in this work.

Lemma D.7 ([22]) . Let W -1 ( x ) = -W -1 ( -e -x ) for all x ≥ 1 , where W -1 is the negative branch of the Lambert W function. The function W -1 is increasing on (1 , + ∞ ) and strictly concave on (1 , + ∞ ) . In particular, W ′ -1 ( x ) = ( 1 -1 W -1 ( x ) ) -1 for all x > 1 . Then, for all y ≥ 1 and x ≥ 1 ,

Moreover, for all x > 1 ,

Lemma D.8 is a standard result to upper bound the expected sample complexity of an algorithm.

Lemma D.8. Let ( E n ) n>K be a sequence of events and T ( δ ) > K be such that for n ≥ T ( δ ) , E n ⊂ { τ δ ≤ n } . Then, E µ [ τ δ ] ≤ T ( δ ) + ∑ n>K P µ ( E ∁ n ) .

Proof.

Technical results Lemma D.9 shows that a linear lower bound on the number of samples allocated to the challenger, i.e. L n,i ⋆ -N i ⋆ n,i ⋆ ⪆ (1 -β )( n -1) .

Lemma D.9. Let w ∈ (0 , 1 -β ) and ( E 1 ,n ) n as in (17) . For all n ∈ N ⋆ , under E 1 ,n ,

Proof. Using Lemma D.3, we obtain that, under E 1 ,n ,

where the last inequality uses Lemma D.1 for t = n and w ∈ (0 , 1 -β ) .

Lemma D.10 gives an explicit upper bound on the constant C 0 ( w -) defined implicitly in Appendix D.2.

Lemma D.10. Let w -> 0 and C 0 ( w -) as in (8) . Then, we have

where h 1 : R ⋆ + → R ⋆ + is an increasing function for x ≥ 2 + K/β defined as

Proof. Using the definition of g u and Lemma D.7, direct manipulations yields

where x = n 1 /α /c µ , d µ = log( c µ ) + (2 + K/β ) /c µ and c µ = 4 α 2 (1 + s ) H ( µ ) /β . For the last equivalence we used that we are only interested in x ≥ 1 (small values are not relevant for upper bounds). Since W -1 ( d µ ) ≥ 1 , this condition is neither restrictive nor informative as we obtain the final condition x ≥ W -1 ( d µ ) . Moreover, we can show

where the last part is true since 2+ K/β ≥ 1 and min x ∈ R x (log( x ) -1) = -1 by direct computations. Therefore, for all n ≥ h 1 ( c µ ) α , we have

Given the definition of C 0 ( w -) as in (8), this concludes the proof.

Since x → W -1 ( x ) is a positive and increasing function, i.e. W -1 ( x ) > 0 and W ′ -1 ( x ) > 0 , a sufficient condition for h 1 to be increasing is to have f : x → log( x ) + 2+ K/β x increasing since

Lemma D.11 gives an explicit upper bound on the constant C 1 ( w -) defined implicitly in Appendix D.2.

Lemma D.11. Let w -> 0 and C 1 ( w -) as in (9) . Then, we have

where

Proof. Using manipulations conducted in Lemma D.10, we obtain that, for all n ≥ h 1 ( 4 α 2 (1 + s ) H ( µ ) /β ) α ,

Given the definition of C 1 ( w -) as in (9), this concludes the proof.

Lemma D.12 gives an asymptotic upper bound for times that are defined implicitly. For example, it can be used with T 0 ( δ ) and T 1 ( δ ) defined in Theorem D.4 and Theorem D.6.

Lemma D.12. Let C > 0 , D > 0 , c ( n, δ ) as in (2) and

Then, we have lim sup δ → 0 T ( δ,C ) log(1 /δ ) ≤ C .

Proof. Let γ > 0 . Direct manipulations yield that

where N γ = sup { n ∈ N ⋆ | ( √ n -1 - √ CD log n ) 2 -4 log ( 4 + log n 2 ) > (1 + γ ) n } . Therefore, we have

Using that C G ( x ) ≈ x +log( x ) (Lemma E.1), we obtain directly that, for all γ > 0 ,

which yields the result by letting γ go to zero.

Lemma D.13 gives a non-asymptotic upper bound for times that are defined implicitly. For example, when considering the idealized choice c ( n, δ ) = log(1 /δ ) , it can be used as a first order approximation of T 0 ( δ ) and T 1 ( δ ) defined in Theorem D.4 and Theorem D.6.

Lemma D.13. Let C > 0 , D > 0 and T ( δ ) := sup { n ∈ N ⋆ | n -1 ≤ C log(1 /δ ) + D log n } . Then, we have

for log(1 /δ ) ≥ D -D log( D ) -1 C , else T ( δ ) < D .

Proof. Direct manipulations yield that

where y = n/D and c D = 1 /D +log D . For the last equivalence, we used that C D log(1 /δ )+ c D ≥ 1 if and only if log(1 /δ ) ≥ D C (1 -c D ) = D -D log( D ) -1 C . When C D log(1 /δ ) + c D < 1 , which means that the δ parameter is large, we have directly that y -log y ≥ 1 > C D log(1 /δ ) + c D for y ≥ 1 . Therefore, the condition becomes n ≥ D .

E Concentration

The proof of Lemma 2.1 is given in Appendix E.1. In Appendix E.2, we show concentration results for the UCB leader (both with g u and g m ) and the TC challenger.

E.1 Proof of Lemma 2.1

Proving δ -correctness of a GLR stopping rule is done by leveraging concentration results. In particular, we build upon Theorem 9 of [27], which is restated below.

Lemma E.1. Consider Gaussian bandits with means µ ∈ R K and unit variance. Let S ⊆ [ K ] and x > 0 .

where C G is defined in [27] by C G ( x ) = min λ ∈ ]1 / 2 , 1] g G ( λ )+ x λ and

where ζ is the Riemann ζ function and C G ( x ) ≈ x +log( x ) .

Since ˆ ı n = i ⋆ ( µ n ) , standard manipulations yield that for all k =ˆ ı n

̸

Let i ⋆ = i ⋆ ( µ ) . Using the stopping rule (1) and the above manipulations, we obtain

̸

̸

where the second inequality is obtained with ( k, x, y ) = ( i ⋆ , µ i , µ i ⋆ ) , and the third by union bound. By concavity of x ↦→ log(4 + log( x )) and N n,i ⋆ + N n,i ≤ ∑ k ∈ [ K ] N n,k = n -1 , we obtain

̸

̸

̸

Combining the above with Lemma E.1 for all i = i ⋆ , we obtain P µ ( τ δ < + ∞ , ˆ ı = i ⋆ ) ≤ ∑ i = i ⋆ δ K -1 = δ .

E.2 Sampling rule

In Lemmas E.2 and E.3, we prove that the UCB leader using g u and g m is truly an upper confidence bounds for the unknown mean parameters, when a certain concentration event occurs. Then, when another concentration event occurs, we show a lower bound on the 'transportation' costs used by the TC challenger in Lemma E.5. Lemma E.6 upper bounds the probability of not being in the intersection of the two above sequence of concentration events.

UCB Leader Lemma E.2 proves that the bonus g u is sufficient to have upper confidence bounds on the unknown mean µ for Gaussian observations. The proof uses a simple union bound argument over the time.

Lemma E.2. Let α > 1 and s > 1 . For all n > K , let g u ( n ) = 2 α (1 + s ) log( n ) and

Then, for all n > K , P ( E ∁ 1 ,n ) ≤ Kn -s .

̸

̸

̸

Proof. Let ( X s ) s ∈ [ n ] be Gaussian observations from one distribution with unit variance. By union bound over [ K ] and using that n ≤ t α , we obtain

where we used that µ t,k -µ k = 1 N t,k ∑ t s =1 1 ( I s = k ) X s,k and concentration results for Gaussian observations.

Lemma E.3 proves that the bonus g m is sufficient to have upper confidence bounds on the unknown mean µ for Gaussian observations. The proof uses a more sophisticated argument based on mixture of martingales.

Lemma E.3. Let α > 1 and s > 1 . For all x ≥ 1 , let W -1 ( x ) = -W -1 ( -e -x ) where W -1 is the negative branch of the Lambert W function. For all n > K , let g m ( n ) = W -1 (2 sα log( n ) + 2 log(2 + α log n ) + 2) and

Then, for all n > K , P ( E ∁ 3 ,n ) ≤ Kn -s .

Proof. Let ( X s ) s ∈ [ t ] the observations from a standard normal distributions and denote S t = ∑ s ∈ [ t ] X s . To derived concentration result, we use peeling.

Let η > 0 and D = ⌈ log( n ) log(1+ η ) ⌉ . For all i ∈ [ D ] , let γ i > 0 and N i = (1 + η ) i -1 . For all i ∈ [ D ] , we define the family of priors f N i ,γ i ( x ) = √ γ i N i 2 π exp ( -x 2 γ i N i 2 ) with weights w i = 1 D and process

which satisfies M (0) = 1 . It is direct to see that M ( t ) = exp ( xS t -1 2 x 2 t ) is a non-negative martingale. By Tonelli's theorem, then M ( t ) is also a non-negative martingale of unit initial value.

Let i ∈ [ D ] and consider t ∈ [ N i , N i +1 ) . For all x ,

Direct computations shows that

Minoring M ( t ) by one of the positive term of its sum, we obtain

Using Ville's maximal inequality, we have that with probability greater than 1 -δ , log M ( t ) ≤ log (1 /δ ) . Therefore, with probability greater than 1 -δ , for all i ∈ [ D ] and t ∈ [ N i , N i +1 ) ,

Since this upper bound is independent of t , we can optimize it and choose γ i as in Lemma E.4 for all i ∈ [ D ] .

Lemma E.4 (Lemma A.3 in [11]) . For a, b ≥ 1 , the minimal value of f ( η ) = (1+ η )( a +log( b + 1 η )) is attained at η ⋆ such that f ( η ⋆ ) ≤ 1 -b + W -1 ( a + b ) . If b = 1 , then there is equality.

Therefore, with probability greater than 1 -δ , for all i ∈ [ D ] and t ∈ [ N i , N i +1 ) ,

The second inequality is obtained since D ≤ 1+ log n log(1+ η ) . The last equality is obtained for the choice η ⋆ = e 2 -1 , which minimizes η ↦→ log(1 + η ) -2 log(log(1 + η )) . Since [ n ] ⊆ ⋃ i ∈ [ D ] [ N i , N i +1 ) and N t,k ( µ t,k -µ k ) = ∑ s ∈ [ N t,k ] X s,k (unit-variance), this yields

Since 3 -2 log 2 ≤ 2 and W -1 is increasing, taking δ = n -s and restricting to m ∈ [ n 1 /α , n ] yields

Using n ≤ m α and doing a union bound over arms yield the result.

TCchallenger Lemma E.5 lower bounds the difference between the empirical gap and the unknown gap.

Lemma E.5. Let α > 1 and s > 1 . For all n > K , let

̸

Then, for all n > K , P ( E ∁ 2 ,n ) ≤ ( K -1) n -s .

Proof. Let ( X s ) s ∈ [ n ] and ( Y s ) s ∈ [ n ] be Gaussian observations from two distributions with unit variance. Then 1 m 1 ∑ m 1 i =1 X i -1 m 2 ∑ m 2 i =1 Y i is Gaussian with variance 1 m 1 + 1 m 2 . By union bound and

using that n ≤ t α , we obtain

̸

̸

̸

where we used that ( µ t,i ⋆ -µ i ⋆ ) -( µ t,k -µ k ) = 1 N t,i ⋆ ∑ t s =1 1 ( I s = i ⋆ ) X s,i ⋆ -1 N t,k ∑ t s =1 1 ( I s = k ) X s,k and concentration results for Gaussian observations.

Using a mixture of martingale arguments, we could improve on Lemma E.5 similarly as g u improved on g m . This will impact second order terms of our non-asymptotic theoretical guarantees, at the price of less explicit non-asymptotic terms.

Concentration event Lemma E.6 upper bounds the summed probabilities of the complementary events.

Lemma E.6. Let ζ be the Riemann ζ function. Let ( E 1 ,n ) n>K , ( E 2 ,n ) n>K and ( E 3 ,n ) n>K as in (17) , (19) and (18) . For all n > K , let E n = E 1 ,n ∩ E 2 ,n and ˜ E n = E 3 ,n ∩ E 2 ,n . Then,

Proof. Using Lemmas E.2 and E.5, direct union bound yields

The same proof trivially holds for ˜ E n by using Lemmas E.3 and E.5.

F Asymptotic analysis

Based solely on Theorem D.4, it is not possible to obtain asymptotic β -optimality due to the multiplicative factor 1 /β . Building on the unified analysis proposed in [21], we prove Theorem 2.3.

Our main technical contribution for this proof lies in the use of tracking instead of sampling. Given that the cumulative probability of being sampled is the expectation of the random empirical counts, it is not surprising that tracking-based Top Two algorithms enjoy the same theoretical guarantees as their sampling counterpart. As we will see, the analysis to obtain similar result is even simpler (Lemmas F.6, F.8 and F.10). Apart from this technical subtlety, the proof of Theorem 2.3 boils down to showing that the UCB leader satisfies the two sufficient properties highlighted by previous work (Lemmas F.4 and F.7).

Using the fact that x → √ 2 x is increasing, it is direct to see that the TC challenger (4) coincides with the definition used by [39, 21], i.e.

̸

̸

̸

̸

F.1 Proof of Theorem 2.3

Let µ ∈ D K such that min i = j | µ i -µ j | > 0 , and let i ⋆ = i ⋆ ( µ ) be the unique best arm. Let β ∈ (0 , 1) and w ⋆ β be the unique allocation β -optimal allocation satisfying w ⋆ β,i > 0 for all i ∈ [ K ] (Lemma C.2), i.e. w ⋆ β ( µ ) = { w ⋆ β } where

̸

̸

̸

Let ε > 0 . Following [36, 39, 21], we aim at upper bounding the expectation of the convergence time T ε β , which is a random variable quantifies the number of samples required for the empirical allocations N n / ( n -1) to be ε -close to w ⋆ β :

Lemma F.1 shows that a sufficient condition for asymptotic β -optimality is to show E µ [ T ε β ] < + ∞ for all ε small enough.

Lemma F.1. Let ( δ, β ) ∈ (0 , 1) 2 . Assume that there exists ε 1 ( µ ) > 0 such that for all ε ∈ (0 , ε 1 ( µ )] , E µ [ T ε β ] < + ∞ . Combining the stopping rule (1) with threshold as in (2) yields an algorithm such that, for all µ ∈ R K with | i ⋆ ( µ ) | = 1 ,

Proof. While the first result in the spirit of Lemma F.1 was derived by [36] for Gaussian distributions, a proof holding for more general distributions is given by Theorem 2 in [21]. The sole criterion on the stopping threshold is to be asymptotically tight (Definition F.2).

Definition F.2. A threshold c : N × (0 , 1] → R + is said to be asymptotically tight if there exists α ∈ [0 , 1) , δ 0 ∈ (0 , 1] , functions f, ¯ T : (0 , 1] → R + and C independent of δ satisfying: (1) for all δ ∈ (0 , δ 0 ] and n ≥ ¯ T ( δ ) , then c ( n, δ ) ≤ f ( δ ) + Cn α , (2) lim sup δ → 0 f ( δ ) / log(1 /δ ) ≤ 1 and (3) lim sup δ → 0 ¯ T ( δ ) / log(1 /δ ) = 0 .

Since C G defined in (16) satisfies C G ≈ x +log( x ) , it is direct to see that

is asymptotically tight, e.g. by taking ( α, δ 0 , C ) = (1 / 2 , 1 , 4) , f ( δ ) = 2 C G ( 1 2 log ( K -1 δ )) and ¯ T ( δ ) = 1 . This concludes the proof.

Throughout the proof, we will use a concentration result of the empirical mean (Lemma F.3) Since this is a standard result for Gaussian observations (see Lemma 5 of [36]), we omit the proof.

Lemma F.3. There exists a sub-Gaussian random variable W µ such that almost surely, for all i ∈ [ K ] and all n such that N n,i ≥ 1 ,

In particular, any random variable which is polynomial in W µ has a finite expectation.

Sufficient exploration To upper bound the expected convergence time, as prior work we first establish sufficient exploration. Given an arbitrary threshold L ∈ R ∗ + , we define the sampled enough set and its arms with highest mean (when not empty) as

̸

Since min i = j | µ i -µ j | > 0 , I ⋆ n is a singleton. We define the highly and the mildly under-sampled sets √

[21] identifies the properties that the leader and the challenger should satisfy to ensure sufficient exploration. Lemma F.4 show that the desired property for the UCB leader defined in (3) with bonus g u ( n ) = 2 α (1 + s ) log n or

̸

Lemma F.4. There exists L 0 with E µ [( L 0 ) α ] < + ∞ for all α > 0 such that if L ≥ L 0 , for all n (at most polynomial in L ) such that S L n = ∅ , B UCB n ∈ S L n implies B UCB n ∈ I ⋆ n and B UCB n ∈ arg max i ∈ S L n µ n,i .

̸

Proof. Let ε > 0 and ∆ min = min i = j | µ i -µ j | . Let g denote either g u or g m . Let L ≥ L 0 , where L 0 will be specified later, and n (at most polynomial in L ) such that S L n = ∅ . Then, there exists a polynomial function P such that n ≤ P ( L ) . By considering arms that are sampled more than L , we can show that for all k ∈ S L n \ I ⋆ n ,

̸

where the last inequality is obtained for L ≥ L 0 with

Since W -1 ( x ) ≈ x +log( x ) , both g u and g m have a logarithmic behavior, hence g ( P ( L )) = L → + ∞ o ( L ) . Since L 0 is polynomial in W µ , Lemma F.3 yields E µ [( L 0 ) α ] < + ∞ for all α > 0 . Moreover, for all k ∈ I ⋆ n ,

Assume that B UCB n ∈ S L n and that B UCB n / ∈ I ⋆ n . Since B UCB n = arg max k ∈ [ K ] { µ n,k + √ g ( n ) N n,k } , taking ε < ∆ min / 3 in the above yields a direct contradiction.

We can use the proof of [21] to obtain Lemma F.5. While their result accounts for the randomization of the sampling procedure, the argument is direct for tracking since it removes the need for a concentration argument.

̸

Lemma F.5 (Lemma 19 in [21]) . Let J ⋆ n = arg max i ∈ V L n µ i . There exists L 1 with E µ [ L 1 ] < + ∞ such that if L ≥ L 1 , for all n (at most polynomial in L ) such that U L n = ∅ , B UCB n / ∈ V L n implies C TC n ∈ V L n ∪ ( J ⋆ n \ { B UCB n }) .

Lemma F.6 proves sufficient exploration for the TTUCB sampling rule. It builds on the same reasoning than the one used in the proofs introduced by [39], and generalized by [21].

̸

Lemma F.6. Assume min j = i | µ i -µ j | > 0 . Under the TTUCB sampling rule, there exist N 0 with E µ [ N 0 ] < + ∞ such that for all n ≥ N 0 and all i ∈ [ K ] , N n,i ≥ √ n/K .

̸

Proof. Let L 0 and L 1 as in Lemmas F.4 and F.5. Therefore, for L ≥ L 2 := max { L 1 , L 4 / 3 0 } , for all n such that U L n = ∅ , B UCB n ∈ V L n or C TC n ∈ V L n since |J ⋆ n | = 1 is implied by min j = i | µ i -µ j | > 0 . We have E µ [ L 2 ] < + ∞ . There exists a deterministic L 4 such that for all L ≥ L 4 , ⌊ L ⌋ ≥ KL 3 / 4 . Let L ≥ max { L 2 , L 4 } .

̸

Suppose towards contradiction that U L ⌊ KL ⌋ is not empty. Then, for any 1 ≤ t ≤ ⌊ KL ⌋ , U L t and V L t are non empty as well. Using the pigeonhole principle, there exists some i ∈ [ K ] such that

N ⌊ L ⌋ ,i ≥ L 3 / 4 . Thus, we have ∣ ∣ ∣ V L ⌊ L ⌋ ∣ ∣ ∣ ≤ K -1 . Our goal is to show that ∣ ∣ ∣ V L ⌊ 2 L ⌋ ∣ ∣ ∣ ≤ K -2 . A sufficient condition is that one arm in V L ⌊ L ⌋ is pulled at least L 3 / 4 times between ⌊ L ⌋ and ⌊ 2 L ⌋ -1 .

Case 1. Suppose there exists i ∈ V L ⌊ L ⌋ such that L ⌊ 2 L ⌋ ,i -L ⌊ L ⌋ ,i ≥ L 3 / 4 β +3 / (2 β ) . Using Lemma D.3, we obtain

hence i is sampled L 3 / 4 times between ⌊ L ⌋ and ⌊ 2 L ⌋ -1 .

Case 2. Suppose that, for all i ∈ V L ⌊ L ⌋ , L ⌊ 2 L ⌋ ,i -L ⌊ L ⌋ ,i < L 3 / 4 β +3 / (2 β ) . Then,

Combining all the above, we obtain

where the last inequality is obtained for L ≥ L 5 with

The l.h.s. summation is exactly the number of times where an arm i ∈ V L ⌊ L ⌋ was leader but wasn't sampled, hence

For any ⌊ L ⌋ ≤ t ≤ ⌊ 2 L ⌋ -1 , U L t is non-empty, hence we have B UCB t ∈ V L ⌊ L ⌋ ⊆ V L t implies C TC t ∈ V L t ⊆ V L ⌊ L ⌋ . Therefore, we have shown that

Therefore, there is at least one arm in V L ⌊ L ⌋ that is sampled L 3 / 4 times between ⌊ L ⌋ and ⌊ 2 L ⌋ -1 .

In summary, we have shown ∣ ∣ ∣ V L ⌊ 2 L ⌋ ∣ ∣ ∣ ≤ K -2 . By induction, for any 1 ≤ k ≤ K , we have ∣ ∣ ∣ V L ⌊ kL ⌋ ∣ ∣ ∣ ≤ K -k , and finally U L ⌊ KL ⌋ = ∅ for all L ≥ L 3 . This concludes the proof.

Convergence towards β -optimal allocation Provided sufficient exploration holds (Lemma F.6), [21] identifies the properties that the leader and the challenger should satisfy to obtain convergence towards the β -optimal allocation w ⋆ β . Lemma F.7 show that the desired property for the UCB leader defined in (3) with bonus g u ( n ) = 2 α (1 + s ) log n or

Lemma F.7. There exists N 1 with E µ [ N 1 ] < + ∞ such that for all n ≥ N 1 , B UCB n = i ⋆ ( µ ) .

̸

Proof. Let ∆ = min i = i ⋆ | µ i ⋆ -µ i | . Let g denote either g u or g m . Let N 0 as in Lemma F.6. For all n ≥ N 2 (to be specified later), we obtain for all k = i ⋆ ,

̸

where the last inequality is obtained for n ≥ N 1 where

Since W -1 ( x ) ≈ x +log( x ) , both g u and g m have a logarithmic behavior, hence g ( n ) = n → + ∞ o ( √ n ) . Since N 1 is polynomial in W µ , Lemma F.3 yields E µ [( N 1 ) α ] < + ∞ for all α > 0 .

Moreover

Therefore, taking ε < ∆ / 3 , yields the result that B UCB n = i ⋆ .

Combining Lemma F.7 with tracking properties (Lemma D.3), we obtain convergence towards the β -optimal allocation for the best arm (Lemma F.8).

Lemma F.8. Let ε > 0 . Under the TTUCB sampling rule, there exists N 2 with E µ [ N 2 ] < + ∞ such that for all n ≥ N 2 ,

Proof. Let N 0 as in Lemma F.6 and C 1 as in Lemma F.7. For all n ≥ max { N 1 , N 0 } , we have B n = i ⋆ . Let M ≥ max { N 1 , N 0 } . Then, we have L n,i ⋆ ≥ n -M and ∑ k = i ⋆ N k n,i ⋆ ≤ M -1 for all n ≥ M +1 . Using Lemma D.3, we have

̸

̸

Taking N 2 = max { N 1 , N 0 , 1 / 2+2 β ( M -1) ε +1 }

We can use the proof of [21] to obtain Lemma F.9. While their result accounts for the randomization of the sampling procedure, the argument is direct for tracking since it removes the need for a concentration argument.

̸

Lemma F.9 (Lemma 20 in [21]) . Let ε > 0 . Under the TTUCB sampling rule, there exists N 3 with E µ [ N 3 ] < + ∞ such that for all n ≥ N 3 and all i = i ⋆ ( µ ) ,

̸

Combining Lemma F.9 with tracking properties (Lemma D.3), we obtain convergence towards the β -optimal allocation for all arms (Lemma F.10).

Lemma F.10. Let ε > 0 . Under the TTUCB sampling rule, there exists N 4 with E µ [ N 4 ] < + ∞ such that for all n ≥ N 4 ,

̸

Proof. Let N 0 , C 1 , N 2 and N 3 as in Lemmas F.6, F.7, F.8 and F.9. For all n ≥ max { N 0 , N 1 , N 2 , N 3 } , we have B UCB n = i ⋆ , ∣ ∣ ∣ N n,i ⋆ n -1 -β ∣ ∣ ∣ ≤ ε and for all i = i ⋆ ,

̸

Let M ≥ max { N 0 , N 1 , N 2 , N 3 } and n ≥ N 5 = max { M -1 ε + 1 , M } . Let t n -1 ,i ( ε ) = max { t ≤ n | N t,i n -1 ≤ w ⋆ β,i + ε } . Since N t,i n -1 ≤ N t,i t -1 for t ≤ n , we have

As a similar upper bound is shown in the proof of Lemma F.8, we obtain N n,i n -1 ≤ w ⋆ β,i +2 ε for all i ∈ [ K ] and all n ≥ N 4 := max { N 0 , N 1 , N 2 , N 3 , N 5 } . Since N n,i n -1 and w ⋆ β,i sum to 1 , we obtain for all n ≥ N 4 and all i ∈ [ K ] ,

̸

̸

Therefore, for all n ≥ N 4 and all i ∈ [ K ] , ∣ ∣ ∣ N n,i n -1 -w ⋆ β,i ∣ ∣ ∣ ≤ 2( K -1) ε . Since E µ [ N 4 ] < + ∞ and we showed the result for all ε , this concludes the proof.

Lemma F.11 shows that E µ [ T ε β ] < + ∞ , it is a direct consequence of Lemma F.10.

LemmaF.11. Let ε > 0 and T ε β as in (20) . Under the TTUCB sampling rule, we have E µ [ T ε β ] < + ∞ .

Theorem 2.3 is a direct consequence of Lemma F.1 and Lemma F.11.

G Implementation details and additional experiments

The implementations details and supplementary experiments are detailed in Appendix G.1 and Appendix G.2.

G.1 Implementation details

Top Two sampling rules Existing Top Two algorithms are based on a sampling procedure to choose between the leader B n and the challenger C n , namely sample B n with probability β , else sample C n . The difference lies in the choice of the leader and the challenger themselves.

TTTS [38] uses a TS (Thompson Sampling) leader and a RS (Re-Sampling) challenger based on a sampler Π n . For Gaussian bandits, the sampler Π n is the posterior distribution × i ∈ [ K ] N ( µ n,i , 1 /N n,i ) given the improper prior Π 1 = ( N (0 , + ∞ )) K . The TS leader is B TS n ∈ arg max i ∈ [ K ] θ i where θ ∼ Π n . The RS challenger samples vector of realizations θ ∈ Π n until B n / ∈ arg max i ∈ [ K ] θ i , then it is defined as C RS n arg max i ∈ [ K ] θ i for this specific vector of realization. When the posterior Π n and the leader B n have almost converged towards the Dirac distribution on µ and the best arm i ⋆ ( µ ) respectively, the event B n / ∈ arg max i ∈ [ K ] θ i becomes very rare. The experiments in [21] reveals that computing the RS challenger can require more than millions of re-sampling steps. Therefore, the RS challenger can become computationally intractable even for Gaussian distribution where sampling from Π n can be done more efficiently.

T3C [39] combines the TS leader and the TC challenger. β -EB-TCI [21] combines the EB leader with the TCI challenger defined as

̸

Since the TC and TCI challenger can re-use computations from the stopping rule, those two challengers have no additional computational cost which makes it very attractive for larger sets of arms.

Each tracking procedure has a computational and memory cost in O (1) , hence total cost of O ( K ) for the K independent procedures. Each UCB index has a computational and memory cost in O (1) , hence total cost of O ( K ) to compute B UCB n as in (3). Each TC index (or stopping rule index) has a computational and memory cost in O (1) , hence total cost of O ( K ) to compute C TC n as in (4). Therefore, the per-round computational and memory cost of TTUCB is in O ( K ) .

Other sampling rules At each time n , Track-and-Stop (TaS) [17] computes the optimal allocation for the current empirical mean, w n = w ⋆ ( µ n ) . Given w n ∈ △ K , it uses a tracking procedure to obtain an arm I n to sample. On top of this tracking a forced exploration is used to enforce convergence towards the optimal allocation for the true unknown parameters. The optimization problem defining w ⋆ ( µ ) can be rewritten as solving an equation ψ µ ( r ) = 0 , where

̸

̸

̸

The function ψ µ is decreasing, and satisfies lim r → + ∞ ψ µ ( r ) = -1 and lim y → 1 / min i = i ⋆ ( µ i ⋆ -µ i ) 2 F µ ( y ) = + ∞ . For the practical implementation of the optimal allocation, we use the approach of [17] and perform binary searches to compute the unique solution of ψ µ ( r ) = 0 . A faster implementation based on Newton's iterates was proposed by [6] after proving that ψ µ is convex. While this improvement holds only for Gaussian distributions, the binary searches can be used for more general distributions.

DKM[12] view T ⋆ ( µ ) -1 as a min-max game between the learner and the nature, and design saddlepoint algorithms to solve it sequentially. At each time n , a learner outputs an allocation w n , which is used by the nature to compute the worst alternative mean parameter λ n . Then, the learner is updated based on optimistic gains based on λ n .

FWS [42] alternates between forced exploration and Frank-Wolfe (FW) updates.

LUCB [24] samples and stops based on upper/lower confidence indices for a bonus function g . For Gaussian distributions, it rewrites for all i ∈ [ K ] as

̸

̸

At each time n , it samples ˆ ı n and arg max i =ˆ ı n U n,i and stops when L n, ˆ ı n ≥ max i =ˆ ı n U n,i . The β -LUCB algorithm samples ˆ ı n with probability β , else it samples arg max i =ˆ ı n U n,i . The stopping time is the same as for LUCB.

̸

Adaptive proportions [44] propose IDS, whcih is an update mechanism for β based on the optimality conditions for the problem underlying T ⋆ ( µ ) . For Gaussian with known homoscedastic variance, IDS can be written as

where the second equality is obtained by direct computations which uses that

Reproducibility Our code is implemented in ❏✉❧✐❛ ✶✳✼✳✷ , and the plots are generated with the ❙/a116❛/a116/a115/a80❧♦/a116/a115✳❥❧ package. Other dependencies are listed in the ❘❡❛❞♠❡✳♠❞ . The ❘❡❛❞♠❡✳♠❞ file also provides detailed julia instructions to reproduce our experiments, as well as a /a115❝/a114✐♣/a116✳/a115❤ to run them all at once. The general structure of the code (and some functions) is taken from the tidnabbil library. 1

G.2 Supplementary experiments

Running time The CPU running time corresponding to the experiment displayed in Figure 1(a) are reported in Table 4. They match our discussion on computational cost detailed in Appendix G.1. The slowest algorithm is TaS, followed closely by FWS and TTTS. All remaining algorithms have similar computational cost: TTUCB, β -EB-TCI, T3C, LUCB and uniform sampling. It is slightly higher for DKM.

We emphasize that this is a coarse empirical comparison of the CPU running time in order to grasp the different orders of magnitude. More efficient implementation could (and should) be used by practitioners. As an example, the computational cost of TaS can be improved for Gaussian distributions by using the algorithm from [6] based on Newton's iterates. However, we doubt that the faster implementation of TaS will match the computational cost of DKM or FWS.

̸

Empirical errors before stopping At the exception of LUCB, all the considered algorithms are anytime algorithms (see [23] for a definition) since they are not using δ in their sampling rule. While all those algorithms are δ -correct, none enjoy theoretical guarantees on the probability of error before stopping, i.e. upper bounds on P µ (ˆ ı n = i ⋆ ( µ )) for n < τ δ . In Figure 3, we display their averaged empirical errors at time n < τ δ ( i.e. 1 (ˆ ı n = i ⋆ ) ) corresponding to the experiment displayed in Figure 1(a), with their associated Wilson Score Intervals [43]. To avoid an unfair comparison between algorithms having different stopping time, we restrict our plots to the median of the observed empirical stopping time. Therefore, even the fastest algorithm will average its empirical errors on at least 2500 instances.

̸

Based on Figure 3, we see that uniform sampling and DKM perform the worst in terms of empirical error. At all times, the smallest empirical errors are achieved by β -EB-TCI, TTTS and LUCB. While TTUCB is at first as bad as FWS, its empirical error tends to match the one of the best algorithms for larger time. For TaS and T3C, the trend is reversed.

Tracking versus sampling The TTUCB sampling rule uses tracking instead of sampling. Since both approaches aim at doing the same with either a deterministic or a randomized approach, it is interesting to assess whether they lead to different empirical performance. Therefore, we compare both approaches for four Top Two sampling rules. Figure 4 reveals that the algorithmic choice of tracking or sampling has a negligible impact on the empirical stopping time.

In light of this experiment, the choice of tracking or sampling is mostly a question of theoretical analysis. Since the analysis of a randomized sampling requires to control of the randomness of the allocation, we choose a deterministic tracking for analytical simplicity. Moreover, this choice is natural when both the leader and challenger are deterministic.

Larger sets of arms In addition to the results presented in Section 4, we also evaluate the performance of our algorithm on the two other benchmarks used in [20]. The ' α = 0 . 3 ' scenarios consider µ i = 1 -( i -1 K -1 ) α for all i ∈ [ K ] , with hardness H ( µ ) ≈ 3 K . The ' α = 0 . 6 ' scenarios consider µ i = 1 -( i -1 K -1 ) α for all i ∈ [ K ] , with hardness H ( µ ) ≈ 12 K 1 . 2 . The observations from Figure 5 are consistent with the ones in Figure 1(b). Overall, T3C performs the best for larger sets of arms.

G.2.1 Adaptive proportions

The ratio T ⋆ 1 / 2 ( µ ) /T ⋆ ( µ ) seems to reach its highest value r K = 2 K/ (1 + √ K -1) 2 for 'equal means' instances (Lemma C.6), i.e. µ i = µ i ⋆ -∆ for all i = i ⋆ with ∆ > 0 . To best observe differences between Top Two algorithms with β = 1 / 2 and their adaptive version, we consider such instances with K = 35 ( r K ≈ 3 / 2 ). Figure 6 reveals that adaptive proportions yield better empirical performance, with an empirical speed-up close to r K ≈ 3 / 2 . We also compare them with three asymptotically optimal BAI algorithms (TaS, FWS and DKM). Even on those hard instances, Top

̸

Two algorithms with fixed β = 1 / 2 are outperforming the asymptotically optimal algorithms for all the values δ ∈ { 0 . 1 , 0 . 01 , 0 . 001 } . While being only 1 / 2 asymptotically optimal, Top Two algorithms (with fixed β = 1 / 2 ) can obtain significantly better empirical performances compared to existing asymptotically optimal in the finite-confidence regime. The gap between the empirical performance of the adaptive and the fixed β Top Two algorithms is increasing with δ decreasing. Interestingly, TTUCB appears to be slightly more robust to decreasing confidence δ compared to other Top Two algorithms.

Wealso compare Top Two algorithms using a fixed proportion β = 1 / 2 with their adaptive counterpart using β n = N n,C n / ( N n,B n + N n,C n ) at time n on random instances.

̸

Experiments are conducted on 5000 random Gaussian instances with K = 10 such that µ 1 = 0 . 6 and µ i ∼ U ([0 . 2 , 0 . 5]) for all i = 1 . Figure 7 shows that their performances are highly similar, with a slim advantage for adaptive algorithms. For instances with multiple close competitors, the same phenomenon appears (see Figure 8 below). This is expected as T ⋆ 1 / 2 ( µ ) /T ⋆ ( µ ) is close to one when sub-optimal arms have significantly distinct means (see Figure 2).

̸

Then, we assess the performance on 5000 random Gaussian instances with K = 10 such that µ 1 = 0 . 6 and µ i = µ 1 -∆ i for i = 1 where ∆ i = 1 20 ( 995 1000 + u i 100 ) for all i ∈ { 2 , 3 , 4 , 5 , 6 } and ∆ i = 1 10 ( 995 1000 + u i 100 ) for all i ∈ { 7 , 8 , 9 , 10 } with u i ∼ U ([0 , 1]) . Numerically, we observe w ⋆ ( µ ) i ⋆ ≈ 0 . 28276 ± 0 . 0003 (mean ± std). Both plots in Figure 8 show the same phenomena than the ones observed in Figures 1(b) and 7.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors proposed a new UCB-based Top-Two algorithm (TTUCB) for the best-arm identification problem (BAI) and derived a non-asymptotic upper bound on the expected sample complexity holding for any instance having a unique best arm. The authors also demonstrated that the TTUCB algorithm achieves competitive empirical performance compared to other algorithms, including some existing Top-Two methods.\n\nSTRENGTHS:\nThere have been some existing works that proposed top-two algorithms and analyzed the expected sample complexity for vanishing error level $\\delta$, i.e., $\\delta \\to 0$ (asymptotic upper bound). However, this work proposed a new top-two algorithm (TTUCB) and provided a non-asymptotic upper bound on the sample complexity, i.e., a bound that holds for any error level $\\delta$. The main (and new) contribution of this paper is to provide analytical techniques to obtain a non-asymptotic bound for the expected sample complexity for the TTUCB.\n\nWEAKNESSES:\nHowever, this paper contains quite many weaknesses as follows.\n\n+  The tightness of this bound is not verified in this paper, especially for non-vanishing $\\delta$. \n+  The obtained non-asymptotic upper bound in Theorem 2.4. does not match the lower bound (Lemma 1.1) as $\\delta \\to 0$, which shows that the non-asymptotic upper bound is not tight enough at small $\\delta$.  Note that Theorem 2.3 shows that TTUCB is $\\beta$-optimal as $\\delta \\to 0$. \n+  In the moderate regime $(\\delta=0.1)$, the experiment (see Fig. 1) shows that the TTUCB algorithm can be better or worse than other existing ones. For example, in Fig. 1a, TTUCB looks worse than T3C although it is better than the others.  Hence, it seems to me that although other existing algorithms are developed for $\\delta \\to 0$, they can be better than your algorithm (i.e., TTUCB) as moderate $\\delta$.\n\nQUESTIONS:\n+ Why don't you compare the performance of TTUCB with the non-asymptotic upper bound in Theorem 2.4 to check whether this bound is tight or not at different values of $\\delta$?\n+  From my second comment in the Originality section, what is the motivation for developing TTUCB for non-vanishing $\\delta$? Why don't we perform non-asymptotic analysis for the existing top-two algorithms? Did you mean that you would like to develop a new top-two algorithm (more specifically, TTUCB) such that the non-asymptotic analysis for the upper bound can be performed easier than the others?\n+ What is the main challenge when changing an asymptotic upper bound to a non-asymptotic one? More specifically, why is Theorem 2.3 optimal, but Theorem 2.4 not optimal as $\\delta \\to 0$?\n\nLIMITATIONS:\nThis is a theoretical work. The authors already finished the checklist as well as stated the limitations of theorems and results.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

comments: SUMMARY:
This paper presents a non-asymptotic upper bound on the expected sample complexity of a TT sampling rule-based algorithm for any error level. Numerical results also verify the performance of TTUCB. This is nice work

STRENGTHS:
1. The paper is well-written and organized. The insights behind the main results are well-presented.
2. The tracking rule is simple and easy to implement.
3. I appreciate that the authors include several simulations.

WEAKNESSES:
One weakness is that there is still room for further improvement in the bound.

QUESTIONS:
1. Any comments on how to choose the function $g_u(n)$? Are there any required properties?
2. Can you provide any suggestions on how to choose $\beta$ in practice?"

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main objective of the paper is to establish the first non-asymptotic bound on the expected sample complexity for a Top two sampling policy in the context of best arm identification. To achieve this, the paper introduces a novel algorithm called TTUCB, which is proven to be $\\beta$-optimal in asymptotic scenarios and also satisfies a non-asymptotic upper bound. This upper bound is compared with the existing upper bounds of established non-Top two algorithms. The paper supports its claims with synthetic simulation experiments.\n\nOne notable contribution of the paper is the development of a generic framework that enables regret minimization algorithms to effectively tackle the best arm identification problem while providing optimal guarantees. Additionally, the paper offers guidelines on how other regret minimization algorithms can be adapted to choose a leader at each round. Overall, the paper provides significant insights into the problem of best-arm identification and presents a promising algorithmic solution along with empirical evidence to support its efficacy.\n\nSTRENGTHS:\nThe paper demonstrates exceptional writing, with clear claims and well-defined objectives. It successfully fills a significant gap in the literature by addressing the challenge of providing non-asymptotic guarantees for Top-two algorithms, resulting in original and valuable contributions to the broader bandit literature.\n\nThe problem formulation is thoroughly justified, establishing a solid foundation for the subsequent theoretical results. The discussions on the theoretical guarantees of the proposed TTUCB algorithm are insightful and comprehensive, effectively identifying areas for potential improvement. Additionally, the paper highlights the low computational complexity of the approach, enhancing its practical applicability.\n\nThe upper bounds achieved by the proposed algorithm are comparable to those of well-established methods in the field, as discussed aptly within the paper. By providing relevant comparisons, the paper strengthens the credibility and applicability of its results.\n\nFurthermore, the paper successfully contextualizes its work within the broader scope of the bandit literature, illustrating the connections between the theoretical analysis and the use of regret-minimizing algorithms in the best arm identification problem. This contextualization is particularly valuable as it addresses a scenario that differs from the traditional exploitation-exploration trade-off associated with regret minimization.\n\nOverall, the paper stands out for its clarity, strong theoretical discussions, well-justified approach, and relevant connections, making it a valuable contribution to the field of bandit problems.\n\nWEAKNESSES:\nOne potential weakness is the tightness of the upper bound provided in the paper. The proposed upper bound incorporates various parameters and constants. While there is a comparison with the upper bounds of other algorithms, it remains uncertain whether this upper bound represents the best achievable result for TTUCB or Top-two algorithms in general. Further exploration into the tightness of the upper bound would contribute to a more comprehensive understanding of the algorithm's performance.\n\nAnother concern is the exclusive use of synthetic data in the experimental setup, without considering real-world datasets. It would be beneficial to address the potential challenges and limitations of applying the proposed approach to real-world datasets. Discussing any potential issues related to data collection, noise, or other factors specific to real-world settings would enhance the paper's practical applicability and broaden its impact.\n\nQUESTIONS:\n1) Could you provide more details on the challenges encountered within the proof technique that led to the decision of adopting a tracking approach instead of randomization in TTUCB? It would be helpful to understand the specific difficulties that arose and how they influenced this decision.\n\n2) Additionally, has there been any simulation-based evidence to demonstrate the comparative performance of TTUCB using tracking versus a randomization-based approach? It would be valuable to assess the effectiveness of TTUCB in comparison to algorithms utilizing random\n\nLIMITATIONS:\nThe authors talk about the limitations and places of potential improvement of the work. Societal impact is not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the problem of best-arm identification in the unstructured multi-armed bandit problem. More precisely, they study the problem in the fixed-budget setting in the moderate confidence regime. They propose a top two algorithm based on UCB which they call TTUCB and establish that the algorithm is asymptotically $\\beta$-optimal. Here, by $\\beta$-optimal, the authors mean that the algorithm samples sub-optimal arms optimally under the constraint that the best arm is sampled a proportion of times that is fixed at $\\beta$. More importantly, they also provide a non-asymptotic guarantee for their algorithm that is valide for all confidence levels. The authors complement their theoretical findings with empirical results.\n\nSTRENGTHS:\n- Overall, the paper is well written and easy to follow. The related work is well discussed and the proofs appear to be sound to the best of my knowledge.\n- The non-asymptotic study of best arm identification in the fixed confidence setting is a fascinating problem and remains largely unexplored. The authors provide a non-trivial contribution to this problem by shedding some  insight into the non-asymptotic behaviour of a UCB-based TOP TWO algorithm. \n-  The proposed algorithm TTUCB seems to enjoy the best non-asymptotic guarantee up to universal constants universal constants.\n\nWEAKNESSES:\n- The theoretical results are limited to the case of Gaussian arms with known variances. Moreover, The authors only provide a algorithm that is asymptotically $\\beta$-optimal but not asymptotically optimal. This is somewhat unsatisfactory!\n- The authors address the non-asymptotic behaviour of best-arm identification but do not provide any non-asymptotic lower bound. As such it is unclear to me what is the best non-asymptotic bound. In fact, when the authors compare the different bounds, I believe they should also clarify (in the table) that TTUCB has an additional linear dependence on the number of arms $K$ at the finite confidence behaviour. \n-  The authors could provide experiments which exhibit more the finite confidence behaviour of their algorithm in contrast with other algorithms. For instance, experiments that show how the sample complexity varies as $\\delta$ varies.\n\nQUESTIONS:\n- In Lemma 3.1, the definition of $D_0$ is not given which, according to Appendix D.2, is equal to $(1+\\varepsilon^2)/\\beta^2$. My first comment is that this needs to be precised in the statement of Lemma 3.1. Secondly, if you were to adaptively seek to achieve the optimal allocation with $\\beta^\\star$, then $D_0$ is not anymore a universal constant and might depend in a non-trivial way on the problem instance $\\mu$, in which case, your proof might not work to obtain a non-asymptotic guarantee. Would this be a limitation of your proof techniques when trying to attain the optimal constant $T_\\star$? I am not sure using IDS might work either. Can you comment on this? \n-  The result of Lemma 3.1, appear to be very specific to the Gaussian case, can the author comment on how can this generalize to the general case?\n-  The authors mention that $ T_{1/2}^\\star(\\mu)\\le  2T^\\star(\\mu)$ in the worst case, can you comment on how tight is this bound? Also according to your experiment it seems TTUCB outperform asymptotically optimal algorithms like track-and-stop and FWS. Does this mean that in the experiment you considered,  $T_{1/2}^\\star(\\mu)$ and $T^\\star(\\mu)$ are roughly the same? Can you run your experiments on a more challenging example where $T_{1/2}^\\star(\\mu)\\approx 2 T^\\star(\\mu)$?\n-  In the experiments section, the authors present plots in a scenario where $\\delta=0.1$. Why is this value of $\\delta$ considered to be in the moderate regime? Is there a clear definition if what we refer to as moderate regime? shouldn't this depend on  the number of arms $K$?\n\nLIMITATIONS:\nI believe the authors have discussed the limitations of their work accordingly.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}