Title: Best Arm Identification with Fixed Budget: A Large Deviation Perspective
Venue: NeurIPS 2023
Accepted: yes
Keywords: Best arm identification

Large deviation
Abstract: We consider the problem of identifying the best arm in stochastic Multi-Armed Bandits (MABs) using a fixed sampling budget. Characterizing the minimal instance-specific error probability for this problem constitutes one of the important remaining open problems in MABs. When arms are selected using a static sampling strategy, the error probability decays exponentially with the number of samples at a rate that can be explicitly derived via Large Deviation techniques. Analyzing the performance of algorithms with adaptive sampling strategies is however much more challenging. In this paper, we establish a connection between the Large Deviation Principle (LDP) satisfied by the empirical proportions of arm draws and that satisfied by the empirical arm rewards. This connection holds for any adaptive algorithm, and is leveraged (i) to improve error probability upper bounds of some existing algorithms, such as the celebrated SR (Successive Rejects) algorithm \cite{audibert2010best}, and (ii) to devise and analyze new algorithms. In particular, we present CR (Continuous Rejects), a truly adaptive algorithm that can reject arms in {\it any} round based on the observed empirical gaps between the rewards of various arms. Applying our Large Deviation results, we prove that CR enjoys better performance guarantees than existing algorithms, including SR. Extensive numerical experiments confirm this observation.

Best Arm Identification with Fixed Budget: A Large Deviation Perspective

Abstract

We consider the problem of identifying the best arm in stochastic Multi-Armed Bandits (MABs) using a fixed sampling budget. Characterizing the minimal instance-specific error probability for this problem constitutes one of the important remaining open problems in MABs. When arms are selected using a static sampling strategy, the error probability decays exponentially with the number of samples at a rate that can be explicitly derived via Large Deviation techniques. Analyzing the performance of algorithms with adaptive sampling strategies is however much more challenging. In this paper, we establish a connection between the Large Deviation Principle (LDP) satisfied by the empirical proportions of arm draws and that satisfied by the empirical arm rewards. This connection holds for any adaptive algorithm, and is leveraged (i) to improve error probability upper bounds of some existing algorithms, such as the celebrated SR (Successive Rejects) algorithm (Audibert et al., 2010), and (ii) to devise and analyze new algorithms. In particular, we present CR (Continuous Rejects), a truly adaptive algorithm that can reject arms in any round based on the observed empirical gaps between the rewards of various arms. Applying our Large Deviation results, we prove that CR enjoys better performance guarantees than existing algorithms, including SR . Extensive numerical experiments confirm this observation.

1 Introduction

Westudy the problem of best-arm identification in stochastic bandits in the fixed budget setting. In this problem, abbreviated by BAI-FB, a learner faces K distributions or arms ν 1 , . . . , ν K characterized by their unknown means µ = ( µ 1 , . . . , µ K ) (we restrict our attention to distributions taken from a one-parameter exponential family). She sequentially pulls arms and observes samples of the corresponding distributions. More precisely, in round t ≥ 1 , she pulls an arm A t = k selected depending on previous observations and observes X k ( t ) a sample of a ν k -distributed random variable. ( X k ( t ) , t ≥ 1 , k ∈ [ K ]) are assumed to be independent over rounds and arms. After T arm draws, the learner returns ˆ ı , an estimate of the best arm 1( µ ) := arg max k µ k . We assume that the best arm is unique, and denote by Λ the set of parameters µ such that this assumption holds. The objective is to devise an adaptive sampling algorithm minimizing the error probability P µ [ˆ ı = 1( µ )] . This learning task is one of the most important problems in stochastic bandits, and despite recent research efforts, it remains largely open (Qin, 2022). In particular, researchers have so far failed

̸

at characterizing the minimal instance-specific error probability. This contrasts with other basic learning tasks in stochastic bandits such as regret minimization (Lai and Robbins, 1985) and BAI with fixed confidence (Garivier and Kaufmann, 2016), for which indeed, asymptotic instance-specific performance limits and matching algorithms have been derived. In BAI-FB, the error probability typically decreases exponentially with the sample budget T , i.e., it scales as exp( -R ( µ ) T ) where the instance-specific rate R ( µ ) depends on the sampling algorithm. Maximizing this rate over the set of adaptive algorithms is an open problem.

Instance-specific error probability lower bound. To guess the maximal rate at which the error probability decays, one may apply the same strategy as that used in regret minimization or BAI in the fixed confidence setting: (i) derive instance-specific lower bound for the error probability for some notion of uniformly good algorithms; (ii) devise a sampling strategy mimicking the optimal proportions of arm draws identified in the lower bound. Here the notion of uniformly good algorithms is that of consistent algorithms. Under such an algorithm, for any µ ∈ Λ , P µ [ˆ ı = 1( µ )] → 1 as T →∞ . (Garivier and Kaufmann, 2016) conjectures the following asymptotic lower bound satisfied by any consistent algorithm (refer to Appendix J for details): as T →∞ ,

̸

where Σ is the ( K -1) -dimensional simplex, Ψ( λ , ω ) = ∑ K k =1 ω k d ( λ k , µ k ) , Alt ( µ ) = { λ ∈ Λ : 1( µ ) = 1( λ ) } is the set of confusing parameters (those for which 1( µ ) is not the best arm), and d ( x, y ) denotes the KL divergence between two distributions of parameters x and y . Interestingly, the solution ω ⋆ ∈ Σ of the optimization problem max ω ∈ Σ inf λ ∈ Alt ( µ ) Ψ( λ , ω ) provides the best static proportions of arm draws. More precisely, an algorithm selecting arms according to the allocation ω ⋆ , i.e., selecting arm k ω ⋆ k T times and returning the best empirical arm after T samples, has an error rate matching the lower bound (1). This is a direct consequence of the fact that, under a static algorithm with allocation ω , the empirical reward process { ˆ µ ( t ) } t ≥ 1 satisfies a LDP with rate function λ ↦→ Ψ( λ , ω ) , see (Glynn and Juneja, 2004) and refer to Section 3 for more details.

̸

Adaptive sampling algorithms and their analysis. The optimal allocation ω ⋆ depends on the instance µ and is initially unknown. We may devise an adaptive sampling algorithm that (i) estimates ω ⋆ and (ii) tracks this estimated optimal allocation. In the BAI with fixed confidence, such tracking scheme exhibits asymptotically optimal performance (Garivier and Kaufmann, 2016). Here however, the error made estimating ω ⋆ would inevitably impact the overall error probability of the algorithm. To quantify this impact or more generally to analyze the performance of adaptive algorithms, one would need to understand the connection between the statistical properties of the arm selection process and the asymptotic statistics of the estimated expected rewards.

̸

To be more specific, any adaptive algorithm generates a stochastic process { Z ( t ) } t ≥ 1 = { ( ω ( t ) , ˆ µ ( t )) } t ≥ 1 . ω ( t ) = ( ω 1 ( t ) , . . . , ω K ( t )) represents the allocation realized by the algorithm up to round t ( ω k ( t ) = N k ( t ) /t and N k ( t ) denotes the number of times arm k has been selected up to round t ). ˆ µ ( t ) = (ˆ µ 1 ( t ) , . . . , ˆ µ K ( t )) denotes the empirical average rewards of the various arms up to round t . Now assuming that at the end of round T , the algorithm returns the arm with the highest empirical reward, the error probability is P µ [ˆ ı = 1( µ )] = P µ [ ˆ µ ( T ) ∈ Alt ( µ )] . Assessing the error probability at least asymptotically requires understanding the asymptotic behavior of ˆ µ ( t ) as t grows large. Ideally, one would wish to establish the Large Deviation properties of the process { Z ( t ) } t ≥ 1 . This task is easy for algorithms using static allocations (Glynn and Juneja, 2004), but becomes challenging and open for adaptive algorithms. Addressing this challenge is the main objective of this paper.

Contributions. In this paper, we develop and leverage tools towards the analysis of adaptive sampling algorithms for the BAI-FB problem. More precisely, our contributions are as follows.

(a) We establish a connection between the LDP satisfied by the empirical proportions of arm draws { ω ( t ) } t ≥ 1 and that satisfied by the empirical arm rewards. This connection holds for any adaptive algorithm. Specifically, we show that if the rate function of { ω ( t ) } t ≥ 1 is lower bounded by ω ↦→ I ( ω ) , then that of (ˆ µ ( t )) t ≥ 1 is also lower bounded by λ ↦→ min ω ∈ Σ max { Ψ( λ , ω ) , I ( ω ) } . This result has interesting interpretations and implies the following asymptotic upper bound on the error probability of the algorithm considered: as T →∞ ,

̸

The above formula, when compared to the lower bound (1), quantifies the price of not knowing ω ⋆ initially, and relates the error probability to the asymptotic statistics of the sampling process used by the algorithm.

(b) We show that by simply applying our generic Large Deviation result, we may improve the error probability upper bounds of some existing algorithms, such as the celebrated SR algorithm (Audibert et al., 2010). Our result further opens up opportunities to devise and analyze new algorithms with a higher level of adaptiveness. In particular, we present CR (Continuous Rejects), an algorithm that, unlike SR , can eliminate arms in each round. This sequential elimination process is performed by comparing the empirical rewards of the various candidate arms using continuously updated thresholds. Leveraging the LDP tools developed in (a), we establish that CR enjoys better performance guarantees than SR . Hence CR becomes the algorithm with the lowest instance-specific and guaranteed error probability. We illustrate our results via numerical experiments, and compare CR to other BAI algorithms.

2 Related Work

̸

We distinguish two main classes of algorithms to solve the best arm identification problem in the fixed budget setting. Algorithms from the first class, e.g. Successive Rejects ( SR ) (Audibert et al., 2010) and Sequential Halving ( SH ) (Karnin et al., 2013), split the sampling budget into phases of fixed durations, and discard arms at the end of each phase. Algorithms from the second class, e.g. UCB-E (Audibert et al., 2010) and UGapE (Gabillon et al., 2012) sequentially sample arms based on confidence bounds of their empirical rewards. It is worth mentioning that algorithms from the second class usually require some prior knowledge about the problem, for example, an upper bound of H = ∑ k =1( µ ) 1 ( µ 1( µ ) -µ k ) 2 . Without this knowledge, the parameters can be chosen in a heuristic way, but the performance gets worse.

̸

Algorithms from the first class exhibit better performance numerically and are also those with the best instance-specific error probability guarantees. SR had actually the best performance guarantees so far: for example, when the reward distributions are supported on [0 , 1] , the error probability of SR satisfies: lim inf T →∞ 1 T log 1 P µ [ˆ ı =1( µ )] ≥ 1 H 2 log K , where H 2 = max k =1( µ ) k ( µ 1( µ ) -µ k ) 2 . In this paper, we strictly improve this guarantee (see Section 3.4). Recently, (Barrier et al., 2022) also refined and extended the analysis of (Audibert et al., 2010) by replacing, in the analysis, Hoeffding's inequality by a large deviation result involving KL-divergences. Again here, we further improve this new guarantee. We also devise CR , an algorithm with error probability provably lower than our improved guarantees for SR .

̸

Fundamental limits on the error probability have also been investigated. In the minimax setting, (Carpentier and Locatelli, 2016) established that for any algorithm, there exists a problem within the class of instances with given complexity H such that the error probability is greater than exp( -400 T H log K ) ≥ exp( -400 T H 2 log K ) . Up to a universal constant (here 400), SR is hence minimax optimal. This lower bound was also recently revisited in (Ariu et al., 2021; Degenne, 2023; Wang et al., 2023) to prove that the instance-specific lower bound (1) cannot be achieved on all instances by a single algorithm . Deriving tight instance-specific lower bounds remains open (Qin, 2022).

We conclude this section by mentioning two interesting algorithms. In (Komiyama et al., 2022), the authors propose DOT , an algorithm trying to match minimax error probability lower bounds. To this aim, the algorithm requires to periodically call an oracle able to determine an optimal allocation, solution of an optimization problem with high and unknown complexity. DOT has minimax guarantees but is computationally challenging if not infeasible (numerically, the authors cannot go beyond simple instances with 3 arms). Finally, researchers have also looked at the best arm identification problem from a Bayesian perspective. For example, (Russo, 2016) devise variants of the celebrated Thompson Sampling algorithm, that could potentially work well in practice. Nevertheless, as discussed in (Komiyama, 2022), Bayesian algorithms cannot be analyzed nor provably perform well in the frequentist setting.

3 Large Deviation Analysis of Adaptive Sampling Algorithms

In this section, we first recall key concepts in Large Deviations (refer to the classical textbooks (Budhiraja and Dupuis, 2019; Dembo and Zeitouni, 2009; Dupuis and Ellis, 2011; Varadhan, 2016) for a more detailed exposition). We then apply these concepts to the performance analysis of adaptive sampling algorithms. Finally, we exemplify the analysis and apply it to improve existing performance guarantees for the SR algorithm (Audibert et al., 2010).

3.1 Large Deviation Principles

Consider the stochastic process { Y ( t ) } t ≥ 1 with values in a separable complete metric space (i.e., a Polish space) Y . Large Deviations are concerned with the probabilities of rare events related to { Y ( t ) } t ≥ 1 that decay exponentially in the parameter t . The asymptotic decay rate is characterized by the rate function I : Y → R + defined so that essentially -1 t log P [ Y ( t ) ∈ B ] converges to min x ∈ B I ( x ) for any Borel set B . We provide a more rigorous definition below.

Definition 1. [Large Deviation Principle (LDP)] The stochastic process { Y ( t ) } t ≥ 1 satisfies a LDP with rate function I if:

(i) I is lower semicontinuous, and ∀ s ∈ [0 , ∞ ] , the set K s = { y ∈ Y : I ( y ) ≤ s } is compact; (ii) for every closed (resp. open) set C ⊂ Y (resp. O ⊂ Y ),

LDPs have been derived earlier in stochastic bandit literature. (Glynn and Juneja, 2004) have used Gärtner-Ellis Theorem (Ellis, 1984; Gärtner, 1977) to establish that under a static sampling algorithm with allocation ω ∈ Σ (i.e., each arm k is selected ω k T times up to round T ), the process { ˆ µ ( T ) } T ≥ 1 satisfies an LDP with rate function λ ↦→ Ψ( λ , ω ) = ∑ K k =1 ω k d ( λ k , µ k ) . Our objective in the next subsection is to investigate how to extend this result to the case of adaptive sampling algorithms.

3.2 Analysis of adaptive sampling algorithms

̸

An adaptive sampling algorithm generates a stochastic process { Z ( t ) } t ≥ 1 = { ( ω ( t ) , ˆ µ ( t )) } t ≥ 1 . When the sampling budget T is exhausted, should the algorithm returns the arm with the highest empirical reward, the error probability is P µ [ˆ ı = 1( µ )] = P µ [ ˆ µ ( T ) ∈ Alt ( µ )] . To assess the rate at which this probability decays with the budget, we may try to establish a LDP for the empirical reward process { ˆ µ ( t ) } t ≥ 1 . Due to the intricate dependence between the sampling and the empirical reward processes, deriving such an LDP is very challenging. Instead, we establish a connection between the LDPs satisfied by these processes. This connection will be enough for us to derive tight upper bounds on the error probability. We present our main result in the following theorem.

Theorem 1. Assume that under some adaptive sampling algorithm, { ω ( t ) } t ≥ 1 satisfies the LDP upper bound (3) with rate function I . Then { ˆ µ ( t ) } t ≥ 1 satisfies the LDP upper bound (3) with rate function λ ↦→ min ω ∈ Σ max { Ψ( λ , ω ) , I ( ω ) } . Moreover, we have: for any bounded Borel subset S of R K and any Borel subset W of Σ ,

where F S ( ω ) := inf λ ∈ cl ( S ) Ψ( λ , ω ) , and cl ( S ) denotes the closure of S .

Before proving the above theorem, we make the following remarks and provide a simple corollary that will lead to improved upper bound on the error probability of the SR algorithm.

(a) Not a complete LDP. To upper bound the error probability of a given algorithm, we do not actually need to establish that { ˆ µ ( t ) } t ≥ 1 satisfies a complete LDP. Instead, deriving a LDP upper bound is enough. Theorem 1 provides such an upper bound, but does not yield a complete LDP. We conjecture if { ω ( t ) } t ≥ 1 satisfies an LDP with rate function I , { ˆ µ ( t ) } t ≥ 1 satisfies an LDP with rate function λ ↦→ inf ω ∈ W max { Ψ( λ , ω ) , I ( ω ) } . The conjecture holds for static sampling algorithms as shown

below. If it holds for adaptive algorithms, we show, in Appendix I, that when K > 3 , no algorithm can attain the instance-specific lower bound (1) for all parameters.

(b) Theorem 1 is tight for static sampling algorithms. When the sampling rule is static, namely ω ( t ) = ω ∈ Σ , then { ω ( t ) } t ≥ 1 satisfies a LDP with rate function I defined as I ( ω ) = 0 and ∞ elsewhere. Theorem 1 with W = Σ states that { ˆ µ ( t ) } t ≥ 1 satisfies the LDP upper bound (3) with rate function F S . In fact, as shown by (Glynn and Juneja, 2004), { ˆ µ ( t ) } t ≥ 1 satisfies a complete LDP with this rate function.

(c) A useful corollary. From Theorem 1, we have:

From there, we will be able to improve the performance guarantee for SR .

3.3 Proof of Theorem 1

Proof. Observe that when S or W is empty, the result holds. Now recall that F S ( · ) = inf λ ∈ cl ( S ) Ψ( λ , · ) is the infimum of a family of linear functions on a compact set, Σ , hence it is upper bounded. Denote u > 0 such an upper bound. F S ( · ) is also continuous (see Appendix F for details). For each integer N ∈ N , we define a collection of closed sets:

We observe that:

Taking the logarithm on both sides and dividing them by -t yields that

where the last inequality follows from Lemma 1. Since for all n ∈ [ N ] ,

the r.h.s. of (6) is equal to

where the first inequality is due to (5). As N can be taken arbitrarily large, we conclude this theorem.

Lemma 1. For any N ∈ N , n ∈ [ N ]

Proof. Recall F S ( · ) = inf λ ∈ cl ( S ) Ψ( λ , · ) . We deduce that

where X denotes t Ψ(ˆ µ ( t ) , ω ( t )) for short. Let α ∈ (0 , 1) , Markov's inequality implies that

✶ where the last inequality uses the definition of W N n (see (5)). By applying Hölder's inequality with p, q , where p ∈ [1 , 1 /α ) and q = p/ ( p -1) on r.h.s. of (7), we deduce that log P µ [ ˆ µ ( t ) ∈ S , ω ( t ) ∈ W N n ] is at most

✶ As αp ∈ (0 , 1) , Lemma 2 in Appendix B shows that the first term above is o ( t ) . Using definition of the rate function, (3) with C = W N n , on the second term yields that lim t →∞ 1 t log 1 P µ [ ˆ µ ( t ) ∈S , ω ( t ) ∈ W n N ] is lower bounded by

As p can be arbitrarily close to 1 /α , we get the lower bound (1 -α ) I ( ω ) + αu ( n -1) N . Further choosing α close to either 1 or 0 , the proof is completed.

3.4 Improved analysis of the Successive Rejects algorithm

In SR , the set of candidate arms is initialized as C K = [ K ] . The budget of samples is partitioned into K -1 phases, and at the end of each phase, SR discards the empirical worst arm from the candidate set. In each phase, SR uniformly samples the arms in candidate set. The lengths of phases are set as follows. Define log K := 1 2 + ∑ K k =2 1 k . The candidate set is denoted by C j when it has j > 2 arms. In the corresponding phase, (i) each arm in C j is sampled until the round t when min k ∈C j N k ( t ) reaches T/ ( j log K ) (recall that N k ( t ) is the number of times arm k has been sampled up to round t ); (ii) the empirical worst arm, denoted by ℓ j , is then discarded, i.e., C j -1 = C j \ { ℓ j } . During the last phase, the algorithm equally samples the two remaining arms and finally recommends ˆ ı , the arm with higher empirical mean in C 2 . The pseudo code is presented in Algorithm 1.

Algorithm 1: SR

initialization C K ← [ K ] , j ← K ;

for

(

do

, . . . , T

)

t

if

= 1

(

j >

and

2

ℓ

j

end sample

end

ℓ 2 ← arg min k ∈C 2 ˆ µ k ( T ) and return ˆ ı ← arg max k ∈C 2 ˆ µ k ( T ) (tie broken arbitrarily).

We apply the corollary (c) in Section 3.2 to improve the existing performance guarantees of SR . To simplify the presentation, we assume wlog that µ 1 > µ 2 ≥ . . . ≥ µ K . For j = 2 , . . . , K , define

where J = { J ⊆ [ K ] : | J | = j, 1 ∈ J } .

A

min

k

argmin

∈C

j

k

j

∈C

t

N

k

ˆ

µ

argmin

(

k

(

k

∈C

t

)

t

T

j

log

K

then

)

)

j

(tie broken arbitrarily),

N

j

1

-

C

k

(

t

)

←C

j

(tie broken arbitrarily), update

\ {

ℓ

{

N

j

k

}

, and

(

t

j

)

}

k

∈C

j

j

and

-

1

ˆ

(

µ

;

t

)

;

̸

Theorem 2. Let µ ∈ Λ . Under SR , we have: for j = 2 , . . . , K , lim T →∞ 1 T log 1 P µ [ ℓ j =1] ≥ Γ j j log K . Hence, the error probability of SR is upper bounded by lim T →∞ 1 T log 1 P µ [ˆ ı =1] ≥ min j =1 Γ j j log K .

Proof of Theorem 2. Fix j ∈ { 2 , . . . , K } . Observe that

which implies that

as |J | < ∞ .

Since ℓ j is selected at the θT -th round 1 where θ = (1 + ∑ K k = j +1 1 k ) / log K , the event { ℓ j = 1 , C j = J } implies that { ˆ µ ( θT ) ∈ S , ω ( θT ) ∈ W } , where

In other words, P µ [ ℓ j = 1 , C j = J ] ≤ P µ [ ˆ µ ( θT ) ∈ S , ω ( θT ) ∈ W ] . Applying (c) in Section 3.2 with the above S and W yields that lim T →∞ 1 θT log 1 P µ [ ˆ µ ( θT ) ∈S , ω ( θT ) ∈ W ] is larger than

where the first inequality uses the fact that KL-divergences and the components of ω are nonnegative, and the second one is due to the definition (8) of Γ j . Combining (9) and (10) completes the proof. □

The upper bound derived in Theorem 2 is tighter than those recently derived in (Barrier et al., 2022). Indeed, for any J ∈ J , since | J | = j , one can find at least one index in J at least larger than j , say k J . Hence,

The r.h.s. in the previous inequality corresponds to the upper bounds derived by (Barrier et al., 2022).

To simplify the presentation and avoid rather intricate computations involving the KL-divergences, in the remaining of the paper, we restrict our attention to specific classes of reward distributions.

̸

Assumption 1. The rewards are bounded with values in (0 , 1) . The reward distributions ν 1 , . . . , ν K are Bernoulli distributions such that ν a is of mean a , and for any a = b , d ( a, b ) ≥ 2( a -b ) 2 (this is a consequence of Pinsker's inequality as rewards are in (0 , 1) ).

Under Assumption 1, we have Γ j ≥ 2 ξ j (a direct consequence of Proposition 3 in Appendix D.1), where for j = 2 , . . . , K ,

̸

We give an explicit expression of ξ j in Proposition 1, presented in Appendix D.1. Moreover, 2 ξ j is clearly larger than 2 inf λ 1 ≤ λ j { ( λ 1 -µ 1 ) 2 +( λ j -µ j ) 2 } = ( µ 1 -µ j ) 2 , and hence min j =1 Γ j / ( j log K ) ≥ min j =1 ( µ 1 -µ j ) 2 / ( j log K ) . This implies that our error probability upper bound is better than that derived in (Audibert et al., 2010).

̸

Example 1. To illustrate the improvement brought by Theorem 2 on the performance guarantees of SR , consider the simple example with 3 Bernoulli arms and µ = (0 . 9 , 0 . 1 , 0 . 1) . Then min j =1 ( µ 1 -µ j ) 2 / ( j log3) = 0 . 16 for the upper bound presented in (Audibert et al., 2010). From Proposition 1, instead we get min j =1 2 ξ j / ( j log3) = 0 . 21 .

̸

̸

̸

4 Continuous Rejects Algorithms

In this section, we present CR , a truly adaptive algorithm that can discard an arm in any round. We propose two variants of the algorithm, CR-C using a conservative criterion to discard arms and CR-A discarding arms more aggressively. Using the Large Deviation results of Theorem 1, we establish error probability upper bounds for both CR-C and CR-A .

4.1 The CR-C and CR-A algorithms

As SR , CR initializes its candidate set as C K = [ K ] . For j ≥ 2 , C j denotes the candidate set when it is reduced to j arms. When j > 2 , the algorithm samples arms in the candidate set C j uniformly until a discarding condition is met. The algorithm then discards the empirically worst arm ℓ j ∈ C j , i.e., C j -1 ← C j \ { ℓ j } . More precisely, in round t , if there are j candidate arms remaining and if ℓ ( t ) denotes the empirically worst candidate arm, the discarding condition is N ℓ ( t ) ( t ) > max k/ ∈C j N k ( t ) , ( ∀ k ∈ C j , N ℓ ( t ) ( t ) = N k ( t )) 2 , and

̸

̸

where G ( β ) = 1 / √ β -1 for all β > 0 . The idea behind (11) is to keep the probability of discarding the best arm at most smaller than that of SR while using less budget. Note that (12) is easier to achieve than (11). CR-A is hence more aggressive than CR-C , and reduces the set of arms to C 2 faster, but at the expense of a higher risk. After discarding ℓ 3 , CR will sample the arms in C 2 evenly, and recommend the empirical best arm in C 2 when the budget is exhausted. The pseudo-code of CR is presented in Algorithm 2.

Algorithm 2: CR-C and CR-A

Input: θ 0 ∈ (0 , ) ∩ Q independent of T

1 log K (can be chosen as small as one wishes)

initialization

C K ← [ K ] , j ← K , sample each arm k ∈ [ K ] once, update { N k ( t ) } k ∈C K and ˆ µ ( t ) ; for t = K +1 , . . . , ⌊ θ 0 T ⌋ do sample A t ← argmin k ∈C j N k ( t ) (tie broken arbitrarily), update { N k ( t ) } k ∈C j and ˆ µ ( t ) ;

end for

(

t

=

⌊

θ

0

T

⌋

+1

, . . . , T

)

do

ℓ ( t ) ← argmin k ∈C j ˆ µ k ( t ) (tie broken arbitrarily);

sample A t ← argmin k ∈C j N k ( t ) (tie broken arbitrarily), update { N k ( t ) } k ∈C j and ˆ µ ( t ) ; end

ℓ 2 ← ℓ ( T ) ; return ˆ ı ← argmax k ∈C 2 ˆ µ k ( T ) (tie broken arbitrarily).

4.2 Analysis of CR-C and CR-A

As in Section 3.4, µ 1 > µ 2 ≥ . . . ≥ µ K is assumed wlog and we further define µ K +1 = 0 . We introduce the following instance-specific quantities needed to state our error probability upper bounds. For j ∈ { 2 , . . . , K } , define

Here we remark ¯ ξ j ≥ ξ j and ¯ ψ j ≥ ψ j . These inequalities are proven in Proposition 3 and Proposition 5 in Appendix D.

̸

Theorem 3. Let µ ∈ [0 , 1] K . Under CR-C , lim T →∞ 1 T log 1 P µ [ˆ ı =1] is larger than

̸

where α j ∈ R is the real number such that 2 ξ j (1 -α j ) j log j = [((1 + ζ j ) √ α j -√ 1 ( j +1)log( j +1) ) + ] 2 .

̸

Theorem 4. Let µ ∈ [0 , 1] K . Under CR-A , lim T →∞ 1 T log 1 P µ [ˆ ı =1] is larger than

̸

where α j ∈ R is the real number such that ψ j (1 -α j ) j log j = j j +1 [((1 + φ j ) √ α j -√ 1 ( j +1)log( j +1) ) + ] 2 .

Note that Theorem 3 implies that CR-C enjoys better performance guarantees than SR , and hence has for now the best known error probability upper bounds.

Proof sketch. The complete proof of Theorems 3 and 4 are given in Appendices C.1 and C.2. We sketch that of Theorem 3. The proof consists in upper bounding P µ [ ℓ j = 1] for j ∈ { 2 , . . . , K } . We focus here on the most challenging case where j ∈ { 3 , . . . , K -1 } (the analysis is simpler when j = K , since the only possible allocation is uniform, and when j = 2 , since the only possible round deciding ℓ 2 is the last round).

To upper bound P µ [ ℓ j = 1] using Theorem 1, we will show that it is enough to study the large deviations of the process { ω ( θT ) } T ≥ 1 for any fixed θ ∈ [ θ 0 , 1] and to define a set S ⊆ [0 , 1] K under which ℓ j = ℓ ( θT ) = 1 . We first observe that ℓ j = ℓ ( θT ) restricts the possible values of ω ( θT ) : ω ( θT ) ∈ X j := { x ∈ Σ : ∃ σ ∈ [ K ] 2 s.t. x σ (1) = . . . = x σ ( j ) > x σ ( j +1) > . . . > x σ ( K ) > 0 } . We can hence just derive the LDP satisfied by { ω ( θT ) } T ≥ 1 on X j . This is done in Appendix E, and we identify by I θ a rate function leading to an LDP upper bound. By defining

As it is shown in Appendix E.1 that I θ ( x ) = ∞ if x ∈ X j,i ( θ ) for i ≥ j , we may further restrict to X j \ ∪ K i = j X j,i ( θ ) .

Next, we explain how to apply Theorem 1 to upper bound P µ [ ℓ j = 1] . Let J = { J ⊆ [ K ] : | J | = j, 1 ∈ J } as defined in Section 3.4. For all β, θ ∈ (0 , 1] and J ∈ J , we introduce the sets

̸

Assume that in round t , ℓ j = ℓ ( t ) = 1 , C j = J and let τ = ∑ k/ ∈ J N k ( t ) ≤ t be the number of times arms outside J are pulled. While ω ( t ) / ∈ X j,j ( t/T ) , we have β = ( t -τ )log j T -τ ∈ (0 , 1] . Using the criteria (11), we observe that

To upper bound the r.h.s. in the above inequality, we combine the results of Theorem 1 and a partitioning technique (presented in Appendix G). This gives:

The proof is completed by providing lower bounds of θF S J ( β ) ( z ) and θI θ ( z ) for a fixed z ∈ Z J ( θ, β ) with various J . Such bounds are derived in Appendix D.1 and E.3.1, respectively. □

Example 2. To conclude this section, we just illustrate through a simple example the gain in terms of performance guarantees brought by CR compared to SR . Assume we have 50 Bernoulli arms with µ 1 = 0 . 95 , µ 2 = 0 . 85 , µ 3 = 0 . 2 , and µ k = 0 for k = 4 , . . . , 50 . For SR , Theorem 2 states that with a budget of 5000 samples, the error probability of SR does not exceed 1 . 93 × 10 -3 . With the same budget, Theorems 3 and 4 state that the error probabilities of CR-C and CR-A do not exceed 6 . 40 × 10 -4 and 6 . 36 × 10 -4 , respectively.

We note that in general, we cannot say that one of our two algorithms, CR-C or CR-A , has better guarantees than the other. This is demonstrated in the problem instances presented in Appendix K.1 and K.4.

5 Numerical Experiments

We consider various problem instances to numerically evaluate the performance of CR . In these instances, we vary the number of arms from 5 to 55; we use Bernoulli distributed rewards, and vary the shape of the arm-to-reward mapping. For each instance, we compare CR to SR , SH , and UGapE .

Most of our numerical experiments are presented in Appendix K. Due to space constraints, we just provide an example of these results below. In this example, we have 55 arms with convex arm-to-reward mapping. The mapping has 10 steps, and the m -th step consists of m arms with same average reward, equal to 3 4 · 3 -m 10 . Table 1 presents the error probabilities averaged over 40 , 000 independent runs. Observe that CR-A performs better than CR-C (being aggressive when discarding arms has some benefits), and both versions of CR perform better than SR and all other algorithms.

6 Conclusion

In this paper, we have established, in MAB problems, a connection between the LDP satisfied by the sampling process (under any adaptive algorithm) and that satisfied by the empirical average rewards of the various arms. This connection has allowed us to improve the performance analysis of existing best arm identification algorithms, and to devise and analyze new algorithms with an increased level of adaptiveness. We show that one of these algorithms CR-C has better performance guarantees than existing algorithms and that it performs also better in practice in most cases.

Future research directions include: (i) developing algorithms with further improved performance guarantees - can the discarding conditions of CR be further optimized? (ii) Enhancing the Large Deviation analysis of adaptive algorithms - under which conditions, can we establish a complete LDP of the process { Z ( t ) } t ≥ 1 = { ( ω ( t ) , ˆ µ ( t )) } t ≥ 1 ? Answering this question would constitute a strong step towards characterizing the minimal instance-specific error probability for best arm identification with fixed budget. (iii) Extending our approach to other pure exploration tasks: topm arm identification problems (Bubeck et al., 2013), best arm identification in structured bandits (Yang and Tan, 2022; Azizi et al., 2022), or best policy identification in reinforcement learning.

References

Ariu, K., Kato, M., Komiyama, J., McAlinn, K., and Qin, C. (2021). Policy choice and best arm identification: Asymptotic analysis of exploration sampling.

Audibert, J.-Y., Bubeck, S., and Munos, R. (2010). Best arm identification in multi-armed bandits. In COLT , pages 41-53. Citeseer.

Azizi, M., Kveton, B., and Ghavamzadeh, M. (2022). Fixed-budget best-arm identification in structured bandits. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22 , pages 2798-2804.

Barrier, A., Garivier, A., and Stoltz, G. (2022). On best-arm identification with a fixed budget in non-parametric multi-armed bandits. arXiv preprint arXiv:2210.00895 .

Berge, C. (1997). Topological Spaces: including a treatment of multi-valued functions, vector spaces, and convexity . Courier Corporation.

Boyd, S., Boyd, S. P., and Vandenberghe, L. (2004). Convex optimization . Cambridge university press.

Bubeck, S., Wang, T., and Viswanathan, N. (2013). Multiple identifications in multi-armed bandits. In International Conference on Machine Learning , pages 258-265. PMLR.

Budhiraja, A. and Dupuis, P. (2019). Analysis and approximation of rare events. Representations and Weak Convergence Methods. Series Prob. Theory and Stoch. Modelling , 94.

Carpentier, A. and Locatelli, A. (2016). Tight (lower) bounds for the fixed budget best arm identification bandit problem. In Conference on Learning Theory , pages 590-604. PMLR.

Combes, R., Magureanu, S., and Proutiere, A. (2017). Minimal exploration in structured stochastic bandits. In Proc. of NeurIPS .

Degenne, R. (2023). On the existence of a complexity in fixed budget bandit identification. In Proc. of COLT .

Degenne, R. and Koolen, W. M. (2019). Pure exploration with multiple correct answers. In Proc. of NeurIPS .

Dembo, A. and Zeitouni, O. (2009). Large deviations techniques and applications , volume 38. Springer Science & Business Media.

Dupuis, P. and Ellis, R. S. (2011). A weak convergence approach to the theory of large deviations . John Wiley & Sons.

Ellis, R. S. (1984). Large deviations for a general class of random vectors. The Annals of Probability , 12(1):1-12.

Gabillon, V., Ghavamzadeh, M., and Lazaric, A. (2012). Best arm identification: A unified approach to fixed budget and fixed confidence. Advances in Neural Information Processing Systems , 25.

Garivier, A. and Kaufmann, E. (2016). Optimal best arm identification with fixed confidence. In Proc. of COLT .

Garivier, A., Ménard, P., and Stoltz, G. (2019). Explore first, exploit next: The true shape of regret in bandit problems. Mathematics of Operations Research , 44(2):377-399.

Gärtner, J. (1977). On large deviations from the invariant measure. Theory of Probability & Its Applications , 22(1):24-39.

Glynn, P. and Juneja, S. (2004). A large deviations perspective on ordinal optimization. In Proceedings of the 2004 Winter Simulation Conference, 2004. , volume 1. IEEE.

Karnin, Z., Koren, T., and Somekh, O. (2013). Almost optimal exploration in multi-armed bandits. In International Conference on Machine Learning , pages 1238-1246. PMLR.

Kaufmann, E., Cappé, O., and Garivier, A. (2016). On the complexity of best-arm identification in multi-armed bandit models. JMLR .

Kaufmann, E. and Koolen, W. (2018). Mixture martingales revisited with applications to sequential tests and confidence intervals. arXiv preprint arXiv:1811.11419 .

Komiyama, J. (2022). Bayes optimal algorithm is suboptimal in frequentist best arm identification. arXiv preprint arXiv:2202.05193 .

Komiyama, J., Tsuchiya, T., and Honda, J. (2022). Minimax optimal algorithms for fixed-budget best arm identification. In Advances in Neural Information Processing Systems .

Lai, T. L. and Robbins, H. (1985). Asymptotically efficient adaptive allocation rules. Advances in applied mathematics .

Magureanu, S., Combes, R., and Proutiere, A. (2014). Lipschitz bandits: Regret lower bound and optimal algorithms. In Conference on Learning Theory , pages 975-999. PMLR.

Qin, C. (2022). Open problem: Optimal best arm identification with fixed-budget. In Conference on Learning Theory , pages 5650-5654. PMLR.

Russo, D. (2016). Simple bayesian algorithms for best arm identification. In Annual Conference on Learning Theory . PMLR.

Varadhan, S. S. (2016). Large deviations , volume 27. American Mathematical Soc.

Wang, P.-A., Ariu, K., and Proutiere, A. (2023). On uniformly optimal algorithms for best arm identification in two-armed bandits with fixed budget. arXiv preprint arXiv:2308.12000 .

Wang, P.-A., Tzeng, R.-C., and Proutiere, A. (2021). Fast pure exploration via frank-wolfe. Advances in Neural Information Processing Systems , 34.

Yang, J. and Tan, V. (2022). Minimax optimal fixed-budget best arm identification in linear bandits. Advances in Neural Information Processing Systems , 35:12253-12266.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the problem of best arm identification with fixed budget in the $K$-arm bandit problem. Using tools from the large deviation literature, they propose a new methodology to analyse algorithms for this problem. They apply this method to a state of the art algorithm and obtain sharper asymptotical bounds on the error probability of this algorithm. They propose two new algorithms and establish bounds on their error probability, showing that one of them is asymptotically better that existing algorithms. Finally, they illustrate the good performances of their algorithms by extensive numerical simulations.\n\nSTRENGTHS:\nClarity : This article is very well written and enjoyable to read. The literature review is complete and well presented. Methods and results are clearly introduced and discussed, and the sketches of proofs are insightful. The level of detail in the evidence greatly helps the reader, and the authors' efforts to make this article self-contained are commendable.\n\nQuality : This article introduces important new methodological tools and theoretical results, that are supported by rigorous proofs.\n\nOriginality : The authors study a central problem in the bandit literature from a new perspective, leveraging tools from large deviation literature to obtain sharper results and develop new algorithms.\n\nSignificance : The error bounds in this paper improve over existing results on a central question, and the methodology used to obtain them could be applied to study other related problems.\n\nWEAKNESSES:\nWhy the authors discuss the motivations behind the two algorithms they introduce, they do not compare their theoretical guarantees.\n\nQUESTIONS:\nCould you please discuss more the theoretical guarantees of the two algorithms? Is there settings where one would perform better than the other, and vice-versa?\n\nCould you perhaps add an introduction to your Appendix, to present its content and the different aspects of the problem that have been postponed there?\n\nLIMITATIONS:\nThe authors adequately address the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-23"}

comments: SUMMARY:
This paper studies the problem of best arm identification (BAI) with fixed budget (FB), with bandit feedback on K arms. The asymptotically optimal sample complexity for this problem has been of significant interest recently and has proved surprisingly difficult to understand compared to the fixed confidence problem. 

This paper gives a new upper bound on the failure probability of a given algorithm, and uses it to study specific algorithms: successive rejects and continuous rejects.

STRENGTHS:
The paper achieves new large deviation bounds for the failure probability of specific algorithms using a new LDP bound, and shows their superior empirical performance at minimizing failure probability. The analysis of the successive rejects algorithm is nice and improves over previous results. The required analyses for the continuous rejections (in the appendix) seem to be quite involved, but show even better performance guarantees.

The writing is pretty good too.

WEAKNESSES:
The result seem a little incremental as they most concern specific algorithms; the paper does not seem like it will help much toward an eventual solution to  the general FB-BAI problem. But getting better non-sharp bounds is still nice (and a general solution might be abstract enough not to subsume them).

QUESTIONS:
I didn't see the relevant Polish spaces specified anywhere.

I suggest using i_* instead of \hat i, since it is hard to see the latter.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

comments: SUMMARY:
This paper considers the problem of best arm identification with fixed budget, where the learner pulls an arm in each round for $T$ rounds, then outputs a candidate for the arm with the largest mean. The objective is to minimize the probability of mis-identification. Characterizing the instance specific complexity for this problem is an open question. The authors present a more refined analysis based on large deviation principles resulting on a refined upper bound for the successive rejects (SR) algorithm. Further, the authors present two algorithms based on the previous analysis where arms elimination is done following conditions rather than at pre-specified rounds as in SR.

STRENGTHS:
The paper is well-organized, and the discussions are interesting. Using LDP for best arm identification appears to be novel.

WEAKNESSES:
The presented bounds are not easy to read (especially in Section 4). It would be nicer to provide a more detailed discussion about the comparisons with bounds in the literature.

Some theorem statements (Proposition and Theorem 1) don't clearly state the assumptions made.

QUESTIONS:
Minor comment/question: line 147, about returning the arm with the highest empirical reward: can't we have a scenario in SR algorithm where the at the end (when the budget is exhausted), the empirical mean of the wining arm is smaller than the empirical mean of the first eliminated arm (which was computed only during the first epoch) ?  If true, the latter statement is not valid.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
This paper contributes an important tool geared towards closing the performance gap for best arm identification problems under the fixed budget setting, a well-known open problem in the area. In particular, the result is a large deviation bound on the sample means of the arm rewards as a function of any large deviation bound on the empirical means of the number of times each of the arms is pulled under a fixed policy (which could very well be adaptive). Since for popular policies such as successive rejects, the latter bound could be easier to specify due to their structure (batched pulls, etc.), the result can be used to derive tighter upper bounds on the probabilities of misidentification under such policies. The paper illustrates this power by deriving a tighter upper bound on the probability of misidentification for the successive rejects (SR) policy and derives a new policy called continuous rejects (CR) that achieves a smaller probability of error.

STRENGTHS:
-  While it doesn't seem surprising that such a result would hold, I think this is a solid result that I anticipate being of importance not only for the best arm-identification problem but as a general tool in the bandit analysis toolkit.
- The paper is very well written.

WEAKNESSES:
- I don't think there are any major weaknesses except stating that the considered problem is the "last fundamental open problem in MABs" is quite presumptuous.

QUESTIONS:
Hasn't it been shown, as stated in the open problem paper by Chao Qin, that the conjecture in Equation (1) is false? Am I missing something?

LIMITATIONS:
The paper addresses a specific problem in bandits that does not seem to have any more nefarious use cases than typical optimization models.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05