Title: Asymptotically Optimal Quantile Pure Exploration for Infinite-Armed Bandits
Venue: NeurIPS 2023
Accepted: yes
Keywords: pure exploration

multi-armed bandits

Fisher information
Abstract: We study pure exploration with infinitely many bandit arms generated \iid from an unknown distribution. Our goal is to efficiently select a single high quality arm whose average reward is, with probability $1-\delta$, within $\varepsilon$ of being with the top $\eta$-fraction of arms; this is a natural adaptation of the classical PAC guarantee for infinite action sets. We consider both the fixed confidence and fixed budget settings, aiming respectively for optimal \emph{expected} and \emph{fixed} sample complexity.


For fixed confidence, we give an algorithm with expected sample complexity $O\left(\frac{\log (1/\eta)\log (1/\delta)}{\eta\varepsilon^2}\right)$. This is optimal except for the $\log (1/\eta)$ factor, and the $\delta$-dependence closes a quadratic gap in the literature. For fixed budget, we show the asymptotically optimal sample complexity as $\delta\to 0$ is $c^{-1}\log(1/\delta)\big(\log\log(1/\delta)\big)^2$ to leading order; equivalently, the optimal failure probability with exactly $N$ samples decays as 
$\exp\big(-(1\pm o(1))\frac{cN}{\log^2 N}\big)$.
The value of $c$ depends explicitly on the problem parameters (including the unknown arm distribution) through a certain Fisher information distance. Even the strictly super-linear dependence on $\log(1/\delta)$ was not known and resolves a question of Grossman-Moshkovitz (FOCS 2015).

Asymptotically Optimal Quantile Pure Exploration for Infinite-Armed Bandits

Abstract

We study pure exploration with infinitely many bandit arms generated i.i.d. from an unknown distribution. Our goal is to efficiently select a single high quality arm whose average reward is, with probability 1 -δ , within ε of being with the top η -fraction of arms. For fixed confidence, we give an algorithm with expected sample complexity O ( log(1 /δ ) log(1 /η ) ηε 2 ) which matches a known lower bound up to the log(1 /η ) factor. In particular the δ -dependence is optimal and closes a quadratic gap. For fixed budget, we show the asymptotically optimal sample complexity as δ → 0 is log(1 /δ ) ( log log(1 /δ ) ) 2 /c . The value of c depends explicitly on the problem parameters (including the unknown arm distribution) through a certain Fisher information distance. Even the strictly super-linear dependence on log(1 /δ ) was not known and resolves a question of [GM20].

1 Introduction

In many learning problems, one faces the classical exploration versus exploitation tradeoff. A central example is the (stochastic) multi-armed bandit [LR85, BF85], where an agent is presented with a set of arms each of which when played gives a stochastic reward from an unknown and arm-dependent distribution. The performance of a bandit algorithm is most commonly determined by its regret , i.e. the difference between its average reward and the expected reward from the best arm. Multiarmed bandits and extensions have been applied in many settings including medical trials [BE95], online advertising [LCLS10], cognitive radio [AMTS11], and information retrieval [LPB17]. Optimal algorithms for the multi-armed bandit, including UCB, Thompson sampling, EXP3, and various forms of mirror descent, all make a principled tradeoff between exploration and exploitation.

In this work we focus on pure exploration bandit problems, a setting motivated by situations where the learning procedure consists of an initial exploration phase followed by a choice of policy to deploy. This is the case in hyperparameter optimization [GM20, LJD + 17] as well as reinforcement learning from simulated environments. As there is no longer a competing need to exploit, optimal algorithms for pure exploration differ from the more common regret setting.

Pure exploration problems were introduced in [EDMM02, MT04, EDMM06] in the probablyapproximately-correct (PAC) model. Here given K arms, one adaptively obtains samples until choosing one of the arms to output - the goal is to ensure that with probability 1 -δ , this arm has average reward within ε of the best arm, with minimum possible sample complexity depending on ε and δ . The early works above focused on the fixed confidence setting in which one aims to minimize the expected sample complexity. Many subsequent works have also considered the fixed budget problem where the sample complexity is uniformly bounded.

While sharp results are known for pure exploration and other bandit problems with K arms, for many applications such as advertising there are far too many arms to explore. This motivated the study of infinite-armed bandit problems in e.g. [BCZ + 97, WAM08]; the pure exploration version

was first studied in [AAKA18]. The main contribution of our work is to obtain near-optimal sample complexity for pure exploration problems with infinitely many arms in both the fixed confidence and fixed budget settings.

1.1 Problem Formulation

We now precisely formulate the infinite-armed pure exploration setting. Let S = { a 1 , a 2 , . . . } be a countably infinite set of stochastic bandit arms indexed by i = 1 , 2 , . . . . When an arm i is sampled, it returns a { 0 , 1 } -valued reward with mean p i . The values p i are drawn i.i.d. from an arbitrary reservoir distribution µ supported in [0 , 1] (which is unknown to the player). We define the cumulative distribution function of µ , and its (left-continuous) inverse

Finally let µ ∗ = G -1 µ (1) denote the essential supremum of µ , i.e. the maximum of its support.

An algorithm A interacts with S in the following way. At each time step t = 1 , 2 , . . . , T the algorithm samples an arm i t ∈ S , and observes a corresponding Bernoulli reward r t ∼ Ber ( p i t ) . The reward r t is independent of previous actions and feedback. Eventually at some time T , A chooses an arm a i ∗ to output. If the time-horizon T = N is fixed, we say A has a fixed budget constraint. If E [ T ] ≤ N is bounded only in expectation, we say A has a fixed confidence constraint.

For η, ε, δ > 0 , we say A is ( η, ε, δ ) -PAC if

and set to be the target quantile value. We emphasize that while η is known, α may not be as it depends on the unknown µ . The definition (1.1) stems from [AAKA18]. As brief justification for the parameter η , note that in an infinite-armed setting the reservoir µ could give ε -optimal arms with arbitrarily small probability. Thus it is impossible to give a non-asymptotic classical ( ε, δ ) -PAC in our setting without assumptions on µ . Taking η > 0 as above ensures that a positive fraction of arms are 'good enough' and will enable such guarantees. One also cannot set ε to zero: for example if µ is supported in [0 . 5 -e -N 4 , 0 . 5] , the quantile value of the output arm will be essentially uniform for any N -sample algorithm.

The purpose of this paper is to give ( η, ε, δ ) -PAC algorithms whose sample complexity N is minimal. We can now state our main results. Let us emphasize that unless explicitly stated, no assumptions on the reservoir distribution µ are made, nor does the algorithm have any prior knowledge about µ . Our main result in the fixed confidence case is as follows.

Theorem 1.1. For any ( η, ε, δ ) , there exists a ( η, ε, δ ) -PAC algorithm with expected sample complexity O ( log(1 /δ ) log(1 /η ) ηε 2 ) .

In the fixed budget setting, our interest is especially in the high-confidence regime δ → 0 , where we obtain the following. The following statement is a slightly informal combination of Theorems 3.1 and 3.2. We note that while the main statement requires α to be given, this is not essential in several cases as discussed extensively in the Appendix. For example if α ≥ 1+ ε 2 then α does not need to be given.

Theorem 1.2 (Informal) . For any fixed ( η, ε ) , let α = G -1 (1 -η ) be given and set β = α -ε . Then as δ → 0 , the optimal ( η, ε, δ ) -PAC algorithm under fixed budget has sample complexity

An equivalent statement is that given exactly N samples, the optimal failure probability δ to have p i ∗ ≥ G -1 (1 -η ) -ε decays as exp ( -N ( c ± o (1)) log 2 ( N ) ) . Indeed once η and ε are fixed, the question of minimizing the sample complexity N = N ( δ ) (given a target confidence δ ) is equivalent to minimizing the failure probability δ = δ ( N ) (given a sample complexity N ). These viewpoints are equivalent in both settings we study, and we switch between them at times.

Interestingly the value η makes no appearance in Theorem 1.2, so it is asymptotically irrelevant for the δ → 0 regime of fixed budget pure exploration. In fact the value arccos(1 -2 α ) -arccos(1 -2 β ) appearing in the definition of c α,β is the Fisher-information distance between α and β in the exponential family of Bernoulli random variables via the formula (1.3). See just below Theorem 3.1 for a brief explanation of why η does not enter the asymptotic sample complexity.

Remark 1.1. In our problem formulation above we assumed rewards are Bernoulli, i.e. lie in { 0 , 1 } . In fact as long as the quality of arm i is measured by its mean reward, this assumption loses no generality and is just a technical convenience: our results extend verbatim to [0 , 1] -valued rewards.

Indeed any arm with [0 , 1] -valued rewards can be transformed into a Bernoulli arm with { 0 , 1 } -valued rewards and the same mean: simply turn reward r ∈ [0 , 1] into reward 1 with probability r , and 0 with probability 1 -r . Note that this reduction (used also in Section 1.2 of [AG12]) might increase the instance-dependent sample complexity of some reservoir distributions, but our results only refer to the distribution of arm means under the reservoir which is unchanged by the reduction.

1.2 Further Notation

We use the convention that algorithms collect 1 sample per unit time until terminating, so the time t equivalently denotes the number of total samples collected so far. Denote by n i,t the number of samples of arm a i collected by time t . The n -th time a i is sampled, its reward is r i,n ∈ [0 , 1] . The total reward of arm i up to time t is

The corresponding average reward is ˆ p i,t = ˆ p i ( n i,t ) = R i,t n i,t . We use C to indicate a universal constant independent of all parameters in this paper, and o n (1) and o N (1) to denote quantities tending to 0 as n → ∞ or N → ∞ , with other parameters implicitly held constant. However in Section C we use e.g. Ω α,ϱ to indicate an asymptotic lower bound with implicit constant factor depending on the values of α, ϱ , which are treated as fixed. In all our uses of these notations it is n or N which is tending to infinity while other parameters are always treated as fixed.

1.3 Related Work

As discussed above, this work belongs to the area of pure exploration for multi-armed bandit problems. Unlike ordinary bandit problems where one aims to minimize the regret compared to the best arm [BCB12, Sli19], in pure exploration all that matters is the final arm selected by the algorithm. We survey several existing results below, with an emphasis on the high-probability regime of small δ . See e.g. Chapter 33 of [LS20] for a more detailed survey.

Pure exploration was first studied in [EDMM02, MT04, EDMM06] in the probably-approximatelycorrect model. Here given K arms, one adaptively obtains samples until choosing one of the arms to output - the goal is to ensure that with probability 1 -δ , this arm has average reward within ε of the best arm. These works showed that the optimal fixed confidence sample complexity is Θ ( K ε 2 log 1 δ ) .

̸

Later, [BMS09] considered the simple regret of pure exploration problems, namely the regret incurred at the final timestep. [ABM10] studied the closely related problem of identifying the best arm, obtaining nearly tight sample complexity bounds in terms of the the sum of the squared inverse suboptimality-gaps H = ∑ i = i ∗ ∆ -2 i . Further upper and lower sample complexity bounds have been obtained in several works. For example [CL15, KCG16] show that for fixed confidence, the sample complexity scales as Θ( H log(1 /δ )) as δ → 0 . The fixed budget setting, in which the number of adaptive samples is upper-bounded almost surely rather than in expectation, turns out to be more difficult. [CL16] proved that the optimal fixed budget sample complexity can be Θ( H log( K ) log(1 /δ )) as δ → 0 , i.e. the fixed budget constraint may lead to an additional log( K )

factor. However it reverts to Θ( H log(1 /δ )) when the value of H is known beforehand. Many recent works have studied other aspects of pure exploration, for example by incorporating structured feedback; see [JMNB14, CGL + 17, KSJ20, KG21, TRMD21, ACD21, ZKSN22, AAJ + 22].

Infinite-armed bandits have also much received previous study, e.g. [BCZ + 97, WAM08]. Since near-optimal arms may be arbitrarily rare, it is natural to instead compare with a quantile of the arm distribution. For example [CK18] aims to minimize regret relative to such a quantile.

The ( η, ε, δ ) -PAC guarantees we address in this paper were first studied in [AAKA18], for infinitely many arms in the fixed confidence setting. Their approach was to sample K ≍ log(1 /δ ) η arms and then apply a PAC algorithm for K -armed pure exploration. As discussed at the beginning of Section 2, the resulting algorithm 'pays twice' for the high confidence level 1 -δ which leads to a suboptimal O (log 2 (1 /δ )) sample complexity upper bound. Topk extensions were also studied in [RLS19, CK19]; the log 2 (1 /δ ) scaling is still present in their results.

Of particular note is the work [dHCMC21] which considers also both fixed budget and confidence settings and obtains somewhat similar looking results. However they restrict attention to a special class of reservoir distributions with supremum achieved by an atom, which must be ∆ -larger than the rest of the support. This structural assumption of a ∆ -gap intrinsically reduces fixed budget sample complexity: their result (see Theorem 4 therein) is actually better than the lower bound we show in Theorem 3.2 as there is no appearance of log log(1 /δ ) (i.e. log( T ) in their notation).

From their fixed budget estimate, [dHCMC21] deduce (at the end of Section 1 therein) the same bound as Theorem 1.1 in their setting for the special case ε = ∆ . Our Theorem 3.2 shows that for the general reservoirs we consider, passing from fixed budget to fixed confidence is inherently suboptimal: the factors of log log(1 /δ ) would remain, but are extraneous for fixed confidence. This underscores that Theorem 1.1 is genuinely new despite the superficial similarity with [dHCMC21], since their proof cannot work in our setting.

Finally [GM20] studied the infinite-arm pure exploration problem where α is given, also focusing on the δ → 0 asymptotics. They proposed an algorithm with fixed budget sample complexity O ( log(1 /δ ) ( log log(1 /δ ) ) 2 ) , and asked whether the log log(1 /δ ) factors are necessary. Theorem 3.2 shows their bound is optimal up to constant factors in terms of δ and in fact obtains the tight constant. Interestingly [GM20] were motivated by complexity theoretic applications to amplification and derandomization , where bandit arms correspond to random seeds.

We remark that the analysis in [GM20] seems to be technically incomplete. In particular in Lemma 4.5 of (the cited, journal version of) the paper, they neglect to take a union bound over sequences ( T 1 , . . . , T k ) summing to T but only estimate the probability of each fixed sequence ( T 1 , . . . , T k ) . This is a serious gap since the number of such sequences is exponentially large in T . However their idea to use a moving sequence of rejection thresholds was fundamentally correct. It is similar to the main phase of our Algorithm 3, for which we give a fully rigorous, supermartingale-based analysis.

2 The Fixed Confidence Setting

Our fixed confidence algorithms proceeds in two phases. The first phase aims to estimate the target quantile value, which we recall depends on the unknown α . The second phase then aims to find a single arm which is almost as good as this estimate with high probability.

Focusing on the δ -dependence, a challenge with infinitely many arms is that to succeed with probability 1 -δ , it is necessary both to sample log(1 /δ ) arms to ensure a good arm is ever observed, and to sample an arm log(1 /δ ) times before outputting it as i ∗ . This is why the approach of [AAKA18] requires O (log 2 (1 /δ )) samples: they obtain O (log(1 /δ )) samples each of O (log(1 /δ )) arms. However in our algorithm, the first phase samples O (log(1 /δ )) arms O (1) times each, while the second phase samples O (1) arms O (log(1 /δ )) times each. This allows us to satisfy both necessary conditions above without paying twice for the confidence level.

2.1 The Algorithm for Fixed Confidence

We first give in Alg. 1 a simple procedure to estimate the top η quantile, allowing an ε/ 3 error as well as an η/ 2 error in the quantile itself. Alg. 1 obtains O ( log(1 /η ) ε 2 ) samples from each of the

Algorithm 1: Output ˆ α ∈ [ G -1 (1 -η ) -ε , G -1 ( 1 -η ) + ε ] with probability 1 -δ

3 2 3 2 1 input: arm set S = ( a 1 , a 2 , . . . ) and parameters η, ε, δ ∈ (0 , 1) . 2 initialize: K = C log(1 /δ ) η . 3 for i = 1 , 2 , . . . , K do 4 Collect n = C log(1 /η ) ε 2 samples of arm i . Set ˆ p i = ˆ p i ( n ) to be the average observed reward. 5 end 6 Let ˆ α be the k -th largest value in { ˆ p 1 , . . . , ˆ p K } for k = ⌈ 3 Kη 4 ⌉ . 7 Return ˆ α

Algorithm 2: Output a ∗ such that p ∗ ≥ ˆ α -ε with probability 1 -δ

first K = O ( log(1 /δ ) η ) arms a 1 , . . . , a K . The resulting estimator ˆ α is the 1 -3 η 4 quantile of the empirical average rewards ˆ p 1 , . . . , ˆ p k . Its main guarantee is below.

Proposition 2.1. Fix 0 ≤ η, ε, δ ≤ 1 . With probability at least 1 -δ 2 , the output ˆ α of Alg. 1 satisfies

Moreover, Alg. 1 has sample complexity

Next Alg. 2 repeatedly chooses a new arm a i and obtains O ( log(1 /ηδ ) ε 2 ) samples. It accepts if the sample mean was at least ˆ α -ε 3 , and otherwise moves on to the next arm. If C log(1 /δ ) η arms have been tried without success, then Alg. 2 outputs no arm, thus declaring failure. This termination condition is necessary to avoid incurring huge sample complexity when Alg. 1's estimate ˆ α of α is inaccurate. Since typically Ω( η ) of arms will be good enough to usually succeed, this also preserves the 1 -δ confidence level. We now give the following more detailed restatement of Theorem 1.1.

Theorem 2.1. Apply Algorithm 1 followed by Algorithm 2 using the resulting value ˆ α . This combined algorithm has expected sample complexity O ( log(1 /η ) log(1 /δ ) ηε 2 ) . Moreover its output a i ∗ satisfies

Proposition 2.1 and Theorem 2.1 are proved in Appendix A. We note that in analyzing Algorithm 2, typically the returned arm a i has i ≤ K + O (1 /η ) . This is because each arm a i in the top η/ 2 quantile has a good chance to pass the test of Algorithm 2. In particular, the expected sample complexity calculation never multiplies two log(1 /δ ) terms together. However, continuing for K + C log(1 /δ ) η steps is important to ensure a 1 -δ success probability. Algorithm 2 stops after O (log(1 /δ ) /η ) arms instead of continuing forever in order to guard against erroneous estimates from Algorithm 1.

Remark 2.1. Our fixed confidence algorithm, given by combining Alg. 1 with Alg. 2 as above, requires only O (1) batches in expectation. Here a batched algorithm operates in a small number of batched phases. At the start of each phase, such an algorithm chooses b arms to sample exactly s times each, where b, s can both depend adaptively on the previous feedback, but cannot be

changed during the current phase. Minimizing the number of required batches is often desirable, see e.g. [PRCS16, GHRZ19]. In particular Alg. 1 uses a single batch with s 1 = C log(1 /η ) ε 2 samples of b 1 = C log(1 /δ ) η arms. Then Alg. 2 can be implemented in a batched way with s 2 = C log(1 /ηδ ) ε 2 and a sequence of batch sizes b 2 ,i = 2 i η for 1 ≤ i ≤ log 2 ( C log(1 /δ )) . (In the latter phase, one stops after finding an arm to accept.) While this construction uses O (1) batches in expectation , it could be interesting to explore pure exploration with an exactly fixed number of batches, which is more analogous to the fixed budget setting.

2.2 Near-Optimality in the Fixed Confidence Regime

Here we explain why the guarantee of Theorem 2.1 is nearly optimal. For comparison, recall from the important work [MT04] that Θ ( K log(1 /δ ) ε 2 ) samples are necessary and sufficient for ( ε, δ ) -PAC pure exploration in the K -armed bandit problem with fixed confidence. Intuitively, one expects this problem to be related to ours via η ≈ 1 /K . In fact the following infinite-arm analog was later shown. 1 It follows that the guarantee of Theorem 1.1 is optimal up to the log(1 /η ) factor.

Proposition 2.2 (Theorem 1 and Remark 2 of [AAKA18]) . There exists an absolute constant c > 0 such that the following holds. For any 1 / 4 ≤ α ≤ 3 / 4 and η, δ ≤ 1 / 10 and for any pure exploration algorithm A with expected sample complexity N ≤ c log(1 /δ ) ηε 2 , there exists a reservoir distribution such that A fails to be ( η, ε, δ ) -PAC.

Letting L = log(1 /δ ) ηε 2 ≥ 1 /η be the lower bound from Proposition 2.2, the expected sample complexity of our algorithm is at most O ( L log(1 /η )) ≤ O ( L log L ) . Hence Theorem 1.1 is always nearly optimal compared to the lower bound. Prior to our work there was a quadratic gap as the best upper bound for general reservoirs (Theorem 6 in [AAKA18]) was proportional to log 2 (1 /δ ) . We note also that Theorem 5 of [dHCMC21] showed a similar result to Proposition 2.2.

3 Fixed Budget

Our algorithm and lower bounds for the fixed budget setting are much more technical, with full details given in the Appendix. Here we carefully state the results and give outlines of the proofs.

3.1 Precise Results for Fixed Budget

We first state the results in the easier case that α is given. Theorems 3.1 and 3.2 below give rigorous statements of Theorem 1.2. As there, we will write β for the target value α -ε when α is given, since then the value ε plays no role.

Theorem 3.1. For any fixed 0 < β < α < 1 , there is a sequence ( A N ) N ≥ 1 of N -sample algorithms given explicitly by Algorithm 3 such that for any η ∈ (0 , 1) and any sequence of reservoir distributions µ N with G -1 µ N (1 -η ) ≥ α , with c α,β as in (1.2) :

Conversely, the following lower bound applies for any quantile η ∈ (0 , 1) , and holds even when α is known (which only makes pure exploration easier). It implies that η is asymptotically irrelevant for fixed budget sample complexity, i.e. the sample complexity of approximating the η = 0 . 01 -quantile and η = 0 . 99 quantile in fixed budget pure exploration depends only on the quantile values themselves as δ → 0 . Some intuition for this is as follows. To succeed in pure exploration, one should have sampled the eventually outputted arm at least Ω(log 1 /δ ) times. The main obstacle to success in the fixed budget case is that any arm we obtain many samples of might gradually degrade over time. The probability of this degradation is essentially given by small probabilities coming from Chernoff-bound type events, which dominate the prior probability that the arm is in a top quantile.

Theorem 3.2. For any 0 < η < 1 and 0 < β < α < 1 there exists a sequence of reservoir distributions µ N with α = G -1 µ N (1 -η ) such that for any sequence of N -sample algorithms A N ,

Remark 3.1. As discussed in the Appendix, the requirement that α be known in Theorem 3.1 is technically necessary to deal with potential discontinuity of α as η varies, but can be removed under mild conditions. In Theorems B.1, B.2, and B.3 below we give three concrete formulations under which the guarantee (3.1) can be achieved without knowledge of α . Informal descriptions (any 1 of which suffices on its own) are:

2. α is redefined as the average of G -1 µ ( η ) for η ranging over an interval.

3. µ N = µ is independent of N , and the target value is

for fixed ε 1 > ε . (Recall µ ∗ is the maximum value in the support of µ .)

Remark 3.2. In fact (see Theorem C.9 in the Appendix), Theorem 3.1 holds even if the algorithms A N must output log N distinct arms i ∗ 1 , . . . , i ∗ log N , all of which must satisfy p i ∗ j ≥ β . I.e. for suitable ( A N ) N ≥ 1 ,

3.2 Algorithm for Fixed Budget

Here we present our Algorithm 3 for the fixed budget problem. We note that this algorithm is given as input the value of α . See Appendix C.1 for first-stage algorithms which estimate α with sufficiently high accuracy and low sample complexity in the three scenarios of Remark 3.1. We use an explore-and-discard approach: at each time, it is currently exploring some arm i , having permanently discarded arms 1 , 2 , . . . , i -1 and not yet interacted with arms i + 1 , i + 2 , . . . . It turns out to be very convenient to operate in a 'B-batch-compressed' way for an increasing integer sequence B = ( b 1 , b 2 , . . . ) . This means that at all times, the current arm i has been sampled b k times for some k , and the samples between b k and b k +1 are compressed into a single decision. The sequence b k increases geometrically at a small rate, i.e. b k +1 /b k ≈ 1 + ϱ for small ϱ once k is mildly large. This batch-compression turns out to be without loss of generality up to a factor 1 + ϱ in the sample complexity, and helps the analysis operate on the proper geometric time-scales.

We now give a precise description as well as pseudo-code. Let 0 < ϱ ≪ ϱ 1 ≪ ϱ 2 ≪ 1 be suitably small constants (i.e. choose ϱ 2 sufficiently small, then ϱ 1 , then ϱ ). Wedefine b k and other parameters as follows:

The outer loop dictates the arm i under consideration. While exploring arm i , the main phase (shown in the last for loop) consists of collecting a new batch of b k 0 + j -b k 0 + j -1 ≈ ϱb k 0 + j -1 samples, and rejecting if the new empirical mean reward ˆ p i,b k 0 + j drops below

Let us explain the point of this formula. First ignoring the function θ for now, we see that the rejection threshold for ˆ p i,b k steadily decreases with k . This threshold is tuned so that when arm i has been sampled say Ω( N/ log N ) times, we have k ≍ log( N ) /ϱ which results in a threshold slightly larger than β . Thus an arm which survives for such a long time will be prepared for acceptance as a new-optimal arm.

This strategy can be motivated as follows. The chief worry in fixed budget exploration is that arms might slowly degrade after many samples have been invested into them, which suggests a gradually decreasing rejection threshold. We designed this threshold to drop by a constant divided by log( N ) each time the number b k of samples doubles. This ensures that for arm i to be rejected at time b k +1 , the last b k +1 -b k samples must have behaved atypically (compared to their past behavior) by roughly √

b k +1 log( N ) standard deviations. The probability of this rare event is roughly exp( -Cb k +1 / log 2 ( N )) for some constant C by a Chernoff bound. Therefore if all N samples are used on eventually rejected arms, the product of these rare event probabilities will be roughly exp( -CN/ log 2 ( N )) since each b k +1 counts the number of samples used on an individual arm.

The use of the non-linear function θ above is essential to achieve the optimal constant c α,β in Theorem 3.1. At a high-level, θ balances the p (1 -p ) -dependence of optimal Chernoff bounds on the underlying probability p of the Binomial random variable. If the optimal constant is not desired, then Theorem 3.1 can be simplified somewhat; the nonlinear θ is not needed and one can double the number of samples at each step rather than multiplying by 1 + ϱ for small ϱ . The earlier for loops in Algorithm 3 are technically important to handle small sample sizes for each given arm, before the required Chernoff bounds have kicked in asymptotically.

3.3 Analysis of Algorithm 3

The analysis of Algorithm 3 goes by controlling the tail distribution for the rejection time of a given arm (this time is taken to be zero if no rejection ever happens). Intuitively, we are most worried about arms which slowly degrade, thus wasting many samples. The following lemma, proved in the Appendix by a supermartingale argument, is key to rigorize this idea.

Lemma 3. Suppose ( Y i ) i ≥ 1 are i.i.d. random variables with non-negative integer values, and E [ Y c i ] ≤ 1 holds for some constant c ≥ 0 . Then with

we have P [ M ≥ A ] ≤ A -c .

We apply Lemma 3 in the following way. Let X i be the number of samples used by arm a i before rejection, and I i ∈ { 0 , 1 } be the indicator of the event that a i is ever rejected (even if Algorithm 3 were to continue past time N and sample arm i an infinite number of times). We set Y i = e X i · I i . The bulk of the analysis thus reduces to proving that E [ Y c i ] ≤ 1 for a suitable exponent c . Once this is shown, Lemma 3 ensures that M is small with high probability. Since log( M ) is exactly the total number of samples used on eventually-rejected arms, if say log M ≤ N ( 1 -1 log N ) then the last arm i ∗ to be selected must have passed enough rejection thresholds to have p i ∗ ≥ β with sufficiently high probability.

The early behavior of Algorithm 3 and b k are specifically designed so that small values of Y i contribute little in expectation. The tail behavior of Y i (corresponding to rejecting arm i after a long time) is controlled by a technical analysis involving many adjacent time-scales in Subsection C.4 of the Appendix. Interestingly this tail analysis of Y i never explicitly models the reward probability p i . This is because we are able to argue that a rejection requires the early and late time behaviors of arm i to differ from each other , which is unlikely since the rewards form an i.i.d. sequence.

3.4 On the Lower Bound Proof

Weconsider the lower bound Theorem 3.2 to be the technical highlight of this paper. In it we face the challenge of proving a very small lower bound on the failure probability of any fixed budget algorithm. To do so we construct an online adversary with bounded probabilistic strength to distort rewards. The setup follows; recall B -batch-compressed algorithms as defined in Subsection 3.2.

Definition 3.1. An adaptive randomness distorting adversary A interacts with a B -batchcompressed algorithm A in the following way. Suppose A chooses to increase the number of samples of arm a i from b k to b k +1 . Then A may restrict the set of possible outcomes of these b k +1 -b k samples. Additionally, when A outputs an arm a i ∗ , the adversary can restrict the possible values of p i ∗ .

We will refer to adversarial actions as declarations . Thus when A chooses a batch of samples, A may declare that some property holds for the observed rewards.

As defined above, an adversary A can do anything. We will limit the power of A to make lowprobability declarations. To formalize this, we charge A per 'bit' of probabilistic distortion, and give A a deterministic 'budget' for doing so. We measure this budget according to the algorithm's filtration which we refer to the 'jointly Bayesian' viewpoint. One should think that the reservoir distribution µ N is known to both A and A , but neither has any information on the true reward probabilities p i beyond the observed rewards. Thus A and A share at any time t the posterior distribution µ t . In particular recalling (B.9), µ t determines the distribution for the outcome of the next batch of b k +1 -b k samples.

Definition 3.2. Suppose that at time t , the declaration of A has probability P t to hold according to µ t . Let the sum

be the total cost of A up to time t , and Cost N the total cost of A . We say strength ( A ) ≤ Cost holds for some Cost ∈ R if the bound Cost N ≤ Cost almost surely.

The next key lemma shows that to obtain a lower bound for the failure probability of an algorithm, it suffices to prevent success using a low strength A .

Lemma 4. Suppose there is a randomness distorting adversary A of strength Cost whose declarations ensure that any algorithm A outputs i ∗ satisfying p i ∗ ≤ β almost surely. Then the true failure probability of A is

Proof. At each step t , let the random variable E t denote the minimum possible conditional probability of the event p i ∗ ≤ β for any algorithm (in the algorithm's jointly Bayesian filtration). We claim that conditioned on A 's declarations holding, for any A the quantity M t ≡ E t ∏ s ≤ t P s evolves as a supermartingale in this filtration. This suffices because it implies E 0 = M 0 ≥ E [ M T ] ≥ e -Cost . (Here T is the random number of total batches used.)

Indeed suppose we are at time t and the next batch has been declared but not sampled. (Identical arguments apply when the adversary restricts p i ∗ in the last stage.) Let the σ -field F t denote all information up to this point including the declaration of the next batch. Let E denote an expectation where the samples from the next batch is distributed according to µ t . Let ˜ E denote an expectation where A 's declaration is conditioned to hold on the next batch. The dynamic programming principle implies that

for any A (with equality for the optimal A ). Moreover since A 's declaration has µ t -probability P t ,

P t is F t measurable, so ˜ E [ P t E t +1 | F t ] ≤ E t . This establishes the claim and ends the proof.

We prove Theorem 3.2 by constructing a Bayesian adversary who slowly degrades the empirical performance of each arm a i . This adversary declares for each batch of samples that the empirical average reward ˆ p i ( n i,t ) of arm i will drop by Ω ( ϱ log( N ) ) , at least once the sample size n i,t ≥ N ϱ is large. This degradation schedule ensures that the average reward of any arm is smaller than β once it has been sampled Ω( N 1 -ϱ ) times. Moreover it follows from reverse Chernoff estimates that this adversary pays O ( 1 log 2 ( N ) ) cost per sample, leading to a failure probability lower bound of exp( -O ( N/ log 2 N )) from Lemma 4. As with the upper bound, sharp constants can be tracked with more work and indeed our optimal adversary uses the same function θ as in Algorithm 3.

Conclusion Our aim in this paper was to understand the sample complexity of pure exploration with infinitely many arms. We showed that, surprisingly, the behavior of fixed confidence and fixed budget problems is provably very different. In the former setting, there is a nearly optimal algorithm which precisely balances between sampling enough distinct arms (to estimate the quantile) and obtaining enough samples of a single arm (to be output). In the latter, the optimal algorithm must repeatedly decide whether to continue with the current arm or switch to a fresh one, via a gradually decreasing sequence of rejection thresholds.

Several interesting questions remain. One is that our fixed budget analysis is tailored to the δ → 0 setting, and does not apply if ε, η, δ all tend to zero at comparable rates. Hence other behaviors could be present in such parameter regimes. Additionally, a key conceptual feature of infinitearmed bandits is the possibility that no 'good' arms are among those sampled by the algorithm. By definition, this simply cannot happen in K -armed bandits. It would be interesting to identify a natural problem setting that interpolates between them. Finally high probability bounds on the fixed confidence sample complexity would interpolate between the two settings we studied.

References

[AAJ + 22] Maryam Aziz, Jesse Anderton, Kevin Jamieson, Alice Wang, Hugues Bouchard, and Javed Aslam. Identifying new podcasts with high general appeal using a pure exploration infinitely-armed bandit strategy. In Proceedings of the 16th ACM Conference on Recommender Systems , pages 134-144, 2022.

[AAKA18] Maryam Aziz, Jesse Anderton, Emilie Kaufmann, and Javed Aslam. Pure exploration in infinitely-armed bandit models with fixed-confidence. In Algorithmic Learning Theory , pages 3-24. PMLR, 2018.

[ABM10] Jean-Yves Audibert, S´ ebastien Bubeck, and R´ emi Munos. Best arm identification in multi-armed bandits. In COLT , pages 41-53. Citeseer, 2010.

[ACD21] Ayya Alieva, Ashok Cutkosky, and Abhimanyu Das. Robust pure exploration in linear bandits with limited budget. In International Conference on Machine Learning , pages 187-195. PMLR, 2021.

[AG12] Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Conference on learning theory , pages 39-1. JMLR Workshop and Conference Proceedings, 2012.

[AMTS11] Animashree Anandkumar, Nithin Michael, Ao Kevin Tang, and Ananthram Swami. Distributed algorithms for learning and cognitive medium access with logarithmic regret. IEEE Journal on Selected Areas in Communications , 29(4):731-745, 2011.

[BCB12] S´ ebastien Bubeck and Nicolo Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends® in Machine Learning , 5(1):1-122, 2012.

[BCZ + 97] Donald Berry, Robert Chen, Alan Zame, David Heath, and Larry Shepp. Bandit problems with infinitely many arms. Annals of Statistics , 25(5):2103-2116, 1997.

[BE95] Donald A Berry and Stephen G Eick. Adaptive assignment versus balanced randomization in clinical trials: a decision analysis. Statistics in medicine , 14(3):231-246, 1995.

[BF85] Donald Berry and Bert Fristedt. Bandit problems: sequential allocation of experiments. London: Chapman and Hall , 5(71-87):7-7, 1985.

[BMS09] S´ ebastien Bubeck, R´ emi Munos, and Gilles Stoltz. Pure exploration in multi-armed bandits problems. In International conference on Algorithmic learning theory , pages 23-37. Springer, 2009.

[CGL + 17] Lijie Chen, Anupam Gupta, Jian Li, Mingda Qiao, and Ruosong Wang. Nearly optimal sampling algorithms for combinatorial pure exploration. In Conference on Learning Theory , pages 482-534. PMLR, 2017.

[CK18] Arghya Roy Chaudhuri and Shivaram Kalyanakrishnan. Quantile-regret minimisation in infinitely many-armed bandits. In UAI , pages 425-434, 2018.

[CK19] Arghya Roy Chaudhuri and Shivaram Kalyanakrishnan. PAC Identification of Many Good Arms in Stochastic Multi-Armed Bandits. In International Conference on Machine Learning , pages 991-1000. PMLR, 2019.

[CL15] Lijie Chen and Jian Li. On the optimal sample complexity for best arm identification. arXiv preprint arXiv:1511.03774 , 2015.

[CL16] Alexandra Carpentier and Andrea Locatelli. Tight (Lower) Bounds for the Fixed Budget Best Arm Identification Bandit Problem. In Conference on Learning Theory , pages 590-604, 2016.

[DA92] Alejandro De Acosta. Moderate deviations and associated Laplace approximations for sums of independent random vectors. Transactions of the American Mathematical Society , 329(1):357-375, 1992.

[dHCMC21] Rianne de Heide, James Cheshire, Pierre M´ enard, and Alexandra Carpentier. Bandits with many optimal arms. Advances in Neural Information Processing Systems , 34:22457-22469, 2021.

[EDMM02] Eyal Even-Dar, Shie Mannor, and Yishay Mansour. PAC bounds for multi-armed bandit and Markov decision processes. In International Conference on Computational Learning Theory , pages 255-270. Springer, 2002.

[EDMM06] Eyal Even-Dar, Shie Mannor, and Yishay Mansour. Action Elimination and Stopping Conditions for the Multi-Armed Bandit and Reinforcement Learning Problems. Journal of machine learning research , 7(6), 2006.

[GHRZ19] Zijun Gao, Yanjun Han, Zhimei Ren, and Zhengqing Zhou. Batched Multi-Armed Bandits Problem. Advances in Neural Information Processing Systems , 32, 2019.

[GM14] Spencer Greenberg and Mehryar Mohri. Tight Lower Bound on the Probability of a Binomial Exceeding its Expectation. Statistics & Probability Letters , 86:91-98, 2014.

[GM20] Ofer Grossman and Dana Moshkovitz. Amplification and Derandomization without Slowdown. SIAM Journal on Computing , 49(5):959-998, 2020.

[Hoe94] Wassily Hoeffding. Probability inequalities for sums of bounded random variables. In The collected works of Wassily Hoeffding , pages 409-426. Springer, 1994.

[JMNB14] Kevin Jamieson, Matthew Malloy, Robert Nowak, and S´ ebastien Bubeck. lil'UCB: An Optimal Exploration Algorithm for Multi-Armed Bandits. In Conference on Learning Theory , pages 423-439. PMLR, 2014.

[KCG16] Emilie Kaufmann, Olivier Capp´ e, and Aur´ elien Garivier. On the complexity of bestarm identification in multi-armed bandit models. The Journal of Machine Learning Research , 17(1):1-42, 2016.

[KG21] Tom´ as Koc´ ak and Aur´ elien Garivier. Epsilon best arm identification in spectral bandits. In IJCAI , pages 2636-2642, 2021.

[KSJ20] Julian Katz-Samuels and Kevin Jamieson. The true sample complexity of identifying good arms. In International Conference on Artificial Intelligence and Statistics , pages 1781-1791. PMLR, 2020.

[LCLS10] Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web , pages 661-670, 2010.

[LJD + 17] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research , 18(1):6765-6816, 2017.

[LP14] Kyle Luh and Nicholas Pippenger. Large-deviation bounds for sampling without replacement. The American Mathematical Monthly , 121(5):449-454, 2014.

[LPB17] David E Losada, Javier Parapar, and Alvaro Barreiro. Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems. Information Processing & Management , 53(5):1005-1025, 2017.

[LR85] Tze Leung Lai and Herbert Robbins. Asymptotically efficient Adaptive Allocation Rules. Advances in Applied Mathematics , 6:4-22, 1985.

[LS20] Tor Lattimore and Csaba Szepesv´ ari. Bandit Algorithms . Cambridge University Press, 2020.

[MNS16] Amit Moscovich, Boaz Nadler, and Clifford Spiegelman. On the exact Berk-Jones statistics and their p -value calculation. Electronic Journal of Statistics , 10(2):23292354, 2016.

[MT04] Shie Mannor and John N Tsitsiklis. The Sample Complexity of Exploration in the Multi-Armed Bandit Problem. Journal of Machine Learning Research , 5(Jun):623648, 2004.

[MU17] Michael Mitzenmacher and Eli Upfal. Probability and computing: Randomization and probabilistic techniques in algorithms and data analysis . Cambridge university press, 2017.

[Nie20] Frank Nielsen. An Elementary Introduction to Information Geometry. Entropy , 22(10):1100, 2020.

[PRCS16] Vianney Perchet, Philippe Rigollet, Sylvain Chassang, and Erik Snowberg. Batched Bandit Problems. The Annals of Statistics , pages 660-681, 2016.

[RLS19] Wenbo Ren, Jia Liu, and Ness B Shroff. Exploring k out of Top ρ Fraction of Arms in Stochastic Bandits. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 2820-2828. PMLR, 2019.

[Sli19] Aleksandrs Slivkins. Introduction to Multi-Armed Bandits. Foundations and Trends® in Machine Learning , 12(1-2):1-286, 2019.

[TRMD21] Parth K Thaker, Nikhil Rao, Mohit Malu, and Gautam Dasarathy. Pure Exploration in Multi-armed Bandits with Graph Side Information. arXiv:2108.01152 , 2021.

[WAM08] Yizao Wang, Jean-Yves Audibert, and R´ emi Munos. Algorithms for infinitely manyarmed bandits. Advances in Neural Information Processing Systems , 21, 2008.

[ZKSN22] Yinglun Zhu, Julian Katz-Samuels, and Robert Nowak. Near instance optimal model selection for pure exploration linear bandits. In International Conference on Artificial Intelligence and Statistics , pages 6735-6769. PMLR, 2022.

A Proofs from Section 2

1 input: arm set S = ( a 1 , a 2 , . . . ) and parameters ( η 1 , η 2 , ε, δ ) ∈ (0 , 1) with η 2 < η 1 .

2 initialize: K = Cη 1 log(1 /δ ) η 2 2 .

3 for i = 1 , 2 , . . . , K do

4

Collect

n

=

5 end

6 Let ˆ α be the k -th largest value in { ˆ p 1 , . . . , ˆ p K } for k = ⌈ K ( η 1 -η 2 2 )⌉ .

7 Return ˆ α

We show the following generalization of Proposition 2.1.

Proposition A.1. Fix 0 ≤ η 1 , η 2 , ε, δ ≤ 1 with η 2 ≤ η 1 . With probability at least 1 -δ 2 , the output ˆ α of Alg. 4 satisfies

Moreover, Alg. 4 has sample complexity

Proof. The sample complexity is clear so we focus on the first statement. First observe that by a Chernoff estimate, for each i ∈ [ K ] ,

Let N ( ε ) be the number of i ∈ [ K ] such that | p i -ˆ p i | ≥ ε 3 . Applying a second Chernoff estimate (of multiplicative form, see e.g. Theorem 4.5 in [MU17]) on these events as i varies and noting that Kη 2 ≥ C log(1 /δ ) , (A.1) implies

We next show that with probability at least 1 -δ 4 ,

With p i the (true) mean reward from arm a i , let

denote the number of the K tested arms which satisfy p i > α . By definition, N α is stochastically dominated by a Bin ( K,η 1 -9 η 2 10 ) random variable, and η 1 -3 η 2 4 = Θ( η 1 ) since η 2 ≤ η 1 . Note that

Therefore another multiplicative Chernoff estimate implies

When both N ( ε ) ≤ Kη 2 6 and N α ≤ K ( η 1 -3 η 2 4 ) hold, it follows by definition that ˆ α ≤ α + ε 3 . Hence recalling (A.2) above, we conclude that

C

log(1

ε

2

/η

2

)

samples of arm

i

. Set

ˆ

p

i

= ˆ

p

i

(

n

)

to be the average observed reward.

establishing (A.3). The other direction is similar. With α = G -1 (1 -η 1 ) as usual, we set

This time, N α stochastically dominates a Bin( K,η 1 ) random variable. Yet another Chernoff estimate yields

Using (A.2) in the same way as above, we find

This concludes the proof.

Proof of Theorem 2.1. First we analyze the expected sample complexity. On the event that

we claim that Alg. 2 terminates with probability η/ 4 for each a i . Indeed, if

then termination always happens by definition. This has probability at least 1 / 4 if p i ≥ G -1 ( 1 -η 2 ) by Theorem 1 in [GM14], and the latter condition holds with probability at least η/ 2 by definition. It follows that when (A.5) holds, the expected sample complexity of Alg. 2 is O ( log(1 /ηδ ) ηε 2 ) . On the other hand, (A.5) fails to hold with probability less than δ . Because of the explicit termination condition in Alg. 2, this yields a additional sample complexity contribution of smaller order O ( δ log(1 /δ ) log(1 /ηδ ) ηε 2 ) . Finally Alg. 4 has sample complexity

which clearly forms the dominant contribution. This completes the proof of the sample complexity bound and we now turn to proving correctness with probability 1 -δ . First, it is easy to see that Alg. 4 outputs some arm a i with probability at least 1 -δ 2 . It therefore suffices to show that for any fixed ˆ α satisfying (A.5), conditioned on the event ˆ p i ≥ ˆ α -ε 3 , the conditional probability that p i ≥ α -ε is at least 1 -δ 2 .

We do this using Bayes' rule. If p i ≥ G -1 (1 -η 2 ) , then as above Theorem 1 in [GM14] implies

This event hence contributes probability at least η/ 4 to the event p i ≥ G -1 (1 -η ) . On the other hand, if p i ≤ G -1 (1 -η ) -ε ≤ ˆ α -2 ε 3 , then

for an absolute constant C . Combining these via Bayes' rule implies the desired result.

B Lower Bound for Fixed Budget

Fixed Budget with Unknown α

Before giving the proof, we give some qualitative discussion of the role of unknown α . We consider Theorem 3.2 to be a definitive lower bound, since e.g. being given the value of α only makes the result stronger. When α is unknown, it is possible to give an essentially matching algorithm, but more care is required when stating the result. This is inherent and stems from the fact that the value α = G -1 µ (1 -η ) can be difficult or even impossible to estimate, yet determines the constant c α,β in the desired rate.

Let us illustrate the issue by a counterexample. Consider µ N defined by:

Similarly define ˜ µ N by:

Then µ N and ˜ µ N are not distinguishable using N samples, yet G -1 µ (1 / 2) = 0 . 4 while G -1 ˜ µ (1 / 2) = 0 . 3 . Using non-distinguishability it follows that the lower bound of Theorem 3.2 applies to ˜ µ N with threshold α = G -1 µ N (1 / 2) = 0 . 4 , as opposed to the direct application using G -1 ˜ µ N (1 / 2) = 0 . 3 . It is 1

not hard to show using monotonicity of √ x (1 -x ) that

for all ε ≤ 0 . 3 . As a result, it is information-theoretically impossible to achieve the rate (3.1) for ˜ µ N if the target quantile value α is not given. The core reason is that the value G -1 ˜ µ (1 / 2) = 0 . 3 is too sensitive to the choice η = 1 / 2 of quantile.

Fortunately, this issue is more of an annoyance than a real difficulty. It can be fixed in several ways. In Theorems B.1, B.2, and B.3 below we give three concrete formulations under which the guarantee (3.1) can be achieved, as mentioned in the main body.

Theorem B.1. For fixed η 1 , η 2 , ε , there is a sequence ( A N ) N ≥ 1 of N -sample algorithms outputting a i ∗ such that the following holds for any sequence ( µ N ) N ≥ 1 of reservoir distributions. Letting

be a quantile average of µ N , we have

Theorem B.2. For fixed η, ε , there is a sequence ( A N ) N ≥ 1 of N -sample algorithms outputting a i ∗ such that for any sequence of reservoir distributions µ N satisfying

we have

Theorem B.3. For any fixed ε 1 > ε , there is a sequence ( A N ) N ≥ 1 of N -sample algorithms outputting a i ∗ such that for any fixed reservoir distribution µ with µ ∗ > ε ,

We emphasize that the rate (3.1) is optimal in all cases since the lower bound of Theorem 3.2 is for an easier problem. The first formulation above may be the most principled choice. The idea is that an averaged quantile depends continuously on µ , and can in fact be estimated by applying Proposition A.1 for several pairs ( η 1 , η 2 ) and computing a Riemann sum. The second formulation requires only the mild condition that α ≥ 1+ ε 2 and uses monotonicity of c α,α -ε on this set. (In other words, if the average reward values p appearing in (B.1), (B.2) were larger than 0 . 5 , there would be no counterexample.) The third formulation allows us to almost send η all the way down to 0 . It uses the fact that

for some η ′ = η ′ ( µ, ε 1 , ε ) > 0 . These results show that (3.1) is achievable even without knowledge of α , up to a choice of technical modification to sidestep the counterexample discussed above.

Remark B.1. In fact uniformity in ( α, β ) holds in the following sense. For any sequence ( α N , β N ) N ≥ 1 of pairs with min ( β N , α N -β N , 1 -α N ) uniformly bounded below, there is a sequence ( A N ) N ≥ 1 of N -sample algorithms such that for any η ∈ (0 , 1) and any sequence of reservoir distributions µ N with G -1 µ N (1 -η ) ≥ α N ,

This can be shown identically to Theorem 3.1, though we don't give the proof in this generality. It is useful for the reduction arguments in Theorems B.1, B.2, and B.3.

B.1 Preparation for the Proof

Here we prove Theorem 3.2. For any α, β, η, ϱ > 0 we construct a reservoir µ = µ α,β,η,ϱ such that

holds for any sequence of N -sample algorithms A N , and where lim ϱ → 0 λ ( ϱ ) = 0 for fixed α, β, η .

B.2 Admissible Reservoirs and Bayesian Perspective

In proving Theorem 3.2, we will use reservoir distributions µ of a specific form. Namely, we require each µ to be supported on an interval [ γ, γ ] , where

In fact we define γ, γ explicitly (recall that ϱ > 0 is a small constant which we eventually send to 0 ) by

Wesay µ is ( γ, γ, f, f ) admissible if µ has density µ ( dx ) = f ( x ) dx for a Borel measurable function f and satisfies for constants 0 < f < f < ∞ ,

Towards proving Theorem 3.2, we fix throughout this section some ( γ, γ, f, f ) admissible µ such that G -1 µ ( α ) = η holds, for appropriate constants ( f, f ) depending only on ( η, ε, α, β, γ, γ ) . It is easy to see that this is always possible.

An admissible µ is roughly comparable to the uniform distribution on an interval. Using admissible reservoirs gives each a i the potential to slowly degrade in observed quality over time. We remark that while it is more convenient to work with reservoirs supported away from the boundaries, i.e. in [ γ, γ ] ⊆ (0 , 1) , we do not expect this to be essential.

It will be helpful throughout this section to take a Bayesian point of view. We treat µ N as known to A N , since A N is in fact allowed to depend on µ N . Thus at each time t , each p i has a posterior probability distribution which we denote by µ i,t . Note that each µ i,t depends only on ( n i,t , ˆ p i,t ) and is initialized at µ i, 0 = µ . We denote by

the sequence of posterior distributions µ i,t . Since arms are independent, µ t is the full timet posterior of the algorithm.

B.3 Batched Algorithms and Adversaries

In pure exploration problems, it is possible to significantly simplify the structure of any algorithm at the cost of a small multiplicative increase in the sample complexity. We carry this out using the notion of a batch-compressed algorithm.

Definition B.1. Given an increasing sequence B = ( b 1 , b 2 , . . . ) of positive integers, an algorithm A is B -batch-compressed if A can only act by increasing the number of times n i that a i has been sampled from b k to b k +1 , so that n i ∈ B holds at all times. B is ϱ -slowly increasing if

Finally if A is B -batch-compressed and B is ϱ -slowly increasing, we say that A is ϱ -batchcompressed.

Unlike the batched algorithms studied in [PRCS16, GHRZ19], batch-compression is only important for us as an analysis technique. Indeed the following proposition shows that it does not fundamentally affect pure exploration algorithms.

Proposition B.2. If B is ϱ -slowly increasing, then for any N -sample algorithm A , there exists an B -batch-compressed ⌊ N (1 + ϱ ) ⌋ algorithm A ′ with the same output.

Proof. We show how to simulate A using the B -batch-compressed A ′ , assuming that the sequence of rewards for each a i is fixed. Each time A samples arm i for the n i = ( a k + 1) -st time for a k ∈ A , A ′ samples arm i until n i = a k +1 . Then A ′ has all the information of A at all times, hence can simulate the behavior and output of A . Moreover by the definition of ϱ -slowly increasing, the sample complexity of A ′ is larger than that of A by at most a factor (1 + ϱ ) .

We will use the above with ϱ → 0 slowly as N →∞ . Then the sample complexity increase 1 + ϱ is absorbed into the 1 + o (1) factor in Theorem 3.2. As a result it suffices to establish (B.7) under the additional assumption that A N is ϱ -batch-compressed.

B.4 Fisher Information Distance

Determining the tight constant c α,β requires significant care. In particular the adversary must decrease the empirical average rewards ˆ p i,t at a precise rate depending on n i,t . This rate turns out to involve the Fisher information distance . For a, b ∈ [0 , 1] we define the Fisher information distance d F ( a, b ) between a and b to be

This agrees with the more general Fisher information metric when each a ∈ [0 , 1] is identified with the corresponding Bernoulli distribution. We refer the reader to [Nie20] for a survey on information geometry. In short, the Fisher information yields a natural Riemannian metric on families of probability distributions which are parametrized by smooth manifolds. However we will use only elementary properties of d F .

We parametrize [0 , 1] using the function θ : [0 , 1] → [0 , π ] defined by

In particular,

and so d F (0 , 1) = π . The main property of θ that we will use is the resulting differential equation

In our case, θ -1 parametrizes a 'constant speed' path through the space of Bernoulli variables, viewing the Fisher information. Correspondingly, our adversary will ensure that θ (ˆ p i ( n i,t )) decreases linearly in log( n i,t ) .

B.5 Preliminary Lemmas from Moderate Deviations

Recall that for positive integers a and b , the Beta( a, b ) distribution has probability density function

for x ∈ [0 , 1] . We now recall a moderate deviations principle for the binomial distribution and a central limit theorem for the beta distribution.

Lemma 4 (Theorem 2.2 in [DA92]) . For any 0 < q < q < 1 and constant ϱ > 0 there exists ∆ 0 ( q, q, ϱ ) and M 0 ( q, q, ϱ ) such that the following holds for all p ∈ [ q, q ] . For n ≥ n 0 ( q, q, ϱ ) sufficiently large and any 1 ∆ 0 √ n ≤ ∆ ≤ ∆ 0 we have

Lemma 5 (Lemma A.1 in [MNS16]) . Let { a n } n ≥ n 0 be a sequence satisfying

Then the Beta( n -a n +1 , a n +1) distribution on [0 , 1] obeys a central limit theorem with mean a n n and standard deviation √ ( a n /n )(1 -( a n /n )) n in the sense that for any bounded sequence ( w n ) n ≥ n 0 of real numbers and with Φ the normal CDF,

In the next two lemmas, we lower bound the probability that ˆ p i,t changes significantly when the number n i,t of samples for a i increases by a factor (1 + ϱ ) .

Lemma 6. Assume µ is ( γ, γ, f, f ) -admissible. Suppose that arm i 's average reward ˆ p i,t after n = n i,t samples satisfies

Then for n ≥ C ( γ, γ, f, f, β ) sufficiently large,

Proof. Let R i,t = n ˆ p i,t be the total reward from arm i so far. The posterior distribution µ i,t for p i takes the form

For x ∈ [ γ, γ ] we estimate

The right-hand side is the density of a beta variable with parameters ( R i,t +1 , n -R i,t +1) . We conclude that

For n sufficiently large, it follows from Lemma 5 and (B.12) that

Therefore P µ i,t [ p i ≤ ˆ p i,t ] ≥ 1 3 , proving (B.13).

Lemma 7. Assume µ is ( γ, γ, f, f ) -admissible and that (B.12) holds. For n = n i,t , let ˜ n ≥ 1 satisfy | ˜ n -ϱn | ≤ 2 . Let

be the average reward from the ( n +1) -th through ( n + ˜ n ) -th samples of arm i . Then as n →∞ , for any sequence ∆ n = Θ(1 / log n ) ,

Proof. Stochastic monotonicity implies that

is a decreasing function of p ∈ [0 , 1] . Combining with Lemma 6, it follows that

Since θ is smooth with smooth inverse on [ γ, γ ] and ∆ n ≤ o n (1) , we have

The result now follows from Lemma 4, where we absorb the factor f/ (3 f ) into the o n (1) .

B.6 Proof of Theorem 3.2

Recall the definition (B.8) of γ and γ . We require A to be B -batch-compressed for B = B ( N,ϱ ) containing:

All positive integers at most N 2 ϱ .

All positive multiples of ⌊ N ϱ ⌋ at most N 6 ϱ .

Integers of the form ⌊ N 6 ϱ (1 + ϱ ) j ⌋ for j ≥ 0 .

It is easy to see that B thus defined is ϱ -slowly increasing for any ϱ > 0 and N sufficiently large. We denote b k = ⌊ N 6 ϱ (1 + ϱ ) k ⌋ so that | b k +1 -(1 + ϱ ) b k | ≤ 2 . (This choice of indexing differs from that of Definition B.1, which will not be used in the sequel.)

We next construct our randomness distorting adversary A = A ( N,ϱ ) . For each arm i , the adversary A acts as follows depending on the current number of samples n i,t .

If n i,t ≤ N 2 ϱ , then A does nothing.

When N 2 ϱ ≤ n i,t < N 6 ϱ increases by N ϱ , A declares that the average reward of this batch of N ϱ samples is at most γ -N -ϱ .

When n increases from b ≥ N 6 ϱ to b

i,t k k +1 : (a) If ˆ p i ( b k ) > β holds, then A declares that

(b) If ˆ p i ( b k ) ≤ β holds, then A declares that

When the A chooses the arm a i ∗ to output, A declares that p i ∗ < β .

Due to step 4, the declarations made by A ensure that p i ∗ < β . Recalling Lemma 4 and Proposition B.2, it remains to show the upper bound

for a constant C ∗ = C ∗ ( γ, γ, f, f, β, α ) independent of ϱ (and N ). We show this bound in several parts. Recalling (3.4), we refer to the cost of a step above as the contribution to Cost from the corresponding declarations by A . The most important parts are Lemmas 10 and 11, which bound the cost of the main step 3a and form the dominant contribution to Cost . Note that throughout the analysis below, all cost upper bounds hold almost surely and we assume that all of A 's declarations hold true .

Lemma 8. The total cost from step 2 is at most C ∗ N 1 -ϱ , for N ≥ C ( γ, γ, f, f, β, α, ϱ ) sufficiently large.

Proof. The probability for each such declaration by A is at least

since p i ≤ γ almost surely. Recall that a Bin( N 2 ϱ , γ ) random variable obeys a central limit theorem centered at γN 2 ϱ with standard deviation at least C ( γ ) N ϱ . Therefore the probability in (B.16) is at least 1 3 for N is sufficiently large depending on ϱ . Hence each such declaration costs at most C ∗ for N sufficiently large. Moreover such declarations can occur only N 1 -ϱ times because each one involves N ϱ samples, and the base algorithm A is an N -sample algorithm. This completes the proof.

Lemma 9. The total cost from step 3b is at most C ∗ N 1 -6 ϱ as long as N ≥ C ( γ, γ, f, f, ϱ ) .

Proof. It suffices to show that the cost per step 3b declaration is at most C ∗ . This follows from (B.13) and stochastic monotonicity.

Lemma 10. The total cost from step 3a is at most

Proof. We claim that the cost from a single instance of step 3a when increasing from b k to b k +1 samples is at most

This implies the desired result since A N is an N -sample algorithm. Taking ∆ = (1 + 10 ϱ ) d F ( α, β ) / log( N ) in Lemma 7, we find that the declared event has probability at least

This implies the desired claim and completes the proof.

Lemma 11. For any a i sampled b 0 = ⌊ N 6 ϱ ⌋ times, ˆ p i ( b 0 ) ≤ γ .

Proof. By definition of A ,

In the last step we used the fact that

This completes the proof.

We now combine the lemmas above to conclude Theorem 3.1 via (B.7).

Proof of Theorem 3.1. Let C ′ ∗ be a larger constant depending on the same parameters. Then by Lemmas 8, 9, and 13, the total cost from Steps 2, 3b, 4 combines to C ′ ∗ N 1 -ϱ ) ≤ o N ( N/ log 2 N ) . The main cost contribution of

comes from Lemma 10, and all other terms are of strictly smaller order. We have thus constructed a reservoir sequence ( µ N ( ϱ )) N ≥ 1 satisfying (B.7) for arbitrary ϱ > 0 , completing the proof.

C An Optimal Algorithm with Fixed Budget

Here we provide an asymptotically optimal algorithm which establishes Theorems B.1, B.2, and B.3. In the next subsection in which we show how to reduce the other results mentioned to Theorem 3.1 (in which α is given) using Proposition A.1. Our main focus will then be to prove Theorem 3.1.

We will fix ϱ > 0 small and construct a sequence of N -sample algorithms ( A ( N,ϱ )) satisfying the slightly relaxed guarantee

for a (possibly different) function λ satisfying lim ϱ → 0 λ ( ϱ ) = 0 (for fixed α, β, η ). Here ( µ N ) N ≥ 1 is any sequence of reservoir distributions satisfying G -1 µ N (1 -η ) = α . An elementary diagonalization argument then implies Theorem 3.1. Thus it suffices to construct algorithms satisfying (C.1) for any desired ϱ > 0 .

for any ϱ > 0 if N is sufficiently large.

Lemma 12. For ϱ ∈ (0 , 1 / 100) , if n i,t ≥ N 1 -ϱ and the declarations of A hold, then ˆ p i,t ≤ β .

Proof. We analyze the rate at which the adversary forces θ (ˆ p i ( b k )) to decrease. From (B.15) and (11) it follows that for k with b k ≥ N 1 -ϱ , we have

Here we used the fact that log(1 + ϱ ) ≤ ϱ and (1 + 10 ϱ )(1 -8 ϱ ) ≥ 1 for ϱ ∈ (0 , 1 / 100) . Since θ is increasing, this shows that ˆ p i,t = ˆ p i ( b k ) < β for b k ≥ N 1 -ϱ , completing the proof.

Lemma 13. The cost from step 4 is at most C ∗ ( N 1 -ϱ +1 ) .

Proof. First, if ˆ p i ∗ ,N ≤ β then the cost from step 4 is at most C ∗ . On the other hand if ˆ p i ∗ ,N > β , then Lemma 11 implies n i ∗ ,N ≤ N 1 -ϱ . Since the prior µ is supported in [ γ, γ ] , the likelihood ratio of updates from N 1 -ϱ samples is almost surely bounded by e C ∗ N 1 -ϱ . Therefore

C.1 Reduction to Known α

We explain why Theorems B.1, B.2, and B.3 all follow from Theorem 3.1 (more precisely, the uniform statement given in Remark B.1). We begin with Theorem B.1, where

It is easy to see that η ( j +1) -η ( j ) ≤ η ( j ) for all j . Wenext apply Alg. 4 on ( η ( j ) , η ( j +1) -η ( j ) , ε ′ , δ ′ ) for 0 ≤ j ≤ J -1 , with:

Let J = ⌈ 6 ε ( η 1 -η 2 ) ⌉ and define

This requires sample complexity

Let ˆ α j be the resulting output. With probability 1 -Jδ , we have for each 0 ≤ j ≤ J -1 ,

Note that the function G -1 µ is increasing and [0 , 1] -valued. Therefore if (C.3) holds for each j , then

Therefore the estimator satisfies

Finally, c α,α -ε ≤ π < 10 for any α, ε ∈ [0 , 1] (see (B.10)). Therefore the δ ′ = e -10 N log 2 ( N ) failure probability above has a negligible contribution in Theorem B.1. It follows that applying Theorem 3.1 with α = ˆ α A as above and N ′ = N -N A implies Theorem B.1.

We now turn to Theorem B.2, where µ N is required to satisfy G -1 µ N (1 -η ) ≥ 1+ ε 2 . We run Alg. 4 with parameters

The sample complexity N B again satisfies N B ≤ o ( N ) exactly as in (C.2). Let ˆ α B + ε ′ be the resulting output. Then with probability at least 1 -e -10 N log 2 ( N ) ,

and so with ε ′′ = ε -2 ε ′ , we have

Moreover, also with probability at least 1 -e -10 N log 2 ( N ) ,

It follows that applying the algorithm of Theorem 3.1 with

suffices to recover Theorem B.2, since η 2 and ε ′ tend to 0 as N → ∞ . As in our discussion of Theorem B.1 above, the failure probability e -10 N log 2 ( N ) is negligible compared to the relevant rate in Theorem B.2.

Finally, Theorem B.3 relies on the simple fact

Recall that µ ∗ ∈ [0 , 1] denotes the maximum value in the support of µ . We run Alg. 4 on ( η 1 , η 2 , ε ′ , δ ′ ) where:

It follows from Proposition A.1 that the resulting output ˆ α C + ε 1 -ε 2 is computed using O ( N log log( N ) log( N ) ) ≤ o ( N ) samples as in the previous cases. Moreover for N sufficiently large:

Since ε 1 > ε , this means for N ≥ N 0 ( µ, c ′ , . . . ) large enough,

Note that Alg. 4 also ensures that with probability 1 -e -10 N log 2 ( N ) ,

for some η ′ ( µ, ε 1 , ε ) > 0 . It follows that applying Theorem 3.1 with

implies Theorem B.3.

C.2 The Fixed Budget Algorithm

We now present Algorithm 3 for the fixed budget problem (recall the informal discussion in Section 3). Algorithm 3 studies one arm a i at a time, moving to a i +1 if a i is rejected. Similarly to the previous section, some details are needed while n t,i is small, since large deviation asymptotics may not have kicked in yet. As explained at the start of the section, we choose a small constant ϱ > 0 . In fact, we will eventually choose small constants

which all tend to 0 as ϱ → 0 . These constants will be defined throughout the proof. More formally, these values can be obtained by choosing ϱ 5 > 0 arbitrarily small, then ϱ 4 > 0 sufficiently small depending on ϱ 5 , and so on.

Algorithm 3 operates in a batch-compressed way, for a sequence ( b 1 , b 2 , . . . ) defined as follows:

Note in particular that b k 0 ≥ log 4 ( N ) . We denote by ˆ p i,t the empirical average reward collected by a i from its first t samples.

The role of the values b j is as follows. When an arm a i reaches b k samples for some k ≥ 0 , it is checked for possible rejection by comparing its empirical average reward to the threshold τ k . Algorithm 3 rejects arm i and moves to arm a i +1 if the empirical average ˆ p i,b k of arm a i drops below a moving threshold τ k . The threshold τ k begins close to α and gradually decreases until reaching β + ϱ by the time τ k ≥ Ω( N ) .

So for, our informal description of Alg. 3 also applies to the algorithm proposed in [GM20]. We now highlight two important differences. The first is that our algorithm is defined more carefully during the 'early' phases when an arm has been sampled at most N O ( ϱ ) times. This is crucial for carrying out a rigorous analysis. The second difference is that in the main phase, we increase the sample size for a given arm in powers of 1+ ϱ rather than powers of 2 , and also move the rejection thresholds τ k based on the Fisher information distance via the function θ . The latter ingredients allow us to obtain the optimal constant factor.

We begin the analysis of Alg 3 by proving Lemma 3.

Proof of Lemma 3. Let M j = ∏ 1 ≤ i ≤ j Y i and observe that M c j is a positive supermartingale with M 0 = 0 . The result follows by Doob's maximal inequality.

We will apply Lemma 3 in the following way. Let X i be the number of samples used by arm a i before rejection, and I i ∈ { 0 , 1 } be the indicator of the event that a i is ever rejected, even if Algorithm 3 were to continue past time N and sample arm i an infinite number of times. We set

With M defined from ( Y i ) i ≥ 1 as in Lemma 3, it follows that log( M ) is at most the amount of time spent on eventual rejections before the first eventually accepted arm. Therefore if log( M ) ≤ N (1 -ϱ ) , we conclude that the last arm to be studied was sampled at least Nϱ times. Since it was not rejected during that time, we can conclude this arm has p i ≥ β with probability 1 -e -Ω ϱ ( N ) . The main contribution to the failure probability of Algorithm 3 comes from the event { M ≥ A } above, for suitable A . Correspondingly, the main work will be to verify E [ Y c i ] ≤ 1 for suitable c .

Note that Y i ∈ { 0 } ∪ [1 , ∞ ) almost surely for each i . Therefore a necessary first step in showing E [ Y c i ] ≤ 1 is to lower bound P [ Y i = 0] , the probability that Algorithm 3 never rejects a i . We now give a sufficient lower bound from the event p i ≥ α .

Proposition C.1. Let x 1 , x 2 , . . . be an i.i.d. Bernoulli ( p ) sequence for p ≥ α , and let S k = ∑ k i =1 x i and set

Then S ≥ α -ϱ holds with probability at least c ( α, ϱ ) > 0 . Thus E [ I i ] ≤ 1 -c ( α, ϱ ) .

Proof. Since the probability that S ≥ α -ϱ is increasing in p it suffices to take p = α and show the probability is positive for any ϱ > 0 . Assume not. Then by restarting the indexing every time S k ≤ k ( α -ϱ ) holds, we find that

This contradicts the strong law of large numbers, thus completing the proof of the first assertion. The second assertion follows since if S k /k ≥ α -ϱ for all k where x 1 , . . . are the rewards of arm i , then arm i will never be rejected by Algorithm 3.

Based on Proposition C.1 above, to show

(which is essentially what we want in light of Lemma 3), it suffices to show that

We let I t i = I i · 1 X i = t be the event that arm i was rejected after exactly t steps. Since Alg 3 can only reject after b j samples, we have

We use this to break the left-hand side of (C.5) into three separate parts and estimate the parts separately. The parts correspond to b 0 , b 1 through b k 0 , and b k 0 +1 onward. The first two parts are easier and handled in Subsection C.3 below. The final term is the main contribution and is handled in Subsection C.4.

C.3 Analysis of Algorithm 3 in the Small and Medium Sample Phases

Proposition C.2 bounds the contribution to (C.5) from the small sample phase , i.e. the first rejection condition in line 7 of Alg 3.

Proposition C.2. For any α, ϱ there is ϱ 1 > 0 sufficiently small that with b 0 as defined above, and with N sufficiently large,

Proof. It suffices to observe that for fixed α, ϱ and ϱ 1 small and N sufficiently large, we have

Proposition C.3 bounds the contribution to (C.5) from the medium sample phase , i.e. the second rejection condition in line 12 of Alg 3.

Proposition C.3. For any α, ϱ, ϱ 1 and for N sufficiently large,

Proof. The event I b k i requires | ˆ p i,b k -ˆ p i,b k -1 | ≥ 1 √ log N . Hence by a standard Chernoff estimate, regardless of the true reward probability p i ,

Since by construction b 0 ≥ ϱ 1 log 2 N , we have

Since k 0 ≤ O (log N ) , summing gives the desired conclusion.

Propositions C.2 and C.3 imply that the total contribution from rejections in the small and medium sample phases is at most c ( α, ϱ ) / 2 . It remains to analyze the large sample phase in the following subsection.

C.4 Analysis of Algorithm 3 in the Large Sample Phase

Similarly to the previous section, the main part of the analysis concerns the large sample phases b k 0 + j for j ≥ 1 . Our goal is to precisely estimate the rejection probability at each time b k 0 + j . Note that these estimates should not depend on the true average rewards p i .

Our approach is based on exchangeability and avoids any consideration of p i . For a given value j and a large constant L = L ( ϱ ) , consider the sequence of times

and the associated sequence of empirical average rewards

It follows from the algorithm description that for I b k 0 + j i to occur, we must have

This is clear for j > L , but it holds also for 0 ≤ j ≤ L as for N sufficiently large,

By exchangeability, conditioned on the future values ˆ p i,b k 0 + j , . . . , ˆ p i,b k 0 + j -ℓ the law of ˆ p i,b k 0 + j -ℓ -1 depends only on ˆ p i,b k 0 + j -ℓ and is given explicitly by a hypergeometric variable. Recalling that

R i,t = n i,t ˆ p i,t is the total reward from the first n i,t samples of arm i , R i,b k 0 + j -ℓ -1 has hypergeometric conditional law given by:

We will refer to this as the HyperGeom ( b k 0 + j -ℓ , b k 0 + j -ℓ -1 , R k 0 + j -ℓ ) distribution. Importantly, this distribution is independent of µ . We exploit this below to control the probability of a given sequence ( ˆ p i,b k 0 + j -L , ˆ p i,b k 0 + j -L +1 , . . . , ˆ p i,b k 0 + j ) of empirical average rewards. The following useful result states that hypergeometric variables automatically inherit tail bounds from the corresponding binomial random variables.

Lemma 1 ([LP14, Hoe94]) . Fix non-negative integers A ≥ B,C and let X ∼ HyperGeom( A,B,C ) and Y ∼ Bin( B,C/A ) . Then for any convex function f : R → R ,

Lemma 2. For any 0 < q < q < 1 and constants ϱ > 0 there exists ∆ 0 ( q, q, ϱ ) and N 0 ( q, q, ϱ ) such that the following holds for all p ∈ [ q, q ] . For n ≥ n 0 sufficiently large and 1 ∆ 0 √ n ≤ ∆ ≤ ∆ 0 ,

Proof. The corresponding binomial result Lemma 4 is proved in Theorem 2.2 in [DA92] by upper bounding an exponential moment. The same proof applies here by Lemma 1.

It will be convenient to define a restricted set of good sequences ( q L , q L -1 , . . . , q 0 ) . These satisfy the key properties of empirical average reward sequences (C.6) for which I b k 0 + j i holds. We say such a length L +1 sequence is good if the following conditions are satisfied:

For each 1 ≤ ℓ ≤ L :

The third condition above is necessary for I b k 0 + j ,i i = 1 , and these together imply the first condition. Indeed for fixed q, q and small ϱ ∈ (0 , 1 / 10) one always has

for large enough N and any j . Hence it suffices to take q = β (1 -2 ϱ ) L and q = 1 -(1 -α )(1 -2 ϱ ) L . With this choice, if

is not good and I b k 0 + j i = 1 , then the second condition must be the only violated one. The following easy lemma controls the failure probability of the second condition. Recall from (C.8) that conditioning on ˆ p i,b k 0 + j determines the joint conditional law of the previous conditional rewards, regardless of µ .

Lemma 3. All sequences violating only the second condition (C.9) above have probability at most

even after conditioning on an arbitrary value for ˆ p i,b k 0 + j .

Proof. The claim follows by an elementary Chernoff estimate for hypergeometric variables, which hold just as for binomial variables by Lemma 1. Indeed the assumption implies that some adjacent difference | ˆ p i,b k 0 + j -ℓ -ˆ p i,b k 0 + j -ℓ +1 | has size Ω(1 / √ log N ) . (Note for applying the Chernoff bound that L is a constant independent of N , and so b k 0 + j -L ≥ Ω L,ϱ ( b k 0 + j ) . )

We now focus on upper-bounding the probability of any good sequence ( q L , . . . , q 0 ) appearing, conditionally on q 0 .

Lemma 4. For any good sequence ( q L , q L -1 , . . . , q 0 ) and j ≥ 0 ,

Proof. It suffices to show that

This follows by applying Lemma 2 to the hypergeometric random variable

The fact that

leads to the factor of ϱ in the denominator of the desired result.

Lemma 5. For fixed problem parameters and N large, any good sequence ( q L , . . . , q 0 ) satisfies

Proof. Recall that θ ′ ( q ) = 1 √ q (1 -q ) and that θ is smooth on [ q, q ] ⊆ (0 , 1) . By Item 2 above, all q ℓ are within o N (1) of each other, so the result follows from the inverse function theorem. (Notice that the factor (1 -ϱ 2 ) changed to (1 -2 ϱ 2 ) above.)

Lemma 6. For 1 ≤ m ≤ L and any good sequence ( q L , . . . , q 0 ) , we have

Proof. The result follows from Lemma 5 and Cauchy-Schwarz in the form

Lemma 7. For any good sequence ( q L , . . . , q 0 ) and j ≥ 0 , we have

Proof. We break the sum into parts and apply Lemma 6 to each one. We have:

For L = L ( ϱ ) = O ( ϱ -1 log( ϱ -1 ) ) sufficiently large,

Substituting and recalling that ϱ ≪ ϱ 2 completes the proof.

Combining with Lemma 4 yields the second inequality below (the first is trivial).

Corollary C.4. For any µ and q 0 , we have

Lemma 8. Let j 0 be the largest j such that b k 0 + j ≤ N . Then for N sufficiently large,

Proof. Recall that c α,β = d F ( α,β ) 2 2 , and observe that the number of total sequences ( q L , . . . , q 0 ) ∈ [0 , 1] L +1 with b k 0 + j + ℓ q ℓ ∈ Z is at most N L +1 for each j ≤ j 0 . Combining Lemma 3 and Corollary C.4 and noting that the latter always gives the main contribution, we find for each j ≤ j 0 ,

so long as ϱ 3 is chosen so that ϱ 3 ≫ max( ϱ, ϱ 2 ) . In the last line we used the fact that b k 0 + j ≥ b k 0 ≥ log 4 N to absorb the factor N L +1 ≤ e ϱ log 3 / 2 N for large N . Summing over j gives the

desired result, since for ϱ 4 = Ω( ϱ 3 ) and N sufficiently large,

We now use Lemma 3 to conclude.

Proof that Algorithm 3 achieves the guarantee of Theorem 3.1. By combining Lemma 8 with the previous Propositions C.2 and C.3, it follows that

Lemma 3 now implies that the total amount of time spent on eventually rejected arms is at most N (1 -ϱ ) with probability

On this event, the output arm i ∗ satisfies n i ∗ ,N ≥ ϱN by definition. Since i ∗ was not rejected, for j 1 be the largest value such b k 0 + j 1 ≤ ϱN we have

The probability for this to hold if p i ≤ β is at most e -Ω ϱ ( N ) . Altogether we find that

for ϱ 5 arbitrarily small. This concludes the analysis of Algorithm 3 (since the last error term is negligible).

C.5 Finding Many Good Arms with a Fixed Budget

In this final subsection we observe that Algorithm 3 can be modified to output as many as log N distinct arms each of which satisfies the same ( η, ε, δ ) -PAC guarantee 2 , with no degradation in the asymptotic failure probability. With other parameters fixed, we denote the N -sample version of Algorithm 3 by A N to emphasize the dependence on N . In particular, N both equals the number of steps in A N and appears (via its logarithm) in the description of A N 's individual steps.

Let ˜ N = N + ⌈ 2 N log 1 / 2 ( N ) ⌉ . We consider a modified algorithm ˜ A ˜ N which mimicks the behavior of A N with two changes:

˜ A ˜ N is a ˜ N -sample algorithm.

If an arm a i has not yet been rejected after M = ⌈ N/ log 3 / 2 ( N ) ⌉ samples, then ˜ A ˜ N accepts a i and continues to a i +1 . In particular, ˜ A ˜ N may accept several arms instead of just one.

Theorem C.9. With probability 1 -exp ( -( c α,β -ϱ 5 -o N (1)) N log 2 N ) , ˜ A ˜ N accepts at least log( N ) distinct arms a i , all of which satisfy p i ≥ β .

The change from N to ˜ N is almost irrelevant in the actual statement of Theorem C.9 since log( N ) ≥ log( ˜ N ) -o N (1) . In particular, ˜ A ˜ N is a ˜ N -sample algorithm which outputs at least log( ˜ N ) -1 arms with probability 1 -exp ( -( c α,β -ϱ 5 -o ˜ N (1)) ˜ N log 2 ˜ N ) . It is certainly not really necessary to use the value log( N ) rather than log( ˜ N ) to describe the individual steps taken by ˜ A ˜ N . However introducing ˜ N streamlines the proof below by letting us treat A N as a blackbox.

Proof. To show that all accepted arms a i satisfy p i ≥ β with sufficiently high probability, it suffices to consider (C.10) with the final term replaced by e -Ω ϱ ( N/ log 3 / 2 ( N )) . In particular, observe that the main term does not change, even after multiplying the failure probability by O ( log 3 / 2 ( N ) ) (the maximum possible number of arms accepted by ˜ A ˜ N . Thus we focus on showing that ˜ A ˜ N outputs at least log( N ) arms with high probability.

Consider yet another N -sample algorithm ̂ A N which deletes each arm independently with probability 1 /N and follows A N on the set of non-deleted arms in order of increasing index. (Like A N , ̂ A N never accepts arms before time N .) We simulate ˜ A ˜ N and ̂ A N on the same reward sequences, i.e. we couple them so that the t -th sample of arm a i always gives the same result for each ( t, i ) . We claim that in this coupling, conditioned on ˜ A ˜ N failing to accept log( N ) arms within the first ˜ N samples, ̂ A N has probability Ω( N -log( N ) ) to fail (i.e. output a i with p i < β ) when run for N samples.

First let us assume the claim and deduce Theorem C.9. Denote by p ( N ) the probability for A N to fail. Note that ̂ A N has the same failure probability p ( N ) , having in fact the same behavior as A N in distribution (as the set of deleted arms is independent of everything else). Moreover let ˜ p ( ˜ N,k ) denote the probability that ˜ A ˜ N fails to accept at least k arms. The claim above implies that

It remains to prove the above claim. Let us say the infinite i.i.d. reward sequence ( r i,n ) n ≥ 1 of arm a i is acceptable if A N would not reject a i within M samples, i.e. ˜ A ˜ N will either accept a i or run out of samples before doing so. We take the point of view that each a i is either acceptable or not (by randomly fixing the reward sequences at the start). Then with probability Ω( N -log( N ) ) , the first log( N ) acceptable arms are skipped by ̂ A , and the first ˆ N unacceptable arms are not skipped. On this event, the first ˆ N -M ≥ N samples obtained by ̂ A N , i.e. all N of its samples, are drawn from unacceptable arms. On this event, ̂ A N fails with constant probability, which establishes the claim and completes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a problem of pure exploration in the infinite-armed bandit setting : in this problem, each arm has mean reward $p_i$ drawn i.i.d. from a distribution with quantile function $G^{-1}$, and the aim is to output an arm $a_{i*}$ after sampling $N$ times an action, such that $\\mathbb{P}\\left(p_{i^*} \\geq G^{-1}(1 - \\eta) - \\epsilon\\right)\\geq 1 - \\delta$ for some given $\\eta$, $\\epsilon$, and $\\delta$. The authors first propose an algorithm for the setting where $G^{-1}(1 - \\eta)$ is unknown that is almost optimal in terms of expected sample complexity $\\mathbb{E}[N]$, and show that is satisfies the $(\\eta, \\epsilon, \\delta)$-PAC constraint and derive an upper bound on the sample complexity. In a second part of the article, they consider the fixed budget setting when $G^{-1}(1 - \\eta)$ is unknown, and propose an asymptotically optimal algorithm (i.e. an algorithm with asymptotically optimal failure rate). They derive upper bound on its failure rate, and obtain a lower bound for the failure rate of any algorithm.\n\nSTRENGTHS:\nOriginality : The author consider in this paper a vary natural problem.\n\nClarity : This paper is very well written and easy to follow. The literature review is thorough and well presented, the algorithms are rather well explained. The comments on the results are interesting, and the authors manage to highlight the different aspects of the problem.\n\nQuality : This paper provides interesting theoretical results.\n\nSignificance : This article makes significant contributions to a natural problem. They improve on existing results, and close open questions in both settings by characterizing optimal sample complexity and optimal failure rate in the fixed confidence and fixed budget setting.\n\nWEAKNESSES:\nResults are sometimes lacking in clarity, and sometimes even in rigor. Bellow are some examples :\n\nThe algorithms make use of some undefined constant $C$ (the same holds for the constants $\\rho, \\rho_1, \\rho_2$). While I agree that the results are true for some specific constant, they are not true in general. The authors should define these constants more carefully, and propagate the numerical constants appearing in their proof instead of using symbols $\\Omega$. The authors make heavy use of these symbols  in their proof, making them harder to read, and less rigourous.\n\nI find the \"proof\" of Lemma 4 not convincing.\n\nQUESTIONS:\nPlease define explicitly the constant to use in the algorithms so that your claims hold.\n\nPlease postpone the proof of Lemma 4 to the Appendix and develop your arguments further.\n\nFor the sake of completeness, please recall the results that you are using (Chernoff bounds (A.1) and (A.2), Thm 4.5 in [MU17], Thm 1 in [GM14], etc).\n\nPlease refrain from using symbols as $\\Omega$ in the proofs, and use instead explicit numerical constants, so that the validity of your arguments is easier to assess.\n\nThere are somme remaining typos :\n- Theorem 1.2: the constant in the definition of $N$ should be $c_{\\alpha, \\beta}$\n- Algorithms 1, 2: $C$ is not defined.\n- Algorithm 3: $\\alpha$, $\\rho$, and maybe somme other parameters should be considered as input to the algorithm\n- line 167 : what is $\\eta_1$? Is it $\\eta$?\n- Algorithm 1 : why not define $K$ as $K = C\\log(1/\\delta)/\\eta$?\n- line 199: do you mean Theorem 2.1?\n- Appendix, line 474 : there is a \"with\" missing\n- Appendix, line 490 : \"qualitative\"\n- Appendix, Lemma 4: there are extra parentheses.\n- Appendix, line 621 : please define the event $E$\n\nI had trouble understanding the sentence in line 197 (I think the word therefore is misleading); could you please rephrase it?\n\nThe meaning of the number of \"batches\" is not very easy to understand. Could you please rephrase the corresponding discussions?\n\nIt would be nice to include an outline and a conclusion in this article.\n\nTheorem 2.1 and Algorithm 1 overlap, which is annoying. Please correct the layout.\n\nYou introduce constants $\\rho_1, \\rho_2$ which do not appear in Algorithm 3. Please correct that.\n\nLIMITATIONS:\nThe authors have not discussed limitations of their results.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers pure exploration problem in multi-armed bandits with infinite number of arms. Its goal is to return an arm with sufficiently high expected reward (within the top $\\eta$ region) with high probability (at least $1-\\delta$). A naive method could lead to a complexity upper bound of $O({\\log(1/\\eta\\delta)\\log(1/\\delta) \\over \\eta\\epsilon^2})$, where $\\epsilon$ is the error tolerance rate. The authors make some improvement, design a new learning algorithm, and show that it achieves a complexity upper bound of $O({\\log(1/\\eta)\\log(1/\\delta) \\over \\eta\\epsilon^2})$. They also consider the fixed budget case, and show that their proposed algorithm can achieve a failure probability approximately $\\exp(-N/\\log^2 N)$, where $N$ is the fixed budget.\n\nSTRENGTHS:\nThe model setting is well-motivated, and reducing one $\\log(1/\\delta)$ factor seems to be an important step in this kind of problems.\n\nWEAKNESSES:\nHere are some of my questions.\n\n1) An $(\\eta, \\epsilon, \\delta)$-PAC is defined as Eq. (1.1), i.e., $\\Pr[p_{i^*} \\ge G^{-1}(1-\\eta) -\\epsilon] \\ge 1 - \\delta$. I can understand that this is a traditional definition for $\\epsilon$-PAC in pure exploration problems, but I am still wondering what if we change this definition to be $\\Pr[p_{i^*} \\ge G^{-1}(1-\\eta - \\epsilon)] \\ge 1 - \\delta$. How would this change influence all the complexity bounds?\n\n2) In Theorem 1.2, does the $\\alpha \\ge \\eta$ mean $\\alpha \\ge G^{-1}(1-\\eta) $? Is the $c$ in equation before Eq. (1.2) the same as the $c_{\\alpha, \\beta}$ in Eq. (1.2)?\n\n3) I do not really understand why $\\eta$ does not appear (in the statement of line 75). In my opinion, a natural lower bound should be $(1-\\eta)^N$, since this is the probability that you try $N$ arms and no one satisfy the constraint that $p_i \\ge G^{-1}(1-\\eta)$. Is this hidden by the factor $c$ (or $c_{\\alpha, \\beta}$)?\n\n4) In Algorithm 1 line 2, why we are writing an $\\eta$ factor on both the numerator and the denominator?\n\n5) In Algorithm 2, each arm should be pulled for $O({\\log(1/\\eta\\delta) \\over \\epsilon^2})$ times, and try $O({\\log(1/\\delta) \\over \\eta})$ arms. This leads to a straightforward $O({\\log(1/\\eta\\delta)\\log(1/\\delta) \\over \\eta\\epsilon^2})$ complexity bound. I guess the reason that the complexity bound in Theorem 2.1 is $O({\\log(1/\\eta)\\log(1/\\delta) \\over \\eta\\epsilon^2})$ is because the expected number of tried arms (and then find one good arm to output) is about $O({1 \\over \\eta})$, but this seems not leading to the complexity bound in Theorem 2.1. Since  no proof sketch about this theorem is given in the main text, I think at least you need to explain the high-level ideas for this proof clear to make sure that the readers can understand why your theorem is true. Besides, is your complexity bound only an expected one or a high-probability one?\n\n6) In line 281, do you mean $\\log M \\le N(1-1/\\log N)$ instead of $M \\le N(1-1/\\log N)$?\n\nQUESTIONS:\nPlease see the above \"Weaknesses\"\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies pure exploration problem in bandits with infinite arms, where the mean reward of each arm is i.i.d. according to some unknown distribution. In fixed-confidence setting, this paper proposes an algorithm that nearly matches the optimal sample complexity. In fixed-budget setting, this paper proposes an asymptotic lower bound and proposes an algorithm that nearly matches it.\n\nSTRENGTHS:\n- The fixed-confidence algorithm is nearly optimal.\n- The lower bound proof for fixed-budget setting contains novel construction of adversaries.\n- The fixed-budget algorithm is asymptotically optimal and the limitation on the knowledge of $\\alpha$ is appropriately addressed.\n\nWEAKNESSES:\nIs that possible to empirically evaluate the proposed algorithms, especially the fixed-budget algorithm, and compare them with previous work?\n\n### Suggestions on Writing\n- line 213, \"known\" -> \"unknown\".\n- It may be better to mention the low probability of faliure when discussing Algorithm 2.\n- It may be better to replace $\\mathbb{P}$ in equations (3.1) and (3.2) by $\\mathbb{P}_{\\mu_N}$.\n- It seems $\\hat{p}_{i, b_k}$ has a slight abuse of notation compared to the definition in Section 1.2 since $b_k$ in Algorithm 3 is not the time index from $t=1$. Maybe some clarification can be added here.\n- What does $\\partial v$ mean in the definition 3.2??\n\nQUESTIONS:\n- Is that possible to build some connection with the instance-dependent fixed-budget complexity in pure exploration with finite arms?\n- Since $\\eta$ does not appear in the error probability of fixed-budget setting, can we just take $\\eta=0$?\n- Is that possible to intuitively explain why Algorithm 3 will be optimal?\n- Why does the ceiling function appear in $b_{k_0+j}$ but not in $b_k$?\n- Why will an adversary with smaller strength make the true failure probability of $\\mathcal{A}$ larger?\n\nLIMITATIONS:\nThe limitations (in particular, the requirement of the knowledge of $\\alpha$) has been adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**[Setting]** This paper studies best arm identification among countably infinite arms in both fixed confidence and fixed budget setting. The mean rewards for these arms are drawn iid from a distribution $\\mu$. Let $\\alpha$ be the $1 - \\eta$ percentile of $\\mu$. The goal is to find an arm with mean reward at least $\\alpha - \\epsilon$. Note that the algorithm knows $\\eta$ and not $\\alpha$.\n\n**[Algorithm - fixed confidence]** - In the fixed confidence setting, the authors propose an algorithm that identifies the required arm with high probability using $O(\\log(1/\\delta) \\log (1/\\eta) / \\eta \\epsilon^2)$ samples in expectation. This algorithm first tries to estimate $\\alpha$ by pulling $O(\\log (1/\\delta) / \\eta)$ arms $O(\\log(1/\\eta) / \\epsilon^2)$ times, and then uses the estimated value to find the target arm by pulling $O(1)$ arms (in expectation) $O(\\log (1 / \\eta \\delta) / \\epsilon^2)$ times.\n\n**[Guarantee - fixed confidence]** - The bound matches matches a known lower bound up to $\\log (1 / \\eta)$ factor. This is an improvement over previous works that include a $(\\log (1 / \\delta))^2$ term.\n\n**[Algorithm - fixed budget]** - The authors propose an algorithm that goes though arms in a sequence. For each arm $i$, samples are collected in batches until this arm is discarded because its mean falls ``reasonably'' below $\\alpha$. The threshold for discarding an arm keeps decreasing across batches, but always stays above $\\alpha - \\epsilon$. This ensures that arms with mean < $\\alpha - \\epsilon$ are discarded early on using few samples, but arms with mean > $\\alpha - \\epsilon$ have to increasingly pass a harder test to be retained. The last un-discarded arm is returned as the best arm.\n\n**[Guarantee - fixed confidence]** - The authors focus on the case where $\\delta \\rightarrow 0$ and derive matching lower and upper bounds showing that an optimal algorithm in the fixed budget setting needs $O(\\log (1/\\delta) (\\log \\log(1 / \\delta))^2 / c)$ samples, where $c$ is a problem dependent parameter.\n\nSTRENGTHS:\n**Originality** - The paper closes a gap (w.r.t $\\delta$) between the upper and lower bounds in the fixed confidence setting. This requires new ideas in the algorithm. It also proposes the first asymptotically optimal algorithm in the fixed budget setting.\n\n**Quality** - I must admit that I did not fully understand the results in the fixed budget setting. For the fixed confidence setting, the ideas appear technically sound to me, though I have a few questions listed later. This is a complete study in my opinion where the derived bounds are tight, and the claims are well supported with theoretical evidence.\n\n**Clarity** - The paper is positioned well with respect to other works. Most parts are very well written and organised (barring a few basic questions listed later).\n\n**Significance** - Given that the paper closes the gap between upper and lower bounds in both fixed confidence and fixed budget setting, it would be a valuable addition to the literature. It also provides a good starting point for further explorations like getting high probability bounds on the sample complexity in the fixed confidence setting.\n\nWEAKNESSES:\n**Quality** - \n\n1. As the authors have pointed out in L212-215, the derived bounds do not depend on $\\eta$ in the fixed confidence setting. However, the intuition for why this is the case is missing from the paper.\n\n2. L206 says that the case of known $\\alpha$ is the easier case. Remark 3.1 gives conditions under which one would not need $\\alpha$ in advance. However, it appears that Algorithm 3 always requires $\\alpha$ as input (see L6 for example). This must be clarified. \n\n\n**Clarity** - The paper can be made more accessible in many places:\n\n1. It is perhaps just me, but the sample complexity bounds for the fixed budget setting are a bit confusing. Isn't it true that people usually fix a budget $N$ and try to minimise the probability of error with this budget? What does the case $\\delta \\rightarrow 0$ mean then? Why is there a bound on $N$ again (which is supposed to be fixed)? \n\n2. The idea of batches in Remark 2.1 can be explained more clearly. In the worst case, Algorithm 2 does sample $O(\\log(1/\\delta))$ arms. Using batches probably ensures that only $O(1)$ arms are queried in expectation.\n\n3. The notation is confusing at several places. For example, $\\eta_1$ in L167 is undefined. L2 in Algorithm 1 uses $\\eta$ in both numerator and denominator. $\\eta_2$ in L184 should probably be explicitly defined.\n\n4. Should there be an $\\eta$ in the denominator inside the $\\log$ in L185? This will make $b_{2, i}\n = C \\log(1/\\delta) / \\eta^2$, which is different from the range of the for loop in Algorithm 2.\n\nQUESTIONS:\nPlease address points 1 and 2 under quality and points 1 and 4 under clarity in the weakness section.\n\nLIMITATIONS:\nI believe that point 2 under clarity in the weaknesses section should be addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}