SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Letting  $L = \frac{\log(1 / \delta)}{\eta\varepsilon^2} \geq 1 / \eta$  be the lower bound from Proposition 2.2, the expected sample complexity of our algorithm is at most  $O(L\log (1 / \eta)) \leq O(L\log L)$ . Hence Theorem 1.1 is always nearly optimal compared to the lower bound. Prior to our work there was a quadratic gap as the best upper bound for general reservoirs (Theorem 6 in [AAKA18]) was proportional to  $\log^2 (1 / \delta)$ . We note also that Theorem 5 of [dHCMC21] showed a similar result to Proposition 2.2.
3 Fixed Budget
Our algorithm and lower bounds for the fixed budget setting are much more technical, with full details given in the Appendix. Here we carefully state the results and give outlines of the proofs.
3.1 Precise Results for Fixed Budget
We first state the results in the easier case that  $\alpha$  is given. Theorems 3.1 and 3.2 below give rigorous statements of Theorem 1.2. As there, we will write  $\beta$  for the target value  $\alpha - \varepsilon$  when  $\alpha$  is given, since then the value  $\varepsilon$  plays no role.
Theorem 3.1. For any fixed  $0 < \beta < \alpha < 1$ , there is a sequence  $(\mathcal{A}_N)_{N\geq 1}$  of  $N$ -sample algorithms given explicitly by Algorithm 3 such that for any  $\eta \in (0,1)$  and any sequence of reservoir distributions  $\mu_N$  with  $G_{\mu_N}^{-1}(1 - \eta) \geq \alpha$ , with  $c_{\alpha,\beta}$  as in (1.2):
$$
\lim  _ {N \rightarrow \infty} \frac {\left(- \log \mathbb {P} _ {\mu N} [ p _ {i ^ {*}} <   \beta ]\right) \cdot \log^ {2} N}{N} \geq c _ {\alpha , \beta}. \tag {3.1}
$$
Conversely, the following lower bound applies for any quantile  $\eta \in (0,1)$ , and holds even when  $\alpha$  is known (which only makes pure exploration easier). It implies that  $\eta$  is asymptotically irrelevant for fixed budget sample complexity, i.e. the sample complexity of approximating the  $\eta = 0.01$ -quantile and  $\eta = 0.99$  quantile in fixed budget pure exploration depends only on the quantile values themselves as  $\delta \to 0$ . Some intuition for this is as follows. To succeed in pure exploration, one should have sampled the eventually outputted arm at least  $\Omega (\log 1 / \delta)$  times. The main obstacle to success in the fixed budget case is that any arm we obtain many samples of might gradually degrade over time. The probability of this degradation is essentially given by small probabilities coming from Chernoff-bound type events, which dominate the prior probability that the arm is in a top quantile.
$^{1}$  [AAKA18] states the result more generally. Specializing to  $\mathbb{P}[p_i = \alpha] = \eta$  and  $\mathbb{P}[p_i = \alpha - \varepsilon] = 1 - \eta$  for any  $1/4 \leq \alpha \leq 3/4$  recovers the statement of Proposition 2.2.
Theorem 3.2. For any  $0 < \eta < 1$  and  $0 < \beta < \alpha < 1$  there exists a sequence of reservoir distributions  $\mu_N$  with  $\alpha = G_{\mu_N}^{-1}(1 - \eta)$  such that for any sequence of  $N$ -sample algorithms  $\mathcal{A}_N$ ,
$$
\lim  _ {N \rightarrow \infty} \frac {\left(- \log \left(\mathbb {P} _ {\mu_ {N}} [ p _ {i ^ {*}} <   \beta ]\right)\right) \cdot \log^ {2} N}{N} \leq c _ {\alpha , \beta}. \tag {3.2}
$$
Remark 3.1. As discussed in the Appendix, the requirement that  $\alpha$  be known in Theorem 3.1 is technically necessary to deal with potential discontinuity of  $\alpha$  as  $\eta$  varies, but can be removed under mild conditions. In Theorems B.1, B.2, and B.3 below we give three concrete formulations under which the guarantee (3.1) can be achieved without knowledge of  $\alpha$ . Informal descriptions (any 1 of which suffices on its own) are:
1.  $\mu_N$  obeys  $G_{\mu_N}^{-1}(1 - \eta) \geq \frac{1 + \varepsilon}{2}$ .
2.  $\alpha$  is redefined as the average of  $G_{\mu}^{-1}(\eta)$  for  $\eta$  ranging over an interval.
3.  $\mu_N = \mu$  is independent of  $N$ , and the target value is
$$
\beta = \mu^ {*} - \varepsilon_ {1}
$$
for fixed  $\varepsilon_1 > \varepsilon$ . (Recall  $\mu^*$  is the maximum value in the support of  $\mu$ .)
Remark 3.2. In fact (see Theorem C.9 in the Appendix), Theorem 3.1 holds even if the algorithms  $\mathcal{A}_N$  must output  $\log N$  distinct arms  $i_1^*,\ldots ,i_{\log N}^*$ , all of which must satisfy  $p_{i_j^*}\geq \beta$ . I.e. for suitable  $(\mathcal{A}_N)_{N > 1}$ ,
$$
\limsup_{N\to \infty}\frac{(-\log\mathbb{P}\bigl[ \min_{j\in[\log N]}p_{i_{j}^{*}} <   \beta \bigr])\cdot\log^{2}N}{N}\geq c_{\alpha ,\beta}.
$$
3.2 Algorithm for Fixed Budget
Here we present our Algorithm 3 for the fixed budget problem. We note that this algorithm is given as input the value of  $\alpha$ . See Appendix C.1 for first-stage algorithms which estimate  $\alpha$  with sufficiently high accuracy and low sample complexity in the three scenarios of Remark 3.1. We use an explore-and-discard approach: at each time, it is currently exploring some arm  $i$ , having permanently discarded arms  $1,2,\ldots,i-1$  and not yet interacted with arms  $i+1,i+2,\ldots$ . It turns out to be very convenient to operate in a "B-batch-compressed" way for an increasing integer sequence  $B = (b_{1},b_{2},\ldots)$ . This means that at all times, the current arm  $i$  has been sampled  $b_{k}$  times for some  $k$ , and the samples between  $b_{k}$  and  $b_{k+1}$  are compressed into a single decision. The sequence  $b_{k}$  increases geometrically at a small rate, i.e.,  $b_{k+1}/b_{k} \approx 1+\varrho$  for small  $\varrho$  once  $k$  is mildly large. This batch-compression turns out to be without loss of generality up to a factor  $1+\varrho$  in the sample complexity, and helps the analysis operate on the proper geometric time-scales.
We now give a precise description as well as pseudo-code. Let  $0 < \varrho \ll \varrho_{1} \ll \varrho_{2} \ll 1$  be suitably small constants (i.e. choose  $\varrho_{2}$  sufficiently small, then  $\varrho_{1}$ , then  $\varrho$ ). We define  $b_{k}$  and other parameters as follows:
$$
\theta (a) = \operatorname {a r c c o s} (1 - 2 a) \quad \forall a \in [ 0, 1 ],
$$
$$
b _ {0} = \left\lceil \varrho_ {1} \log^ {2} (N) \right\rceil ,
$$
$$
k _ {0} = \lceil \log_ {1 + \varrho} \left(\log^ {4} (N) / b _ {0}\right) \rceil
$$
$$
b _ {k} = \left\lceil b _ {0} (1 + \varrho) ^ {k} \right\rceil , \quad k \leq k _ {0}
$$
$$
b _ {k _ {0} + j} = \lceil (1 + \varrho) ^ {j} b _ {k _ {0}} \rceil , \quad j \geq 1 \tag {3.3}
$$
$$
\tau_ {k} = \alpha - \varrho - \frac {k}{\sqrt {\log N}}, \quad k \leq k _ {0}
$$
$$
\tau_ {k _ {0} + j} = \theta (\alpha - 2 \varrho) - j \cdot \frac {\left(\theta (\alpha) - \theta (\beta)\right) \varrho (1 - \varrho_ {2})}{\log N}, \quad j \geq 1.
$$
The outer loop dictates the arm  $i$  under consideration. While exploring arm  $i$ , the main phase (shown in the last for loop) consists of collecting a new batch of  $b_{k_0 + j} - b_{k_0 + j - 1} \approx \varrho b_{k_0 + j - 1}$  samples, and rejecting if the new empirical mean reward  $\hat{p}_{i,b_{k_0 + j}}$  drops below
$$
\theta^ {- 1} \left(\theta (\alpha - 2 \varrho) - j \cdot \frac {\left(\theta (\alpha) - \theta (\beta)\right) \varrho (1 - \varrho_ {2})}{\log N}\right).
$$
Algorithm 3: Output arm with  $p_i \geq \beta$  using  $N$  samples with high probability
1 input: parameters  $N,\alpha ,\beta$  , and an infinite sequence of arms  $i = 1,2$    
2 initialize: parameters from (3.3) and  $i = 0$    
3 while fewer than  $N$  samples have been collected do   
4  $\begin{array}{rl} & i\gets i + 1\\ & \mathrm{Get~}b_{0}\mathrm{~samples~of~arm~}i.\\ & \mathrm{if~}\hat{p}_{i,b_0}\leq \alpha -\varrho \mathrm{then}\\ & \mathrm{|Reject~arm~}i\\ & \mathrm{end}\\ & \mathrm{for~}k = 1,2,\ldots ,k_0\mathrm{~do}\\ & \mathrm{Get~}b_k - b_{k - 1}\mathrm{~samples~of~arm~}i\mathrm{(total~}b_k).\\ & \mathrm{if~}\hat{p}_{i,b_k}\leq \alpha -\varrho -\frac{k}{\sqrt{\log N}}\mathrm{~then}\\ & \mathrm{|Reject~arm~}i;\\ & \mathrm{end}\\ & \mathrm{end}\\ & \mathrm{for~}j = 1,2,\ldots \mathrm{~do}\\ & \mathrm{Get~}b_{k_0 + j} - b_{k_0 + j - 1}\mathrm{~samples~of~arm~}i\mathrm{(total~}b_{k_0 + j}).\\ & \mathrm{if~}\theta (\hat{p}_{i,b_{k_0 + j}})\leq \theta (\alpha -2\varrho) - j\cdot \frac{\left(\theta(\alpha) - \theta(\beta)\right)\varrho(1 - \varrho_2)}{\log N}\mathrm{~then}\\ & \mathrm{|Reject~arm~}i\\ & \mathrm{end}\\ & \mathrm{end}\\ & \mathrm{end}\\ & \mathrm{Return~arm~}i. \end{array}$    
5   
6 if  $\hat{p}_{i,b_0}\leq \alpha -\varrho$  then   
7 Reject arm i   
8 end   
9 for  $k = 1,2,\dots ,k_0$  do   
10 Get  $b_{k} - b_{k - 1}$  samples of arm i (total  $b_{k}$  . if  $\hat{p}_{i,b_k}\leq \alpha -\varrho -\frac{k}{\sqrt{\log N}}$  then   
12 |Reject arm i;   
13 end   
14 end   
15 for  $j = 1,2,\ldots$  do   
16 Get  $b_{k_0 + j} - b_{k_0 + j - 1}$  samples of arm i (total  $b_{k_0 + j}$  . if  $\theta (\hat{p}_{i,b_{k_0 + j}})\leq \theta (\alpha -2\varrho) - j\cdot \frac{\left(\theta(\alpha) - \theta(\beta)\right)\varrho(1 - \varrho_2)}{\log N}$  then   
18 |Reject arm i   
19 end   
20 end   
21 end   
22 Return arm i.
Let us explain the point of this formula. First ignoring the function  $\theta$  for now, we see that the rejection threshold for  $\hat{p}_{i,b_k}$  steadily decreases with  $k$ . This threshold is tuned so that when arm  $i$  has been sampled say  $\Omega (N / \log N)$  times, we have  $k\asymp \log (N) / \varrho$  which results in a threshold slightly larger than  $\beta$ . Thus an arm which survives for such a long time will be prepared for acceptance as a new-optimal arm.
This strategy can be motivated as follows. The chief worry in fixed budget exploration is that arms might slowly degrade after many samples have been invested into them, which suggests a gradually decreasing rejection threshold. We designed this threshold to drop by a constant divided by  $\log (N)$  each time the number  $b_{k}$  of samples doubles. This ensures that for arm  $i$  to be rejected at time  $b_{k + 1}$ , the last  $b_{k + 1} - b_{k}$  samples must have behaved atypically (compared to their past behavior) by roughly  $\frac{\sqrt{b_{k + 1}}}{\log(N)}$  standard deviations. The probability of this rare event is roughly  $\exp (-Cb_{k + 1} / \log^2 (N))$  for some constant  $C$  by a Chernoff bound. Therefore if all  $N$  samples are used on eventually rejected arms, the product of these rare event probabilities will be roughly  $\exp (-CN / \log^2 (N))$  since each  $b_{k + 1}$  counts the number of samples used on an individual arm.
The use of the non-linear function  $\theta$  above is essential to achieve the optimal constant  $c_{\alpha, \beta}$  in Theorem 3.1. At a high-level,  $\theta$  balances the  $p(1 - p)$ -dependence of optimal Chernoff bounds on the underlying probability  $p$  of the Binomial random variable. If the optimal constant is not desired, then Theorem 3.1 can be simplified somewhat; the nonlinear  $\theta$  is not needed and one can double the number of samples at each step rather than multiplying by  $1 + \varrho$  for small  $\varrho$ . The earlier for loops in Algorithm 3 are technically important to handle small sample sizes for each given arm, before the required Chernoff bounds have kicked in asymptotically.
3.3 Analysis of Algorithm 3
The analysis of Algorithm 3 goes by controlling the tail distribution for the rejection time of a given arm (this time is taken to be zero if no rejection ever happens). Intuitively, we are most worried about arms which slowly degrade, thus wasting many samples. The following lemma, proved in the Appendix by a supermartingale argument, is key to rigorize this idea.
Lemma 3. Suppose  $(Y_{i})_{i\geq 1}$  are i.i.d. random variables with non-negative integer values, and  $\mathbb{E}[Y_i^c ]\leq 1$  holds for some constant  $c\geq 0$ . Then with
$$
M = \sup_{j\geq 0}\prod_{1\leq i\leq j}Y_{i}
$$
we have  $\mathbb{P}[M\geq A]\leq A^{-c}$
We apply Lemma 3 in the following way. Let  $X_{i}$  be the number of samples used by arm  $a_{i}$  before rejection, and  $I_{i} \in \{0,1\}$  be the indicator of the event that  $a_{i}$  is ever rejected (even if Algorithm 3 were to continue past time  $N$  and sample arm  $i$  an infinite number of times). We set  $Y_{i} = e^{X_{i}} \cdot I_{i}$ . The bulk of the analysis thus reduces to proving that  $\mathbb{E}[Y_i^c ] \leq 1$  for a suitable exponent  $c$ . Once this is shown, Lemma 3 ensures that  $M$  is small with high probability. Since  $\log (M)$  is exactly the total number of samples used on eventually-rejected arms, if say  $\log M \leq N\left(1 - \frac{1}{\log N}\right)$  then the last arm  $i^{*}$  to be selected must have passed enough rejection thresholds to have  $p_{i^{*}} \geq \beta$  with sufficiently high probability.
The early behavior of Algorithm 3 and  $b_{k}$  are specifically designed so that small values of  $Y_{i}$  contribute little in expectation. The tail behavior of  $Y_{i}$  (corresponding to rejecting arm  $i$  after a long time) is controlled by a technical analysis involving many adjacent time-scales in Subsection C.4 of the Appendix. Interestingly this tail analysis of  $Y_{i}$  never explicitly models the reward probability  $p_{i}$ . This is because we are able to argue that a rejection requires the early and late time behaviors of arm  $i$  to differ from each other, which is unlikely since the rewards form an i.i.d. sequence.
3.4 On the Lower Bound Proof
We consider the lower bound Theorem 3.2 to be the technical highlight of this paper. In it we face the challenge of proving a very small lower bound on the failure probability of any fixed budget algorithm. To do so we construct an online adversary with bounded probabilistic strength to distort rewards. The setup follows; recall  $B$ -batch-compressed algorithms as defined in Subsection 3.2.
Definition 3.1. An adaptive randomness distorting adversary  $\mathbb{A}$  interacts with a  $B$ -batch-compressed algorithm  $\mathcal{A}$  in the following way. Suppose  $\mathcal{A}$  chooses to increase the number of samples of arm  $a_{i}$  from  $b_{k}$  to  $b_{k + 1}$ . Then  $\mathbb{A}$  may restrict the set of possible outcomes of these  $b_{k + 1} - b_{k}$  samples. Additionally, when  $\mathcal{A}$  outputs an arm  $a_{i^{*}}$ , the adversary can restrict the possible values of  $p_{i^{*}}$ .
We will refer to adversarial actions as declarations. Thus when  $\mathcal{A}$  chooses a batch of samples,  $\mathbb{A}$  may declare that some property holds for the observed rewards.
As defined above, an adversary  $\mathbb{A}$  can do anything. We will limit the power of  $\mathbb{A}$  to make low-probability declarations. To formalize this, we charge  $\mathbb{A}$  per "bit" of probabilistic distortion, and give  $\mathbb{A}$  a deterministic "budget" for doing so. We measure this budget according to the algorithm's filtration which we refer to the "jointly Bayesian" viewpoint. One should think that the reservoir distribution  $\mu_{N}$  is known to both  $\mathcal{A}$  and  $\mathbb{A}$ , but neither has any information on the true reward probabilities  $p_i$  beyond the observed rewards. Thus  $\mathcal{A}$  and  $\mathbb{A}$  share at any time  $t$  the posterior distribution  $\pmb{\mu}^{t}$ . In particular recalling (B.9),  $\pmb{\mu}^{t}$  determines the distribution for the outcome of the next batch of  $b_{k + 1} - b_{k}$  samples.
Definition 3.2. Suppose that at time  $t$ , the declaration of  $\mathbb{A}$  has probability  $P_t$  to hold according to  $\mu^t$ . Let the sum
$$
\operatorname {C o s t} _ {t} = \sum_ {s \leq t} \log \left(1 / P _ {s}\right) \tag {3.4}
$$
be the total cost of  $\mathbb{A}$  up to time  $t$ , and  $\mathrm{Cost}_N$  the total cost of  $\mathbb{A}$ . We say strength  $(\mathbb{A}) \leq \mathrm{Cost}$  holds for some  $\mathrm{Cost} \in \mathbb{R}$  if the bound  $\mathrm{Cost}_N \leq \mathrm{Cost}$  almost surely.
The next key lemma shows that to obtain a lower bound for the failure probability of an algorithm, it suffices to prevent success using a low strength  $\mathbb{A}$ .
Lemma 4. Suppose there is a randomness distorting adversary  $\mathbb{A}$  of strength Cost whose declarations ensure that any algorithm  $\mathcal{A}$  outputs  $i^{*}$  satisfying  $p_{i^{*}}\leq \beta$  almost surely. Then the true failure probability of  $\mathcal{A}$  is
$$
\mathbb {P} ^ {\mu_ {N}, \mathcal {A}} [ p _ {i ^ {*}} \leq \beta ] \geq e ^ {- \text {C o s t}}.
$$
Proof. At each step  $t$ , let the random variable  $E_{t}$  denote the minimum possible conditional probability of the event  $p_{i^{*}} \leq \beta$  for any algorithm (in the algorithm's jointly Bayesian filtration). We claim that conditioned on  $\mathbb{A}$ 's declarations holding, for any  $\mathcal{A}$  the quantity  $M_{t} \equiv E_{t} \prod_{s \leq t} P_{s}$  evolves as a supermartingale in this filtration. This suffices because it implies  $E_{0} = M_{0} \geq \mathbb{E}[M_{T}] \geq e^{-\mathrm{Cost}}$ . (Here  $T$  is the random number of total batches used.)
Indeed suppose we are at time  $t$  and the next batch has been declared but not sampled. (Identical arguments apply when the adversary restricts  $p_{i^*}$  in the last stage.) Let the  $\sigma$ -field  $\mathcal{F}_t$  denote all information up to this point including the declaration of the next batch. Let  $\mathbb{E}$  denote an expectation where the samples from the next batch is distributed according to  $\pmb{\mu}^t$ . Let  $\widetilde{\mathbb{E}}$  denote an expectation where  $\mathbb{A}$ 's declaration is conditioned to hold on the next batch. The dynamic programming principle implies that
$$
\mathbb {E} \left[ E _ {t + 1} \mid \mathcal {F} _ {t} \right] \leq E _ {t}
$$
for any  $\mathcal{A}$  (with equality for the optimal  $\mathcal{A}$ ). Moreover since  $\mathbb{A}$ 's declaration has  $\mu^t$ -probability  $P_t$ ,
$$
\widetilde {\mathbb {E}} \left[ E _ {t + 1} \mid \mathcal {F} _ {t} \right] \leq \mathbb {E} \left[ E _ {t + 1} \mid \mathcal {F} _ {t} \right] / P _ {t} \leq E _ {t} / P _ {t}.
$$
$P_{t}$  is  $\mathcal{F}_t$  measurable, so  $\widetilde{\mathbb{E}}[P_t E_{t+1} \mid \mathcal{F}_t] \leq E_t$ . This establishes the claim and ends the proof.
We prove Theorem 3.2 by constructing a Bayesian adversary who slowly degrades the empirical performance of each arm  $a_{i}$ . This adversary declares for each batch of samples that the empirical average reward  $\hat{p}_i(n_{i,t})$  of arm  $i$  will drop by  $\Omega\left(\frac{\rho}{\log(N)}\right)$ , at least once the sample size  $n_{i,t} \geq N^{\varrho}$  is large. This degradation schedule ensures that the average reward of any arm is smaller than  $\beta$  once it has been sampled  $\Omega(N^{1 - \varrho})$  times. Moreover, it follows from reverse Chernoff estimates that this adversary pays  $O\left(\frac{1}{\log^2(N)}\right)$  cost per sample, leading to a failure probability lower bound of  $\exp(-O(N / \log^2 N))$  from Lemma 4. As with the upper bound, sharp constants can be tracked with more work and indeed our optimal adversary uses the same function  $\theta$  as in Algorithm 3.
Conclusion Our aim in this paper was to understand the sample complexity of pure exploration with infinitely many arms. We showed that, surprisingly, the behavior of fixed confidence and fixed budget problems is provably very different. In the former setting, there is a nearly optimal algorithm which precisely balances between sampling enough distinct arms (to estimate the quantile) and obtaining enough samples of a single arm (to be output). In the latter, the optimal algorithm must repeatedly decide whether to continue with the current arm or switch to a fresh one, via a gradually decreasing sequence of rejection thresholds.
Several interesting questions remain. One is that our fixed budget analysis is tailored to the  $\delta \rightarrow 0$  setting, and does not apply if  $\varepsilon, \eta, \delta$  all tend to zero at comparable rates. Hence other behaviors could be present in such parameter regimes. Additionally, a key conceptual feature of infinite-armed bandits is the possibility that no "good" arms are among those sampled by the algorithm. By definition, this simply cannot happen in  $K$ -armed bandits. It would be interesting to identify a natural problem setting that interpolates between them. Finally high probability bounds on the fixed confidence sample complexity would interpolate between the two settings we studied.
[AAJ+22] Maryam Aziz, Jesse Anderton, Kevin Jamieson, Alice Wang, Hugues Bouchard, and Javed Aslam. Identifying new podcasts with high general appeal using a pure exploration infinitely-armed bandit strategy. In Proceedings of the 16th ACM Conference on Recommender Systems, pages 134–144, 2022.
[AAKA18] Maryam Aziz, Jesse Anderton, Emilie Kaufmann, and Javed Aslam. Pure exploration in infinitely-armed bandit models with fixed-confidence. In Algorithmic Learning Theory, pages 3–24. PMLR, 2018.
[ABM10] Jean-Yves Audibert, Sébastien Bubeck, and Rémi Munos. Best arm identification in multi-armed bandits. In  $COLT$ , pages 41–53. CiteSeer, 2010.
[ACD21] Ayya Alieva, Ashok Cutkosky, and Abhimanyu Das. Robust pure exploration in linear bandits with limited budget. In International Conference on Machine Learning, pages 187-195. PMLR, 2021.
A Proofs from Section 2
Algorithm 4: Output  $\hat{\alpha} \in \left[G^{-1}(1 - \eta_1) - \frac{\varepsilon}{3}, G^{-1}(1 - \eta_1 + \eta_2) + \frac{\varepsilon}{3}\right]$  with probability  $1 - \frac{\delta}{2}$ .
1 input: arm set  $S = (a_{1}, a_{2}, \ldots)$  and parameters  $(\eta_{1}, \eta_{2}, \varepsilon, \delta) \in (0, 1)$  with  $\eta_{2} < \eta_{1}$ .
2 initialize:  $K = \frac{C\eta_1\log(1 / \delta)}{\eta_2^2}$
3 for  $i = 1,2,\ldots ,K$  do
4 Collect  $n = \frac{C\log(1 / \eta_2)}{\varepsilon^2}$  samples of arm  $i$ . Set  $\hat{p}_i = \hat{p}_i(n)$  to be the average observed reward.
5 end
6 Let  $\hat{\alpha}$  be the  $k$ -th largest value in  $\{\hat{p}_1, \dots, \hat{p}_K\}$  for  $k = \left[K\left(\eta_1 - \frac{\eta_2}{2}\right)\right]$ .
7 Return  $\hat{\alpha}$
We show the following generalization of Proposition 2.1.
Proposition A.1. Fix  $0 \leq \eta_1, \eta_2, \varepsilon, \delta \leq 1$  with  $\eta_2 \leq \eta_1$ . With probability at least  $1 - \frac{\delta}{2}$ , the output  $\hat{\alpha}$  of Alg. 4 satisfies
$$
\hat {\alpha} \in \left[ G ^ {- 1} (1 - \eta_ {1}) - \frac {\varepsilon}{3}, G ^ {- 1} (1 - \eta_ {1} + \eta_ {2}) + \frac {\varepsilon}{3} \right].
$$
Moreover, Alg. 4 has sample complexity
$$
O \left(\frac {\eta_ {1} \log (1 / \eta_ {2}) \log (1 / \delta)}{\eta_ {2} ^ {2} \varepsilon^ {2}}\right).
$$
Proof. The sample complexity is clear so we focus on the first statement. First observe that by a Chernoff estimate, for each  $i \in [K]$ ,
$$
\mathbb {P} \left[ | p _ {i} - \hat {p} _ {i} | \geq \frac {\varepsilon}{3} \right] \leq \frac {\eta_ {2}}{8}. \tag {A.1}
$$
Let  $N(\varepsilon)$  be the number of  $i \in [K]$  such that  $|p_i - \hat{p}_i| \geq \frac{\varepsilon}{3}$ . Applying a second Chernoff estimate (of multiplicative form, see e.g. Theorem 4.5 in [MU17]) on these events as  $i$  varies and noting that  $K\eta_2 \geq C\log(1/\delta)$ , (A.1) implies
$$
\mathbb {P} \left[ N (\varepsilon) \leq \frac {K \eta_ {2}}{6} \right] \geq 1 - \frac {\delta}{8}. \tag {A.2}
$$
We next show that with probability at least  $1 - \frac{\delta}{4}$ ,
$$
\hat {\alpha} \leq \bar {\alpha} + \frac {\varepsilon}{3} \equiv G ^ {- 1} \left(1 - \eta_ {1} + \eta_ {2}\right) + \frac {\varepsilon}{3}. \tag {A.3}
$$
With  $p_i$  the (true) mean reward from arm  $a_i$ , let
$$
N _ {\overline {{\alpha}}} \equiv | \{i \in [ K ]: p _ {i} > \overline {{\alpha}} \} |
$$
denote the number of the  $K$  tested arms which satisfy  $p_i > \overline{\alpha}$ . By definition,  $N_{\overline{\alpha}}$  is stochastically dominated by a Bin  $\left(K,\eta_1 - \frac{9\eta_2}{10}\right)$  random variable, and  $\eta_1 - \frac{3\eta_2}{4} = \Theta (\eta_1)$  since  $\eta_2\leq \eta_1$ . Note that
$$
\eta_ {1} - \frac {9 \eta_ {2}}{1 0} \asymp \eta_ {1} - \frac {3 \eta_ {2}}{4} \asymp \eta_ {1},
$$
$$
\frac {\eta_ {1} - \frac {9 \eta_ {2}}{1 0}}{\eta_ {1} - \frac {3 \eta_ {2}}{4}} \geq 1 + \frac {\eta_ {2}}{2 0 \eta_ {1}}.
$$
Therefore another multiplicative Chernoff estimate implies