SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiao-Yue Gong
Carnegie Mellon University  
exiaoyue@andrew.cmu.edu
Mark Sellke
Harvard University
msellke@fas.harvard.edu
We study pure exploration with infinitely many bandit arms generated i.i.d. from an unknown distribution. Our goal is to efficiently select a single high quality arm whose average reward is, with probability  $1 - \delta$ , within  $\varepsilon$  of being with the top  $\eta$ -fraction of arms. For fixed confidence, we give an algorithm with expected sample complexity  $O\left(\frac{\log(1 / \delta)\log(1 / \eta)}{\eta\varepsilon^2}\right)$  which matches a known lower bound up to the  $\log (1 / \eta)$  factor. In particular the  $\delta$ -dependence is optimal and closes a quadratic gap. For fixed budget, we show the asymptotically optimal sample complexity as  $\delta \to 0$  is  $\log (1 / \delta)\big(\log \log (1 / \delta)\big)^{2} / c$ . The value of  $c$  depends explicitly on the problem parameters (including the unknown arm distribution) through a certain Fisher information distance. Even the strictly super-linear dependence on  $\log (1 / \delta)$  was not known and resolves a question of [GM20].
In many learning problems, one faces the classical exploration versus exploitation tradeoff. A central example is the (stochastic) multi-armed bandit [LR85, BF85], where an agent is presented with a set of arms each of which when played gives a stochastic reward from an unknown and arm-dependent distribution. The performance of a bandit algorithm is most commonly determined by its regret, i.e. the difference between its average reward and the expected reward from the best arm. Multi-armed bandits and extensions have been applied in many settings including medical trials [BE95], online advertising [LCLS10], cognitive radio [AMTS11], and information retrieval [LPB17]. Optimal algorithms for the multi-armed bandit, including UCB, Thompson sampling, EXP3, and various forms of mirror descent, all make a principled tradeoff between exploration and exploitation.
In this work we focus on pure exploration bandit problems, a setting motivated by situations where the learning procedure consists of an initial exploration phase followed by a choice of policy to deploy. This is the case in hyperparameter optimization [GM20, LJD+17] as well as reinforcement learning from simulated environments. As there is no longer a competing need to exploit, optimal algorithms for pure exploration differ from the more common regret setting.
Pure exploration problems were introduced in [EDMM02, MT04, EDMM06] in the probably-approximately-correct (PAC) model. Here given  $K$  arms, one adaptively obtains samples until choosing one of the arms to output - the goal is to ensure that with probability  $1 - \delta$ , this arm has average reward within  $\varepsilon$  of the best arm, with minimum possible sample complexity depending on  $\varepsilon$  and  $\delta$ . The early works above focused on the fixed confidence setting in which one aims to minimize the expected sample complexity. Many subsequent works have also considered the fixed budget problem where the sample complexity is uniformly bounded.
While sharp results are known for pure exploration and other bandit problems with  $K$  arms, for many applications such as advertising there are far too many arms to explore. This motivated the study of infinite-armed bandit problems in e.g. [BCZ+97, WAM08]; the pure exploration version
was first studied in [AAKA18]. The main contribution of our work is to obtain near-optimal sample complexity for pure exploration problems with infinitely many arms in both the fixed confidence and fixed budget settings.
1.1 Problem Formulation
We now precisely formulate the infinite-armed pure exploration setting. Let  $\mathcal{S} = \{a_1, a_2, \ldots\}$  be a countably infinite set of stochastic bandit arms indexed by  $i = 1, 2, \ldots$ . When an arm  $i$  is sampled, it returns a  $\{0, 1\}$ -valued reward with mean  $p_i$ . The values  $p_i$  are drawn i.i.d. from an arbitrary reservoir distribution  $\mu$  supported in  $[0, 1]$  (which is unknown to the player). We define the cumulative distribution function
$$
G _ {\mu} (\tau) = \mathbb {P} ^ {p \sim \mu} [ p \leq \tau ]
$$
of  $\mu$ , and its (left-continuous) inverse
$$
G _ {\mu} ^ {- 1} (p) = \inf  \{\tau : G (\tau) \geq p \}.
$$
Finally let  $\mu^{*} = G_{\mu}^{-1}(1)$  denote the essential supremum of  $\mu$ , i.e. the maximum of its support.
An algorithm  $\mathcal{A}$  interacts with  $\mathcal{S}$  in the following way. At each time step  $t = 1,2,\dots,T$  the algorithm samples an arm  $i_t\in S$ , and observes a corresponding Bernoulli reward  $r_t\sim \mathrm{Ber}(p_{i_t})$ . The reward  $r_t$  is independent of previous actions and feedback. Eventually at some time  $T$ ,  $\mathcal{A}$  chooses an arm  $a_{i^*}$  to output. If the time-horizon  $T = N$  is fixed, we say  $\mathcal{A}$  has a fixed budget constraint. If  $\mathbb{E}[T]\leq N$  is bounded only in expectation, we say  $\mathcal{A}$  has a fixed confidence constraint.
For  $\eta, \varepsilon, \delta > 0$ , we say  $\mathcal{A}$  is  $(\eta, \varepsilon, \delta)$ -PAC if
$$
\mathbb {P} \left[ p _ {i ^ {*}} \geq G ^ {- 1} (1 - \eta) - \varepsilon \right] \geq 1 - \delta \tag {1.1}
$$
and set
$$
\alpha \equiv G ^ {- 1} (1 - \eta)
$$
to be the target quantile value. We emphasize that while  $\eta$  is known,  $\alpha$  may not be as it depends on the unknown  $\mu$ . The definition (1.1) stems from [AAKA18]. As brief justification for the parameter  $\eta$ , note that in an infinite-armed setting the reservoir  $\mu$  could give  $\varepsilon$ -optimal arms with arbitrarily small probability. Thus it is impossible to give a non-asymptotic classical  $(\varepsilon, \delta)$ -PAC in our setting without assumptions on  $\mu$ . Taking  $\eta > 0$  as above ensures that a positive fraction of arms are "good enough" and will enable such guarantees. One also cannot set  $\varepsilon$  to zero: for example if  $\mu$  is supported in  $[0.5 - e^{-N^4}, 0.5]$ , the quantile value of the output arm will be essentially uniform for any  $N$ -sample algorithm.
The purpose of this paper is to give  $(\eta, \varepsilon, \delta)$ -PAC algorithms whose sample complexity  $N$  is minimal. We can now state our main results. Let us emphasize that unless explicitly stated, no assumptions on the reservoir distribution  $\mu$  are made, nor does the algorithm have any prior knowledge about  $\mu$ . Our main result in the fixed confidence case is as follows.
Theorem 1.1. For any  $(\eta, \varepsilon, \delta)$ , there exists a  $(\eta, \varepsilon, \delta)$ -PAC algorithm with expected sample complexity  $O\left(\frac{\log(1 / \delta) \log(1 / \eta)}{\eta \varepsilon^2}\right)$ .
In the fixed budget setting, our interest is especially in the high-confidence regime  $\delta \to 0$ , where we obtain the following. The following statement is a slightly informal combination of Theorems 3.1 and 3.2. We note that while the main statement requires  $\alpha$  to be given, this is not essential in several cases as discussed extensively in the Appendix. For example if  $\alpha \geq \frac{1 + \varepsilon}{2}$  then  $\alpha$  does not need to be given.
Theorem 1.2 (Informal). For any fixed  $(\eta, \varepsilon)$ , let  $\alpha = G^{-1}(1 - \eta)$  be given and set  $\beta = \alpha - \varepsilon$ . Then as  $\delta \to 0$ , the optimal  $(\eta, \varepsilon, \delta)$ -PAC algorithm under fixed budget has sample complexity
$$
\begin{array}{l} N = \left(c _ {\alpha , \beta} ^ {- 1} \pm o _ {\delta \rightarrow 0} (1)\right) \log (1 / \delta) \left(\log \log (1 / \delta)\right) ^ {2}; \\ c _ {\alpha , \beta} \equiv \frac {\left(\operatorname {a r c c o s} (1 - 2 \alpha) - \operatorname {a r c c o s} (1 - 2 \beta)\right) ^ {2}}{2} (1.2) \\ = \frac {\left(\int_ {\beta} ^ {\alpha} \frac {d x}{\sqrt {x (1 - x)}}\right) ^ {2}}{2}. (1.3) \\ \end{array}
$$
An equivalent statement is that given exactly  $N$  samples, the optimal failure probability  $\delta$  to have  $p_{i^*} \geq G^{-1}(1 - \eta) - \varepsilon$  decays as  $\exp\left(-\frac{N(c \pm o(1))}{\log^2(N)}\right)$ . Indeed once  $\eta$  and  $\varepsilon$  are fixed, the question of minimizing the sample complexity  $N = N(\delta)$  (given a target confidence  $\delta$ ) is equivalent to minimizing the failure probability  $\delta = \delta(N)$  (given a sample complexity  $N$ ). These viewpoints are equivalent in both settings we study, and we switch between them at times.
Interestingly the value  $\eta$  makes no appearance in Theorem 1.2, so it is asymptotically irrelevant for the  $\delta \rightarrow 0$  regime of fixed budget pure exploration. In fact the value  $\arccos (1 - 2\alpha) - \arccos (1 - 2\beta)$  appearing in the definition of  $c_{\alpha ,\beta}$  is the Fisher-information distance between  $\alpha$  and  $\beta$  in the exponential family of Bernoulli random variables via the formula (1.3). See just below Theorem 3.1 for a brief explanation of why  $\eta$  does not enter the asymptotic sample complexity.
Remark 1.1. In our problem formulation above we assumed rewards are Bernoulli, i.e. lie in  $\{0,1\}$ . In fact as long as the quality of arm  $i$  is measured by its mean reward, this assumption loses no generality and is just a technical convenience: our results extend verbatim to  $[0,1]$ -valued rewards.
Indeed any arm with  $[0,1]$ -valued rewards can be transformed into a Bernoulli arm with  $\{0,1\}$ -valued rewards and the same mean: simply turn reward  $r \in [0,1]$  into reward 1 with probability  $r$ , and 0 with probability  $1 - r$ . Note that this reduction (used also in Section 1.2 of [AG12]) might increase the instance-dependent sample complexity of some reservoir distributions, but our results only refer to the distribution of arm means under the reservoir which is unchanged by the reduction.
1.2 Further Notation
We use the convention that algorithms collect 1 sample per unit time until terminating, so the time  $t$  equivalently denotes the number of total samples collected so far. Denote by  $n_{i,t}$  the number of samples of arm  $a_i$  collected by time  $t$ . The  $n$ -th time  $a_i$  is sampled, its reward is  $r_{i,n} \in [0,1]$ . The total reward of arm  $i$  up to time  $t$  is
$$
R _ {i, t} = \sum_ {n = 1} ^ {n _ {i, t}} r _ {i, n}.
$$
The corresponding average reward is  $\hat{p}_{i,t} = \hat{p}_i(n_{i,t}) = \frac{R_{i,t}}{n_{i,t}}$ . We use  $C$  to indicate a universal constant independent of all parameters in this paper, and  $o_n(1)$  and  $o_N(1)$  to denote quantities tending to 0 as  $n\to \infty$  or  $N\to \infty$ , with other parameters implicitly held constant. However in Section C we use e.g.  $\Omega_{\alpha,\varrho}$  to indicate an asymptotic lower bound with implicit constant factor depending on the values of  $\alpha, \varrho$ , which are treated as fixed. In all our uses of these notations it is  $n$  or  $N$  which is tending to infinity while other parameters are always treated as fixed.
As discussed above, this work belongs to the area of pure exploration for multi-armed bandit problems. Unlike ordinary bandit problems where one aims to minimize the regret compared to the best arm [BCB12, Sli19], in pure exploration all that matters is the final arm selected by the algorithm. We survey several existing results below, with an emphasis on the high-probability regime of small  $\delta$ . See e.g. Chapter 33 of [LS20] for a more detailed survey.
Pure exploration was first studied in [EDMM02, MT04, EDMM06] in the probably-approximately-correct model. Here given  $K$  arms, one adaptively obtains samples until choosing one of the arms to output - the goal is to ensure that with probability  $1 - \delta$ , this arm has average reward within  $\varepsilon$  of the best arm. These works showed that the optimal fixed confidence sample complexity is  $\Theta\left(\frac{K}{\varepsilon^2}\log \frac{1}{\delta}\right)$ .
Later, [BMS09] considered the simple regret of pure exploration problems, namely the regret incurred at the final timestep. [ABM10] studied the closely related problem of identifying the best arm, obtaining nearly tight sample complexity bounds in terms of the sum of the squared inverse suboptimality-gaps  $H = \sum_{i \neq i^*} \Delta_i^{-2}$ . Further upper and lower sample complexity bounds have been obtained in several works. For example [CL15, KCG16] show that for fixed confidence, the sample complexity scales as  $\Theta(H \log(1/\delta))$  as  $\delta \to 0$ . The fixed budget setting, in which the number of adaptive samples is upper-bounded almost surely rather than in expectation, turns out to be more difficult. [CL16] proved that the optimal fixed budget sample complexity can be  $\Theta(H \log(K) \log(1/\delta))$  as  $\delta \to 0$ , i.e. the fixed budget constraint may lead to an additional  $\log(K)$
factor. However it reverts to  $\Theta$  ( $H\log (1 / \delta)$ ) when the value of  $H$  is known beforehand. Many recent works have studied other aspects of pure exploration, for example by incorporating structured feedback; see [JMNB14, CGL+17, KSJ20, KG21, TRMD21, ACD21, ZKSN22, AAJ+22].
Infinite-armed bandits have also much received previous study, e.g.  $\mathrm{[BCZ^{+}97}$ , WAM08]. Since near-optimal arms may be arbitrarily rare, it is natural to instead compare with a quantile of the arm distribution. For example [CK18] aims to minimize regret relative to such a quantile.
The  $(\eta, \varepsilon, \delta)$ -PAC guarantees we address in this paper were first studied in [AAKA18], for infinitely many arms in the fixed confidence setting. Their approach was to sample  $K \asymp \frac{\log(1 / \delta)}{\eta}$  arms and then apply a PAC algorithm for  $K$ -armed pure exploration. As discussed at the beginning of Section 2, the resulting algorithm "pays twice" for the high confidence level  $1 - \delta$  which leads to a suboptimal  $O(\log^2(1 / \delta))$  sample complexity upper bound. Top- $k$  extensions were also studied in [RLS19, CK19]; the  $\log^2(1 / \delta)$  scaling is still present in their results.
Of particular note is the work [dHCMC21] which considers also both fixed budget and confidence settings and obtains somewhat similar looking results. However they restrict attention to a special class of reservoir distributions with supremum achieved by an atom, which must be  $\Delta$ -larger than the rest of the support. This structural assumption of a  $\Delta$ -gap intrinsically reduces fixed budget sample complexity: their result (see Theorem 4 therein) is actually better than the lower bound we show in Theorem 3.2 as there is no appearance of  $\log \log(1/\delta)$  (i.e.  $\log(T)$  in their notation).
From their fixed budget estimate, [dHCMC21] deduce (at the end of Section 1 therein) the same bound as Theorem 1.1 in their setting for the special case  $\varepsilon = \Delta$ . Our Theorem 3.2 shows that for the general reservoirs we consider, passing from fixed budget to fixed confidence is inherently suboptimal: the factors of  $\log \log(1/\delta)$  would remain, but are extraneous for fixed confidence. This underscores that Theorem 1.1 is genuinely new despite the superficial similarity with [dHCMC21], since their proof cannot work in our setting.
Finally [GM20] studied the infinite-arm pure exploration problem where  $\alpha$  is given, also focusing on the  $\delta \rightarrow 0$  asymptotics. They proposed an algorithm with fixed budget sample complexity  $\mathcal{O}\big(\log (1 / \delta)\big(\log \log (1 / \delta)\big)^2\big)$ , and asked whether the  $\log \log (1 / \delta)$  factors are necessary. Theorem 3.2 shows their bound is optimal up to constant factors in terms of  $\delta$  and in fact obtains the tight constant. Interestingly [GM20] were motivated by complexity theoretic applications to amplification and derandomization, where bandit arms correspond to random seeds.
We remark that the analysis in [GM20] seems to be technically incomplete. In particular in Lemma 4.5 of (the cited, journal version of) the paper, they neglect to take a union bound over sequences  $(T_{1},\ldots ,T_{k})$  summing to  $T$  but only estimate the probability of each fixed sequence  $(T_{1},\ldots ,T_{k})$ . This is a serious gap since the number of such sequences is exponentially large in  $T$ . However their idea to use a moving sequence of rejection thresholds was fundamentally correct. It is similar to the main phase of our Algorithm 3, for which we give a fully rigorous, supermartingale-based analysis.
2 The Fixed Confidence Setting
Our fixed confidence algorithms proceed in two phases. The first phase aims to estimate the target quantile value, which we recall depends on the unknown  $\alpha$ . The second phase then aims to find a single arm which is almost as good as this estimate with high probability.
Focusing on the  $\delta$ -dependence, a challenge with infinitely many arms is that to succeed with probability  $1 - \delta$ , it is necessary both to sample  $\log(1/\delta)$  arms to ensure a good arm is ever observed, and to sample an arm  $\log(1/\delta)$  times before outputting it as  $i^*$ . This is why the approach of [AAKA18] requires  $O(\log^2(1/\delta))$  samples: they obtain  $O(\log(1/\delta))$  samples each of  $O(\log(1/\delta))$  arms. However in our algorithm, the first phase samples  $O(\log(1/\delta))$  arms  $O(1)$  times each, while the second phase samples  $O(1)$  arms  $O(\log(1/\delta))$  times each. This allows us to satisfy both necessary conditions above without paying twice for the confidence level.
2.1 The Algorithm for Fixed Confidence
We first give in Alg. 1 a simple procedure to estimate the top  $\eta$  quantile, allowing an  $\varepsilon /3$  error as well as an  $\eta /2$  error in the quantile itself. Alg. 1 obtains  $O\left(\frac{\log(1 / \eta)}{\varepsilon^2}\right)$  samples from each of the
Algorithm 1: Output  $\hat{\alpha} \in \left[G^{-1}(1 - \eta) - \frac{\varepsilon}{3}, G^{-1}\left(1 - \frac{\eta}{2}\right) + \frac{\varepsilon}{3}\right]$  with probability  $1 - \frac{\delta}{2}$
1 input: arm set  $\mathcal{S} = (a_1,a_2,\dots)$  and parameters  $\eta ,\varepsilon ,\delta \in (0,1)$    
2 initialize:  $K = \frac{C\log(1 / \delta)}{\eta}$    
3 for  $i = 1,2,\ldots ,K$  do  
4 Collect  $n = \frac{C\log(1 / \eta)}{\varepsilon^2}$  samples of arm i. Set  $\hat{p}_i = \hat{p}_i(n)$  to be the average observed reward.   
5 end   
6 Let  $\hat{\alpha}$  be the  $k$  -th largest value in  $\{\hat{p}_1,\dots ,\hat{p}_K\}$  for  $k = \left\lceil \frac{3K\eta}{4}\right\rceil$    
7 Return  $\hat{\alpha}$    
Algorithm 2: Output  $a_{i^{*}}$  such that  $p_i^*\geq \hat{\alpha} -\varepsilon$  with probability  $1 - \delta$    
1 input: arm set  $\mathcal{S} = (a_1,a_2,\dots)$  and parameters  $(\eta ,\varepsilon ,\delta ,\hat{\alpha})$    
2 for  $i = K + 1,K + 2,\ldots ,K + \frac{C\log(1 / \delta)}{\eta}$  do  
3 Collect  $\frac{C\log(1 / \eta\delta)}{\varepsilon^2}$  samples of arm i. Set  $\hat{p}_i$  to be the average reward.   
4 if  $\hat{p}_i\geq \hat{\alpha} -\frac{\varepsilon}{3}$  then  
5 | Return  $a_{i}$    
6 end   
7 end
first  $K = O\left(\frac{\log(1 / \delta)}{\eta}\right)$  arms  $a_1, \ldots, a_K$ . The resulting estimator  $\hat{\alpha}$  is the  $1 - \frac{3\eta}{4}$  quantile of the empirical average rewards  $\hat{p}_1, \ldots, \hat{p}_k$ . Its main guarantee is below.
Proposition 2.1. Fix  $0 \leq \eta, \varepsilon, \delta \leq 1$ . With probability at least  $1 - \frac{\delta}{2}$ , the output  $\hat{\alpha}$  of Alg. 1 satisfies
$$
\hat {\alpha} \in \left[ G ^ {- 1} (1 - \eta) - \frac {\varepsilon}{3}, G ^ {- 1} \left(1 - \frac {\eta}{2}\right) + \frac {\varepsilon}{3} \right].
$$
Moreover, Alg. 1 has sample complexity
$$
O \left(\frac {\log (1 / \eta) \log (1 / \delta)}{\eta \varepsilon^ {2}}\right).
$$
Next Alg. 2 repeatedly chooses a new arm  $a_{i}$  and obtains  $O\left(\frac{\log(1 / \eta\delta)}{\varepsilon^{2}}\right)$  samples. It accepts if the sample mean was at least  $\hat{\alpha} -\frac{\varepsilon}{3}$ , and otherwise moves on to the next arm. If  $\frac{C\log(1 / \delta)}{\eta}$  arms have been tried without success, then Alg. 2 outputs no arm, thus declaring failure. This termination condition is necessary to avoid incurring huge sample complexity when Alg. 1's estimate  $\hat{\alpha}$  of  $\alpha$  is inaccurate. Since typically  $\Omega (\eta)$  of arms will be good enough to usually succeed, this also preserves the  $1 - \delta$  confidence level. We now give the following more detailed restatement of Theorem 1.1.
Theorem 2.1. Apply Algorithm 1 followed by Algorithm 2 using the resulting value  $\hat{\alpha}$ . This combined algorithm has expected sample complexity  $O\left(\frac{\log(1 / \eta)\log(1 / \delta)}{\eta\varepsilon^2}\right)$ . Moreover its output  $a_{i^*}$  satisfies
$$
\mathbb {P} [ p _ {i ^ {*}} \geq G ^ {- 1} (1 - \eta) - \varepsilon ] \geq 1 - \delta .
$$
Proposition 2.1 and Theorem 2.1 are proved in Appendix A. We note that in analyzing Algorithm 2, typically the returned arm  $a_i$  has  $i \leq K + O(1 / \eta)$ . This is because each arm  $a_i$  in the top  $\eta / 2$  quantile has a good chance to pass the test of Algorithm 2. In particular, the expected sample complexity calculation never multiplies two  $\log(1 / \delta)$  terms together. However, continuing for  $K + \frac{C \log(1 / \delta)}{\eta}$  steps is important to ensure a  $1 - \delta$  success probability. Algorithm 2 stops after  $O(\log(1 / \delta) / \eta)$  arms instead of continuing forever in order to guard against erroneous estimates from Algorithm 1.
Remark 2.1. Our fixed confidence algorithm, given by combining Alg. 1 with Alg. 2 as above, requires only  $O(1)$  batches in expectation. Here a batched algorithm operates in a small number of batched phases. At the start of each phase, such an algorithm chooses  $b$  arms to sample exactly  $s$  times each, where  $b, s$  can both depend adaptively on the previous feedback, but cannot be
changed during the current phase. Minimizing the number of required batches is often desirable, see e.g. [PRCS16, GHRZ19]. In particular Alg. 1 uses a single batch with  $s_1 = \frac{C\log(1 / \eta)}{\varepsilon^2}$  samples of  $b_{1} = \frac{C\log(1 / \delta)}{\eta}$  arms. Then Alg. 2 can be implemented in a batched way with  $s_2 = \frac{C\log(1 / \eta\delta)}{\varepsilon^2}$  and a sequence of batch sizes  $b_{2,i} = \frac{2^i}{\eta}$  for  $1\leq i\leq \log_2(C\log (1 / \delta))$ . (In the latter phase, one stops after finding an arm to accept.) While this construction uses  $O(1)$  batches in expectation, it could be interesting to explore pure exploration with an exactly fixed number of batches, which is more analogous to the fixed budget setting.
2.2 Near-Optimality in the Fixed Confidence Regime
Here we explain why the guarantee of Theorem 2.1 is nearly optimal. For comparison, recall from the important work [MT04] that  $\Theta\left(\frac{K\log(1 / \delta)}{\varepsilon^2}\right)$  samples are necessary and sufficient for  $(\varepsilon, \delta)$ -PAC pure exploration in the  $K$ -armed bandit problem with fixed confidence. Intuitively, one expects this problem to be related to ours via  $\eta \approx 1 / K$ . In fact the following infinite-arm analog was later shown. It follows that the guarantee of Theorem 1.1 is optimal up to the  $\log (1 / \eta)$  factor.
Proposition 2.2 (Theorem 1 and Remark 2 of [AAKA18]). There exists an absolute constant  $c > 0$  such that the following holds. For any  $1/4 \leq \alpha \leq 3/4$  and  $\eta, \delta \leq 1/10$  and for any pure exploration algorithm  $\mathcal{A}$  with expected sample complexity  $N \leq \frac{c\log(1/\delta)}{\eta\varepsilon^2}$ , there exists a reservoir distribution such that  $\mathcal{A}$  fails to be  $(\eta, \varepsilon, \delta)$ -PAC.