SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Summary of the proof for bound (5): Let  $i$  denote a sub-optimal arm and  $\Upsilon_{i} := \operatorname{Arg}\operatorname{Min}_{j}\Lambda_{ij}$ . First, we show that at round  $t$ , if  $i \in S_t$  then necessarily we have  $\Upsilon_{i} \cap C_{t} \neq \emptyset$ . We proceed by a contradiction argument: assume  $\Upsilon_{i} \cap C_{t} = \emptyset$ , let  $j$  denote the element of  $\Upsilon_{i}$  with the largest mean. Let  $k$  denote the arm that eliminated  $j$  from  $S$  at a round  $s < t$ . Lemma E.3 shows that, since  $\hat{\Delta}_{jk}(s,\delta) > 0$ , we necessarily have  $\log(1/\delta)\Lambda_{jk}/4 \lesssim s$ . Moreover,  $j$  was kept up to round  $82s$  and in this last round we had  $\hat{\Delta}_{ij}(82s,\delta) \leq 0$ , which gives by Lemma E.3:  $82s \lesssim (25/2)\log(1/\delta_{82s})\Lambda_{ij}$ . Combining the two bounds on  $s$ , gives:  $\Lambda_{jk} < \Lambda_{ij}$ . Finally, we use the ultra-metric property satisfied by the quantities  $\Lambda_{uv}$  (Lemma E.6), stating that:  $\Lambda_{ik} \leq \max\{\Lambda_{ij},\Lambda_{jk}\}$ . Combining the last inequality with the latter, we get  $\Lambda_{ik} \leq \Lambda_{ij}$ , which means that  $k \in \Upsilon_{i}$ . The contradiction arises from the fact that  $j$  is the element of the largest mean in  $\Upsilon_{i}$  and arm  $k$  eliminated  $j$  (hence  $\mu_k > \mu_j$ ). We conclude that necessarily  $\Upsilon_{i} \cap C_{t} \neq \emptyset$ , therefore by Lemma E.3, at a round of the order of  $\min_{j}\Lambda_{ij}$ , we will necessarily have  $\hat{\Delta}_{ij}(t,\delta) > 0$  for some  $j \in C_t$ .
Remark 1. From a practical standpoint, to keep sampling an arm eliminated at  $t$  for additional 81 t rounds is a conservative approach. The stated value of 81 is determined by specifics of the proof, but we believe that it can be optimized to a smaller constant based on the insights gained from numerical simulations. Furthermore, even if we omit the oversampling step, as presented in Theorem 4.1, the algorithm is still guaranteed to identify the best arm with probability  $1 - \delta$ . Only the query complexity guarantee is weaker, but the algorithm may still lead to effective arm elimination, although with potentially slightly slower convergence.
Remark 2. The idea of successive elimination based on evaluating the differences between variables was previously introduced in [28], in the context of model selection aggregation. Their analysis allows having a bound slightly looser than bound (6) (with the distances between variables instead of variances). On the other hand, Theorem 4.1 in our paper provides a sharper bound in (5).
First, it is important to note that bound (5) is always sharper (up to a numerical constant factor) than bound (6) because the optimal arm is included in the set over which the minimum in bound (5) is taken. On the other hand, bound (5) can be smaller than bound (6) by a factor of  $1 / K$ . This situation can arise in scenarios where the  $K - 1$  sub-optimal arms, which have close means, are highly correlated, while the optimal arm is independent of the rest. In such a situation, the terms in
the sum in (5) corresponding to the correlated sub-optimal arms are relatively small, except for the arm (denoted as  $i$ ) with the largest mean among the  $K - 1$  correlated suboptimal rewards. That last remaining arm will be eliminated by the optimal arm, incurring a potentially large cost, but for that arm only. On the other hand, each term in the second bound (6) would be of the order of the last cost.
To provide perspective on our guarantees compared to those developed in the independent one query per round setting, it is worth noting that the standard guarantees in that setting provide a sample complexity corresponding to  $\sum_{i\neq i^{*}}\log (1 / \delta) / (\mu_{i^{*}} - \mu_{i})^{2}$ . Since variables are bounded by 1, their variances are bounded by 1. Therefore, in the worst case, we recover the previous guarantees with a numerical constant factor of 2.
A refined adaptive algorithm presented in [25] also utilizes a successive elimination approach using confidence intervals for the arm means based on the empirical Bernstein's inequality. However, unlike our algorithm, they use the concentration inequality to evaluate each arm's mean independently of the other arms. Their approach allows for adaptability to individual arm variances, and is particularly beneficial are small, i.e.,  $\operatorname{Var}(X_i) \ll 1$ . The sample complexity of their algorithm is of the order of:
$$
\log (K \Gamma \delta^ {- 1}) \sum_ {i \neq i ^ {*}} \frac {\mathrm {V a r} (X _ {i}) + \mathrm {V a r} (X _ {i ^ {*}})}{(\mu_ {i ^ {*}} - \mu_ {i ^ {*}}) ^ {2}},
$$
where  $\Gamma := \max_{i \neq i^*} (\operatorname{Var}(X_i) + \operatorname{Var}(X_{i^*}) / (\mu_{i^*} - \mu_{i^*})^2$ . Neglecting numerical constant factors, the last bound is larger than both our bounds (5) and (6). This is because the variance of the differences can be bounded as follows:  $V_{ii^*} \leq 2(\operatorname{Var}(X_i) + \operatorname{Var}(X_{i^*}))$ .
5 Algorithms and main theorem for Gaussian distributions
In this section, we address the scenario where arms are assumed to follow a Gaussian distribution. We consider a setting where the learner has no prior knowledge about the arms' distribution parameters, and we continue using the notation introduced in the previous section. The main difference between this case and the bounded variables setting, other than the form of the sample complexity obtained, is the behavior of the algorithm when variances of differences between variables tend to 0, displayed by the second bound in Theorem 5.1.
Our algorithm relies on the empirical Bernstein inequality, which was originally designed in the literature for bounded variables [3, 24]. We have extended this inequality to accommodate Gaussian variables by leveraging existing Gaussian concentration results. Note that extending such inequalities more generally for sub-Gaussian variables is a non-trivial task. One possible direction is to suppose that arms follow a sub-Gaussian distribution and satisfy a Bernstein moment assumption (such extensions were pointed by works on bounded variables e.g., ?). Given the last class of distributions, we can build on the standard Bernstein inequality with known variance, then plug in an estimate of the empirical variance leveraging the concentration of quadratic forms (see 5). However, it remains uncertain whether an extension for sub-Gaussian variables (without additional assumptions) is practically feasible.
We extend the previous algorithm to the Gaussian case by performing sequential tests between pairs of arms  $(i,j) \in [[K]]$ . We establish a confidence bound for the difference variables  $(X_{i} - X_{j})$  (Lemma D.1 in the appendix) and introduce the following quantity:
$$
\hat {\Delta} _ {i j} ^ {\prime} (t, \delta) := \hat {\mu} _ {j, t} - \hat {\mu} _ {i, t} - \frac {3}{2} \alpha (t, \delta) \sqrt {2 f (\alpha (t , \delta)) \hat {V} _ {i j , t}}, \tag {7}
$$
where the  $f$  is defined by:  $f(x) = \exp (2x + 1)$  if  $x\geq 1 / 3$ , and  $f(x) = 1 / (1 - 2x)$  otherwise.
Using the empirical confidence bounds on the differences between arms, we apply the same procedure as in the case of bounded variables. We make one modification to Algorithm 2: we use the quantities  $\hat{\Delta}_{ij}'(t,\delta)$  instead of  $\hat{\Delta}_{ij}(t,\delta)$ .
By following the same analysis as in the bounded setting, we establish that the sample complexity required for the comparison tests between arms  $i$  and  $j$  is characterized by the quantity  $\frac{V_{ij}}{(\mu_i - \mu_j)^2} \vee 1$ . The following theorem provides guarantees on the algorithm presented above:
Theorem 5.1. Suppose Assumptions 1 and  $2 - \mathcal{G}$  holds. Consider the algorithm described above, with input  $\delta \in (0,1)$ . We have with probability at least  $1 - \delta$ : the algorithm identifies the best arm
and the total number of queries  $N$  satisfies:
$$
N \leq c \log \left(K \Lambda \delta^ {- 1}\right) \sum_ {i \in [   [ K ]   ] \backslash \{i ^ {*} \}} \min  _ {\substack {j \in [   [ K ]   ]; \\ \mu_ {j} > \mu_ {i}}} \left\{\frac {V _ {i j}}{(\mu_ {i} - \mu_ {j}) ^ {2}} \vee 1 \right\}, \tag{8}
$$
where  $\Lambda = \max_{i\in [[K]]}\min_{\substack{j\in [[K]]:\\ \mu_j > \mu_i}}\left\{\frac{V_{ij}}{(\mu_i - \mu_j)^2}\lor 1\right\}$  and  $c$  is a numerical constant.
$$
N \leq 3 K + c \log \left(\frac {K \delta^ {- 1}}{\log (1 + 1 / \Lambda)}\right) \sum_ {i \in [   [ K ]   ] \backslash \{i ^ {*} \}} \frac {1}{\log \left(1 + \frac {\left(\mu_ {i ^ {*}} - \mu_ {i}\right) ^ {2}}{V _ {i i ^ {*}}}\right)}, \tag {9}
$$
where  $\Lambda = \max_{i\neq i^{*}}V_{ii^{*}} / (\mu_{i^{*}} - \mu_{i})^{2}$  is defined above and  $c$  is a numerical constant.
Remarks. On the sample complexity cost of being adaptive to the variance: If the sample size is larger than  $\log(1/\delta)$ , the cost of plugging in the empirical variance estimate into the Chernoff's concentration inequality is only a multiplicative constant slightly larger than one (nearly  $1 + 2\sqrt{\log(1/\delta)/n}$ ). However, in the case of a small sample regime ( $n < \log(1/\delta)$ ), the cost is a multiplicative factor of  $\exp(\sqrt{\log(1/\delta)/n} + 1/2)$  due to the nature of the left tail of the chi-squared distribution (see Sections D and K of the appendix for detailed calculations). For most natural regimes, the number of queries made for each arm is larger than  $\log(1/\delta)$ , hence the last described effect does not arise. However, in some specific regimes (such as the case of very small variances of the arms) an optimal algorithm should query less than  $\log(1/\delta)$  samples, which necessitates introducing the exponential multiplicative term above into the concentration upper bound. This translates into a different form of guarantee presented in Theorem 5.1, inequality (9). It is important to note the cost in this regime cannot be avoided as highlighted by our lower bound presented in Theorem 6.2.
Theorem 5.1 above shows that Algorithm 2 is applied to Gaussian distribution guarantees that each sub-optimal arm  $i$  is eliminated after roughly  $\min_j V_{ij}^2 / (\mu_i - \mu_j)^2$  queries. Bounds (8) and (9) derived from our algorithm are smaller than the standard complexity bound in the independent case, which is given by  $\sum_{i \neq i^*} \sigma^2 / (\mu_i - \mu_{i^*})^2$  when all arms have variances smaller than  $\sigma^2$ . It is important to note that unlike most existing procedures in the literature that achieve the standard complexity bound, our algorithm does not require knowledge of the parameter  $\sigma^2$ .
About the upper bound (9): This bound can be further bounded by  $\log (\delta^{-1})\sum_{i\neq i^{*}}\frac{V_{ii^{*}}}{(\mu_i - \mu_i^{*})^2}\vee 1$  The form presented in Theorem 5.1 is particularly sharp when the variances  $V_{ii^{*}}$  tend to zero, resulting in a constant upper bound. Recently, in the independent setting where variances are unknown, [17] analyzed the comparison of two arms  $i$  and  $j$  . They derived a complexity of the order  $1 / \log \left(1 + \frac{(\mu_i - \mu_j)^2}{(\sigma_i^2 + \sigma_j^2)}\right)$  , where  $\sigma_i^2$  and  $\sigma_j^2$  are the variances of arms  $i$  and  $j$  , respectively. This result is reflected in our bound (9), where instead of the sum of variances, our bound considers the variance of the difference, leading to a sharper bound. In Section 6, we provide a lower bound that nearly matches (9).
6 Lower bounds
In this section, we present lower bounds for the problem of best arm identification with multiple queries per round, following Protocol 1. We provide lower bounds for both the bounded distributions setting (Theorem 6.1) and the Gaussian distributions setting (Theorem 6.2). It is important to note that our lower bounds are derived considering a class of correlated arm distributions. Therefore, the results obtained in the standard one-query-per-round setting [20] do not hold in our setting.
Our first lower bound is derived for the case where arms follow a Bernoulli distribution. Let  $\pmb{\mu} = (\mu_i)_{i\in [[K]]}$  be a sequence of means in  $[1 / 4,3 / 4]$ , denote by  $i^{*}$  the index of the largest mean. Consider a sequence of positive numbers  $\mathbf{V} = (V_{ii^*})_{i\in [[K]]}$ . We define  $\mathbb{B}_K(\pmb {\mu},\pmb {V})$  as the set of Bernoulli arm distributions such that: (i)  $\mathbb{E}[X_i] = \mu_i$ ; (ii)  $\mathrm{Var}(X_i - X_{i^*})\leq V_{ii^*}$  for all  $i\in [[K]]$ .
For Bernoulli distributions, the variances and means are linked. Specifically, Lemma K.8 demonstrates that for any pair of Bernoulli random variables  $(B_{1}, B_{2})$  with means  $(b_{1}, b_{2})$ , the following inequality holds:  $(b_{1} - b_{2}) - (b_{1} - b_{2})^{2} \leq \operatorname{Var}(B_{1} - B_{2}) \leq \min \{2 - (b_{1} + b_{2}); b_{1} + b_{2}\} - (b_{1} - b_{2})^{2}$ .
We assume that the sequences  $\mathbf{V}$  and  $\pmb{\mu}$  satisfy the aforementioned condition for  $(X_{i}, X_{i^{*}})_{i \in [K]}$ , as otherwise the class  $\mathbb{B}_K(\pmb{\mu}, \pmb{V})$  would be an empty set. Theorem 6.1 provides a lower bound on the sample complexity required for best arm identification in the worst-case scenario over the class  $\mathbb{B}_K(\pmb{\mu}, \pmb{V})$ .
Theorem 6.1. Let  $K \geq 2$  and  $\delta \in (0,1)$ . For any  $\delta$ -sound algorithm, we have
$$
\max  _ {\mathcal {B} \in \mathbb {B} _ {K} (\boldsymbol {\mu}, \boldsymbol {V})} \mathbb {E} _ {\mathcal {B}} [ N ] \geq \frac {1}{8} \log (1 / 4 \delta) \sum_ {i \in [ K ] \backslash \{i ^ {*} \}} \max  \left\{\frac {V _ {i i ^ {*}}}{(\mu_ {i ^ {*}} - \mu_ {i}) ^ {2}}, \frac {1}{\mu_ {i ^ {*}} - \mu_ {i}} \right\},
$$
where  $N$  is the total number of queries.
The presented lower bound takes into account the correlation between arms by incorporating the quantities  $V_{ii^*}$  as upper bounds for the variances of the differences between arm  $i$  and the optimal arm  $i^*$ . This lower bound indicates that for class  $\mathbb{B}_K(\pmb{\mu},\pmb{V})$ , Algorithm 2 is nearly optimal.
Next, we provide a lower bound in the Gaussian case. Let  $\pmb{\mu} = (\mu_i)_{i\in [[K]]}$  be a sequence of means, where  $i^{*}$  denotes the index of the largest mean. We also consider a sequence of positive numbers  $\mathbf{V} = (V_{ii^*})_{i\in [[K]]}$ . We define  $\mathbb{G}_K(\pmb {\mu},\pmb {V})$  as the set of Gaussian arm distributions satisfying the following conditions: (i)  $\mathbb{E}[X_i] = \mu_i$ , (ii)  $\operatorname {Var}(X_i - X_{i^*}) = V_{ii^*}$ , and (iii)  $\operatorname {Var}(X_i)\geq 1$  for all  $i\in [[K]]$ . Theorem 6.2 provides a lower bound on the sample complexity required for best arm identification in the worst-case scenario over the class  $\mathbb{G}_K(\pmb {\mu},\pmb {V})$ .
Theorem 6.2. Let  $K \geq 2$  and  $\delta \in (0,1)$ . For any  $\delta$ -sound algorithm, we have
$$
\max _ {\mathcal {G} \in \mathbb {G} _ {K} (\boldsymbol {\mu}, \boldsymbol {V})} \mathbb {E} _ {\mathcal {G}} [ N ] \geq 2 \log (1 / 4 \delta) \sum_ {i \in [ K ] \backslash \{i ^ {*} \}} \frac {1}{\log \left(1 + \frac {(\mu_ {i} - \mu_ {i ^ {*}}) ^ {2}}{V _ {i i ^ {*}} ^ {2}}\right)},
$$
Theorem 6.2 demonstrates that our algorithm achieves near-optimal performance over  $\mathbb{G}_K(\pmb {\mu},\pmb {V})$  up to a logarithmic factor.
7 Numerical simulations
We consider the Gaussian rewards scenario. We compare our algorithm Pairwise-BAI (Algorithm 2) to 3 benchmark algorithms: Hoeffding race [23], adapted to the Gaussian setting (consisting of successive elimination based on Chernoff's bounds) and LUCB [19], which is an instantiation of the upper confidence bound (UCB) method. We assume that the last two algorithms have a prior knowledge on the variances of the arms. The third benchmark algorithm consists of using a successive elimination approach using the empirical estimates of the variances. We evaluated two variations of our algorithm. The first one, Pairwise-BAI+, implemented Algorithm 2 for Gaussian variables. In this instance, we modified line 12 by continuing to sample sub-optimal arms that were eliminated at round  $t$  until round  $2t$  instead of  $82t$ . We stress that both variants guarantee a  $\delta$ -sound decision on the optimal arm (see Theorem 5.1). The second instance involved removing the last instruction, meaning we directly stopped querying sub-optimal arms. Figure 1 displays the average sample complexities for each considered algorithm. As expected, the larger the correlation between arms, the better Pairwise-BAI performs.
The second experiment aims to demonstrate that Algorithm 2 adapts to the covariance between suboptimal arms, as indicated by bound (8) in Theorem 4.1, rather than solely adapting to the correlation with the optimal arm as shown by (9). In this experiment, we consider that the arms are organized into clusters, where each pair of arms within the same cluster exhibits a high correlation (close to 1), while arms from different clusters are independent. According to bound (8) (see also discussion after Theorem 4.1) we expect to observe a gain of up to a factor corresponding to the number of arms per cluster compared to algorithms that do not consider covariance. (Since the total number of arms is kept fixed, the number of arms per cluster scales as the inverse of the number of clusters.)
Figure 1 illustrates the results, showing that the performance of Pairwise-BAI improves as the number of clusters decreases (indicating a larger number of correlated arms). This suggests that increasing
Figure 1: Left figure: the average sample complexities, in 4 scenarios with  $K = 10$  arms with means  $\mu_{i} = i / 10$ . The covariance matrix  $C$  is defined as follows: for  $i \in [[K]]$ ,  $C_{ii} = 1$  and for  $i \neq j$ :  $C_{ij} = \rho$ . We consider 4 scenarios with the correlation:  $\rho \in \{0,0.5,0.7,0.9\}$ . Right figure: the average sample complexities with  $K = 16$  arms with means  $\mu_{i} = i / 10$ . Arms in the same cluster have a correlation of 0.99, and arms from different clusters are independent. We consider 4 scenarios with different number of clusters:  $n_{\mathrm{cl}} \in \{8,4,2,1\}$ . All clusters are of the same size.
the number of correlated sub-optimal arms, which are independent from the optimal arm, still leads to significant performance improvement. These findings support the idea that Pairwise-BAI and Pairwise-BAI+ exhibit behavior that aligns more closely with bound (8) in Theorem 4.1, rather than bound (9).
In both experiments, we observe that Pairwise-BAI+ performs worse compared to Pairwise-BAI, indicating that, empirically, in the given scenarios, continuing to sample sub-optimal arms does not contribute to improved performance. While this modification provides better theoretical guarantees, it may not lead to empirical performance improvements in general scenarios.
8 Conclusion and future directions
This work gives rise to several open questions. Firstly, the presented lower bounds take into account partially the covariance between arms. It would be interesting to explore the development of a more precise lower bound that can adapt to any covariance matrix of the arms. Additionally, in terms of the upper bound guarantees, our focus has been on pairwise comparisons, along with an algorithm that compares candidate arms with convex combinations of the remaining arms (Section B). An interesting direction for further research would involve extending this analysis to an intermediate setting, involving comparisons with sparse combinations.
Acknowledgements: This work is supported by ANR-21-CE23-0035 (ASCAI) and ANR-19-CHIA-0021-01 (BISCOTTE). This work was conducted while EM Saad was in INRAE Montpellier.
[19] Shivaram Kalyanakrishnan, Ambuj Tewari, Peter Auer, and Peter Stone. PAC subset selection in stochastic multi-armed bandits. In ICML, volume 12, pages 655–662, 2012.
[20] Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On the complexity of best-arm identification in multi-armed bandit models. The Journal of Machine Learning Research, 17(1):1-42, 2016.
[21] Che-Yu Liu and Sébastien Bubeck. Most correlated arms identification. In Conference on Learning Theory, pages 623-637. PMLR, 2014.
[22] Shie Mannor and John N Tsitsiklis. The sample complexity of exploration in the multi-armed bandit problem. Journal of Machine Learning Research, 5(Jun):623-648, 2004.
[23] Oded Maron and Andrew Moore. Hoeffding races: Accelerating model selection search for classification and function approximation. Advances in neural information processing systems, 6, 1993.
[24] Andreas Maurer and Massimiliano Pontil. Empirical Bernstein bounds and sample-variance penalization. In COLT 2009 - The 22nd Conference on Learning Theory, Montreal, Quebec, Canada, June 18-21, 2009, 2009.
[25] Volodymyr Mnih, Csaba Szepesvář, and Jean-Yves Audibert. Empirical Bernstein stopping. In Proceedings of the 25th international conference on Machine learning, pages 672-679, 2008.
[26] Andrew W Moore and Mary S Lee. Efficient algorithms for minimizing cross validation error. In Machine Learning Proceedings 1994, pages 190-198. Elsevier, 1994.
[27] Pierre Perrault, Michal Valko, and Vianney Perchet. Covariance-adapting algorithm for semibandits with application to sparse outcomes. In Conference on Learning Theory, pages 3152-3184. PMLR, 2020.
[28] El Mehdi Saad and Gilles Blanchard. Fast rates for prediction with limited expert advice. Advances in Neural Information Processing Systems, 34, 2021.
[29] Alexandre B Tsybakov. Optimal rates of aggregation. In Learning theory and kernel machines, pages 303-313. Springer, 2003.
[30] Nicolas Verzelen. Minimax risks for sparse regressions: Ultra-high dimensional phenomenons. Electronic Journal of Statistics, 6:38 - 90, 2012.
[31] Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge University Press, 2019.