SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Upper bound on the sample complexity  $\tau_{\delta}$  in probability  $(\S)$  or in expectation  $(\dagger)$ . The notation  $\mathcal{O}$  displays the dominating term when  $\delta \to 0$  for the asymptotic regime, and when  $H(\mu) \to +\infty$  (or  $\Delta_i \to 0$ ) for the finite-confidence one. The notation  $\tilde{\mathcal{O}}$  hides polylogarithmic factors. (*) Upper bound on  $\mathbb{E}_{\mu}[\tau_{\delta} \mathbb{1}(\mathcal{E})]$  where  $\mathbb{P}[\mathcal{E}^{\mathbb{G}}] \leq \gamma$ . (**) The asymptotic upper bound holds for instances having all distinct means, while the non-asymptotic one doesn't require this assumption.
Algorithm	Asymptotic behavior	Finite-confidence behavior
LUCB1† [24]	O (H(μ) log(1/δ))	O (H(μ) log H(μ))
Exp-Gap§ [25]	O (H(μ) log(1/δ))	O(∑i≠i* Δi-2 log log Δi-1)
lil' UCB§ [19]	O (H(μ) log(1/δ))	O(∑i≠i* Δi-2 log log Δi-1)
DKM† [12]	T*(μ) log(1/δ) + ŷ(√log(1/δ))	O (KT*(μ)2)
Peace§ [26]	O (T*(μ) log(1/δ))	O (H(μ) log(K/Δmin))
FWS† [42]	T*(μ) log(1/δ) + O(log log(1/δ))	O (eK H(μ)19/2)
EBS† [6]*	T*(μ) log(1/δ) + o(1)	O (KH(μ)4/w2min)
TTUCB**	T*(μ) log(1/δ) + O(log log(1/δ))	O ((H(μ) log H(μ))α) with α > 1
Since it yields a larger  $T_0(\delta)$ , we are trading-off asymptotic terms for improved non-asymptotic ones. We illustrate this with two archetypal instances. For the "1-sparse" instance, in which  $\mu_1 > 0$  and  $\mu_i = 0$  for all  $i \neq 1$ , we have by symmetry that  $2w_{1/2}^\star(\mu)_i = 1/(K-1)$  for all  $i \neq 1$ . Therefore, we have  $C_0(w_0) = \mathcal{O}(K/\varepsilon)$  since  $d_\mu(w_0) = 0$  for all  $w_0 \in [0,1/(K-1)]$ . The "almost dense" instance is such that  $\mu_1 = 1$ ,  $\mu_K = 0$  and  $\mu_i = 1 - \gamma$  for all  $i \notin \{1,K\}$ . By symmetry, there exists a function  $h:[0,1) \to [0,(K-1)^{-1})$  with  $\lim_{\gamma \to 0} h(\gamma) = 0$ , such that  $2w_{1/2}^\star(\mu)_K = h(\gamma)$  and  $2w_{1/2}^\star(\mu)_i = (1 - h(\gamma))/(K-2)$  for all  $i \notin \{1,K\}$ . While  $\lim_{\gamma \to 0} C_0(0) = +\infty$ , we obtain  $\lim_{\gamma \to 0} C_0(w_0) = \mathcal{O}(K/\varepsilon)$  by taking  $w_0 = (1 - h(\gamma))/(K-2)$  since  $d_\mu(w_0) = 1$ .
Comparison with existing upper bounds Table 1 summarizes the asymptotic and non-asymptotic scalings of the upper bound on the sample complexity of existing BAI algorithms. Among the class of asymptotically  $(\beta)$ -optimal algorithms, very few of them also enjoy non-asymptotic guarantees, e.g. the analyses of Track-and-Stop and Top Two algorithms are asymptotic. The gamification approach of [12] is the first attempt to provide both. Their non-asymptotic upper bound on  $\mathbb{E}_{\mu}[\tau_{\delta}]$  involves an implicit time  $T_{1}(\delta)$  which scales with  $KT^{\star}(\mu)^{2}$  and is only valid for  $\log (1 / \delta) \gtrsim KT^{\star}(\mu)$  (see Lemma 2, with constants in Appendix D.7). Let  $T_{\delta}^{\star} \coloneqq T^{\star}(\mu) \log (1 / \delta)$ . As a first order approximation, they obtain  $T_{1}(\delta) \approx T_{\delta}^{\star} + \Theta \left( \sqrt{T_{\delta}^{\star} \log T_{\delta}^{\star}} \right)$ , and we obtain  $T_{0}(\delta) \approx \Theta \left( T_{\delta}^{\star} + \log T_{\delta}^{\star} \right)$  (Lemma D.13). [42] were the first to obtain an upper bound on  $\mathbb{E}_{\mu}[\tau_{\delta}]$  of the form  $\Theta (T_{\delta}^{\star} + \log \log (1 / \delta))$ . While they improved the second-order  $\delta$ -dependent term, the  $\delta$ -independent term scales with  $e^{K}H(\mu)^{19 / 2}$  (see their Theorem 2 for  $\varepsilon^{-1} \gtrsim T^{\star}(\mu)$ , with constants given by Appendix N). The algorithm proposed by [6] has a non-asymptotic upper bound on  $\mathbb{E}_{\mu}[\tau_{\delta} \mathbb{1}(\mathcal{E})]$  of the form  $(1 + \varepsilon)T_{\delta}^{\star} + f(\mu, \delta)$  which is valid for  $\log (1 / \delta) \gtrsim w_{\min}^{-2} K / \Delta_{\min}$ , where  $\mathcal{E}$  is such that  $\mathbb{P}_{\mu}(\mathcal{E}^{\complement}) \leq \gamma$ . Since  $f(\mu, \delta) = \delta \rightarrow 0$  o(1), they obtain a better  $\delta$ -dependency. However,  $f(\mu, \delta)$  is arbitrarily large when  $w_{\min} \coloneqq \min_{i \in [K]} w^{\star}(\mu)_i$  is arbitrarily small since it scales with  $KH(\mu)^{4} / w_{\min}^{2}$ . Therefore, they suffer from the pitfall which we avoided by clipping. In light of Table 1, TTUCB enjoys the best scaling when  $H(\mu) \to +\infty$  in the class of asymptotically  $(\beta)$ -optimal BAI algorithms.
The LUCB1 algorithm [24] has a structure similar to a Top Two algorithm, with the difference that LUCB samples both the leader and the challenger instead of choosing one. As LUCB1 satisfies  $\mathbb{E}_{\mu}[\tau_{\delta}] \leq 292H(\mu)\log (H(\mu) / \delta) + 16$ , it enjoys better scaling when  $H(\mu) \to +\infty$  than TTUCB. Since the empirical allocation of LUCB1 is not converging towards  $w_{1/2}^{\star}(\mu)$ , it is not asymptotically 1/2-optimal. The Peace algorithm [26] has a non-asymptotic upper bound on  $\tau_{\delta}$  of the form  $\mathcal{O}((T_{\delta}^{\star} + \gamma^{\star}(\mu))\log (K / \Delta_{\mathrm{min}}))$  holding with probability  $1 - \delta$ . The term  $\gamma^{\star}(\mu)$  is a Gaussian-width which originates from concentration on the suprema of Gaussian processes and satisfies  $\gamma^{\star}(\mu) = \Omega(H(\mu))$ .
Another class of BAI algorithms focuses on the dependency in the gaps  $\Delta_{i} \coloneqq \mu_{i^{\star}} - \mu_{i}$ , and derive non-asymptotic upper bound on  $\tau_{\delta}$  holding with high probability. [25, 19, 8, 9] gives  $\delta$ -PAC algorithms with an upper bound of the form  $\mathcal{O}(H(\mu)\log (1 / \delta) + \sum_{i\neq i^{\star}}\Delta_{i}^{-2}\log \log \Delta_{i}^{-1})$ , and [19] shows that
for two arms the dependency  $\Delta^{-2}\log \log \Delta^{-1}$  is optimal when  $\Delta \to 0$ . While those algorithms obtain the best scaling when  $H(\mu)\rightarrow +\infty$ , they are not asymptotically ( $\beta$ -)optimal.
3 Non-asymptotic analysis
3.1 Proof sketch of Theorem 2.4
Existing analyses of Top Two algorithms are asymptotic in nature and requires too much control on the empirical means and proportions to yield any meaningful information in the finite-confidence regime. Therefore, we adopt a different approach which resembles the non-asymptotic analysis of [12]. We first define concentration events to control the deviations of the random variables used in the UCB leader and the TC challenger. For all  $n > K$ , let  $\mathcal{E}_n \coloneqq \bigcap_{i\in [K]}\bigcap_{t\in [n^{5 / 6},n]}(\mathcal{E}_{t,i}^1\cap \mathcal{E}_{t,i}^2)$  where
$$
\mathcal {E} _ {t, i} ^ {1} := \left\{\sqrt {N _ {t , i}} | \mu_ {t, i} - \mu_ {i} | <   \sqrt {6 \log t} \right\} \quad \text {a n d} \quad \mathcal {E} _ {t, i} ^ {2} := \left\{\frac {\left(\mu_ {t , i ^ {\star}} - \mu_ {t , i}\right) - \left(\mu_ {i ^ {\star}} - \mu_ {i}\right)}{\sqrt {1 / N _ {t , i ^ {\star}} + 1 / N _ {t , i}}} > - \sqrt {8 \log t} \right\}.
$$
Using Lemmas D.8 and E.6, the proof boils down to constructing a time  $T(\delta)$  after which  $\mathcal{E}_n \subset \{\tau_\delta \leq n\}$  for  $n > T(\delta)$  since it would yield that  $\mathbb{E}_{\mu}[\tau_{\delta}] \leq T(\delta) + 12K$ .
Let  $n > K$  such that  $\mathcal{E}_n \cap \{n < \tau_{\delta}\}$  holds true, and  $t \in [n^{5/6}, n]$  such that  $B_t^{\mathrm{UCB}} = i^\star$ . Using that  $t \leq n < \tau_{\delta}$ , under  $\bigcap_{i \neq i^\star} \mathcal{E}_{t,i}^2$ , the stopping condition yields that
$$
\sqrt {2 c (n - 1 , \delta)} \geq ((\mu_ {i ^ {\star}} - \mu_ {C _ {t} ^ {\mathrm {T C}}}) (1 / N _ {t, i ^ {\star}} ^ {i ^ {\star}} + 1 / N _ {t, C _ {t} ^ {\mathrm {T C}}} ^ {i ^ {\star}}) ^ {- 1 / 2} - \sqrt {8 \log n}) _ {+}.
$$
Let  $w_{1/2}^{\star}$  be the unique element of  $w_{1/2}^{\star}(\mu)$ . Lemma 3.1 links the empirical proportions  $N_{t,i}^{i^{\star}} / (t - 1)$  to  $w_{1/2,i}^{\star}$  for  $i \in \{i^{\star}, C_t^{\mathrm{TC}}\}$ . It is the key technical challenge of our non-asymptotic proof strategy.
Lemma 3.1. Let  $\varepsilon \in (0,1]$ . There exist  $T_{\mu} > 0$  such that for all  $n > T_{\mu}$  such that  $\mathcal{E}_n \cap \{n < \tau_\delta\}$  holds true, there exists  $t \in [n^{5/6}, n]$  with  $B_t^{UCB} = i^\star$ , which satisfies
$$
(n - 1) \left(1 / N _ {t, i ^ {\star}} ^ {i ^ {\star}} + 1 / N _ {t, C _ {t} ^ {T C}} ^ {i ^ {\star}}\right) \leq (1 + \varepsilon) ^ {2} \left(2 + 1 / w _ {1 / 2, C _ {t} ^ {T C}} ^ {\star}\right) / \beta .
$$
Before proving Lemma 3.1, we conclude the proof of Theorem 2.4. Let  $\varepsilon, T_{\mu}$  and  $t$  as in Lemma 3.1 and  $T(\delta) := \sup \{n \mid n - 1 \leq T_{1/2}^{\star}(\mu)(1 + \varepsilon)^2 (\sqrt{c(n - 1,\delta)} + \sqrt{4\log n})^2 / \beta\}$ . For all  $n > \max \{T_{\mu}, T(1)\}$ , we have  $\sqrt{c(n - 1,\delta)} + \sqrt{4\log n} \geq \sqrt{\beta(n - 1)T_{1/2}^{\star}(\mu)^{-1}(1 + \varepsilon)^{-2}}$ . Therefore, we have proved that  $\mathcal{E}_n \cap \{n < \tau_{\delta}\} = \emptyset$  for all  $n > \max \{T_{\mu}, T(\delta)\}$ . This concludes the proof.
Provided that  $B_{t} = i^{\star}$ , the above only used the stopping condition and the TC challenger, and no other properties of the leader. Lemma 3.2 shows that  $\bar{B}_{t}^{\mathrm{UCB}} = i^{\star}$ , except for a sublinear number of times. Section 3.3 exhibits sufficient conditions on a regret minimization algorithm to obtain a non-asymptotic upper bound.
Lemma 3.2. Under the event  $\bigcap_{k\in [K]}\bigcap_{t\in [n^{5 / 6},n]} \mathcal{E}_{t,k}^1$ , we have  $L_{n,i^{\star}} \geq n - 1 - 24H(\mu) \log n - 2K$ .
Proof sketch of Lemma 3.1 The key technical challenge is to link  $N_{t,C_t^{\mathrm{TC}}}^{i^\star} / (n - 1)$  with  $w_{1/2,C_t^{\mathrm{TC}}}^\star$ . We adopt the approach used to analyze of APT [30]: consider an arm being over-sampled and study the last time this arm was pulled. By the pigeonhole principle, at time  $n$ ,
$$
\exists k _ {1} \neq i ^ {\star}, \text {s . t .} N _ {n, k _ {1}} ^ {i ^ {\star}} \geq 2 \left(L _ {n, i ^ {\star}} - N _ {n, i ^ {\star}} ^ {i ^ {\star}}\right) w _ {1 / 2, k _ {1}} ^ {\star}. \tag {5}
$$
Let  $t_1$  be the last time at which  $B_t^{\mathrm{UCB}} = i^\star$  and  $C_t^{\mathrm{TC}} = k_1$ , hence  $N_{t_1,k_1}^{i^\star} \geq N_{n,k_1}^{i^\star} - 1$ . Using Lemmas 2.2 and 3.2, we show that  $N_{t_1,k_1}^{i^\star} \gtrsim w_{1/2,k_1}^\star (n-1)$ , hence  $t_1 \geq n^{5/6}$  for  $n$  large enough (see Appendix D.2). Then, we need to link  $N_{t_1,i^\star}^{i^\star}$  to  $(n-1)/2$ . When  $w_{1/2,k_1}^\star$  is small, (5) can be true at  $t_1 = n^{5/6}$ , hence there is no hope to show that  $t_1 = n - o(n)$ . To circumvent this problem, we link  $N_{t_1,i^\star}^{i^\star}$  to  $N_{t_1,k_1}^{i^\star}$  thanks to Lemma 2.2, and use that
$$
\frac {n - 1}{N _ {t _ {1} , i ^ {\star}} ^ {i ^ {\star}}} + \frac {n - 1}{N _ {t _ {1} , k _ {1}} ^ {i ^ {\star}}} \leq \left(2 + \frac {n - 1}{N _ {t _ {1} , k _ {1}} ^ {i ^ {\star}}}\right) \left(\frac {N _ {t _ {1} , k _ {1}} ^ {i ^ {\star}}}{N _ {t _ {1} , i ^ {\star}} ^ {i ^ {\star}}} + 1\right) \leq 2 (1 + \varepsilon) ^ {2} (2 + 1 / w _ {1 / 2, k _ {1}} ^ {\star}),
$$
for  $n > T_{\mu}(w_{-})$  with  $T_{\mu}(w_{-}) \leq \max \{C_{\mu}^{1,2}, (2 / (\varepsilon w_{-}) + 1)^{6}, (2 / \varepsilon)^{1.2}\}$  (Lemmas D.10 and D.11), where  $w_{-} = \min_{i \neq i^{\star}} w_{1/2,i}^{\star} > 0$  lower bounds  $w_{1/2,k_1}^{\star}$ . This concludes the proof for  $w_{0} = 0$ . The (sub-optimal) multiplicative factor 2 in  $T_{0}(\delta)$  comes from the inequality (6). To remove it, we need to control the deviation between the empirical proportion of arm  $i$  and  $w_{1/2,i}^{\star}$  for all  $i \in [K]$ . Nevertheless, TTUCB is asymptotically 1/2-optimal (Theorem 2.3).
Refined analysis For  $w_0 \in (0, (K - 1)^{-1}]$ , we clip  $\min_{i \neq i^*} w_{1/2,i}^\star$  by  $w_0 / 2$  (see Appendix D). Our method can be used to analyze other algorithms, and it improves existing results on APT.
3.2 Beyond Gaussian distributions
Theorems 2.3 and 2.4 hold for sub-Gaussian r.v. thanks to direct adaptations of concentration results (Lemmas 2.1, E.2 and E.5). The situation is akin to the regret bound of UCB: it holds for any sub-Gaussian, but it is close to optimality in a distribution-dependent sense only for Gaussians. However, if the focus is on asymptotically  $\beta$ -optimal algorithms, then it is challenging to express the characteristic time  $T^{\star}(\mu)$  for the non-parametric class of sub-Gaussian distributions.
The TTUCB algorithm can also be defined for more general distributions such as single-parameter exponential families or bounded distributions. It is only a matter of adapting the definition of the UCB leader and the TC challenger. For bounded distributions, the UCB leader was studied in [1] and the TC challenger was analyzed in [21]. Leveraging their unified analysis of Top Two algorithms with our tracking-based results, we can show asymptotic  $\beta$ -optimality of TTUCB for bounded distributions and single-parameter exponential families with sub-exponential tails. We believe that non-asymptotic guaranties could be obtained for more general distributions, but it will come at the price of more technical arguments and less explicit non-asymptotic terms.
3.3 Generic regret minimizing leader
Our non-asymptotic analysis highlights that any regret minimization algorithm that selects the arm  $i^{\star}$  except for a sublinear number of times (Property 1) can be used as leader with the TC challenger.
Property 1. There exists  $(\tilde{\mathcal{E}}_n)_n$  with  $\sum_{n}\mathbb{P}_{\mu}(\tilde{\mathcal{E}}_n^{\mathbb{C}}) < +\infty$  and a function  $h$  with  $h(n) = \mathcal{O}(n^{\gamma})$  for some  $\gamma \in (0,1)$  such that under event  $\tilde{\mathcal{E}}_n$ ,  $L_{n,i^{\star}}\geq n - 1 - h(n)$ .
For asymptotic guarantees, the sufficient properties on the leader from [21] are weaker since they are even satisfied by the greedy choice  $B_{n} = \hat{i}_{n}$ . While Top Two algorithms were introduced by [38] to adapt Thompson Sampling to BAI, we have shown that other regret minimization algorithms can be used: the Top Two method is a generic wrapper to convert any regret minimization algorithm into a best arm identification strategy.
The regret of an algorithm at time  $n$ ,  $\bar{R}_n = \sum_{i \neq i^{\star}} \Delta_i N_{n,i}$ , is almost always studied through its expectation  $\mathbb{E}[\bar{R}_n]$ . This is however not sufficient for our application. We need to prove that with high probability,  $N_{n,i}$  is small for all arm  $i \neq i^{\star}$ . Such guarantees are known for UCB [3] and ETC [29], but are yet unknown for Thompson Sampling. We cannot in general obtain a good enough bound on  $N_{n,i}$  from a bound on  $\mathbb{E}[\bar{R}_n]$ . However, we can if we have high probability bounds on  $\bar{R}_n$ . Suppose that a regret minimization algorithm  $\mathrm{Alg}_1$  satisfies Property 2 and is independent of the horizon  $n$ .
Property 2. There exists  $s > 1$ ,  $\gamma \in (0,1)$ ,  $(\mathcal{E}_{n,\delta})_{(n,\delta)}$  with  $\sum_{n}\mathbb{P}_{\mu}[\mathcal{E}_{n,n^{-s}}^{\mathsf{C}}] < +\infty$  and a function  $h$  with  $h(n,n^{-s}) = \mathcal{O}(n^{\gamma})$  such that under event  $\mathcal{E}_{n,\delta}$ ,  $\bar{R}_n \leq h(n,\delta)$ .
Let  $\mathrm{Alg}_2$  be the algorithm  $\mathrm{Alg}_1$  used in a Top Two procedure, but which uses only the observations obtained at times  $n$  such that  $I_n = B_n$  and discards the rest. Let  $\tilde{\mathcal{E}}_n = \mathcal{E}_{n,n^{-s}}$  and  $\Delta_{\min} = \min_{i \neq i^*} \Delta_i$ . Then, under  $\tilde{\mathcal{E}}_n$ ,  $\mathrm{Alg}_2$  satisfies  $\sum_{i \neq i^*} N_{n,i}^i \leq h(n,n^{-s}) / \Delta_{\min}$  and Lemma 2.2 yields  $N_{n,i^*}^{i^*} \geq \beta (n - 1) - h(n,n^{-s}) / \Delta_{\min} - K / 2$ . Therefore, Property 1 holds for  $\tilde{\mathcal{E}}_n$  and  $h(n) = (h(n,n^{-s}) / \Delta_{\min} + K / 2 + 1) / \beta$ . Given a specific algorithm, a finer analysis could avoid discarding information by using  $\mathrm{Alg}_1$  with every observations.
3.4 Adaptive proportions
Given a fixed allocation  $\beta$ , any Top Two algorithm can at best be asymptotically  $\beta$ -optimal. Since the optimal allocation  $\beta^{\star} \in \arg \min_{\beta \in (0,1)} T_{\beta}^{\star}(\mu)$  is unknown, it should be learned from the observations by a Top Two algorithm using an adaptive proportion  $\beta_{n}$  at time  $n$ . Recently, [44] proposes IDS to choose  $\beta_{n}$  in an adaptive manner. For BAI with Gaussian observations, IDS yields  $\beta_{n} = N_{n,C_{n}} / (N_{n,B_{n}} + N_{n,C_{n}})$ . Let  $\bar{\beta}_{n}^{i} \coloneqq \frac{1}{L_{n,i}} \sum_{t \in [n-1]} \beta_{t} \mathbb{1}(B_{t} = i)$  be the average proportion when arm  $i$  was the leader before time  $n$ . Tracking with IDS requires to use  $\bar{\beta}_{n+1}^{B_{n}}$  instead of  $\beta$ . Using the analysis of [44], it is reasonable to believe that one could obtain asymptotic optimality of TTUCB with IDS. However, it is not clear how to adapt the non-asymptotic analysis since it heavily relies on  $\beta$  being fixed and bounded away from  $\{0,1\}$ . Experiments with IDS are available in Appendix G.2.1.
In the moderate regime ( $\delta = 0.1$ ), we assess the empirical performance of TTUCB with bonus  $g_{m}$  and concentration parameters  $s = \alpha = 1.2$ . As benchmarks, we compare our algorithm with three sampling-based Top Two algorithms: TTTS, T3C and  $\beta$ -EB-TCI. In addition, we consider Track-and-Stop (TaS) [17], FWS [42], DKM [12], LUCB [24] and uniform sampling. At time  $n$ , the LUCB algorithm computes a leader and a challenger, then sample them both (see Appendix G.1). To provide a clear comparison with Top Two algorithms, we define a new  $\beta$ -LUCB algorithm which sample the leader with probability  $\beta$ , else sample the challenger. At the exception of LUCB and  $\beta$ -LUCB which have their own stopping rule, all algorithms use the stopping rule (1) with the heuristic threshold  $c(n,\delta) = \log((1 + \log n)/\delta)$ . Even though this choice is not sufficient to prove  $\delta$ -correctness, it yields an empirical error which is several orders of magnitude lower than  $\delta$ . Top Two algorithms and  $\beta$ -LUCB use  $\beta = 1/2$ . To allow for a fair numerical comparison, LUCB and  $\beta$ -LUCB use  $\sqrt{2c(n - 1,\delta)/N_{n,i}}$  as bonus, which is too tight to yield valid confidence intervals. Supplementary experiments are available in Appendix G.
Random instances We assess the performance on 5000 random Gaussian instances with  $K = 10$  such that  $\mu_1 = 0.6$  and  $\mu_i \sim \mathcal{U}([0.2, 0.5])$  for all  $i \neq 1$ . Numerically, we observe  $w^{\star}(\mu)_{i^{\star}} \approx 1/3 \pm 0.02$  (mean ± std). In Figure 1(a), we see that TTUCB performs on par with existing Top Two algorithms, and slightly outperforms TaS and FWS. Our algorithm achieves significantly better result than DKM, LUCB, 1/2-LUCB and uniform sampling. The CPU running time is reported in Table 4, and the observed empirical errors before stopping is displayed in Figure 3 (Appendix G.2).
Larger sets of arms We evaluate the impact of larger number of arms. The "1-sparse" scenario of [20] sets  $\mu_{1} = 1 / 4$  and  $\mu_{i} = 0$  for all  $i\neq 1$ , i.e.  $H(\mu) = 32(K - 1)$  (see Appendix G.2 for other instances). We consider algorithms with low computational cost. In Figure 1(b), all algorithms have the same linear scaling in  $K$  (i.e. in  $H(\mu)$ ). Faced with an increase in the number of arms, the TS leader used in T3C appears to be more robust than the UCB leader in TTUCB. This is a common feature of UCB algorithms which have to overcome the bonus of sub-optimal arms.
Figure 1: Empirical stopping time on (a) random instances  $(K = 10)$  and (b) "1-Sparse" instances.
In this paper, we have shown the first non-asymptotic upper bound on the expected sample complexity of a Top Two algorithm, which holds for any error level and for any instance having a unique best arm. Furthermore, we have demonstrated that the TTUCB algorithm achieves competitive empirical performance compared to other algorithms, including Top Two methods.
While our guarantees hold for a fixed proportion  $\beta$  allocated to the leader, [44] recently introduced IDS to define an adaptive proportion  $\beta_{n}$  at time  $n$  and show asymptotic optimality for Gaussian distributions. Deriving guarantees for IDS for single-parameter exponential families is a challenging open problem. Finally, Top Two algorithms are a promising method to tackle complex settings. While heuristics exist for some structured bandits such as Top- $k$ , it would be interesting to efficiently adapt Top Two methods to deal with sophisticated structure, e.g. linear bandits.
Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr). This work has been partially supported by the THIA ANR program "AI_PhD@Lille". The authors acknowledge the funding of the French National Research Agency under the project FATE (ANR22-CE23-0016-01).
[1] S. Agrawal, S. K. Juneja, and W. M. Koolen. Regret minimization in heavy-tailed bandits. In Conference on Learning Theory, 2021.
[2] K. Ariu, M. Kato, J. Komiyama, K. McAlinn, and C. Qin. Policy choice and best arm identification: Asymptotic analysis of exploration sampling under posterior weighted policy regret. arXiv preprint arXiv:2109.08229, 2021.
[3] J.-Y. Audibert, R. Munos, and C. Szepesvári. Exploration-exploitation trade-off using variance estimates in multi-armed bandits. Theoretical Computer Science, 410(19), 2009.
[4] J.-Y. Audibert, S. Bubeck, and R. Munos. Best Arm Identification in Multi-armed Bandits. In Conference on Learning Theory, 2010.
[5] P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2):235-256, 2002.
[6] A. Barrier, A. Garivier, and T. Kocák. A non-asymptotic approach to best-arm identification for gaussian bandits. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, 2022.
[7] K. Chaloner and I. Verdinelli. Bayesian experimental design: A review. Statistical Science, 1995.
[8] L. Chen, J. Li, and M. Qiao. Towards instance optimal bounds for best arm identification. Conference on Learning Theory, 2017.
[9] L. Chen, J. Li, and M. Qiao. Nearly instance optimal sample complexity bounds for top-k arm selection. In Artificial Intelligence and Statistics, 2017.
[10] H. Chernoff. Sequential design of Experiments. The Annals of Mathematical Statistics, 30(3): 755-770, 1959.
[11] R. Degenne. Impact of structure on the design and analysis of bandit algorithms. PhD thesis, Université de Paris, 2019.
[12] R. Degenne, W. M. Koolen, and P. Menard. Non-asymptotic pure exploration by solving games. In Advances in Neural Information Processing Systems (NeurIPS), 2019.