SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $M \geq \max \{N_0, N_1, N_2, N_3\}$  and  $n \geq N_5 = \max \{\frac{M - 1}{\varepsilon} + 1, M\}$ . Let  $t_{n - 1,i}(\varepsilon) = \max \left\{t \leq n \mid \frac{N_{t,i}}{n - 1} \leq w_{\beta,i}^{\star} + \varepsilon \right\}$ . Since  $\frac{N_{t,i}}{n - 1} \leq \frac{N_{t,i}}{t - 1}$  for  $t \leq n$ , we have
$$
\begin{array}{l} \frac {N _ {n , i}}{n - 1} \leq \frac {M - 1}{n - 1} + \frac {1}{n - 1} \sum_ {t = M} ^ {n} \mathbb {1} \left(B _ {t} ^ {\mathrm {U C B}} = i ^ {\star}, C _ {t} ^ {\mathrm {T C}} = i, I _ {t} = i\right) \\ \leq \varepsilon + \frac {1}{n - 1} \sum_ {t = M} ^ {n} \mathbb {1} \left(\frac {N _ {t , i}}{n - 1} \leq w _ {\beta , i} ^ {\star} + \varepsilon\right) \mathbb {1} \left(B _ {t} ^ {\mathrm {U C B}} = i ^ {\star}, C _ {t} ^ {\mathrm {T C}} = i, I _ {t} = i\right) \\ \leq \varepsilon + \frac {N _ {t _ {n - 1 , i} (\varepsilon) , i}}{n - 1} \leq w _ {\beta , i} ^ {\star} + 2 \varepsilon . \\ \end{array}
$$
As a similar upper bound is shown in the proof of Lemma F.8, we obtain  $\frac{N_{n,i}}{n - 1} \leq w_{\beta,i}^{\star} + 2\varepsilon$  for all  $i \in [K]$  and all  $n \geq N_4 \coloneqq \max \{N_0, N_1, N_2, N_3, N_5\}$ . Since  $\frac{N_{n,i}}{n - 1}$  and  $w_{\beta,i}^{\star}$  sum to 1, we obtain for all  $n \geq N_4$  and all  $i \in [K]$ ,
$$
\frac {N _ {n , i}}{n - 1} = 1 - \sum_ {k \neq i} \frac {N _ {n , k}}{n - 1} \geq 1 - \sum_ {k \neq i} \left(w _ {\beta , k} ^ {\star} + 2 \varepsilon\right) = w _ {\beta , i} ^ {\star} - 2 (K - 1) \varepsilon .
$$
Therefore, for all  $n \geq N_4$  and all  $i \in [K]$ ,  $\left|\frac{N_{n,i}}{n - 1} - w_{\beta,i}^{\star}\right| \leq 2(K - 1)\varepsilon$ . Since  $\mathbb{E}_{\mu}[N_4] < +\infty$  and we showed the result for all  $\varepsilon$ , this concludes the proof.
Lemma F.11 shows that  $\mathbb{E}_{\mu}[T_{\beta}^{\varepsilon}] < +\infty$ , it is a direct consequence of Lemma F.10.
Lemma F.11. Let  $\varepsilon > 0$  and  $T_{\beta}^{\varepsilon}$  as in (20). Under the TTUCB sampling rule, we have  $\mathbb{E}_{\mu}[T_{\beta}^{\varepsilon}] < +\infty$ .
Theorem 2.3 is a direct consequence of Lemma F.1 and Lemma F.11.
G Implementation details and additional experiments
The implementations details and supplementary experiments are detailed in Appendix G.1 and Appendix G.2.
G.1 Implementation details
Top Two sampling rules Existing Top Two algorithms are based on a sampling procedure to choose between the leader  $B_{n}$  and the challenger  $C_n$ , namely sample  $B_{n}$  with probability  $\beta$ , else sample  $C_n$ . The difference lies in the choice of the leader and the challenger themselves.
TTTS [38] uses a TS (Thompson Sampling) leader and a RS (Re-Sampling) challenger based on a sampler  $\Pi_{n}$ . For Gaussian bandits, the sampler  $\Pi_{n}$  is the posterior distribution  $\times_{i\in [K]}\mathcal{N}(\mu_{n,i},1 / N_{n,i})$  given the improper prior  $\Pi_1 = (\mathcal{N}(0, + \infty))^K$ . The TS leader is  $B_n^{\mathrm{TS}}\in \arg \max_{i\in [K]}\theta_i$  where  $\theta \sim \Pi_{n}$ . The RS challenger samples vector of realizations  $\theta \in \Pi_{n}$  until  $B_{n}\notin \arg \max_{i\in [K]}\theta_{i}$ , then it is defined as  $C_n^{\mathrm{RS}}$  arg  $\max_{i\in [K]}\theta_{i}$  for this specific vector of realization. When the posterior  $\Pi_{n}$  and the leader  $B_{n}$  have almost converged towards the Dirac distribution on  $\mu$  and the best arm  $i^{\star}(\mu)$  respectively, the event  $B_{n}\notin \arg \max_{i\in [K]}\theta_{i}$  becomes very rare. The experiments in [21] reveals that computing the RS challenger can require more than millions of re-sampling steps. Therefore, the RS challenger can become computationally intractable even for Gaussian distribution where sampling from  $\Pi_{n}$  can be done more efficiently.
T3C [39] combines the TS leader and the TC challenger.  $\beta$ -EB-TCI [21] combines the EB leader with the TCI challenger defined as
$$
C _ {n} ^ {\mathrm {T C I}} = \underset {i \neq B _ {n}} {\arg \min } \mathbb {1} \left(\mu_ {n, B _ {n}} > \mu_ {n, i}\right) \frac {\left(\mu_ {n , B _ {n}} - \mu_ {n , i}\right) ^ {2}}{2 \left(1 / N _ {n , B _ {n}} + 1 / N _ {n , i}\right)} + \log (N _ {n, i}).
$$
Since the TC and TCI challenger can re-use computations from the stopping rule, those two challengers have no additional computational cost which makes it very attractive for larger sets of arms.
Each tracking procedure has a computational and memory cost in  $\mathcal{O}(1)$ , hence total cost of  $\mathcal{O}(K)$  for the  $K$  independent procedures. Each UCB index has a computational and memory cost in  $\mathcal{O}(1)$ , hence total cost of  $\mathcal{O}(K)$  to compute  $B_{n}^{\mathrm{UCB}}$  as in (3). Each TC index (or stopping rule index) has a computational and memory cost in  $\mathcal{O}(1)$ , hence total cost of  $\mathcal{O}(K)$  to compute  $C_{n}^{\mathrm{TC}}$  as in (4). Therefore, the per-round computational and memory cost of TTUCB is in  $\mathcal{O}(K)$ .
Other sampling rules At each time  $n$ , Track-and-Stop (TaS) [17] computes the optimal allocation for the current empirical mean,  $w_{n} = w^{\star}(\mu_{n})$ . Given  $w_{n} \in \triangle_{K}$ , it uses a tracking procedure to obtain an arm  $I_{n}$  to sample. On top of this tracking a forced exploration is used to enforce convergence towards the optimal allocation for the true unknown parameters. The optimization problem defining  $w^{\star}(\mu)$  can be rewritten as solving an equation  $\psi_{\mu}(r) = 0$ , where
The function  $\psi_{\mu}$  is decreasing, and satisfies  $\lim_{r\to +\infty}\psi_{\mu}(r) = -1$  and  $\lim_{y\to 1 / \min_{i\neq i^*}(\mu_{i^*} - \mu_i)^2}F_\mu (y) = +\infty$ . For the practical implementation of the optimal allocation, we use the approach of [17] and perform binary searches to compute the unique solution of  $\psi_{\mu}(r) = 0$ . A faster implementation based on Newton's iterates was proposed by [6] after proving that  $\psi_{\mu}$  is convex. While this improvement holds only for Gaussian distributions, the binary searches can be used for more general distributions.
DKM [12] view  $T^{\star}(\mu)^{-1}$  as a min-max game between the learner and the nature, and design saddle-point algorithms to solve it sequentially. At each time  $n$ , a learner outputs an allocation  $w_{n}$ , which is used by the nature to compute the worst alternative mean parameter  $\lambda_{n}$ . Then, the learner is updated based on optimistic gains based on  $\lambda_{n}$ .
FWS [42] alternates between forced exploration and Frank-Wolfe (FW) updates.
LUCB [24] samples and stops based on upper/ lower confidence indices for a bonus function  $g$  .For Gaussian distributions, it rewrites for all  $i\in [K]$  as
At each time  $n$ , it samples  $\hat{i}_n$  and arg  $\max_{i \neq \hat{i}_n} U_{n,i}$  and stops when  $L_{n,\hat{i}_n} \geq \max_{i \neq \hat{i}_n} U_{n,i}$ . The  $\beta$ -LUCB algorithm samples  $\hat{i}_n$  with probability  $\beta$ , else it samples  $\arg \max_{i \neq \hat{i}_n} U_{n,i}$ . The stopping time is the same as for LUCB.
Adaptive proportions [44] propose IDS, which is an update mechanism for  $\beta$  based on the optimality conditions for the problem underlying  $T^{\star}(\mu)$ . For Gaussian with known homoscedastic variance, IDS can be written as
$$
\beta_ {n} = \frac {N _ {n , B _ {n}} d _ {\mathrm {K L}} (\mu_ {n , B _ {n}} , u _ {n} (B _ {n} , C _ {n}))}{N _ {n , B _ {n}} d _ {\mathrm {K L}} (\mu_ {n , B _ {n}} , u _ {n} (B _ {n} , C _ {n})) + N _ {n , C _ {n}} d _ {\mathrm {K L}} (\mu_ {n , C _ {n}} , u _ {n} (B _ {n} , C _ {n}))} = \frac {N _ {n , C _ {n}}}{N _ {n , B _ {n}} + N _ {n , C _ {n}}},
$$
$$
u _ {n} (i, j) = \inf  _ {x \in \mathbb {R}} \left[ N _ {n, i} d _ {\mathrm {K L}} (\mu_ {n, i}, x) + N _ {n, j} d _ {\mathrm {K L}} (\mu_ {n, j}, x) \right] = \frac {N _ {n , i} \mu_ {n , i} + N _ {n , j} \mu_ {n , j}}{N _ {n , i} + N _ {n , j}}.
$$
Reproducibility Our code is implemented in Julia 1.7.2, and the plots are generated with the StatsPlots.jl package. Other dependencies are listed in the Readme.md. The Readme.md file also provides detailed julia instructions to reproduce our experiments, as well as a script.sh to run them all at once. The general structure of the code (and some functions) is taken from the tidnabbil library.<sup>1</sup>
Table 4: CPU running time in seconds on random Gaussian instances  $\left( {K = {10}}\right)$  .
TTUCB	EB-TCI	T3C	TTTS	TaS	FWS	DKM	LUCB	Uniform
Average	0.14	0.10	0.06	0.82	78.38	7.10	0.40	0.06	0.14
Std	0.11	0.30	0.05	0.65	50.34	9.6	0.30	0.09	0.10
Figure 3: Empirical errors at time  $n < \tau_{\delta}$  on random Gaussian instances ( $K = 10$ ).
G.2 Supplementary experiments
Running time The CPU running time corresponding to the experiment displayed in Figure 1(a) are reported in Table 4. They match our discussion on computational cost detailed in Appendix G.1. The slowest algorithm is TaS, followed closely by FWS and TTTS. All remaining algorithms have similar computational cost: TTUCB,  $\beta$ -EB-TCI, T3C, LUCB and uniform sampling. It is slightly higher for DKM.
We emphasize that this is a coarse empirical comparison of the CPU running time in order to grasp the different orders of magnitude. More efficient implementation could (and should) be used by practitioners. As an example, the computational cost of TaS can be improved for Gaussian distributions by using the algorithm from [6] based on Newton's iterates. However, we doubt that the faster implementation of TaS will match the computational cost of DKM or FWS.
Empirical errors before stopping At the exception of LUCB, all the considered algorithms are anytime algorithms (see [23] for a definition) since they are not using  $\delta$  in their sampling rule. While all those algorithms are  $\delta$ -correct, none enjoy theoretical guarantees on the probability of error before stopping, i.e. upper bounds on  $\mathbb{P}_{\mu}(\hat{i}_n \neq i^{\star}(\mu))$  for  $n < \tau_{\delta}$ . In Figure 3, we display their averaged empirical errors at time  $n < \tau_{\delta}$  (i.e.  $\mathbb{1}(\hat{i}_n \neq i^{\star})$ ) corresponding to the experiment displayed in Figure 1(a), with their associated Wilson Score Intervals [43]. To avoid an unfair comparison between algorithms having different stopping time, we restrict our plots to the median of the observed empirical stopping time. Therefore, even the fastest algorithm will average its empirical errors on at least 2500 instances.
Based on Figure 3, we see that uniform sampling and DKM perform the worst in terms of empirical error. At all times, the smallest empirical errors are achieved by  $\beta$ -EB-TCI, TTTS and LUCB. While TTUCB is at first as bad as FWS, its empirical error tends to match the one of the best algorithms for larger time. For TaS and T3C, the trend is reversed.
This library was created by [12], see https://bitbucket.org/wmkoolen/tidnabbil. No license were available on the repository, but we obtained the authorization from the authors.
Figure 4: Empirical stopping time on random Gaussian instances ( $K = 10$ ): tracking (T-) versus sampling (S-).
Tracking versus sampling The TTUCB sampling rule uses tracking instead of sampling. Since both approaches aim at doing the same with either a deterministic or a randomized approach, it is interesting to assess whether they lead to different empirical performance. Therefore, we compare both approaches for four Top Two sampling rules. Figure 4 reveals that the algorithmic choice of tracking or sampling has a negligible impact on the empirical stopping time.
In light of this experiment, the choice of tracking or sampling is mostly a question of theoretical analysis. Since the analysis of a randomized sampling requires control of the randomness of the allocation, we choose a deterministic tracking for analytical simplicity. Moreover, this choice is natural when both the leader and challenger are deterministic.
Figure 5: Influence of the dimension  $K$  on the average empirical stopping time (± standard deviation) for the Gaussian benchmark (a) "  $\alpha = 0.3$  " and (b) "  $\alpha = 0.6$  ".
Larger sets of arms In addition to the results presented in Section 4, we also evaluate the performance of our algorithm on the two other benchmarks used in [20]. The “ $\alpha = 0.3$ ” scenarios consider  $\mu_{i} = 1 - \left(\frac{i - 1}{K - 1}\right)^{\alpha}$  for all  $i \in [K]$ , with hardness  $H(\mu) \approx 3K$ . The “ $\alpha = 0.6$ ” scenarios consider  $\mu_{i} = 1 - \left(\frac{i - 1}{K - 1}\right)^{\alpha}$  for all  $i \in [K]$ , with hardness  $H(\mu) \approx 12K^{1.2}$ . The observations from Figure 5 are consistent with the ones in Figure 1(b). Overall, T3C performs the best for larger sets of arms.
G.2.1 Adaptive proportions
The ratio  $T_{1/2}^{\star}(\mu) / T^{\star}(\mu)$  seems to reach its highest value  $r_K = 2K / (1 + \sqrt{K - 1})^2$  for "equal means" instances (Lemma C.6), i.e.  $\mu_i = \mu_{i^*} - \Delta$  for all  $i \neq i^*$  with  $\Delta > 0$ . To best observe differences between Top Two algorithms with  $\beta = 1/2$  and their adaptive version, we consider such instances with  $K = 35$  ( $r_K \approx 3/2$ ). Figure 6 reveals that adaptive proportions yield better empirical performance, with an empirical speed-up close to  $r_K \approx 3/2$ . We also compare them with three asymptotically optimal BAI algorithms (TaS, FWS and DKM). Even on those hard instances, Top
Figure 6: Empirical stopping time on "equal means" instances  $(K, \mu_{i^{\star}}, \Delta) = (35, 0, 0.5)$  for (top)  $\delta = 0.1$  and (bottom)  $\delta \in \{0.01, 0.001\}$ : constant  $\beta = 1/2$  and adaptive (A-).
Two algorithms with fixed  $\beta = 1/2$  are outperforming the asymptotically optimal algorithms for all the values  $\delta \in \{0.1, 0.01, 0.001\}$ . While being only  $1/2$  asymptotically optimal, Top Two algorithms (with fixed  $\beta = 1/2$ ) can obtain significantly better empirical performances compared to existing asymptotically optimal in the finite-confidence regime. The gap between the empirical performance of the adaptive and the fixed  $\beta$  Top Two algorithms is increasing with  $\delta$  decreasing. Interestingly, TTUCB appears to be slightly more robust to decreasing confidence  $\delta$  compared to other Top Two algorithms.
We also compare Top Two algorithms using a fixed proportion  $\beta = 1 / 2$  with their adaptive counterpart using  $\beta_{n} = N_{n,C_{n}} / (N_{n,B_{n}} + N_{n,C_{n}})$  at time  $n$  on random instances.
Figure 7: Empirical stopping time on random Gaussian instances  $(K = 10)$ : constant  $\beta = 1/2$  and adaptive (A-).
Experiments are conducted on 5000 random Gaussian instances with  $K = 10$  such that  $\mu_1 = 0.6$  and  $\mu_i \sim \mathcal{U}([0.2, 0.5])$  for all  $i \neq 1$ . Figure 7 shows that their performances are highly similar, with a slim advantage for adaptive algorithms. For instances with multiple close competitors, the same phenomenon appears (see Figure 8 below). This is expected as  $T_{1/2}^{\star}(\mu) / T^{\star}(\mu)$  is close to one when sub-optimal arms have significantly distinct means (see Figure 2).
Figure 8: Empirical stopping time on random Gaussian instances ( $K = 10$ ) with multiple close competitors.
Then, we assess the performance on 5000 random Gaussian instances with  $K = 10$  such that  $\mu_{1} = 0.6$  and  $\mu_{i} = \mu_{1} - \Delta_{i}$  for  $i \neq 1$  where  $\Delta_{i} = \frac{1}{20}\left(\frac{995}{1000} +\frac{u_{i}}{100}\right)$  for all  $i \in \{2,3,4,5,6\}$  and  $\Delta_{i} = \frac{1}{10}\left(\frac{995}{1000} +\frac{u_{i}}{100}\right)$  for all  $i \in \{7,8,9,10\}$  with  $u_{i} \sim U([0,1])$ . Numerically, we observe  $w^{\star}(\mu)_i \approx 0.28276 \pm 0.0003$  (mean ± std). Both plots in Figure 8 show the same phenomena than the ones observed in Figures 1(b) and 7.