SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[34] A. Mukherjee and A. Tajer. Sprt-based efficient best arm identification in stochastic bandits. IEEE Journal on Selected Areas in Information Theory, 2023.
[35] F. Pukelsheim. Optimal design of experiments. SIAM, 2006.
[36] C. Qin, D. Klabjan, and D. Russo. Improving the expected improvement algorithm. In Advances in Neural Information Processing Systems 30 (NIPS), 2017.
[37] H. Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 58(5):527-535, 1952.
[38] D. Russo. Simple Bayesian algorithms for best arm identification. In Proceedings of the 29th Conference on Learning Theory (COLT), 2016.
[39] X. Shang, R. de Heide, E. Kaufmann, P. Menard, and M. Valko. Fixed-confidence guarantees for bayesian best-arm identification. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
[40] M. Simchowitz, K. Jamieson, and B. Recht. The simulator: Understanding adaptive sampling in the moderate-confidence regime. In Conference on Learning Theory, 2017.
[41] W. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25:285-294, 1933.
[42] P.-A. Wang, R.-C. Tzeng, and A. Proutiere. Fast pure exploration via frank-wolfe. Advances in Neural Information Processing Systems, 2021.
[43] E. B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209-212, 1927.
[44] W. You, C. Qin, Z. Wang, and S. Yang. Information-directed selection for top-two algorithms. In The Thirty Sixth Annual Conference on Learning Theory, pages 2850–2851. PMLR, 2023.
- Appendix B gathers notation used in this work.
- In Appendix C, we study the link between  $T^{\star}(\mu)$  and  $T_{\beta}^{\star}(\mu)$  for Gaussian distributions.
- The detailed analysis of our non-asymptotic upper bound (Theorem 2.4), sketched in Section 3, is detailed in Appendix D. We also give a non-asymptotic upper bound on the TTUCB using  $g_{m}$  in (3) (Corollary D.5), and on uniform sampling (Theorem D.6).
- We show the asymptotic optimality of our algorithm (Theorem 2.3) in Appendix F.
- Appendix E gathers concentration results used by the stopping rule (Lemma 2.1) and the sampling rule.
- Implementation details and supplementary experiments are detailed in Appendix G.
Table 2: Notation for the setting.
Notation	Type	Description
K	N	Number of arms
μi	R	Mean of arm i ∈ [K]
μ	R^K	Vector of means, μ := (μi)_{i∈[K]}
i^*	R^K → [K]	Best arm operator, i^*(μ) = arg max_{i∈[K]} μi
T^*(μ), T^*_β(μ)	R_+^*	Asymptotic (β-)characteristic time
w^*(μ), w^*_β(μ) = {(w^*_β,i)_{i∈[K]}}	Δ_K	Asymptotic (β-)optimal allocation
We recall some commonly used notation: the set of integers  $[n] := \{1, \dots, n\}$ , the complement  $X^{\mathfrak{C}}$  and interior  $\hat{X}$  of a set  $X$ , Landau's notation  $o$ ,  $\mathcal{O}$ ,  $\Omega$  and  $\Theta$ , the  $(K - 1)$ -dimensional probability simplex  $\triangle_K := \left\{w \in \mathbb{R}_+^K \mid w \geq 0, \sum_{i \in [K]} w_i = 1\right\}$ . While Table 2 gathers problem-specific notation, Table 3 groups notation for the algorithms. We emphasize that  $N_{n,i}^i$  is the number of times where we pulled arm  $i$  as a leader before time  $n$ .
Table 3: Notation for algorithms.
Notation	Type	Description
Bn	[K]	Leader at time n
Cn	[K]	Challenger at time n
In	[K]	Arm sampled at time n
β	(0,1)	Proportion parameter
Xn,In	R	Sample observed at the end of time n, i.e. Xn,In ~ N(μIn, 1)
Fn	History before time n, i.e. Fn := σ(I1, X1,I1, ..., In, Xn,In)
ˆn	[K]	Arm recommended before time n, i.e.ˆn ∈ arg maxi∈[K] μn,i
τδ	N	Sample complexity (stopping time of the algorithm)
ˆi	[K]	Arm recommended by the algorithm
c(n,δ)	R+	Stopping threshold function
Nn,i	N	Number of pulls of arm i before time n, Nn,i := ∑t∈[n-1] 1 (It = i)
μn,i	I	Empirical mean of arm i before time n, μn,i := 1/Nn,i ∑t∈[n-1] Xt,It 1 (It = i)
Ln,i	N	Counts of Bt = i before time n, Ln,i := ∑t∈[n-1] 1 (Bt = i)
Ni/n,j	N	Counts of (Bt, It) = (i, j) before time n, Ni,j := ∑t∈[n-1] 1 ((Bt, It) = (i, j))
C Characteristic times
Let  $\mu \in \mathcal{D}^K$  such that  $i^{\star}(\mu) = \{i^{\star}\}$ . Let  $\beta \in (0,1)$  and  $w_{\beta}^{\star}$  be the unique allocation  $\beta$ -optimal allocation satisfying  $w_{\beta,i}^{\star} > 0$  for all  $i \in [K]$  (Lemma C.2), i.e.  $w_{\beta}^{\star}(\mu) = \{w_{\beta}^{\star}\}$  where
$$
w _ {\beta} ^ {\star} (\mu) := \underset {w \in \triangle_ {K}: w _ {i ^ {\star}} = \beta} {\arg \max} \underset {i \neq i ^ {\star}} {\min} \frac {(\mu_ {i ^ {\star}} - \mu_ {i}) ^ {2}}{2 (1 / \beta + 1 / w _ {i})} = \underset {w \in \triangle_ {K}: w _ {i ^ {\star}} = \beta} {\arg \max} \underset {i \neq i ^ {\star}} {\min} \frac {\mu_ {i ^ {\star}} - \mu_ {i}}{\sqrt {1 / \beta + 1 / w _ {i}}}.
$$
We restate without proof two fundamental results on the characteristic time and the associated allocation, which were first shown in [38]. Lemma C.1 gives an upper bound on  $T_{\beta}^{\star}(\mu) / T^{\star}(\mu)$  and Lemma C.2 shows that the  $(\beta-)$  optimal allocation is unique with strictly positive values. [38] shows that these two results hold for any single-parameter exponential families. [21] extended their proof for the non-parametric family of bounded distributions. Moreover, they argue that these results should hold for more general distributions provided some regularity assumptions are satisfied.
Lemma C.1 ([38]).  $T_{1/2}^{\star}(\mu) \leq 2T^{\star}(\mu)$  and with  $\beta^{\star} = w_{i^{\star}}^{\star}(\mu)$ ,
$$
\frac {T _ {\beta} ^ {\star} (\mu)}{T ^ {\star} (\mu)} \leq \max  \left\{\frac {\beta^ {\star}}{\beta}, \frac {1 - \beta^ {\star}}{1 - \beta} \right\}.
$$
Lemma C.2 ([38]). If  $i^{\star}(\mu)$  is a singleton and  $\beta \in (0,1)$ , then  $w^{\star}(\mu)$  and  $w_{\beta}^{\star}(\mu)$  are singletons, i.e. the optimal allocations are unique, and  $w^{\star}(\mu)_i > 0$  and  $w_{\beta}^{\star}(\mu)_i > 0$  for all  $i \in [K]$ .
Gaussian distributions Since Lemma C.1 is a worst-case inequality holding for general distributions, we expect that tighter inequality can be achieved for Gaussian distributions by leveraging their symmetry. This intuition is fueled by recent results of [6]. Using a rewriting of the optimization problem underlying  $T^{\star}(\mu)$  (Lemma C.3), they provide a better understanding of characteristic times and their optimal allocations (Lemma C.4). In particular, for Gaussian distributions, Lemma C.4 shows that the optimal allocation of arm  $i^{\star}$  is never above  $1 / 2$  and is larger than  $1 / (\sqrt{K - 1} + 1) \geq 1 / K$ .
Lemma C.3 (Proposition 8 in [6]). Let  $\mu \in \mathbb{R}^K$  be a  $K$ -arms Gaussian bandits and  $r(\mu)$  be the solution of  $\psi_{\mu}(r) = 0$ , where
$$
\forall r \in (1 / \min _ {i \neq i ^ {\star}} (\mu_ {i ^ {\star}} - \mu_ {i}) ^ {2}, + \infty), \quad \psi_ {\mu} (r) = \sum_ {i \neq i ^ {\star}} \frac {1}{(r (\mu_ {i ^ {\star}} - \mu_ {i}) ^ {2} - 1) ^ {2}} - 1,
$$
and  $\psi_{\mu}$  is convex and decreasing. Then,
$$
T ^ {\star} (\mu) = \frac {2 r (\mu)}{1 + \sum_ {i \neq i ^ {\star}} \frac {1}{r (\mu) (\mu_ {i} ^ {\star} - \mu_ {i}) ^ {2} - 1}}.
$$
Lemma C.4 (Proposition 10 in [6]). Let  $\mu \in \mathbb{R}^K$  be a  $K$ -arms Gaussian bandit. For  $K = 2$ ,
$$
w ^ {\star} (\mu) = (0. 5, 0. 5) \quad a n d \quad T ^ {\star} (\mu) = 8 \left(\mu_ {1} - \mu_ {2}\right) ^ {2}.
$$
For  $K\geq 3$  ,we have
$$
1 / (\sqrt {K - 1} + 1) \leq w ^ {\star} (\mu) _ {i ^ {*}} \leq 1 / 2,
$$
$$
\max  \left\{\frac {8}{\min  _ {i \neq i ^ {\star}} (\mu_ {i ^ {\star}} - \mu_ {i}) ^ {2}}, 4 \frac {1 + \sqrt {K - 1}}{\overline {{\Delta^ {2}}}} \right\} \leq T ^ {\star} (\mu) \leq 2 \frac {(1 + \sqrt {K - 1}) ^ {2}}{\min  _ {i \neq i ^ {\star}} (\mu_ {i ^ {\star}} - \mu_ {i}) ^ {2}},
$$
where  $\overline{\Delta^2} = \frac{1}{K - 1}\sum_{i\neq i^\star}(\mu_{i^\star} - \mu_i)^2$ . In particular, the equalities  $w^{\star}(\mu)_{i^{\star}} = 1 / (\sqrt{K - 1} +1)$  and  $T^{\star}(\mu) = \frac{2(1 + \sqrt{K - 1})^{2}}{\min_{i\neq i^{\star}}(\mu_{i^{\star}} - \mu_i)^2}$  are reached if and only if  $\mu_{i} = \max_{i\neq i^{\star}}\mu_{i}$  for all  $i\neq i^{\star}$ .
By inspecting the proof of Proposition 8 in [6], we obtain directly the following rewriting of  $T_{\beta}^{\star}(\mu)$ .
Lemma C.5. Let  $\mu \in \mathbb{R}^K$  be a  $K$ -arms Gaussian bandit and  $r_{\beta}(\mu)$  be the solution of  $\varphi_{\mu,\beta}(r) = 0$ , where
$$
\forall r \in (1 / \min _ {i \neq i ^ {\star}} (\mu_ {i ^ {\star}} - \mu_ {i}) ^ {2}, + \infty), \quad \varphi_ {\mu , \beta} (r) = \sum_ {i \neq i ^ {\star}} \frac {1}{r (\mu_ {i ^ {\star}} - \mu_ {i}) ^ {2} - 1} - \frac {1 - \beta}{\beta},
$$
and  $\varphi_{\mu ,\beta}$  is convex and decreasing. Then,
$$
T _ {\beta} ^ {\star} (\mu) = \frac {2 r _ {\beta} (\mu)}{\beta}.
$$
Figure 2: Ratio of characteristic times  $\Omega(x) = T_{1/2}^{\star}(\mu) / T^{\star}(\mu)$  for  $K = 3$  (left) and  $K = 4$  (right). The dashed blue line is  $r_2 = 6 / (1 + \sqrt{2})^2$  (left), and the blue point is  $r_3 = 8 / (1 + \sqrt{3})^2$  (right).
Proof. Using the proof of Proposition 8 in [6], we obtain directly that
$$
2 T _ {\beta} ^ {\star} (\mu) ^ {- 1} = C _ {\beta} (\mu)
$$
where  $C_{\beta}(\mu)$  is the solution of  $\Phi_{\mu}(C) = 0$  where
$$
\Phi_ {\mu} (C) = \sum_ {i \neq i ^ {\star}} \frac {\beta}{\beta (\mu_ {i ^ {\star}} - \mu_ {i}) ^ {2} / C - 1} - (1 - \beta).
$$
The idea behind the above result is that at the equilibrium, i.e. at  $w^{\star}$ , all the transportation costs are equal to  $C$ . Then, the implicit equation defining  $C$  is obtained by using the constraints that  $\sum_{i \neq i^{\star}} w_i^{\star} = 1 - \beta$ . To conclude, we simply use  $r = \beta / C$ .
Since it is the sum of  $K - 1$  convex and decreasing functions,  $\varphi_{\mu ,\beta}$  is also convex and decreasing.
Lemma C.6 aims at improving the worst-case inequality between  $T_{1/2}^{\star}(\mu)$  and  $T^{\star}(\mu)$  in the Gaussian setting. For  $K = 2$ , those two quantities are equal. For  $K \geq 3$ , we showed that  $\max_{\mu: |i^{\star}(\mu)| = 1} T_{1/2}^{\star}(\mu) / T^{\star}(\mu)$  is at least  $r_K$ , which is achieved when all sub-optimal arms have the same mean. As the gradient of the ratio is the null vector for those instances, we conjecture this is the maximum, i.e.  $T_{1/2}^{\star}(\mu) \leq r_K T^{\star}(\mu)$ . Our conjecture is supported by numerical simulations for  $K \geq 3$ . In Figure 2, we plot the ratio of characteristic times  $\Omega(x) = T_{1/2}^{\star}(\mu) / T^{\star}(\mu)$  for  $K \in \{3,4\}$ . We observed that our conjecture is validated empirically, and that  $T_{1/2}^{\star}(\mu) / T^{\star}(\mu)$  is often close to 1.
Lemma C.6. For  $K = 2$ , we have  $T_{1/2}^{\star}(\mu) = T^{\star}(\mu)$ . For  $K \geq 3$ , let  $r_K = 2K / (1 + \sqrt{K - 1})^2$ . Then, for all  $\mu$  such that  $i^{\star}(\mu)$  is unique, we have  $\frac{T_{1/2}^{\star}(\mu)}{T^{\star}(\mu)} = \Omega(x)$  where  $x_j = \frac{\mu_i^{\star} - \mu_j}{\mu_i^{\star} - \mu_j^{\star}} \geq 1$  for all  $j \notin \{i^{\star}, j^{\star}\}$  with  $j^{\star} \in \arg \min_{j \neq i^{\star}} \mu_{i^{\star}} - \mu_j$ . In other words,  $\frac{T_{1/2}^{\star}(\mu)}{T^{\star}(\mu)}$  is independent from  $\mu_{i^{\star}}$  and  $\min_{j \neq i^{\star}} \mu_{i^{\star}} - \mu_j$ . Moreover, we have
Proof. Using Lemma C.4, we have  $T_{1/2}^{\star}(\mu) = T^{\star}(\mu)$  directly for  $K = 2$ .
For  $K \geq 3$ , we want to upper bound  $T_{1/2}^{\star}(\mu) / T^{\star}(\mu)$ . If we denote by  $\Delta_{\min}(\mu) = \min_{i \neq i^{\star}} (\mu_{i^{\star}} - \mu_i)^2$ , it is easy to see that  $T_{1/2}^{\star}(\mu) / T^{\star}(\mu) = T_{1/2}^{\star}(\tilde{\mu}) / T^{\star}(\tilde{\mu})$  when  $\Delta_{\min}(\mu) = \Delta_{\min}(\tilde{\mu})$ . Likewise, this ratio is invariant by translation of all means by a same quantity. Therefore, we consider without restriction an instance  $\mu$  such that  $\Delta_{\min}(\mu) = 1$  and with
$$
\mu_ {1} = 0 > \mu_ {2} = - x _ {2} \geq \dots \geq \mu_ {K} = - x _ {K},
$$
where  $x_{2} = 1$  and  $x_{i}\geq 1$  for all  $i\geq 3$
First, we can rewrite
$$
T _ {1 / 2} ^ {\star} (x) ^ {- 1} = \frac {1}{4} \max _ {q \in \triangle_ {K - 1}} \min _ {i \geq 2} \frac {x _ {i} ^ {2}}{1 + 1 / q _ {i}} \geq \frac {1}{4} \max _ {q \in \triangle_ {K - 1}} \min _ {i \geq 2} \frac {1}{1 + 1 / q _ {i}} = \frac {1}{4 K},
$$
where the inequality is an equality if and only  $x_{i} = 1$  for all  $i \geq 2$ . Using Lemma C.4, we know that
$$
T ^ {\star} (x) = 2 (1 + \sqrt {K - 1}) ^ {2}
$$
if and only  $x_{i} = 1$  for all  $i \geq 2$ . Therefore, we have exhibited an instance such that
$$
\frac {T _ {1 / 2} ^ {\star} (\mu)}{T ^ {\star} (\mu)} = \frac {2 K}{(1 + \sqrt {K - 1}) ^ {2}}.
$$
Applying Lemma C.5 for  $\beta = 1/2$ , we obtain  $T_{1/2}^{\star}(x) = 4C(x)$  where  $C(x)$  is the implicit solution of the equation
$$
\varphi (x, C (x)) = 0 \quad \text {w h e r e} \quad \varphi (x, C) = \frac {1}{C - 1} + \sum_ {i \geq 3} \frac {1}{C x _ {i} ^ {2} - 1} - 1.
$$
Using Lemma C.3, we know that
$$
T ^ {\star} (x) = \frac {2 r (x)}{1 + \frac {1}{r (x) - 1} + \sum_ {i \geq 3} \frac {1}{r (x) x _ {i} ^ {2} - 1}}
$$
where  $r(x)$  is the implicit solution of the equation
$$
\psi (x, r (x)) = 0 \quad \text {w h e r e} \quad \psi (x, r) = \frac {1}{(r - 1) ^ {2}} + \sum_ {i \geq 3} \frac {1}{(r x _ {i} ^ {2} - 1) ^ {2}} - 1.
$$
Therefore, we obtain that
$$
\Omega (x) = \frac {1}{2} \frac {T _ {1 / 2} ^ {\star} (x)}{T ^ {\star} (x)} = \frac {C (x)}{r (x) - 1} + \sum_ {i > 3} \frac {C (x)}{r (x) \left(r (x) x _ {i} ^ {2} - 1\right)},
$$
and our goal is to show that the above quantity is maximum if and only  $x_{i} = 1$  for all  $i \geq 2$ . One way of doing this is by computing the gradient and showing it is negative, which would imply that the function is decreasing. Let  $j \geq 3$ . Using the implicit differentiation theorem, we obtain
$$
\frac {\partial C}{\partial x _ {j}} (x) = - \frac {\frac {\partial \varphi}{\partial x _ {j}} (x , C (x))}{\frac {\partial \varphi}{\partial C} (x , C (x))} = - \frac {\frac {2 C (x) x _ {j}}{(C (x) x _ {j} ^ {2} - 1) ^ {2}}}{\frac {1}{(C (x) - 1) ^ {2}} + \sum_ {i \geq 3} \frac {x _ {i} ^ {2}}{(C (x) x _ {i} ^ {2} - 1) ^ {2}}} \quad \text {a n d} \quad \frac {\partial C}{\partial x _ {j}} (1 _ {K - 2}) = - \frac {2 C (1 _ {K - 2})}{K - 1},
$$
$$
\frac {\partial r}{\partial x _ {j}} (x) = - \frac {\frac {\partial \psi}{\partial x _ {j}} (x , r (x))}{\frac {\partial \psi}{\partial r} (x , r (x))} = - \frac {\frac {2 r (x) x _ {j}}{(r (x) x _ {j} ^ {2} - 1) ^ {3}}}{\frac {1}{(r (x) - 1) ^ {3}} + \sum_ {i \geq 3} \frac {x _ {i} ^ {2}}{(r (x) x _ {i} ^ {2} - 1) ^ {3}}} \quad \text {a n d} \quad \frac {\partial r}{\partial x _ {j}} (1 _ {K - 2}) = - \frac {2 r (1 _ {K - 2})}{K - 1}.
$$
Direct computations yield that
$$
\frac {\partial}{\partial x _ {j}} \left(\frac {C (x)}{r (x) - 1}\right) = \frac {\frac {\partial C}{\partial x _ {j}} (x) (r (x) - 1) - C (x) \frac {\partial r}{\partial x _ {j}} (x)}{(r (x) - 1) ^ {2}} \quad \mathrm {a n d} \quad \frac {\partial}{\partial x _ {j}} \left(\frac {C (x)}{r (x) - 1}\right) _ {x = 1 _ {K - 2}} = \frac {1}{K - 1} \frac {2 C (1 _ {K - 2})}{(r (1 _ {K - 2}) - 1) ^ {2}},
$$
$$
\frac {\partial}{\partial x _ {j}} \left(\frac {C (x)}{r (x) (r (x) x _ {j} ^ {2} - 1)}\right) = \frac {\frac {\partial C}{\partial x _ {j}} (x) r (x) (r (x) x _ {j} ^ {2} - 1) - C (x) \frac {\partial r}{\partial x _ {j}} (x) \left(2 r (x) x _ {j} ^ {2} - 1\right) - 2 C (x) r (x) ^ {2} x _ {j}}{\left(r (x) (r (x) x _ {j} ^ {2} - 1)\right) ^ {2}}
$$
$$
\frac {\partial}{\partial x _ {j}} \left(\frac {C (x)}{r (x) (r (x) x _ {j} ^ {2} - 1)}\right) _ {x = 1 _ {K - 2}} = \left(\frac {1}{K - 1} - 1\right) \frac {2 C (1 _ {K - 2})}{(r (1 _ {K - 2}) - 1) ^ {2}},
$$
and, for  $i \geq 3$  s.t.  $i \neq j$ ,
$$
\frac {\partial}{\partial x _ {j}} \left(\frac {C (x)}{r (x) \left(r (x) x _ {i} ^ {2} - 1\right)}\right) = \frac {\frac {\partial C}{\partial x _ {j}} (x) r (x) \left(r (x) x _ {i} ^ {2} - 1\right) - C (x) \frac {\partial r}{\partial x _ {j}} (x) \left(2 r (x) x _ {i} ^ {2} - 1\right)}{\left(r (x) \left(r (x) x _ {i} ^ {2} - 1\right)\right) ^ {2}}
$$
$$
\frac {\partial}{\partial x _ {j}} \left(\frac {C (x)}{r (x) (r (x) x _ {i} ^ {2} - 1)}\right) _ {x = 1 _ {K - 2}} = \frac {1}{K - 1} \frac {2 C (1 _ {K - 2})}{(r (1 _ {K - 2}) - 1) ^ {2}}.
$$
Then, plugging everything together, we obtained that
$$
\frac {\partial \Omega}{\partial x _ {j}} (1 _ {K - 2}) = \frac {2 C (1 _ {K - 2})}{(r (1 _ {K - 2}) - 1) ^ {2}} \left(\frac {1}{K - 1} + \frac {1}{K - 1} - 1 + \sum_ {i \geq 3, i \neq j} \frac {1}{K - 1}\right) = 0.
$$
Therefore, we have shown that  $\nabla \Omega (1_{K - 2}) = 0_{K - 2}$
D Non-asymptotic analysis
In Appendix D.1, we state and prove one key result for each one of the three main components of the TTUCB sampling rule: the UCB leader (Lemma D.1), the TC challenger (Lemma D.2) and the tracking (Lemma D.3). The proof of Theorem 2.4 is detailed in Appendix D.2, which uses the stopping rule (1) and a proof method from [30]. It is a direct consequence of a more general result (Theorem D.4). In Appendix D.3, we prove a non-asymptotic upper bound for the TTUCB when using  $g_{m}$  instead of  $g_{u}$  (Corollary D.5). We compare our results with uniform sampling in Appendix D.4 (Theorem D.6). Other technicalities are gathered in Appendix D.5.
D.1 Key properties
Before delving in the proof of Theorem D.4 itself, we present the key properties of each component of the TTUCB sampling rule under a some concentration event.
Let  $\alpha > 1$  and  $s > 1$ . Let  $(\mathcal{E}_n)_{n > K}$  be the sequence of concentration events defined as  $\mathcal{E}_n := \mathcal{E}_{1,n} \cap \mathcal{E}_{2,n}$  for all  $n > K$  where  $\mathcal{E}_{1,n}$  and  $\mathcal{E}_{2,n}$  are defined in (17) and (19) as
$$
\begin{array}{l} \mathcal {E} _ {1, n} := \left\{\forall k \in [ K ], \forall t \in [ n ^ {1 / \alpha}, n ], | \mu_ {t, k} - \mu_ {k} | <   \sqrt {\frac {g _ {u} (t)}{N _ {t , k}}} \right\}, \\ \mathcal {E} _ {2, n} := \left\{\forall k \neq i ^ {\star}, \forall t \in [ n ^ {1 / \alpha}, n ], (\mu_ {t, i ^ {\star}} - \mu_ {t, k}) - (\mu_ {i ^ {\star}} - \mu_ {k}) > - \sqrt {2 \alpha (2 + s) \log (t) \left(\frac {1}{N _ {t , i ^ {\star}}} + \frac {1}{N _ {t , k}}\right)} \right\}, \\ \end{array}
$$
with  $g_{u}(n) = 2\alpha (1 + s)\log n$ . In Lemma E.6, it is shown that  $\sum_{n > K}\mathbb{P}(\mathcal{E}_n^{\mathbb{C}})\leq (2K - 1)\zeta (s)$ .
UCB leader Lemma D.1 shows that the UCB leader is different from  $i^{\star}$  for only a sublinear number of times under a certain concentration event. It is slightly more general than Lemma 3.2 presented in Section 2, which follows from  $H_{1}(\mu) \leq H_{1}(\mu) + 2\Delta_{\min}^{-2} = H(\mu)$ .
Lemma D.1. Let  $(\mathcal{E}_{1,n})_n$  and  $g_{u}$  as in Lemma E.2. Let  $H_{1}(\mu) = \sum_{i\neq i^{\star}(\mu)}\frac{2}{(\mu_{i^{\star}(\mu)} - \mu_{i})^{2}}$ . For all  $n > K$ , under the event  $\mathcal{E}_{1,n}$ ,
$$
\forall t \in \left[ n ^ {1 / \alpha}, n \right], \quad L _ {t, i ^ {*}} \geq t - 1 - 2 H (\mu) g _ {u} (t) / \beta - K / \beta , \tag {6}
$$
Let  $(\mathcal{E}_{3,n})_n$  and  $g_{m}$  as in Lemma E.3. Under the event  $\mathcal{E}_{3,n}$ , (6) holds by using  $g_{m}$  instead of  $g_{u}$ .
Proof. Suppose that at time  $t \in [n^{1 / \alpha}, n]$ , the UCB leader is different from  $i^{\star}$ , i.e.  $B_{t} = k \neq i^{\star}$ . Using the event  $\mathcal{E}_{1,n}$  and the definition of  $B_{t}$  yields
$$
\mu_ {i ^ {\star}} \leq \mu_ {t, i ^ {\star}} + \sqrt {\frac {g _ {u} (t)}{N _ {t , i ^ {\star}}}} \leq \mu_ {t, k} + \sqrt {\frac {g _ {u} (t)}{N _ {t , k}}} \leq \mu_ {k} + \sqrt {\frac {4 g _ {u} (t)}{N _ {t , k}}}.
$$
We get that if  $t \geq n^{1 / \alpha}$ , then  $N_{t,k} \leq \frac{4g_u(t)}{(\mu_i\star - \mu_k)^2}$ . Therefore, we obtain the following upper bound on the number of times the leader is different from  $i^{\star}$  up to time  $t$
$$
t - 1 - L _ {t, i ^ {\star}} = \sum_ {k \neq i ^ {\star}} L _ {t, k} \leq \frac {1}{\beta} \sum_ {k \neq i ^ {\star}} N _ {t, k} ^ {k} + \frac {K - 1}{2 \beta} \leq \frac {1}{\beta} \sum_ {k \neq i ^ {\star}} N _ {t, k} + \frac {K - 1}{2 \beta} \leq \sum_ {k \neq i ^ {\star}} \frac {4 g _ {u} (t)}{(\mu_ {i ^ {\star}} - \mu_ {k}) ^ {2} \beta} + \frac {K - 1}{2 \beta},
$$
where we used Lemma D.3 for the second inequality and  $N_{t,k}^{k} \leq N_{t,k}$  for the third. This concludes the proof for  $g_{u}$ . The same reasoning can be applied for  $g_{m}$ .
TC challenger Lemma D.2 shows a lower bound on the "transportation" costs used by the TC challenger provided a certain concentration holds. This lower bound depends only on the empirical counts when the best arm is the leader.
Lemma D.2. For all  $n > K$ , under the event  $\mathcal{E}_{2,n}$ , for all  $t \in [n^{1/\alpha}, n]$  such that  $B_t = i^\star$ ,
$$
\forall k \neq i ^ {\star}, \quad \frac {\mu_ {t , B _ {t}} - \mu_ {t , k}}{\sqrt {1 / N _ {t , B _ {t}} + 1 / N _ {t , k}}} \geq \frac {\mu_ {i ^ {\star}} - \mu_ {k}}{\sqrt {1 / N _ {t , i ^ {\star}} ^ {i ^ {\star}} + 1 / N _ {t , k} ^ {i ^ {\star}}}} - \sqrt {2 \alpha (2 + s) \log n}.
$$
Proof. Under  $\mathcal{E}_{2,n}$ , using  $B_{t} = i^{\star}$  yields
$$
\frac {\mu_ {t , B _ {t}} - \mu_ {t , k}}{\sqrt {1 / N _ {t , B _ {t}} + 1 / N _ {t , k}}} \geq \frac {\mu_ {i ^ {\star}} - \mu_ {k}}{\sqrt {1 / N _ {t , i ^ {\star}} + 1 / N _ {t , k}}} - \sqrt {2 \alpha (2 + s) \log t} \geq \frac {\mu_ {i ^ {\star}} - \mu_ {k}}{\sqrt {1 / N _ {t , i ^ {\star}} ^ {i ^ {\star}} + 1 / N _ {t , k} ^ {i ^ {\star}}}} - \sqrt {2 \alpha (2 + s) \log n},
$$
where the second inequality uses that  $N_{t,k} \geq N_{t,k}^{i^{\star}}$  for all  $k \neq i^{\star}$ .
Tracking Lemma D.3 shows the key property satisfied by the  $K$  independent tracking procedures used by the TTUCB sampling rule. It is slightly more general than Lemma 2.2 presented in Section 2. It is a simple corollary of Theorem 6 in [13].
Lemma D.3. For all  $n > K$  and  $i\in [K]$ , we have  $-1 / 2\leq N_{n,i}^i -\beta L_{n,i}\leq 1$  and for all  $k\neq i$
$$
N _ {n, i} ^ {i} \geq \frac {\beta}{1 - \beta} N _ {n, k} ^ {i} - \frac {1}{2} \frac {1}{1 - \beta}.
$$
Proof. We can rewrite the tracking condition  $N_{n,B_n}^{B_n} \leq \beta L_{n+1,B_n}$  as
$$
N _ {n, B _ {n}} ^ {B _ {n}} - \beta L _ {n + 1, B _ {n}} \leq \left(L _ {n + 1, B _ {n}} - N _ {n, B _ {n}} ^ {B _ {n}}\right) - (1 - \beta) L _ {n + 1, B _ {n}}.
$$
For all  $k \in [K]$ , this corresponds to a two-arms C-Tracking between the leader  $k$  and the challengers with  $w_{n} = (\beta, 1 - \beta)$  for all  $n$  such that  $B_{n} = k$ . The leader's pulling count is  $N_{n,B_{n}}^{B_{n}}$  and the challengers' pulling count is  $L_{n+1,B_{n}} - N_{n,B_{n}}^{B_{n}}$ . We recall that C-Tracking was defined as  $I_{n} = \arg \min_{k \in [K]} N_{n,k} - \sum_{t \in [n]} w_{t,k}$ .
Theorem 6 in [13] yields for all  $n > K$ ,  $-\frac{1}{2} \leq N_{n,B_n}^{B_n} - \beta L_{n,B_n} \leq 1$ . The  $K$  parallel tracking procedures are independent since they are considering counts partitioned on the considered leader. Therefore, the above results hold for all  $i \in [K]$ . This concludes the first part of the proof.
Direct manipulations yield the second part of the result, namely
$$
N _ {n, k} ^ {i} \leq L _ {n, i} - N _ {n, i} ^ {i} \leq (1 - \beta) L _ {n, i} + \frac {1}{2} \quad \text {a n d} \quad N _ {n, i} ^ {i} \geq - \frac {1}{2} + L _ {n, i} \beta \geq \frac {\beta}{1 - \beta} \left(N _ {n, k} ^ {i} - \frac {1}{2}\right) - \frac {1}{2}.
$$