SlowGuess/NeurIPS_2023_OCR huggingface dataset:
which satisfies  $\overline{M}(0) = 1$ . It is direct to see that  $M(t) = \exp \left(xS_{t} - \frac{1}{2} x^{2}t\right)$  is a non-negative martingale. By Tonelli's theorem, then  $\overline{M}(t)$  is also a non-negative martingale of unit initial value. Let  $i \in [D]$  and consider  $t \in [N_{i}, N_{i+1})$ . For all  $x$ ,
$$
f _ {N _ {i}, \gamma} (x) \geq \sqrt {\frac {N _ {i}}{t}} f _ {t, \gamma_ {i}} (x) \geq \frac {1}{\sqrt {1 + \eta}} f _ {t, \gamma_ {i}} (x)
$$
$$
\int f _ {t, \gamma_ {i}} (x) \exp \left(x S _ {t} - \frac {1}{2} x ^ {2} t\right) d x = \frac {1}{\sqrt {1 + \gamma_ {i} ^ {- 1}}} \exp \left(\frac {S _ {t} ^ {2}}{2 (1 + \gamma_ {i}) t}\right).
$$
$$
\overline {{M}} (t) \geq \frac {1}{D} \frac {1}{\sqrt {(1 + \gamma_ {i} ^ {- 1}) (1 + \eta)}} \exp \left(\frac {S _ {t} ^ {2}}{2 (1 + \gamma_ {i}) t}\right),
$$
Using Ville's maximal inequality, we have that with probability greater than  $1 - \delta$ ,  $\log \overline{M}(t) \leq \log (1 / \delta)$ . Therefore, with probability greater than  $1 - \delta$ , for all  $i \in [D]$  and  $t \in [N_i, N_{i+1})$ ,
$$
\frac {S _ {t} ^ {2}}{t} \leq \left(1 + \gamma_ {i}\right) \left(2 \log \left(1 / \delta\right) + 2 \log D + \log \left(1 + \gamma_ {i} ^ {- 1}\right) + \log (1 + \eta)\right).
$$
Since this upper bound is independent of  $t$ , we can optimize it and choose  $\gamma_{i}$  as in Lemma E.4 for all  $i \in [D]$ .
Lemma E.4 (Lemma A.3 in [11]). For  $a, b \geq 1$ , the minimal value of  $f(\eta) = (1 + \eta)(a + \log(b + \frac{1}{\eta}))$  is attained at  $\eta^{\star}$  such that  $f(\eta^{\star}) \leq 1 - b + \overline{W}_{-1}(a + b)$ . If  $b = 1$ , then there is equality.
$$
\mathbb {P} \left(\exists t \leq n, \left| \frac {1}{t} \sum_ {s = 1} ^ {t} X _ {s} \right| \geq \sqrt {\frac {1}{t} \bar {W} _ {- 1} (2 \log (1 / \delta) + 2 \log (2 + \log (n)) + 3 - 2 \log 2)}\right) \leq \delta .
$$
Since  $3 - 2\log 2 \leq 2$  and  $\overline{W}_{-1}$  is increasing, taking  $\delta = n^{-s}$  and restricting to  $m \in [n^{1 / \alpha}, n]$  yields
$$
\mathbb {P} \left(\exists m \in [ n ^ {1 / \alpha}, n ], \sqrt {N _ {m , k}} | \mu_ {m, k} - \mu_ {k} | \geq \sqrt {\overline {{W}} _ {- 1} (2 s \log (n) + 2 \log (2 + \log (n)) + 2)}\right) \leq n ^ {- s}.
$$
Using  $n \leq m^{\alpha}$  and doing a union bound over arms yield the result.
TC challenger Lemma E.5 lower bounds the difference between the empirical gap and the unknown gap.
Lemma E.5. Let  $\alpha > 1$  and  $s > 1$ . For all  $n > K$ , let
$$
\mathcal {E} _ {2, n} := \left\{\forall k \neq i ^ {\star}, \forall t \in \left[ n ^ {1 / \alpha}, n \right], \left(\mu_ {t, i ^ {\star}} - \mu_ {t, k}\right) - \left(\mu_ {i ^ {\star}} - \mu_ {k}\right) > - \sqrt {2 \alpha (2 + s) \log (t) \left(\frac {1}{N _ {t , i ^ {\star}}} + \frac {1}{N _ {t , k}}\right)} \right\}. \tag {19}
$$
Then, for all  $n > K$ ,  $\mathbb{P}(\mathcal{E}_{2,n}^{\complement}) \leq (K - 1)n^{-s}$ .
Proof. Let  $(X_{s})_{s\in [n]}$  and  $(Y_{s})_{s\in [n]}$  be Gaussian observations from two distributions with unit variance. Then  $\frac{1}{m_1}\sum_{i = 1}^{m_1}X_i - \frac{1}{m_2}\sum_{i = 1}^{m_2}Y_i$  is Gaussian with variance  $\frac{1}{m_1} +\frac{1}{m_2}$ . By union bound and
using that  $n \leq t^{\alpha}$ , we obtain
$$
\begin{array}{l} \mathbb {P} \left(\exists k \neq i ^ {\star}, \exists t \in [ n ^ {1 / \alpha}, n ], \frac {\left(\mu_ {t , i ^ {\star}} - \mu_ {t , k}\right) - \left(\mu_ {i ^ {\star}} - \mu_ {k}\right)}{\sqrt {1 / N _ {t , i ^ {\star}} + 1 / N _ {t , k}}} \leq - \sqrt {2 \alpha (2 + s) \log (t)}\right) \\ \leq \sum_ {k \neq i ^ {\star}} \mathbb {P} \left(\exists t \in [ n ^ {1 / \alpha}, n ], \frac {\left(\mu_ {t , i ^ {\star}} - \mu_ {t , k}\right) - \left(\mu_ {i ^ {\star}} - \mu_ {k}\right)}{\sqrt {1 / N _ {t , i ^ {\star}} + 1 / N _ {t , k}}} \leq - \sqrt {2 (2 + s) \log (n)}\right) \\ \leq \sum_ {k \neq i ^ {\star}} \mathbb {P} \left(\exists (m _ {1}, m _ {2}) \in [ n ] ^ {2}, \frac {\frac {1}{m _ {1}} \sum_ {i = 1} ^ {m _ {1}} X _ {i} - \frac {1}{m _ {2}} \sum_ {i = 1} ^ {m _ {2}} Y _ {i}}{\sqrt {1 / m _ {1} + 1 / m _ {2}}} \leq - \sqrt {2 (2 + s) \log (n)}\right) \\ \leq \sum_ {k \neq i ^ {\star}} \sum_ {(m _ {1}, m _ {2}) \in [ n ] ^ {2}} \mathbb {P} \left(\frac {1}{m _ {1}} \sum_ {i = 1} ^ {m _ {1}} X _ {i} - \frac {1}{m _ {2}} \sum_ {i = 1} ^ {m _ {2}} Y _ {i} \leq - \sqrt {2 (2 + s) \log (n) (1 / m _ {1} + 1 / m _ {2})}\right) \\ \leq \sum_ {k \neq i ^ {\star}} \sum_ {(m _ {1}, m _ {2}) \in [ n ] ^ {2}} n ^ {- (2 + s)} = (K - 1) n ^ {- s} , \\ \end{array}
$$
where we used that  $(\mu_{t,i^{\star}} - \mu_{i^{\star}}) - (\mu_{t,k} - \mu_{k}) = \frac{1}{N_{t,i^{\star}}}\sum_{s=1}^{t}\mathbb{1}(I_{s} = i^{\star})X_{s,i^{\star}} - \frac{1}{N_{t,k}}\sum_{s=1}^{t}\mathbb{1}(I_{s} = k)X_{s,k}$  and concentration results for Gaussian observations.
Using a mixture of martingale arguments, we could improve on Lemma E.5 similarly as  $g_{u}$  improved on  $g_{m}$ . This will impact second order terms of our non-asymptotic theoretical guarantees, at the price of less explicit non-asymptotic terms.
Concentration event Lemma E.6 upper bounds the summed probabilities of the complementary events.
Lemma E.6. Let  $\zeta$  be the Riemann  $\zeta$  function. Let  $(\mathcal{E}_{1,n})_{n > K}$ ,  $(\mathcal{E}_{2,n})_{n > K}$  and  $(\mathcal{E}_{3,n})_{n > K}$  as in (17), (19) and (18). For all  $n > K$ , let  $\mathcal{E}_n = \mathcal{E}_{1,n} \cap \mathcal{E}_{2,n}$  and  $\tilde{\mathcal{E}}_n = \mathcal{E}_{3,n} \cap \mathcal{E}_{2,n}$ . Then,
$$
\max  \left\{\sum_ {n > K} \mathbb {P} \left(\mathcal {E} _ {n} ^ {\mathbf {C}}\right), \sum_ {n > K} \mathbb {P} \left(\tilde {\mathcal {E}} _ {n} ^ {\mathbf {C}}\right) \right\} \leq (2 K - 1) \zeta (s).
$$
Proof. Using Lemmas E.2 and E.5, direct union bound yields
$$
\sum_ {n \in \mathbb {N} ^ {\star}} \mathbb {P} \left(\mathcal {E} _ {n} ^ {\mathbb {C}}\right) \leq \sum_ {n \in \mathbb {N} ^ {\star}} \mathbb {P} \left(\mathcal {E} _ {1, n} ^ {\mathbb {C}}\right) + \mathbb {P} \left(\mathcal {E} _ {2, n} ^ {\mathbb {C}}\right) \leq \sum_ {n \in \mathbb {N} ^ {\star}} (2 K - 1) n ^ {- s} = (2 K - 1) \zeta (s).
$$
The same proof trivially holds for  $\tilde{\mathcal{E}}_n$  by using Lemmas E.3 and E.5.
F Asymptotic analysis
Based solely on Theorem D.4, it is not possible to obtain asymptotic  $\beta$ -optimality due to the multiplicative factor  $1 / \beta$ . Building on the unified analysis proposed in [21], we prove Theorem 2.3.
Our main technical contribution for this proof lies in the use of tracking instead of sampling. Given that the cumulative probability of being sampled is the expectation of the random empirical counts, it is not surprising that tracking-based Top Two algorithms enjoy the same theoretical guarantees as their sampling counterpart. As we will see, the analysis to obtain similar result is even simpler (Lemmas F.6, F.8 and F.10). Apart from this technical subtlety, the proof of Theorem 2.3 boils down to showing that the UCB leader satisfies the two sufficient properties highlighted by previous work (Lemmas F.4 and F.7).
Using the fact that  $x \to \sqrt{2x}$  is increasing, it is direct to see that the TC challenger (4) coincides with the definition used by [39, 21], i.e.
$$
C _ {n} ^ {\mathrm {T C}} = \underset {i \neq B _ {n}} {\arg \min} \frac {(\mu_ {n , B _ {n}} - \mu_ {n , i}) _ {+}}{\sqrt {1 / N _ {n , B _ {n}} + 1 / N _ {n , i}}} = \underset {i \neq B _ {n}} {\arg \min} \mathbb {1} (\mu_ {n, B _ {n}} > \mu_ {n, i}) \frac {(\mu_ {n , B _ {n}} - \mu_ {n , i}) ^ {2}}{2 (1 / N _ {n , B _ {n}} + 1 / N _ {n , i})}.
$$
F.1 Proof of Theorem 2.3
Let  $\mu \in \mathcal{D}^K$  such that  $\min_{i\neq j}|\mu_i - \mu_j| > 0$ , and let  $i^{\star} = i^{\star}(\mu)$  be the unique best arm. Let  $\beta \in (0,1)$  and  $w_{\beta}^{\star}$  be the unique allocation  $\beta$ -optimal allocation satisfying  $w_{\beta ,i}^{\star} > 0$  for all  $i\in [K]$  (Lemma C.2), i.e.  $w_{\beta}^{\star}(\mu) = \{w_{\beta}^{\star}\}$  where
Let  $\varepsilon > 0$ . Following [36, 39, 21], we aim at upper bounding the expectation of the convergence time  $T_{\beta}^{\varepsilon}$ , which is a random variable quantifies the number of samples required for the empirical allocations  $N_{n} / (n - 1)$  to be  $\varepsilon$ -close to  $w_{\beta}^{\star}$ :
$$
T _ {\beta} ^ {\varepsilon} := \inf  \left\{T \geq 1 \mid \forall n \geq T, \left\| \frac {N _ {n}}{n - 1} - w _ {\beta} ^ {\star} \right\| _ {\infty} \leq \varepsilon \right\}. \tag {20}
$$
Lemma F.1 shows that a sufficient condition for asymptotic  $\beta$ -optimality is to show  $\mathbb{E}_{\mu}[T_{\hat{\beta}}^{\varepsilon}] < +\infty$  for all  $\varepsilon$  small enough.
Lemma F.1. Let  $(\delta, \beta) \in (0, 1)^2$ . Assume that there exists  $\varepsilon_1(\mu) > 0$  such that for all  $\varepsilon \in (0, \varepsilon_1(\mu)]$ ,  $\mathbb{E}_{\mu}[T_{\beta}^{\varepsilon}] < +\infty$ . Combining the stopping rule (1) with threshold as in (2) yields an algorithm such that, for all  $\mu \in \mathbb{R}^K$  with  $|i^{\star}(\mu)| = 1$ ,
$$
\limsup_{\delta \to 0}\frac{\mathbb{E}_{\mu}[\tau_{\delta}]}{\log(1 / \delta)}\leq T^{\star}_{\beta}(\mu)  .
$$
Proof. While the first result in the spirit of Lemma F.1 was derived by [36] for Gaussian distributions, a proof holding for more general distributions is given by Theorem 2 in [21]. The sole criterion on the stopping threshold is to be asymptotically tight (Definition F.2).
Definition F.2. A threshold  $c:\mathbb{N}\times (0,1]\to \mathbb{R}_+$  is said to be asymptotically tight if there exists  $\alpha \in [0,1)$ ,  $\delta_0\in (0,1]$ , functions  $f,\bar{T}:(0,1]\to \mathbb{R}_{+}$  and  $C$  independent of  $\delta$  satisfying: (1) for all  $\delta \in (0,\delta_0]$  and  $n\geq T(\delta)$ , then  $c(n,\delta)\leq f(\delta) + Cn^{\alpha}$ , (2)  $\lim \sup_{\delta \to 0}f(\delta) / \log (1 / \delta)\leq 1$  and (3)  $\lim \sup_{\delta \to 0}\bar{T} (\delta) / \log (1 / \delta) = 0$ .
Since  $\mathcal{C}_G$  defined in (16) satisfies  $\mathcal{C}_G\approx x + \log (x)$ , it is direct to see that
is asymptotically tight, e.g. by taking  $(\alpha, \delta_0, C) = (1/2, 1, 4)$ ,  $f(\delta) = 2\mathcal{C}_G\left(\frac{1}{2}\log\left(\frac{K-1}{\delta}\right)\right)$  and  $\bar{T}(\delta) = 1$ . This concludes the proof.
Throughout the proof, we will use a concentration result of the empirical mean (Lemma F.3) Since this is a standard result for Gaussian observations (see Lemma 5 of [36]), we omit the proof.
Lemma F.3. There exists a sub-Gaussian random variable  $W_{\mu}$  such that almost surely, for all  $i \in [K]$  and all  $n$  such that  $N_{n,i} \geq 1$ ,
Sufficient exploration To upper bound the expected convergence time, as prior work we first establish sufficient exploration. Given an arbitrary threshold  $L \in \mathbb{R}_+^*$ , we define the sampled enough set and its arms with highest mean (when not empty) as
$$
S _ {n} ^ {L} := \left\{i \in [ K ] \mid N _ {n, i} \geq L \right\} \quad \text {a n d} \quad \mathcal {I} _ {n} ^ {\star} := \underset {i \in S _ {n} ^ {L}} {\arg \max } \mu_ {i}. \tag {21}
$$
Since  $\min_{i\neq j}|\mu_i - \mu_j| > 0,\mathcal{I}_n^\star$  is a singleton. We define the highly and the mildly under-sampled sets
$$
U _ {n} ^ {L} := \left\{i \in [ K ] \mid N _ {n, i} <   \sqrt {L} \right\} \quad \text {a n d} \quad V _ {n} ^ {L} := \left\{i \in [ K ] \mid N _ {n, i} <   L ^ {3 / 4} \right\}. \tag {22}
$$
[21] identifies the properties that the leader and the challenger should satisfy to ensure sufficient exploration. Lemma F.4 show that the desired property for the UCB leader defined in (3) with bonus  $g_{u}(n) = 2\alpha (1 + s)\log n$  or
$$
g _ {m} (n) = \bar {W} _ {- 1} \left(2 s \alpha \log (n) + 2 \log (2 + \alpha \log n) + 2\right).
$$
Lemma F.4. There exists  $L_0$  with  $\mathbb{E}_{\mu}[(L_0)^\alpha] < +\infty$  for all  $\alpha > 0$  such that if  $L \geq L_0$ , for all  $n$  (at most polynomial in  $L$ ) such that  $S_n^L \neq \emptyset$ ,  $B_n^{UCB} \in S_n^L$  implies  $B_n^{UCB} \in \mathcal{I}_n^\star$  and  $B_n^{UCB} \in \arg \max_{i \in S_n^L} \mu_{n,i}$ .
Proof. Let  $\varepsilon > 0$  and  $\Delta_{\min} = \min_{i \neq j} |\mu_i - \mu_j|$ . Let  $g$  denote either  $g_u$  or  $g_m$ . Let  $L \geq L_0$ , where  $L_0$  will be specified later, and  $n$  (at most polynomial in  $L$ ) such that  $S_n^L \neq \emptyset$ . Then, there exists a polynomial function  $P$  such that  $n \leq P(L)$ . By considering arms that are sampled more than  $L$ , we can show that for all  $k \in S_n^L \setminus \mathcal{I}_n^\star$ ,
$$
\mu_ {n, k} + \sqrt {\frac {g (n)}{N _ {n , k}}} \leq \mu_ {k} + W _ {\mu} \sqrt {\frac {\log (e + N _ {n , k})}{1 + N _ {n , k}}} + \sqrt {\frac {g (P (L))}{L}} \leq \mu_ {k} + W _ {\mu} \sqrt {\frac {\log (e + L)}{1 + L}} + \sqrt {\frac {g (P (L))}{L}} \leq \mu_ {k} + 2 \varepsilon ,
$$
where the last inequality is obtained for  $L \geq L_0$  with
$$
L _ {0} = 1 + \sup  \left\{L \in N ^ {\star} \mid W _ {\mu} \sqrt {\frac {\log (e + L)}{1 + L}} > \varepsilon , \sqrt {\frac {g (P (L))}{L}} > \varepsilon \right\}.
$$
Since  $\overline{W}_{-1}(x) \approx x + \log(x)$ , both  $g_u$  and  $g_m$  have a logarithmic behavior, hence  $g(P(L)) = L \to +\infty$  or  $(L)$ . Since  $L_0$  is polynomial in  $W_\mu$ , Lemma F.3 yields  $\mathbb{E}_\mu[(L_0)^\alpha] < +\infty$  for all  $\alpha > 0$ .
Moreover, for all  $k\in \mathcal{I}_n^{\star}$
$$
\mu_ {n, k} + \sqrt {\frac {g (n)}{N _ {n , k}}} \geq \mu_ {k} - W _ {\mu} \sqrt {\frac {\log (e + N _ {n , k})}{1 + N _ {n , k}}} \geq \mu_ {k} - W _ {\mu} \sqrt {\frac {\log (e + L)}{1 + L}} \geq \mu_ {k} - \varepsilon .
$$
Assume that  $B_{n}^{\mathrm{UCB}} \in S_{n}^{L}$  and that  $B_{n}^{\mathrm{UCB}} \notin \mathcal{I}_{n}^{\star}$ . Since  $B_{n}^{\mathrm{UCB}} = \arg \max_{k \in [K]} \left\{ \mu_{n,k} + \sqrt{\frac{g(n)}{N_{n,k}}} \right\}$ , taking  $\varepsilon < \Delta_{\min} / 3$  in the above yields a direct contradiction.
We can use the proof of [21] to obtain Lemma F.5. While their result accounts for the randomization of the sampling procedure, the argument is direct for tracking since it removes the need for a concentration argument.
Lemma F.5 (Lemma 19 in [21]). Let  $\mathcal{J}_n^\star = \arg \max_{i\in \overline{V_n^L}}\mu_i$ . There exists  $L_{1}$  with  $\mathbb{E}_{\mu}[L_1] < +\infty$  such that if  $L\geq L_{1}$ , for all  $n$  (at most polynomial in  $L$ ) such that  $U_{n}^{L}\neq \emptyset$ ,  $B_{n}^{UCB}\notin V_{n}^{L}$  implies  $C_n^{TC}\in V_n^L\cup \left(\mathcal{J}_n^\star \setminus \left\{B_n^{UCB}\right\}\right)$ .
Lemma F.6 proves sufficient exploration for the TTUCB sampling rule. It builds on the same reasoning than the one used in the proofs introduced by [39], and generalized by [21].
Lemma F.6. Assume  $\min_{j\neq i}|\mu_i - \mu_j| > 0$ . Under the TTUCB sampling rule, there exist  $N_0$  with  $\mathbb{E}_{\mu}[N_0] < +\infty$  such that for all  $n\geq N_0$  and all  $i\in [K]$ ,  $N_{n,i}\geq \sqrt{n / K}$ .
Proof. Let  $L_0$  and  $L_1$  as in Lemmas F.4 and F.5. Therefore, for  $L \geq L_2 \coloneqq \max \{L_1, L_0^{4/3}\}$ , for all  $n$  such that  $U_n^L \neq \emptyset$ ,  $B_n^{\mathrm{UCB}} \in V_n^L$  or  $C_n^{\mathrm{TC}} \in V_n^L$  since  $|\mathcal{J}_n^\star| = 1$  is implied by  $\min_{j \neq i} |\mu_i - \mu_j| > 0$ . We have  $\mathbb{E}_{\mu}[L_2] < +\infty$ . There exists a deterministic  $L_4$  such that for all  $L \geq L_4$ ,  $\lfloor L \rfloor \geq KL^{3/4}$ . Let  $L \geq \max \{L_2, L_4\}$ .
Suppose towards contradiction that  $U_{[KL]}^{L}$  is not empty. Then, for any  $1 \leq t \leq [KL]$ ,  $U_{t}^{L}$  and  $V_{t}^{L}$  are non empty as well. Using the pigeonhole principle, there exists some  $i \in [K]$  such that
$N_{[L],i} \geq L^{3/4}$ . Thus, we have  $\left|V_{[L]}^L\right| \leq K - 1$ . Our goal is to show that  $\left|V_{[2L]}^L\right| \leq K - 2$ . A sufficient condition is that one arm in  $V_{[L]}^L$  is pulled at least  $L^{3/4}$  times between  $[L]$  and  $[2L] - 1$ .
Case 1. Suppose there exists  $i \in V_{\lfloor L \rfloor}^{L}$  such that  $L_{\lfloor 2L \rfloor, i} - L_{\lfloor L \rfloor, i} \geq \frac{L^{3/4}}{\beta} + 3/(2\beta)$ . Using Lemma D.3, we obtain
$$
N _ {\lfloor 2 L \rfloor , i} ^ {i} - N _ {\lfloor L \rfloor , i} ^ {i} \geq \beta \left(L _ {\lfloor 2 L \rfloor , i} - L _ {\lfloor L \rfloor , i}\right) - 3 / 2 \geq L ^ {3 / 4},
$$
hence  $i$  is sampled  $L^{3 / 4}$  times between  $\lfloor L\rfloor$  and  $\lfloor 2L\rfloor -1$
Case 2. Suppose that, for all  $i \in V_{\lfloor L \rfloor}^{L}, L_{\lfloor 2L \rfloor, i} - L_{\lfloor L \rfloor, i} < \frac{L^{3/4}}{\beta} + 3 / (2\beta)$ . Then,
$$
\sum_ {i \notin V _ {\lfloor L \rfloor} ^ {L}} \left(L _ {\lfloor 2 L \rfloor , i} - L _ {\lfloor L \rfloor , i}\right) \geq (\lfloor 2 L \rfloor - \lfloor L \rfloor) - K \left(\frac {L ^ {3 / 4}}{\beta} + 3 / (2 \beta)\right)
$$
Using Lemma D.3, we obtain
$$
\left| \sum_ {i \notin V _ {\lfloor L \rfloor} ^ {L}} \left(N _ {\lfloor 2 L \rfloor , i} ^ {i} - N _ {\lfloor L \rfloor , i} ^ {i}\right) - \beta \sum_ {i \notin V _ {\lfloor L \rfloor} ^ {L}} \left(L _ {\lfloor 2 L \rfloor , i} - L _ {\lfloor L \rfloor , i}\right) \right| \leq 3 (K - 1) / 2.
$$
Combining all the above, we obtain
$$
\begin{array}{l} \sum_ {i \notin V _ {\lfloor L \rfloor} ^ {L}} \left(L _ {\lfloor 2 L \rfloor , i} - L _ {\lfloor L \rfloor , i}\right) - \sum_ {i \notin V _ {\lfloor L \rfloor} ^ {L}} \left(N _ {\lfloor 2 L \rfloor , i} ^ {i} - N _ {\lfloor L \rfloor , i} ^ {i}\right) \\ \geq (1 - \beta) \sum_ {i \notin V _ {\lfloor L \rfloor} ^ {L}} \left(L _ {\lfloor 2 L \rfloor , i} - L _ {\lfloor L \rfloor , i}\right) - 3 (K - 1) / 2 \\ \geq (1 - \beta) \left(\left(\lfloor 2 L \rfloor - \lfloor L \rfloor\right) - K \left(\frac {L ^ {3 / 4}}{\beta} + 3 / (2 \beta)\right)\right) - 3 (K - 1) / 2 \geq K L ^ {3 / 4}, \\ \end{array}
$$
where the last inequality is obtained for  $L \geq L_5$  with
$$
L _ {5} = \sup  \left\{L \in \mathbb {N} \mid (1 - \beta) \left(\left(\lfloor 2 L \rfloor - \lfloor L \rfloor\right) - K \left(\frac {L ^ {3 / 4}}{\beta} + 3 / (2 \beta)\right)\right) - 3 (K - 1) / 2 <   K L ^ {3 / 4} \right\}.
$$
The l.h.s. summation is exactly the number of times where an arm  $i \in \overline{V_{[L]}^{L}}$  was leader but wasn't sampled, hence
$$
\sum_ {t = \lfloor L \rfloor} ^ {\lfloor 2 L \rfloor - 1} \mathbb {1} \left(B _ {t} ^ {\mathrm {U C B}} \in \overline {{V _ {\lfloor L \rfloor} ^ {L}}}, I _ {t} = C _ {t} ^ {\mathrm {T C}}\right) \geq K L ^ {3 / 4}
$$
For any  $\lfloor L\rfloor \leq t\leq \lfloor 2L\rfloor -1$ ,  $U_{t}^{L}$  is non-empty, hence we have  $B_{t}^{\mathrm{UCB}}\in \overline{V_{\lfloor L\rfloor}^{L}}\subseteq \overline{V_{t}^{L}}$  implies  $C_t^{\mathrm{TC}}\in V_t^L\subseteq V_{|L|}^L$ . Therefore, we have shown that
$$
\sum_ {t = \lfloor L \rfloor} ^ {\lfloor 2 L \rfloor - 1} \mathbb {1} \left(I _ {t} \in V _ {\lfloor L \rfloor} ^ {L}\right) \geq \sum_ {t = \lfloor L \rfloor} ^ {\lfloor 2 L \rfloor - 1} \mathbb {1} \left(B _ {t} ^ {\mathrm {U C B}} \in \overline {{V _ {\lfloor L \rfloor} ^ {L}}}, I _ {t} = C _ {t} ^ {\mathrm {T C}}\right) \geq K L ^ {3 / 4}.
$$
Therefore, there is at least one arm in  $V_{\lfloor L \rfloor}^{L}$  that is sampled  $L^{3/4}$  times between  $\lfloor L \rfloor$  and  $\lfloor 2L \rfloor - 1$ .
In summary, we have shown  $\left|V_{[2L]}^{L}\right| \leq K - 2$ . By induction, for any  $1 \leq k \leq K$ , we have  $\left|V_{[kL]}^{L}\right| \leq K - k$ , and finally  $U_{[KL]}^{L} = \emptyset$  for all  $L \geq L_3$ . This concludes the proof.
Convergence towards  $\beta$ -optimal allocation Provided sufficient exploration holds (Lemma F.6), [21] identifies the properties that the leader and the challenger should satisfy to obtain convergence towards the  $\beta$ -optimal allocation  $w_{\beta}^{\star}$ . Lemma F.7 show that the desired property for the UCB leader defined in (3) with bonus  $g_{u}(n) = 2\alpha(1 + s)\log n$  or
Lemma F.7. There exists  $N_{1}$  with  $\mathbb{E}_{\mu}[N_1] < +\infty$  such that for all  $n \geq N_1$ ,  $B_n^{UCB} = i^*(\mu)$ .
Proof. Let  $\Delta = \min_{i\neq i^{\star}}|\mu_{i^{\star}} - \mu_{i}|$ . Let  $g$  denote either  $g_{u}$  or  $g_{m}$ . Let  $N_0$  as in Lemma F.6. For all  $n\geq N_2$  (to be specified later), we obtain for all  $k\neq i^{\star}$
$$
\mu_ {n, k} + \sqrt {\frac {g (n)}{N _ {n , k}}} \leq \mu_ {k} + W _ {\mu} \sqrt {\frac {\log (e + N _ {n , k})}{1 + N _ {n , k}}} + K ^ {1 / 4} \sqrt {\frac {g (n)}{\sqrt {n}}} \leq \mu_ {k} + W _ {\mu} \sqrt {\frac {\log (e + \sqrt {\frac {n}{K}})}{1 + \sqrt {\frac {n}{K}}}} + K ^ {1 / 4} \sqrt {\frac {g (n)}{\sqrt {n}}} \leq \mu_ {k} + 2 \varepsilon ,
$$
where the last inequality is obtained for  $n \geq N_1$  where
$$
N _ {1} = 1 + \sup \left\{n \in N ^ {\star} \mid W _ {\mu} \sqrt {\frac {\log (e + \sqrt {\frac {n}{K}})}{1 + \sqrt {\frac {n}{K}}}} > \varepsilon , K ^ {1 / 4} \sqrt {\frac {g (n)}{\sqrt {n}}} > \varepsilon \right\}.
$$
Since  $\overline{W}_{-1}(x) \approx x + \log(x)$ , both  $g_u$  and  $g_m$  have a logarithmic behavior, hence  $g(n) = n \to +\infty$  or  $(\sqrt{n})$ . Since  $N_1$  is polynomial in  $W_\mu$ , Lemma F.3 yields  $\mathbb{E}_\mu[(N_1)^\alpha] < +\infty$  for all  $\alpha > 0$ .
$$
\mu_ {n, i ^ {\star}} + \sqrt {\frac {g (n)}{N _ {n , i ^ {\star}}}} \geq \mu_ {i ^ {\star}} - W _ {\mu} \sqrt {\frac {\log (e + N _ {n , i ^ {\star}})}{1 + N _ {n , i ^ {\star}}}} \geq \mu_ {i ^ {\star}} - W _ {\mu} \sqrt {\frac {\log (e + \sqrt {\frac {n}{K}})}{1 + \sqrt {\frac {n}{K}}}} \geq \mu_ {i ^ {\star}} - \varepsilon .
$$
Therefore, taking  $\varepsilon <  \Delta /3$  yields the result that  $B_{n}^{\mathrm{UCB}} = i^{\star}$
Combining Lemma F.7 with tracking properties (Lemma D.3), we obtain convergence towards the  $\beta$ -optimal allocation for the best arm (Lemma F.8).
Lemma F.8. Let  $\varepsilon > 0$ . Under the TTUCB sampling rule, there exists  $N_2$  with  $\mathbb{E}_{\mu}[N_2] < +\infty$  such that for all  $n \geq N_2$ ,
$$
\left| \frac {N _ {n , i ^ {\star} (\mu)}}{n - 1} - \beta \right| \leq \varepsilon .
$$
Proof. Let  $N_0$  as in Lemma F.6 and  $C_1$  as in Lemma F.7. For all  $n \geq \max \{N_1, N_0\}$ , we have  $B_n = i^\star$ . Let  $M \geq \max \{N_1, N_0\}$ . Then, we have  $L_{n,i^{\star}} \geq n - M$  and  $\sum_{k \neq i^{\star}} N_{n,i^{\star}}^{k} \leq M - 1$  for all  $n \geq M + 1$ . Using Lemma D.3, we have
$$
\left| \frac {N _ {n , i ^ {\star}}}{n - 1} - \beta \right| \leq \frac {| N _ {n , i ^ {\star}} ^ {i ^ {\star}} - \beta L _ {n , i ^ {\star}} |}{n - 1} + \beta \left| \frac {L _ {n , i ^ {\star}}}{n - 1} - 1 \right| + \frac {1}{n - 1} \sum_ {k \neq i ^ {\star}} N _ {n, i ^ {\star}} ^ {k} \leq \frac {1}{2 (n - 1)} + \beta \frac {2 (M - 1)}{n - 1}.
$$
Taking  $N_{2} = \max \{N_{1}, N_{0}, \frac{1/2 + 2\beta(M - 1)}{\varepsilon} + 1\}$  yields the result.
We can use the proof of [21] to obtain Lemma F.9. While their result accounts for the randomization of the sampling procedure, the argument is direct for tracking since it removes the need for a concentration argument.
Lemma F.9 (Lemma 20 in [21]). Let  $\varepsilon > 0$ . Under the TTUCB sampling rule, there exists  $N_3$  with  $\mathbb{E}_{\mu}[N_3] < +\infty$  such that for all  $n \geq N_3$  and all  $i \neq i^{\star}(\mu)$ ,
$$
\frac {N _ {n , i}}{n - 1} \geq w _ {\beta , i} ^ {\star} + \varepsilon \quad \Longrightarrow \quad C _ {n} ^ {T C} \neq i.
$$
Combining Lemma F.9 with tracking properties (Lemma D.3), we obtain convergence towards the  $\beta$ -optimal allocation for all arms (Lemma F.10).
Lemma F.10. Let  $\varepsilon > 0$ . Under the TTUCB sampling rule, there exists  $N_4$  with  $\mathbb{E}_{\mu}[N_4] < +\infty$  such that for all  $n \geq N_4$ ,
$$
\forall i \in [ K ], \quad \left| \frac {N _ {n , i}}{n - 1} - w _ {\beta , i} ^ {\star} \right| \leq \varepsilon .
$$
Proof. Let  $N_0, C_1, N_2$  and  $N_3$  as in Lemmas F.6, F.7, F.8 and F.9. For all  $n \geq \max\{N_0, N_1, N_2, N_3\}$ , we have  $B_n^{\mathrm{UCB}} = i^\star$ ,  $\left|\frac{N_{n,i^\star}}{n-1} - \beta\right| \leq \varepsilon$  and for all  $i \neq i^\star$ ,
$$
\frac {N _ {n - 1 , i}}{n - 1} \geq w _ {\beta , i} ^ {\star} + \varepsilon \Rightarrow C _ {n} ^ {\mathrm {T C}} \neq i.
$$