SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The choice of  $K$  independent tracking procedures was made for two reasons. First, independent procedures are simpler to analyze for theoretical purpose. Second, independent procedures yields better empirical performance since it avoids over-sampling a sub-optimal arm when it is mistakenly chosen as leader. To understand the second argument, let's look at another design with one tracking procedure. Namely we set  $I_{n} = B_{n}$  if  $N_{n,B_{n}} \leq \beta n$ , else  $I_{n} = C_{n}$ . When  $B_{n} \neq i^{\star}$ , then it will (almost always) take  $I_{n} = B_{n}$  since  $N_{n,B_{n}}$  is lower than  $\beta n$ . On the other hand, when  $B_{n} \neq i^{\star}$  with  $K$  independent tracking procedures, both  $N_{n,B_{n}}^{B_{n}}$  and  $L_{n,B_{n}}$  are small, hence it is less systematic.
D.2 Proof of Theorem 2.4
Theorem 2.4 is a direct corollary of Theorem D.4, which holds for any  $\beta \in (0,1)$ ,  $s > 1$  and  $\alpha > 1$ .
Theorem D.4. Let  $(\delta, \beta) \in (0,1)^2$ ,  $s > 1$  and  $\alpha > 1$ . Using the threshold (2) in (1) and  $g_u$  in (3), the TTUCB algorithm satisfies that, for all  $\mu \in \mathbb{R}^K$  such that  $|i^\star(\mu)| = 1$ ,
$$
\mathbb {E} _ {\mu} \left[ \tau_ {\delta} \right] \leq \max  \left\{T _ {0} (\delta), C _ {\mu} ^ {\alpha}, C _ {0} ^ {\frac {\alpha}{\alpha - 1}}, C _ {1} ^ {\alpha} \right\} + C _ {2},
$$
$$
\begin{array}{l} T _ {0} (\delta) = \sup  \left\{n > K \mid n - 1 \leq T _ {\beta} ^ {\star} (\mu) \frac {(1 + \varepsilon) ^ {2}}{\beta (1 - w _ {0}) ^ {d _ {\mu} (w _ {0})}} (\sqrt {c (n - 1 , \delta)} + \sqrt {\alpha (2 + s) \log n}) ^ {2} \right\}, \\ C _ {\mu} = h _ {1} \left(4 \alpha^ {2} (1 + s) H (\mu) / \beta\right), C _ {0} = \frac {2}{\varepsilon a _ {\mu} (w _ {0})} + 1, C _ {1} = 1 / (\beta \varepsilon), C _ {2} = (2 K - 1) \zeta (s) + 1, \\ a _ {\mu} (w _ {0}) = (1 - w _ {0}) ^ {d _ {\mu} (w _ {0})} \max  \{\min  _ {i \neq i ^ {\star} (\mu)} w _ {\beta} ^ {\star} (\mu) _ {i}, (1 - \beta) w _ {0} \}, d _ {\mu} (w _ {0}) = | \{i \in [ K ] \setminus \{i ^ {\star} (\mu) \} | w _ {\beta} ^ {\star} (\mu) _ {i} <   (1 - \beta) w _ {0} \} |, \\ \end{array}
$$
with  $\varepsilon \in (0,1]$ ,  $w_0 \in [0,1/(K-1)]$  and  $\zeta$  is the Riemann  $\zeta$  function. For all  $x > 0$ , the function  $h_1(x) := x\overline{W}_{-1}\left(\log(x) + \frac{2+K/\beta}{x}\right)$  is positive, increasing for  $x \geq 2+K/\beta$ , and satisfies  $h_1(x) \approx x(\log x + \log \log x)$ .
Let  $n > K$  such that  $\mathcal{E}_n$  holds true and the algorithm has not stop yet, i.e.  $\mathcal{E}_n \cap \{n < \tau_{\delta}\}$ . Let  $t \in [n^{1 / \alpha}, n]$  such that  $B_t = i^\star$ . Let  $c(n, \delta)$  as in (2), which satisfies that  $n \mapsto c(n, \delta)$  is increasing. Using the stopping rule (1) and  $t \leq n < \tau_{\delta}$ , we obtain
$$
\sqrt {2 c (n - 1 , \delta)} \geq \sqrt {2 c (t - 1 , \delta)} \geq \min _ {i \neq \hat {i} _ {t}} \frac {\mu_ {t , \hat {i} _ {t}} - \mu_ {t , i}}{\sqrt {1 / N _ {t , \hat {i} _ {t}} + 1 / N _ {t , i}}} \geq \min _ {i \neq B _ {t}} \frac {(\mu_ {t , B _ {t}} - \mu_ {t , i}) _ {+}}{\sqrt {1 / N _ {t , B _ {t}} + 1 / N _ {t , i}}} = \frac {(\mu_ {t , B _ {t}} - \mu_ {t , C _ {t}}) _ {+}}{\sqrt {1 / N _ {t , B _ {t}} + 1 / N _ {t , C _ {t}}}},
$$
The last inequality is an equality when  $B_{t} = \hat{\iota}_{t}$ , and trivially true when  $B_{t} \neq \hat{\iota}_{t}$  since a positive term is higher than zero (already null when taking  $i = \iota_t$ ). Using Lemma D.2, we obtain
$$
\begin{array}{l} \frac {\mu_ {t , B _ {t}} - \mu_ {t , C _ {t}}}{\sqrt {1 / N _ {t , B _ {t}} + 1 / N _ {t , C _ {t}}}} \geq \frac {\mu_ {i ^ {*}} - \mu_ {C _ {t}}}{\sqrt {1 / N _ {t , i ^ {*}} ^ {i ^ {*}} + 1 / N _ {t , C _ {t}} ^ {i ^ {*}}}} - \sqrt {2 \alpha (2 + s) \log n} \\ \geq \sqrt {\frac {1 / \beta + 1 / w _ {\beta , C _ {t}} ^ {\star}}{1 / N _ {t , i ^ {\star}} ^ {i ^ {\star}} + 1 / N _ {t , C _ {t}} ^ {i ^ {\star}}}} \min  _ {i \neq i ^ {\star}} \frac {\mu_ {i ^ {\star}} - \mu_ {i}}{\sqrt {1 / \beta + 1 / w _ {\beta , i} ^ {\star}}} - \sqrt {2 \alpha (2 + s) \log n} \\ \geq \sqrt {\frac {1 / \beta + 1 / w _ {\beta , C _ {t}} ^ {\star}}{1 / N _ {t , i ^ {\star}} ^ {i ^ {\star}} + 1 / N _ {t , C _ {t}} ^ {i ^ {\star}}}} \sqrt {2 T _ {\beta} ^ {\star} (\mu) ^ {- 1}} - \sqrt {2 \alpha (2 + s) \log n}, \\ \end{array}
$$
where the second inequality is obtained by artificially making appear  $1 / \beta + 1 / w_{\beta, C_t}^\star$  and taking the minimum of  $i \neq i^\star$ . The last inequality simply uses the definition of  $w_\beta^{i^\star}$ .
While combining Lemma D.1 and Lemma D.3 links  $N_{t,i^{\star}}^{i^{\star}} / (t - 1)$  with  $\beta$ , we need another argument to compare the empirical allocation  $N_{t,C_t}^{i^{\star}} / (t - 1)$  of the sub-optimal arm  $C_t$  with its  $\beta$ -optimal allocation  $w_{\beta ,C_t}^{\star}$ . Before delving into this key argument, we conclude the proof under an assumption that will be shown to hold later: there exists  $D_0 > 0$  and  $T_{\mu} > 0$  such that for all  $n > T_{\mu}$ , there exists a well chosen  $t \in [n^{1 / \alpha},n]$  with  $B_{t} = i^{\star}$ , which satisfies
$$
1 / N _ {t, i ^ {\star}} ^ {i ^ {\star}} + 1 / N _ {t, C _ {t}} ^ {i ^ {\star}} \leq \frac {D _ {0}}{n - 1} \left(1 / \beta + 1 / w _ {\beta , C _ {t}} ^ {\star}\right). \tag {7}
$$
Let's define
$$
T (\delta , D _ {0}) := \sup  \left\{n > K \mid n - 1 \leq T _ {\beta} ^ {\star} (\mu) D _ {0} \left(\sqrt {c (n - 1 , \delta)} + \sqrt {\alpha (2 + s) \log n}\right) ^ {2} \right\}.
$$
For all  $n > \max \{T_{\mu}, T(1, D_0)\}$ , the lower bound on  $\frac{\mu_{t,B_t} - \mu_{t,C_t}}{\sqrt{1 / N_{t,B_t} + 1 / N_{t,C_t}}}$  is positive, hence we can use that it is upper bounded by  $\sqrt{2c(n - 1,\delta)}$ . Putting everything together, we have shown that
$$
\forall n > \max \{T _ {\mu}, T (1, D _ {0}) \}, \quad n - 1 \leq T _ {\beta} ^ {\star} (\mu) D _ {0} \left(\sqrt {c (n - 1 , \delta)} + \sqrt {\alpha (2 + s) \log n}\right) ^ {2}.
$$
Therefore, we have  $\mathcal{E}_n\cap \{n <   \tau_{\delta}\} = \emptyset$  (i.e.  $\mathcal{E}_n\subset \{\tau_\delta \leq n\}$  ) for all  $n\geq$  max  $\{T(\delta ,D_0),T(1,D_0),T_\mu \} +1$  . Using that  $\delta \to T(\delta ,D_0)$  is an decreasing function (since  $\mathcal{C}_G$  is increasing), we obtain that  $T(\delta ,D_0) = \max \{T(\delta ,D_0),T(1,D_0)\}$
Combining Lemmas D.8 and E.6 yields
$$
\mathbb {E} _ {\mu} \left[ \tau_ {\delta} \right] \leq \max  \left\{T \left(\delta , D _ {0}\right), T _ {\mu} \right\} + 1 + \sum_ {n \geq 1} \mathbb {P} _ {\mu} \left(\mathcal {E} _ {n} ^ {\mathfrak {C}}\right) \leq \max  \left\{T \left(\delta , D _ {0}\right), T _ {\mu} \right\} + 1 + (2 K - 1) \zeta (s).
$$
At this stage, the proof of Theorem D.4 boils down to exhibiting  $D_0 > 0$  and  $T_{\mu} > 0$  such that: for all  $n > T_{\mu}$ , there exists a well chosen  $t \in [n^{1 / \alpha}, n]$  with  $B_t = i^\star$  and such that (7) holds. As mentioned above, the crux of the problem is to relate  $N_{t,C_t}^{i^\star} / (t - 1)$  and  $w_{\beta ,C_t}^\star$ . To do so, we will build on the idea behind the proof for APT from [30]: consider an arm being over-sampled and study the last time this arm was pulled.
By the pigeonhole principle, at time  $n$ , there is an index  $k_{1} \neq i^{\star}$  such that (5) holds, i.e.
$$
N _ {n, k _ {1}} ^ {i ^ {\star}} \geq \frac {w _ {\beta , k _ {1}} ^ {\star}}{1 - \beta} \left(L _ {n, i ^ {\star}} - N _ {n, i ^ {\star}} ^ {i ^ {\star}}\right),
$$
and we take such  $k_{1}$ . Let  $t_{1} := \sup \left\{t < n \mid (B_{t}, C_{t}) = (i^{\star}, k_{1})\right\}$  be the last time at which  $i^{\star}$  was the leader and  $k_{1}$  was the challenger. If  $I_{t_{1}} = B_{t_{1}}$  then  $N_{t_{1},k_{1}}^{i^{\star}} = N_{n,k_{1}}^{i^{\star}}$ , else  $N_{t_{1},k_{1}}^{i^{\star}} = N_{n,k_{1}}^{i^{\star}} - 1$ . In both cases, we have  $N_{t_{1},k_{1}}^{i^{\star}} \geq N_{n,k_{1}}^{i^{\star}} - 1$ . Let  $f_{1}$  as in (14). Combined with Lemma D.9, we obtain
$$
N _ {t _ {1}, k _ {1}} ^ {i ^ {\star}} \geq N _ {n, k _ {1}} ^ {i ^ {\star}} - 1 \geq \frac {w _ {\beta , k _ {1}} ^ {\star}}{1 - \beta} \left(L _ {n, i ^ {\star}} - N _ {n, i ^ {\star}} ^ {i ^ {\star}}\right) - 1 \geq w _ {\beta , k _ {1}} ^ {\star} (n - 1) - f _ {1} (n).
$$
Let  $w_{-} > 0$  be a lower bound on  $w_{\beta, k_1}^{\star}$ , for example consider  $w_{-} = \min_{i \neq i^{\star}} w_{\beta, i}^{\star}$ . Let
$$
C _ {0} \left(w _ {-}\right) = \sup  \left\{n \geq 1 \mid n - 1 <   \frac {1}{w _ {-}} \left(n ^ {1 / \alpha} + f _ {1} (n)\right) \right\}. \tag {8}
$$
Since  $t_1 - 1 \geq N_{t_1, k_1}^{i^{\star}} \geq w_{\beta, k_1}^{\star}(n - 1) - f_1(n)$ , we obtain that  $t_1 \geq n^{1/\alpha}$  for all  $n > N_0(w_-)$ . For instances  $\mu$  such that  $w_{\beta, k_1}^{\star}$  is small, the equation (5) can be satisfied at the very beginning, hence  $t_1$  might be sublinear in  $n$ . Therefore, while combining Lemma D.1 and Lemma D.3 yields  $N_{t_1, i^{\star}}^{i^{\star}} \stackrel{>}{{\approx}} \beta(t_1 - 1)$ , it is not possible to obtain  $N_{t_1, i^{\star}}^{i^{\star}} \stackrel{>}{{\approx}} \beta(n - 1)$ . Due to the missing link between  $t_1$  and  $n$ , we use the following inequality
$$
1 / N _ {t _ {1}, i ^ {\star}} ^ {i ^ {\star}} + 1 / N _ {t _ {1}, k _ {1}} ^ {i ^ {\star}} \leq \frac {1}{n - 1} \left(1 / \beta + \frac {n - 1}{N _ {t _ {1} , k _ {1}} ^ {i ^ {\star}}}\right) \left(\frac {N _ {t _ {1} , k _ {1}} ^ {i ^ {\star}}}{N _ {t _ {1} , i ^ {\star}}} + 1\right),
$$
which is a suboptimal step which artificially introduces  $1 / \beta$ , and is responsible for the multiplicative factor  $1 / \beta$  in Theorem D.4. Improving on this suboptimal step is an interesting question, whose answer still eludes us. One idea would be to leverage the information of the sampled arm at time  $t$  since we have  $N_{t,i^{\star}}^{i^{\star}} \leq \beta L_{t+1,i^{\star}}$  when  $i^{\star} = B_{t} = I_{t}$ , else  $N_{t,i^{\star}}^{i^{\star}} \geq \beta L_{t+1,i^{\star}}$ .
Let  $\varepsilon \in (0,1]$ . It remains to control both terms. First, we obtain
$$
1 / \beta + \frac {n - 1}{N _ {t _ {1} , k _ {1}} ^ {i ^ {\star}}} \leq 1 / \beta + \frac {1}{w _ {\beta , k _ {1}} ^ {\star} - f _ {1} (n) / (n - 1)} \leq (1 + \varepsilon) \left(1 / \beta + 1 / w _ {\beta , k _ {1}} ^ {\star}\right),
$$
for all  $n > C_1(w_-)$ . The last inequality is obtained by definition of
$$
C _ {1} \left(w _ {-}\right) = \sup  \left\{n \geq 1 \mid n - 1 <   \frac {f _ {1} (n)}{w _ {-}} \left(1 + \frac {1}{\varepsilon}\right) \right\}, \tag {9}
$$
which ensures that, for all  $n > C_1(w_-)$ , the last condition of the equivalence
$$
w _ {\beta , k _ {1}} ^ {\star} - f _ {1} (n) / (n - 1) \geq (1 + \varepsilon) ^ {- 1} w _ {\beta , k _ {1}} ^ {\star} \quad \Longleftrightarrow \quad n - 1 \geq \frac {f _ {1} (n)}{w _ {\beta , k _ {1}} ^ {\star}} \left(1 + \frac {1}{\varepsilon}\right)
$$
is satisfied since  $w_{\beta, k_1}^* \geq w_-$  and  $n > C_1(w_-)$ . Second, using Lemma D.3 with  $N_{t_1, k_1}^{i^*} \geq w_{\beta, k_1}^* (n - 1) - f_1(n)$ , we obtain
$$
\frac {N _ {t _ {1} , k _ {1}} ^ {i ^ {\star}}}{N _ {t _ {1} , i ^ {\star}} ^ {i ^ {\star}}} + 1 \leq \left(\frac {\beta}{1 - \beta} - \frac {1}{2 (1 - \beta) \left(w _ {\beta , k _ {1}} ^ {\star} (n - 1) - f _ {1} (n)\right)}\right) ^ {- 1} + 1 \leq (1 + \varepsilon) / \beta ,
$$
for all  $n > C_2(w_-)$ . The last inequality is obtained by definition of
$$
C _ {2} \left(w _ {-}\right) := \sup  \left\{n \in \mathbb {N} ^ {\star} \mid n - 1 <   \frac {1}{w _ {-}} \left(f _ {1} (n) + \frac {1 - \beta + \varepsilon}{2 \beta \varepsilon}\right) \right\}, \tag {10}
$$
which ensures that, for all  $n > C_2(w_-)$ , the last condition of the equivalence
$$
\begin{array}{l} \left(\frac {\beta}{1 - \beta} - \frac {1}{2 (1 - \beta) \left(w _ {\beta , k _ {1}} ^ {\star} (n - 1) - f _ {1} (n)\right)}\right) ^ {- 1} + 1 \leq (1 + \varepsilon) / \beta \\ \Longleftrightarrow \frac {1}{w _ {\beta , k _ {1}} ^ {\star} (n - 1) - f _ {1} (n)} \leq \frac {2 \beta \varepsilon}{1 - \beta + \varepsilon} \Longleftrightarrow n - 1 \geq \frac {1}{w _ {\beta , k _ {1}} ^ {\star}} \left(f _ {1} (n) + \frac {1 - \beta + \varepsilon}{2 \beta \varepsilon}\right) \\ \end{array}
$$
is satisfied since  $w_{\beta ,k_1}^{\star}\geq w_{-}$  and  $n > C_2(w_{-})$  . By comparison between (8) and (10), we notice that
$$
\max  \left\{C _ {0} (w _ {-}), C _ {2} (w _ {-}) \right\} \leq \max  \left\{C _ {0} (w _ {-}), \left(\frac {1 - \beta + \varepsilon}{2 \beta \varepsilon}\right) ^ {\alpha} \right\}
$$
Putting everything together, we have shown that taking  $D_0 = (1 + \varepsilon)^2 / \beta$ , we have for all  $n > \max \{K, C_0(w_{-}), C_1(w_{-}), C_2(w_{-})\}$ , there exists  $t_1 \in [n^{1/\alpha}, n]$  with  $B_{t_1} = i^\star$  and such that (7) holds. Let  $h_1$  defined in (15). Since  $\varepsilon \leq 1$ , using Lemmas D.10 and D.11 with the above yields
$$
\max \{C _ {0} (w _ {-}), C _ {1} (w _ {-}), C _ {2} (w _ {-}) \} \leq \max \left\{h _ {1} \left(4 \alpha^ {2} (1 + s) H (\mu)\right), \left(\frac {2}{\varepsilon w _ {-}} + 1\right) ^ {1 / (\alpha - 1)}, \frac {1}{\beta \varepsilon} \right\} ^ {\alpha}.
$$
Using  $w_{-} = \min_{i\neq i^{\star}}w_{\beta ,i}^{\star}$  yields the first part of Theorem D.4, i.e. the special case of  $w_0 = 0$
Refined non-asymptotic upper bound When considering large  $K$  or instances with unbalanced  $\beta$ -optimal allocation,  $\min_{i \neq i^{\star}} w_{\beta, i}^{\star}$  can become arbitrarily small. Therefore, the dependency in the inverse of  $\min_{i \neq i^{\star}} w_{\beta, i}^{\star}$  is undesired, and we would like to clip it with a value of our choosing which is away from zero.
Let  $w_0 \in (0, 1 / (K - 1)]$  be an allocation threshold and  $d_{\mu}(w_0) \coloneqq |\{i \in [K] \setminus \{i^{\star}\} \mid w_{\beta ,i}^{\star} < (1 - \beta)w_0\}|$  be the number of arms having a  $\beta$ -optimal allocation strictly smaller than  $(1 - \beta)w_0$ . As discussed above, for instances  $\mu$  such that  $w_{\beta ,k_1}^{\star}$  (defined above) is small, the equation (5) can be satisfied at the very beginning for a small empirical allocation  $N_{n,k_1}^{i^{\star}}$ . To provide a more meaningful result, one needs to have a sub-optimal arm  $k_1$  such that either:
- Case 1:  $w_{\beta, k_1}^{\star}$  is not too small, i.e.  $w_{\beta, k_1}^{\star} \geq (1 - \beta) w_0$ .
- Case 2:  $w_{\beta, k_1}^*$  is too small but  $N_{n, k_1}^{i^*}$  is large enough, i.e.  $w_{\beta, k_1}^* < (1 - \beta)w_0$  and  $N_{n, k_1}^{i^*} \geq w_0(L_{n, i^*} - N_{n, i^*}^{i^*})$ .
In case 1, we can conduct the same manipulations as above simply by using  $w_{-} = \max \{(1 - \beta)w_{0},\min_{i\neq i^{\star}}w_{\beta ,i}^{\star}\}$  instead of  $w_{-} = \min_{i\neq i^{\star}}w_{\beta ,i}^{\star}$ , since it is a lower bound for  $w_{\beta ,k_1}^{\star}$ .
In case 2, the above proof can also be applied by conducting the same algebraic manipulations with  $(1 - \beta)w_{0}$  instead of  $w_{\beta ,k_1}^{\star}$ , and using  $w_{-} = (1 - \beta)w_{0} = \max \{(1 - \beta)w_{0},\min_{i\neq i^{\star}}w_{\beta ,i}^{\star}\}$ . To slightly detail the argument, we can show similarly that  $N_{t_1,k_1}^{i^{\star}}\geq (1 - \beta)w_0(n - 1) - f_1(n)$ , where  $f_{1}$  as in (14) since we have  $(1 - \beta)w_{0} < 1 - \beta$ . Then, for all  $n > C_1((1 - \beta)w_0)$ ,
$$
1 / \beta + \frac {n - 1}{N _ {t _ {1} , k _ {1}} ^ {i ^ {\star}}} \leq (1 + \varepsilon) \left(1 / \beta + \frac {1}{(1 - \beta) w _ {0}}\right) \leq (1 + \varepsilon) \left(1 / \beta + 1 / w _ {\beta , k _ {1}} ^ {\star}\right).
$$
The problematic situation happens when we are neither in case 1 nor in case 2:
- Case 3: both  $w_{\beta, k_1}^*$  and  $N_{n, k_1}^{i^*}$  are too small, i.e.  $w_{\beta, k_1}^* < (1 - \beta)w_0$  and  $N_{n, k_1}^{i^*} < w_0(L_{n, i^*} - N_{n, i^*}^{i^*})$ .
In case 3 it is not possible to conclude the proof with the arm  $k_{1}$  without paying the price of an inverse of  $\min_{i\neq i^{\star}}w_{\beta ,i}^{\star}$ . To overcome this price, we need to find another arm  $k$  such that either case 1 or case 2 happens. Since  $N_{n,k_1}^{i^{\star}}$  and  $w_{\beta ,k_1}^{\star}$  are small, we will ignore arm  $k_{1}$  and use the pigeonhole principle on all arm  $i\in [K]\setminus \{i^{\star},k_{1}\}$ . As in (5), we obtain that there exists  $k_{2}$  such that
$$
N _ {n, k _ {2}} ^ {i ^ {\star}} \geq \frac {w _ {\beta , k _ {2}} ^ {\star}}{1 - \beta - w _ {\beta , k _ {1}} ^ {\star}} (L _ {n, i ^ {\star}} - N _ {n, i ^ {\star}} ^ {i ^ {\star}} - N _ {n, k _ {1}} ^ {i ^ {\star}}) \geq \frac {(1 - w _ {0}) w _ {\beta , k _ {2}} ^ {\star}}{1 - \beta} (L _ {n, i ^ {\star}} - N _ {n, i ^ {\star}} ^ {i ^ {\star}}),
$$
where the last inequality is obtained by using  $w_{\beta, k_1}^{\star} > 0$  and  $N_{n, k_1}^{i^{\star}} < w_0(L_{n, i^{\star}} - N_{n, i^{\star}}^{i^{\star}})$ . Based on  $k_2$  the same dichotomy happens: either we can conclude the proof when we are in case 1 or 2 or we cannot since we are in case 3. If case 3 occurs also for  $k_2$ , i.e.  $w_{\beta, k_2}^{\star} < (1 - \beta)w_0$  and  $N_{n, k_2}^{i^{\star}} < w_0(L_{n, i^{\star}} - N_{n, i^{\star}}^{i^{\star}} - N_{n, k_1}^{i^{\star}})$ , we should also ignore it since it is non informative.
The main idea is then to peel off arms that are not informative, till we find an informative one. By induction, we construct a sequence  $(k_{a})_{a\in [d]}$  of such arms, where  $k_{d}$  is the first arm for which either case 1 or case 2 holds. This means that for all  $a\in [d - 1]$ , we have  $w_{\beta ,k_a}^\star < (1 - \beta)w_0$  and
$$
N _ {n, k _ {a}} ^ {i ^ {\star}} <   w _ {0} \left(L _ {n, i ^ {\star}} - N _ {n, i ^ {\star}} ^ {i ^ {\star}} - \sum_ {b \in [ a - 1 ]} N _ {n, k _ {b}} ^ {i ^ {\star}}\right).
$$
The construction, which relies on the pigeonhole principle for  $i \in [K] \setminus \left( \{i^{\star}\} \cup \{k_{a}\}_{a \in [d - 1]} \right)$ , yields that  $k_{d}$  satisfies
$$
N _ {n, k _ {d}} ^ {i ^ {\star}} \geq \frac {w _ {\beta , k _ {d}} ^ {\star}}{1 - \beta - \sum_ {a \in [ d - 1 ]} w _ {\beta , k _ {a}} ^ {\star}} \left(L _ {n, i ^ {\star}} - N _ {n, i ^ {\star}} ^ {i ^ {\star}} - \sum_ {a \in [ d - 1 ]} N _ {n, k _ {a}} ^ {i ^ {\star}}\right) \geq \frac {(1 - w _ {0}) ^ {d - 1} w _ {\beta , k _ {d}} ^ {\star}}{1 - \beta} \left(L _ {n, i ^ {\star}} - N _ {n, i ^ {\star}} ^ {i ^ {\star}}\right),
$$
where the last inequality is obtained since  $\sum_{a\in [d - 1]}w_{\beta ,k_a}^\star >0$  and by a simple recurrence on the arms  $\{k_a\}_{a\in [d - 1]}$ . Since the arm  $k_{d}$  satisfies case 1 or case 2, we can conclude similarly as above. Let  $t_d\coloneqq \sup \left\{t < n\mid (B_t,C_t) = (i^\star ,k_d)\right\}$ .
When  $w_{\beta, k_d}^\star \geq (1 - \beta)w_0$ , the above proof can also be applied by conducting the same algebraic manipulations with  $(1 - w_0)^{d-1}w_{\beta, k_d}^\star$ , and using  $w_- = (1 - w_0)^{d-1}\max \{(1 - \beta)w_0, \min_{i \neq i^\star} w_{\beta, i}^\star\}$ . In more details, we can show that  $N_{t_d, k_d}^{i^\star} \geq (1 - w_0)^{d-1}w_{\beta, k_d}^\star (n-1) - f_1(n)$ , where  $f_1$  as in (14) since we have  $(1 - w_0)^{d-1}w_{\beta, k_d}^\star < 1 - \beta$ . Then, for all  $n > C_1(w_-)$ ,
$$
1 / \beta + \frac {n - 1}{N _ {t _ {d} , k _ {d}} ^ {i ^ {\star}}} \leq (1 + \varepsilon) \left(1 / \beta + \frac {1}{(1 - w _ {0}) ^ {d - 1} w _ {\beta , k _ {d}} ^ {\star}}\right) \leq \frac {1 + \varepsilon}{(1 - w _ {0}) ^ {d - 1}} \left(1 / \beta + 1 / w _ {\beta , k _ {d}} ^ {\star}\right).
$$
This allows to conclude the result with  $D_0 = \frac{(1 + \varepsilon)^2}{\beta(1 - w_0)^{d-1}}$ , hence paying a multiplicative factor of  $1/(1 - w_0)^{d-1}$ .
When  $w_{\beta, k_d}^{\star} < (1 - \beta)w_0$  and  $N_{n, k_a}^{i^{\star}} \geq w_0(L_{n, i^{\star}} - N_{n, i^{\star}}^{i^{\star}} - \sum_{a \in [d-1]}N_{n, k_a}^{i^{\star}})$ , we conclude similarly by manipulating  $(1 - w_0)^{d-1}(1 - \beta)w_0$ , and using  $w_- = (1 - w_0)^{d-1}(1 - \beta)w_0 = (1 - w_0)^{d-1}\max \{(1 - \beta)w_0, \min_{i \neq i^{\star}}w_{\beta, i}^{\star}\}$ . First, we have  $N_{t_d, k_d}^{i^{\star}} \geq (1 - w_0)^{d-1}w_0(n-1) - f_1(n)$ , where  $f_1$  as in (14) since we have  $(1 - \beta)(1 - w_0)^{d-1}w_0 < 1 - \beta$ . Then, for all  $n > C_1(w_-)$ ,
$$
1 / \beta + \frac {n - 1}{N _ {t _ {d} , k _ {d}} ^ {i ^ {\star}}} \leq (1 + \varepsilon) \left(1 / \beta + \frac {1}{(1 - w _ {0}) ^ {d - 1} (1 - \beta) w _ {0}}\right) \leq \frac {1 + \varepsilon}{(1 - w _ {0}) ^ {d - 1}} \left(1 / \beta + 1 / w _ {\beta , k _ {d}} ^ {\star}\right).
$$
This allows to conclude the result with  $D_0 = \frac{(1 + \varepsilon)^2}{\beta(1 - w_0)^{d - 1}}$ .
To remove the dependency in the random variable  $d$ , we consider the worst case scenario where  $\{k_{a}\}_{a\in [d - 1]} = \{i\in [K]\setminus \{i^{\star}\} \mid w_{\beta ,i}^{\star} < (1 - \beta)w_{0}\}$ , i.e.  $d - 1\leq d_{\mu}(w_0)$ . In words, it means that we had to enumerate over all arms with small allocation, such that case 2 didn't hold, before finding an arm with large allocation, i.e. satisfying case 1.
In all the cases considered above, the parameters always satisfied  $w_{-} \geq (1 - w_0)^{d_\mu(w_0)} \max \{(1 - \beta)w_0, \min_{i \neq i^\star} w_{\beta,i}^\star\}$  and  $D_0 \leq \frac{(1 + \varepsilon)^2}{\beta(1 - w_0)^{d_\mu(w_0)}}$ . This yields the second part of Theorem D.4, i.e. for  $w_0 \in (0,1/(K-1)]$ .
D.3 Tighter UCB leader
While Theorem 2.3 holds for the TTUCB sampling rule using  $g_{m}$  and  $g_{u}$ , Theorem D.4 is formulated solely for  $g_{u}$ . Experiments highlight that using  $g_{u}$  leads to worse performances than when using  $g_{m}$ . This is not surprising since  $g_{m}$  is smaller than  $g_{u}$ .
It is direct to see that the proof of Theorem D.4 also holds for  $g_{m}$  up to additional technicalities which we detail below. As the obtained non-asymptotic upper bound is less explicit, we chose not to include it in Theorem 2.3.
Let  $\alpha > 1$  and  $s > 1$ . Let  $(\tilde{\mathcal{E}}_n)_n$  be the sequence of concentration events defined as  $\tilde{\mathcal{E}}_n \coloneqq \mathcal{E}_{3,n} \cap \mathcal{E}_{2,n}$  for all  $n > K$  where  $\mathcal{E}_{3,n}$  is defined in (18) as
$$
\mathcal {E} _ {3, n} := \left\{\forall k \in [ K ], \forall t \in [ n ^ {1 / \alpha}, n ], | \mu_ {t, k} - \mu_ {k} | <   \sqrt {\frac {g _ {m} (t)}{N _ {t , k}}} \right\},
$$
$$
g _ {m} (n) := \bar {W} _ {- 1} \left(2 s \alpha \log (n) + 2 \log (2 + \alpha \log n) + 2\right),
$$
where  $\overline{W}_{-1}(x) = -W_{-1}(-e^{-x})$  for all  $x\geq 1$ , with  $W_{-1}$  is the negative branch of the Lambert  $W$  function.
Let  $n > K$  such that  $\tilde{\mathcal{E}}_n$  holds true and the algorithm has not stop yet, i.e.  $\tilde{\mathcal{E}}_n \cap \{n < \tau_\delta\}$ . Using the second part of Lemma D.1, the vast majority of the proof is unchanged. Modifying the definition of  $f_1$  in (14) of Lemma D.9 to account for  $g_m$ , we define  $f_2(n) \coloneqq 2H(\mu)g_m(n) / \beta + K / \beta + 2$  for all  $n > K$ .
In light of the proofs of the technical Lemmas D.10 and D.11, we can define
$$
\tilde {C} _ {\mu} = \sup  \left\{x \in \mathbb {N} ^ {\star} \mid x <   f _ {2} \left(x ^ {\alpha}\right) \right\},
$$
and obtain directly Corollary D.5 with the same proof as in Appendix D.2.
Corollary D.5. Let  $(\delta, \beta) \in (0,1)^2$ ,  $\varepsilon \in (0,1]$ ,  $s > 1$ ,  $\alpha > 1$ ,  $w_0 \in [0,1/(K - 1)]$ . Combining the stopping rule (1) with threshold (2) and the TTUCB sampling rule using  $g_u$  in (3) yields a  $\delta$ -correct algorithm such that, for all  $\mu \in \mathbb{R}^K$  with  $|i^\star(\mu)| = 1$ ,
$$
\mathbb {E} _ {\mu} \left[ \tau_ {\delta} \right] \leq \max  \left\{T _ {0} (\delta), \tilde {C} _ {\mu} ^ {\alpha}, C _ {0} ^ {\frac {\alpha}{\alpha - 1}}, C _ {1} ^ {\alpha} \right\} + C _ {2},
$$
where  $T_0(\delta), C_0, C_1$  and  $C_2$  are defined in Theorem 2.3, and
$$
\tilde {C} _ {\mu} := \sup  \left\{x \in \mathbb {N} ^ {\star} \mid x <   2 H (\mu) g _ {m} \left(x ^ {\alpha}\right) / \beta + K / \beta + 2 \right\}. \tag {11}
$$