SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. Let  $p \geq 1$ . Let  $n \in \mathbb{N}$  such that  $S_{n}^{p} \neq \emptyset$ , where  $S_{n}^{p}$  as in Equation (27). Let  $(k_{n,a})_{a \in [K]}$  be the phases indices for all arms. Since  $N_{n,a} \geq 2^{p-1}$  for all  $a \in S_{n}^{p}$ , we have  $k_{n,a} \geq p$  and  $\tilde{N}_{k_{n,a},a} \geq 2^{p-2}$  by using Lemma 3. Likewise,  $N_{n,a} < 2^{p-1}$  for all  $a \notin S_{n}^{p}$ , we have  $k_{n,a} < p$  and  $\tilde{N}_{k_{n,a},a} < 2^{p-2}$ . Let  $\overline{\Delta}_{\max} = \min_{a \neq b} |\mu_a - \mu_b|$ , which satisfies  $\overline{\Delta}_{\max} > 0$  by assumption on the instance considered. Using Lemma 7, for all  $a \in S_{n}^{p}$  and  $b \notin S_{n}^{p}$ , we obtain
$$
\begin{array}{l} \frac {\tilde {\mu} _ {k _ {n , a} , a} - \tilde {\mu} _ {k _ {n , b} , b}}{\sqrt {1 / \tilde {N} _ {k _ {n , a} , a} + 1 / \tilde {N} _ {k _ {n , b} , b}}} \leq \sqrt {\tilde {N} _ {k _ {n , b} , b}} (\tilde {\mu} _ {k _ {n, a}, a} - \tilde {\mu} _ {k _ {n, b}, b}) \\ \leq \sqrt {\tilde {N} _ {k _ {n , b} , b}} \left(\mu_ {a} - \mu_ {b}\right) + 2 W _ {\mu} \sqrt {\log \left(e + \tilde {N} _ {k _ {n , b} , b}\right)} + 2 W _ {\epsilon} \frac {\log \left(e + k _ {n , b}\right)}{\sqrt {\tilde {N} _ {k _ {n , b} , b}}} \\ \leq 2 ^ {(p - 2) / 2} \overline {{\Delta}} _ {\max } + 2 W _ {\mu} \sqrt {\log (e + 2 ^ {p - 2})} + 2 W _ {\epsilon} \log (e + p) \\ \end{array}
$$
where we used that  $\tilde{N}_{k_{n,b},b} \geq 1$ ,  $k_{n,b} < p$ ,  $\tilde{N}_{k_{n,b},b} < 2^{p-2} \leq \tilde{N}_{k_{n,a},a}$  and  $x \to \log(e + x)/x$  is decreasing. Taking  $D_{\mu} = \overline{\Delta}_{\max}/2$  yields the result.
Lemma 13 shows that the challenger is mildly undersampled if the leader is not mildly undersampled.
Lemma 13. Let  $V_{n}^{p}$  be as in Equation (28). There exists  $p_2$  with  $\mathbb{E}_{\nu}[\exp(\alpha p_2)] < +\infty$  for all  $\alpha > 0$  such that if  $p \geq p_2$ , for all  $n$  such that  $U_{n}^{p} \neq \emptyset$ ,  $B_{n} \notin V_{n}^{p}$  implies  $C_{n} \in V_{n}^{p}$ .
Proof. Let  $p_2$  to be specified later. Let  $p \geq p_2$ . Let  $n \in \mathbb{N}$  such that  $U_n^p \neq \emptyset$  and  $V_n^p \neq [K]$ , where  $U_n^p \subseteq V_n^p$  are defined in Equation (28). In the following, we suppose that  $B_n \notin V_n^p$ .
Let  $(k_{n,a})_{a\in [K]}$  be the phases indices for all arms. Let  $p_0$  as in Lemma 10. Let  $b_{n}^{\star} = \arg \max_{b\notin V_{n}^{p}}\mu_{b}$ . Then, for all  $p\geq 4p_0 / 3 - 1 / 3$  and all  $n$  such that  $B_{n}\notin V_{n}^{p}$ , Lemma 10 yields that  $B_{n} = b_{n}^{\star} = \arg \max_{a\notin V_{n}^{p}}\tilde{\mu}_{k_{n,a},a}$ .
Let  $p_1$  and  $C_\mu$  as in Lemma 11, and  $D_\mu$  as in Lemma 12. Then, for all  $p \geq 4 \max \{p_0, p_1\} / 3 - 1/3$  and all  $n$  such that  $B_n \notin V_n^p$ , we have  $B_n = b_n^\star$  and
$$
\forall b \notin V _ {n} ^ {p}, \quad \frac {\tilde {\mu} _ {k _ {n , b _ {n} ^ {\star}} , b _ {n} ^ {\star}} - \tilde {\mu} _ {k _ {n , b} , b}}{\sqrt {1 / \tilde {N} _ {k _ {n , b _ {n} ^ {\star}} , b _ {n} ^ {\star}} + 1 / \tilde {N} _ {k _ {n , b} , b}}} \geq 2 ^ {(3 p + 1) / 8} C _ {\mu},
$$
$$
\begin{array}{l} \forall b \in U _ {n} ^ {p}, \frac {\tilde {\mu} _ {k _ {n , b _ {n} ^ {\star}} , b _ {n} ^ {\star}} - \tilde {\mu} _ {k _ {n , b} , b}}{\sqrt {1 / \tilde {N} _ {k _ {n , b _ {n} ^ {\star}} , b _ {n} ^ {\star}} + 1 / \tilde {N} _ {k _ {n , b} , b}}} \leq 2 ^ {(p + 1) / 4} D _ {\mu} + 2 W _ {\mu} \sqrt {\log (e + 2 ^ {(p + 1) / 2 - 2})} \\ + 2 W _ {\epsilon} \log (e + (p + 1) / 2), \\ \end{array}
$$
where we used Lemmas 11 and 12. Let  $p_3 = 16\lceil \log_2(2D_\mu /C_\mu)\rceil +1$ , then we have  $2^{(p - 1) / 16} > \frac{D_{\mu}}{C_{\mu}}$  for all  $p\geq p_3$ . Let  $p_4 = \frac{16}{9}\lceil \log_2X_4\rceil +25$  and  $p_5 = \frac{32}{9}\lceil \log_2X_5\rceil +7$  where
$$
X _ {4} = \sup  \left\{x > 1 \mid x \leq \frac {W _ {\mu} ^ {2}}{C _ {\mu} ^ {2}} \log \left(e + x ^ {8 / 9} 2 ^ {2 5 / 1 8 - 3 / 4}\right) \right\},
$$
$$
X _ {5} = \sup \left\{x > 1 \mid x \leq \frac {2 W _ {\epsilon}}{C _ {\mu}} \log (e + 4 + 3 2 \log_ {2} (x) / 1 8) \right\}.
$$
As in the proof of Lemma 10, using Lemma 7 yields that  $\mathbb{E}_{\nu}[\exp (\alpha p_4)] < +\infty$  and  $\mathbb{E}_{\nu}[\exp (\alpha p_5)] < +\infty$  for all  $\alpha \in \mathbb{R}_+$ . Let  $p_2 = \max \{p_3,p_4,p_5,4\max \{p_0,p_1\} /3 - 1 / 3\} +1$ . Then, we have shown that for all  $p\geq p_2$ , for all  $n$  such that  $B_{n}\notin V_{n}^{p}$ , we have  $B_{n} = b_{n}^{\star}$  and
$$
\min _ {b \notin V _ {n} ^ {p}} \frac {\tilde {\mu} _ {k _ {n , b _ {n} ^ {\star}} , b _ {n} ^ {\star}} - \tilde {\mu} _ {k _ {n , b} , b}}{\sqrt {1 / \tilde {N} _ {k _ {n , b _ {n} ^ {\star}} , b _ {n} ^ {\star}} + 1 / \tilde {N} _ {k _ {n , b} , b}}} > \max _ {b \in U _ {n} ^ {p}} \frac {\tilde {\mu} _ {k _ {n , b _ {n} ^ {\star}} , b _ {n} ^ {\star}} - \tilde {\mu} _ {k _ {n , b} , b}}{\sqrt {1 / \tilde {N} _ {k _ {n , b _ {n} ^ {\star}} , b _ {n} ^ {\star}} + \left. 1 / \tilde {N} _ {k _ {n , b} , b} \right.}},
$$
Therefore, by definition of the TC challenger  $C_n = \arg \min_{b \neq b_n^\star} \frac{\tilde{\mu}_{k_{n,b_n^\star},b_n^\star} - \tilde{\mu}_{k_{n,b},b}}{\sqrt{1 / \tilde{N}_{k_{n,b_n^\star},b_n^\star} + 1 / \tilde{N}_{k_{n,b},b}}}$ , we obtain that  $C_n \in V_n^p$ . Otherwise, there would be a contradiction given that we assumed that  $U_n^p \neq \emptyset$ . Given all the condition exhibited above, it is direct to see that  $\mathbb{E}_{\nu}[\exp(\alpha p_2)] < +\infty$  for all  $\alpha > 0$ . This concludes the proof.
Lemma 14 shows that all the arms are sufficient explored for large enough  $n$ .
Lemma 14. There exists  $N_0$  with  $\mathbb{E}_{\nu}[N_0] < +\infty$  such that for all  $n \geq N_0$  and all  $a \in [K]$ ,
$$
N _ {n, a} \geq \sqrt {n / K} \quad a n d \quad k _ {n, a} \geq \frac {\log (n / K)}{2 \log 2} + 1.
$$
Proof. Let  $p_0$  and  $p_2$  as in Lemmas 10 and 13. Combining Lemmas 10 and 13 yields that, for all  $p \geq p_3 = \max \{p_2, 4p_0/3 - 1/3\}$  and all  $n$  such that  $U_n^p \neq \emptyset$ , we have  $B_n \in V_n^p$  or  $C_n \in V_n^p$ . We have  $\mathbb{E}_{\nu}[2^{p_2}] < +\infty$ . We have  $2^{p-1} \geq K2^{3(p-1)/4}$  for all  $p \geq p_4 = 4\lceil \log_2 K \rceil + 1$ . Let  $p \geq \max \{p_3, p_4\}$ .
Suppose towards contradiction that  $U_{K2^{p-1}}^p$  is not empty. Then, for any  $1 \leq t \leq K2^{p-1}$ ,  $U_t^p$  and  $V_t^p$  are non-empty as well. Using the pigeonhole principle, there exists some  $a \in [K]$  such that  $N_{2^{p-1},a} \geq 2^{3(p-1)/4}$ . Thus, we have  $\left|V_{2^{p-1}}^p\right| \leq K - 1$ . Our goal is to show that  $|V_{2^p}| \leq K - 2$ . A sufficient condition is that one arm in  $V_{2^{p-1}}^p$  is pulled at least  $2^{3(p-1)/4}$  times between  $2^{p-1}$  and  $2^p - 1$ .
Case 1. Suppose there exists  $a \in V_{2^{p-1}}^p$  such that  $L_{2^p, a} - L_{2^{p-1}, a} \geq \frac{2^{3(p-1)/4}}{\beta} + 3 / (2\beta)$ . Using Lemma 6, we obtain
$$
N _ {2 ^ {p}, a} ^ {a} - N _ {2 ^ {p - 1}, a} ^ {a} \geq \beta \left(L _ {2 ^ {p}, a} - L _ {2 ^ {p - 1}, a}\right) - 3 / 2 \geq 2 ^ {3 (p - 1) / 4},
$$
hence  $a$  is sampled  $2^{3(p - 1) / 4}$  times between  $2^{p - 1}$  and  $2^{p} - 1$ .
Case 2. Suppose that for all  $a \in V_{2p-1}^p$ , we have  $L_{2p,a} - L_{2p-1,a} < 2^{3(p-1)/4} / \beta + 3 / (2\beta)$ . Then,
$$
\sum_ {a \notin V _ {2 ^ {p} - 1} ^ {p}} \left(L _ {2 ^ {p}, a} - L _ {2 ^ {p - 1}, a}\right) \geq 2 ^ {p - 1} - K \left(2 ^ {3 (p - 1) / 4} / \beta + 3 / (2 \beta)\right)
$$
Using Lemma 6, we obtain
$$
\left| \sum_ {a \notin V _ {2 p - 1} ^ {p}} \left(N _ {2 ^ {p}, a} ^ {a} - N _ {2 ^ {p - 1}, a} ^ {a}\right) - \beta \sum_ {a \notin V _ {2 p - 1} ^ {p}} \left(L _ {2 ^ {p}, a} - L _ {2 ^ {p - 1}, a}\right) \right| \leq 3 (K - 1) / 2.
$$
$$
\begin{array}{l} \sum_ {a \notin V _ {2 p - 1} ^ {p}} (L _ {2 ^ {p}, a} - L _ {2 ^ {p - 1}, a}) - \sum_ {a \notin V _ {2 p - 1} ^ {p}} (N _ {2 ^ {p}, a} ^ {a} - N _ {2 ^ {p - 1}, a} ^ {a}) \\ \geq (1 - \beta) \sum_ {a \notin V _ {2 p - 1} ^ {p}} \left(L _ {2 ^ {p}, a} - L _ {2 ^ {p - 1}, a}\right) - 3 (K - 1) / 2 \\ \geq (1 - \beta) \left(2 ^ {p - 1} - K \left(2 ^ {3 (p - 1) / 4} / \beta + 3 / (2 \beta)\right)\right) - 3 (K - 1) / 2 \geq K 2 ^ {3 (p - 1) / 4}, \\ \end{array}
$$
where the last inequality is obtained for  $p \geq p_5$  with
$$
p _ {5} = \sup  \left\{p \in \mathbb {N} \mid (1 - \beta) \left(2 ^ {p - 1} - K \left(2 ^ {3 (p - 1) / 4} / \beta + 3 / (2 \beta)\right)\right) - 3 (K - 1) / 2 <   K 2 ^ {3 (p - 1) / 4} \right\}.
$$
The LHS summation is exactly the number of times where an arm  $a \notin V_{2^{p - 1}}^p$  was leader but wasn't sampled, hence
$$
\sum_ {t = 2 ^ {p} - 1} ^ {2 ^ {p} - 1} \mathbb {1} \left(B _ {t} \notin V _ {2 ^ {p} - 1} ^ {p}, I _ {t} = C _ {t}\right) \geq K 2 ^ {3 (p - 1) / 4}
$$
For any  $2^{p - 1} \leq t \leq 2^p - 1$ ,  $U_t^p$  is non-empty, hence we have  $B_t \notin V_{2^{p - 1}}^p$  (hence  $B_t \notin V_t^p$ ) implies  $C_t \in V_t^p \subseteq V_{2^{p - 1}}^p$ . Therefore, we have shown that
$$
\sum_ {t = 2 ^ {p - 1}} ^ {2 ^ {p} - 1} \mathbb {1} \left(I _ {t} \in V _ {2 ^ {p - 1}} ^ {p}\right) \geq \sum_ {t = 2 ^ {p - 1}} ^ {2 ^ {p} - 1} \mathbb {1} \left(B _ {t} \notin V _ {2 ^ {p - 1}} ^ {p}, I _ {t} = C _ {t}\right) \geq K 2 ^ {3 (p - 1) / 4}.
$$
Therefore, there is at least one arm in  $V_{2^{p-1}}^p$  that is sampled  $2^{3(p-1)/4}$  times between  $2^{p-1}$  and  $2^p - 1$ . In summary, we have shown  $|V_{2^p}| \leq K - 2$  for all  $p \geq p_6 = \max\{p_3, p_4, p_5\}$ . By induction, for any  $1 \leq k \leq K$ , we have  $|V_{k2^{p-1}}^p| \leq K - k$ , and finally  $U_{K2^{p-1}}^p = \emptyset$  for all  $p \geq p_6$ . Defining  $N_0 = K2^{p_6-1}$ , we have  $\mathbb{E}_{\nu}[N_0] < +\infty$  by using Lemmas 10 and 13 for  $p_3 = \max\{p_2, 4p_0/3 - 1/3\}$  and  $p_4$  and  $p_5$  are deterministic. For all  $n \geq N_0$ , we let  $2^{p-1} = \frac{n}{K}$ . Then, by applying the above, we have  $U_{K2^{p-1}}^p = U_n^{\log_2(n/K) + 1}$  is empty, which shows that  $N_{n,a} \geq \sqrt{n/K}$  for all  $a \in [K]$ . Using Lemma 3, we obtain that  $k_{n,a} \geq \frac{\log(n/K)}{2\log 2} + 1$  for all  $a \in [K]$ . This concludes the proof.
E.3 Convergence towards  $\beta$ -optimal allocation
The second step of in the generic analysis of Top Two algorithms  $\left[\mathrm{JDB}^{+}22\right]$  is to show that AdaP-TT ensures convergence of its empirical proportions towards the  $\beta$ -optimal allocation. First, we show that the leader coincides with the best arm. Hence, the tracking procedure will ensure that the empirical proportion of time we sample it is exactly  $\beta$ . Second, we show that a sub-optimal arm whose empirical proportion overshoots its  $\beta$ -optimal allocation will not be sampled next as challenger. Therefore, this "overshoots implies not sampled" mechanism will ensure the convergence towards the  $\beta$ -optimal allocation. We emphasise that there are multiple ways to select the leader/challenger pair in order to ensure convergence towards the  $\beta$ -optimal allocation. Therefore, while we conduct the proof for AdaP-TT, other choices of leader/challenger pair would yield similar results. Note that our results heavily rely on having obtained sufficient exploration first.
Convergence for the best arm. Lemma 15 exhibits a random phase which ensures that the leader and the candidate answer are equal to the best arm for large enough  $n$ .
Lemma 15. Let  $N_0$  be as in Lemma 14. There exists  $N_1 \geq N_0$  with  $\mathbb{E}_{\nu}[N_1] < +\infty$  such that, for all  $n \geq N_1$ , we have  $\hat{a}_n = B_n = a^\star$ .
Proof. Let  $k \geq 1$  and  $(T_k(a))_{a \in [K]}$  as in Equation (10). Suppose that  $\mathbb{E}_{\nu}[\max_{a \in [K]} T_k(a)] < +\infty$ . Then, Lemma 3 yields that  $N_{T_k(a), a} = 2^{k-1}$  and  $\tilde{N}_{k,a} = 2^{k-2}$ . Using Lemma 7, we obtain that
$$
\tilde {\mu} _ {k, a ^ {\star}} \geq \mu_ {a ^ {\star}} - W _ {\mu} \sqrt {\frac {\log (e + 2 ^ {k - 2})}{2 ^ {k - 2}}} - W _ {\epsilon} \frac {\log (e + k)}{2 ^ {k - 2}},
$$
$$
\forall a \neq a ^ {\star}, \tilde {\mu} _ {k, a} \leq \mu_ {a} + W _ {\mu} \sqrt {\frac {\log (e + 2 ^ {k - 2})}{2 ^ {k - 2}}} + W _ {\epsilon} \frac {\log (e + k)}{2 ^ {k - 2}}.
$$
Let  $p_1 = \lceil \log_2(X_1 - e) \rceil + 2$  and  $p_2 = \lceil \log_2(X_2 - e - 1) \rceil + 2$  with
$$
X _ {1} = \sup  \left\{x > 1 \mid x \leq 6 4 \Delta_ {\min } ^ {- 2} W _ {\mu} ^ {2} \log x + e \right\} \leq h _ {1} \left(6 4 \Delta_ {\min } ^ {- 2} W _ {\mu} ^ {2}, e\right),
$$
$$
X _ {2} = \sup  \left\{x > 1 \mid x \leq 8 \Delta_ {\min } ^ {- 1} W _ {\epsilon} \log x + e + 1 \right\} \leq h _ {1} \left(8 \Delta_ {\min } ^ {- 1} W _ {\epsilon}, e + 1\right),
$$
$$
X _ {2} \geq \sup  \left\{x > 1 \mid x \leq 8 \Delta_ {\min } ^ {- 1} W _ {\epsilon} \log (e + 2 + \log x) \right\},
$$
where we used Lemma 9, and  $h_1$  defined therein. Then, for all  $k \in \mathbb{N}^K$  such that  $\min_{a \in [K]} k_a > p_0 = \max \{p_1, p_2\}$  such that  $\mathbb{E}_{\boldsymbol{\nu}}[\max_{a \in [K]} T_{k_a}(a)] < +\infty$ , we have  $\tilde{\mu}_{k,a^\star} \geq \mu_{a^\star} - \Delta_{\min} / 4$  and  $\tilde{\mu}_{k,a} \leq \mu_a + \Delta_{\min} / 4$  for all  $a \neq a^\star$ , hence  $a^\star = \arg \max_{a \in [K]} \tilde{\mu}_{k,a}$ . We have, for all  $\alpha \in \mathbb{R}_+$ ,
where we used Lemma 7 and  $h_1(x, e) \sim_{x \to +\infty} x \log x$  to obtain that  $\exp(\alpha p_1)$  is at most polynomial in  $W_\mu$ . Likewise, we obtain that  $\mathbb{E}_\nu[\exp(\alpha p_2)] < +\infty$  for all  $\alpha \in \mathbb{R}_+$ . Therefore, we have  $\mathbb{E}_\nu[\exp(\alpha p_0)] < +\infty$  for all  $\alpha \in \mathbb{R}_+$ .
Let us define the UCB indices by  $I_{k,a} = \tilde{\mu}_{k,a} + \sqrt{k / \tilde{N}_{k,a}} + k / (\epsilon \tilde{N}_{k,a})$ . Using the above, we have
$$
I _ {k, a ^ {\star}} \geq \mu_ {a ^ {\star}} - W _ {\mu} \sqrt {\frac {\log (e + 2 ^ {k - 2})}{2 ^ {k - 2}}} - W _ {\epsilon} \frac {\log (e + k)}{2 ^ {k - 2}} + \frac {k}{\epsilon 2 ^ {k - 2}},
$$
$$
\forall a \neq a ^ {\star}, I _ {k, a} \leq \mu_ {a} + W _ {\mu} \sqrt {\frac {\log (e + 2 ^ {k - 2})}{2 ^ {k - 2}}} + W _ {\epsilon} \frac {\log (e + k)}{2 ^ {k - 2}} + \frac {k}{\epsilon 2 ^ {k - 2}}.
$$
Therefore, we have  $a^{\star} = \arg \max_{a\in [K]}I_{k,a}$  for all  $k\in \mathbb{N}^{K}$  such that  $\min_{a\in [K]}k_a > \max \{p_1,p_2\}$  such that  $\mathbb{E}_{\nu}[\max_{a\in [K]}T_{k_a}(a)] < + \infty$ .
Let  $N_0$  as in Lemma 14. Using Lemma 14, we obtain that, for all  $n \geq N_0$  and all  $a \in [K]$ ,  $k_{n,a} \geq \log_2(n / K) / 2 + 1$ . Therefore, we obtain  $\min_{a \in [K]} k_{n,a} > \max \{p_1, p_2\}$  is implied by  $n \geq N_1 = \max \{K4^{\max \{p_1, p_2\}}, N_0\}$ . Using the above, we conclude that  $\mathbb{E}_{\nu}[N_1] < +\infty$  and  $\hat{a}_n = B_n = a^\star$  for all  $n \geq N_1$ .
Lemma 16 shows that the pulling proportion of the best arm converges towards  $\beta$ , provided the phase defined in Lemma 15 is reached in finite time for all arms.
Lemma 16. Let  $\gamma >0$ , and  $N_{1}$  be as in Lemma 15. There exists a deterministic constant  $C_0\geq 1$  such that, for all  $n\geq C_0N_1$
$$
\left| \frac {N _ {n , a ^ {\star}}}{n - 1} - \beta \right| \leq \gamma .
$$
Proof. Let  $\gamma > 0$ . Let  $N_{1}$  as in Lemma 15. Let  $M \geq N_{1}$ . Using Lemma 15, we obtain  $B_{n} = a^{\star}$  for all  $n \geq M$ . Therefore, we obtain  $L_{n,a^{\star}} \geq n - M$  and  $\sum_{a \neq a^{\star}} N_{n,a^{\star}}^{a} \leq M$  for all  $n \geq M$ . Using Lemma 6 yields that
$$
\begin{array}{l} \left| \frac {N _ {n , a ^ {\star}}}{n - 1} - \beta \right| \leq \frac {\left| N _ {n , a ^ {\star}} ^ {a ^ {\star}} - \beta L _ {n , a ^ {\star}} \right|}{n - 1} + \beta \left| \frac {L _ {n , a ^ {\star}}}{n - 1} - 1 \right| + \frac {1}{n - 1} \sum_ {a \neq a ^ {\star}} N _ {n, a ^ {\star}} ^ {a} \\ \leq \frac {1}{2 (n - 1)} + \beta \frac {2 (M - 1)}{n - 1} \leq \gamma , \\ \end{array}
$$
where the last inequality is obtained by taking  $n \geq \max \{M, (1/2 + 2\beta(M - 1)) / \gamma + 1\}$ .
Convergence for the sub-optimal arms Lemma 17 exhibits a random phase which ensures that if a sub-optimal arm overshoots its  $\beta$ -optimal allocation then it cannot be selected as challenger for large enough  $n$ .
Lemma 17. Let  $\gamma >0$ . Let  $N_{1}$  and  $C_0$  be as in Lemma 15 and 16. There exists  $N_{2}\geq C_{0}N_{1}$  with  $\mathbb{E}_{\nu}[N_2] < + \infty$  such that, for all  $n\geq N_2$
$$
\exists a \neq a ^ {\star}, \quad \frac {N _ {n , a}}{n - 1} \geq \omega_ {\beta , a} ^ {\star} + \gamma \quad \Longrightarrow \quad C _ {n} \neq a.
$$
Proof. Let  $\gamma > 0$  and  $\tilde{\gamma} > 0$ . Let  $N_{1}$  as in Lemma 15 and  $C_0$  as in Lemma 16 for  $\tilde{\gamma}$ . Let  $n \geq C_0 N_1$ .
Let  $a \neq a^{\star}$  such that  $\frac{N_{n,a}}{n-1} \geq \omega_{\beta,a}^{\star} + \gamma$ . Suppose towards contradiction that  $\frac{N_{n,b}}{n-1} > \omega_{\beta,a}^{\star}$  for all  $b \notin \{a^{\star}, a\}$ . Then, for all  $n \geq C_0 N_1$ , we have
$$
1 - \beta + \tilde {\gamma} \geq 1 - \frac {N _ {n , a ^ {\star}}}{n - 1} = \sum_ {b \neq a ^ {\star}} \frac {N _ {n , b}}{n - 1} > \gamma + \sum_ {b \neq a ^ {\star}} \omega_ {\beta , b} ^ {\star} = 1 - \beta + \gamma ,
$$
which yields a contradiction for  $\tilde{\gamma} \leq \gamma$ . Therefore, for all  $n \geq C_0 N_1$ , we have
$$
\exists a \neq a ^ {\star}, \quad \frac {N _ {n , a}}{n - 1} \geq \omega_ {\beta , a} ^ {\star} + \gamma \quad \Longrightarrow \quad \exists b \notin \left\{a ^ {\star}, a \right\}, \quad \frac {N _ {n , b}}{n - 1} \leq \omega_ {\beta , b} ^ {\star}.
$$
$$
\sqrt {\frac {1 + N _ {n , a ^ {\star}} / N _ {n , b}}{1 + N _ {n , a ^ {\star}} / N _ {n , a}}} \geq \sqrt {\frac {1 + (\beta - \tilde {\gamma}) / \omega_ {\beta , b} ^ {\star}}{1 + (\beta + \tilde {\gamma}) / (\omega_ {\beta , a} ^ {\star} + \gamma)}}.
$$
In the following, we use Lemma 7 and similar manipulations as in the proof of Lemma 15. Therefore, we obtain that, for all  $c \neq a^{\star}$ ,
$$
\begin{array}{l} \left| \tilde {\mu} _ {k _ {n, a ^ {\star}}, a ^ {\star}} - \tilde {\mu} _ {k _ {n, c}, c} - \Delta_ {c} \right| \leq W _ {\mu} \left(\sqrt {\frac {\log (e + 2 ^ {k _ {n , a ^ {\star}} - 2})}{2 ^ {k _ {n , a ^ {\star}} - 2}}} + \sqrt {\frac {\log (e + 2 ^ {k _ {n , c} - 2})}{2 ^ {k _ {n , c} - 2}}}\right) \\ + W _ {\epsilon} \left(\frac {\log (e + k _ {n , a ^ {\star}})}{2 ^ {k _ {n , a ^ {\star}} - 2}} + \frac {\log (e + k _ {n , c})}{2 ^ {k _ {n , c} - 2}}\right). \\ \end{array}
$$
Let  $p_3 = \lceil \log_2(X_1 - e) \rceil + 2$  and  $p_2 = \lceil \log_2(X_2 - e - 1) \rceil + 2$  with
$$
X _ {3} = \sup \left\{x > 1 \mid x \leq 1 6 \eta^ {- 2} W _ {\mu} ^ {2} \log x + e \right\} \leq h _ {1} (1 6 \eta^ {- 2} W _ {\mu} ^ {2}, e),
$$
$$
X _ {2} = \sup  \left\{x > 1 \mid x \leq 4 \eta^ {- 1} W _ {\epsilon} \log x + e + 1 \right\} \leq h _ {1} \left(4 \eta^ {- 1} W _ {\epsilon}, e + 1\right),
$$
$$
X _ {2} \geq \sup  \left\{x > 1 \mid x \leq 4 \eta^ {- 1} W _ {\epsilon} \log (e + 2 + \log x) \right\},
$$
where we used Lemma 9, and  $h_1$  defined therein. We have, for all  $\alpha \in \mathbb{R}_+$
$$
\exp (\alpha p _ {3}) \leq e ^ {3 \alpha} (X _ {3} - e) ^ {\alpha / \log 2} \quad \text {h e n c e} \quad \mathbb {E} _ {\boldsymbol {\nu}} [ \exp (\alpha p _ {3}) ] <   + \infty ,
$$
where we used Lemma 7 and  $h_1(x, e) \sim_{x \to +\infty} x \log x$  to obtain that  $\exp(\alpha p_3)$  is at most polynomial in  $W_\mu$ . Likewise, we obtain that  $\mathbb{E}_\nu[\exp(\alpha p_2)] < +\infty$  for all  $\alpha \in \mathbb{R}_+$ .
Using Lemma 14 (with  $C_0N_1 \geq N_1 \geq N_0$ ), we obtain that, for all  $n \geq C_0N_1$  and all  $a \in [K]$ ,  $k_{n,a} \geq \log_2(n / K) / 2 + 1$ . Therefore, we obtain  $\min_{a \in [K]} k_{n,a} > \max \{p_2, p_3\}$  is implied by  $n \geq N_2 = \max \{K4^{\max \{p_3, p_2\}}, C_0N_1\}$ . Using the above, we conclude that  $\mathbb{E}_{\boldsymbol{\nu}}[N_2] < +\infty$  and  $\max_{c \neq a^\star} |\tilde{\mu}_{k_{n,a^\star}, a^\star} - \tilde{\mu}_{k_{n,c}, c} - \Delta_c| \leq \eta$  for all  $n \geq N_2$ .
Then, for all  $n \geq N_2$ , we have  $B_n = a^\star$  and
$$
\frac {\tilde {\mu} _ {k _ {n , a ^ {\star}} , a ^ {\star}} - \tilde {\mu} _ {k _ {n , a} , a}}{\tilde {\mu} _ {k _ {n , a ^ {\star}} , a ^ {\star}} - \tilde {\mu} _ {k _ {n , b} , b}} \sqrt {\frac {1 + N _ {n , a ^ {\star}} / N _ {n , b}}{1 + N _ {n , a ^ {\star}} / N _ {n , a}}} \geq \frac {\Delta_ {a} - \eta}{\Delta_ {b} + \eta} \sqrt {\frac {1 + (\beta - \tilde {\gamma}) / \omega_ {\beta , b} ^ {\star}}{1 + (\beta + \tilde {\gamma}) / (\omega_ {\beta , a} ^ {\star} + \gamma)}} > 1,
$$
where the last inequality is obtained by taking  $\eta$  and  $\tilde{\gamma}$  sufficiently small and by using Equation (25), i.e.
$$
\frac {\Delta_ {a}}{\Delta_ {b}} \sqrt {\frac {1 + \beta / \omega_ {\beta , b} ^ {\star}}{1 + \beta / \omega_ {\beta , a} ^ {\star}}} = 1.
$$
Therefore, we have shown that  $B_{n} = a^{\star}$  and
$$
\frac {\tilde {\mu} _ {k _ {n , a ^ {\star}} , a ^ {\star}} - \tilde {\mu} _ {k _ {n , a} , a}}{\sqrt {1 / N _ {n , a ^ {\star}} + 1 / N _ {n , a}}} > \frac {\tilde {\mu} _ {k _ {n , a ^ {\star}} , a ^ {\star}} - \tilde {\mu} _ {k _ {n , b} , b}}{\sqrt {1 / N _ {n , a ^ {\star}} + 1 / N _ {n , b}}} \quad \text {h e n c e} \quad C _ {n} \neq a.
$$
Lemma 18 shows that the pulling proportion of the best arm converges towards  $\beta$  for large enough  $n$ .
Lemma 18. Let  $\gamma >0$  and  $T_{\mu ,\gamma}$  be as in Equation (26). Then, we have  $\mathbb{E}_{\nu}[T_{\mu ,\gamma}] < + \infty$
Proof. Let  $\gamma > 0$  and  $\tilde{\gamma} > 0$ . Let  $N_{2}$  as in Lemma 17 for  $\tilde{\gamma}$ . Let  $M \geq N_{2}$ . Using Lemmas 15, 16 and 17 for all  $n \geq M$ , we obtain that  $B_{n} = a^{\star}$ ,  $\left|\frac{N_{n,a^{\star}}}{n - 1} - \beta\right| \leq \tilde{\gamma}$  and
$$
\exists a \neq a ^ {\star}, \quad \frac {N _ {n , a}}{n - 1} \geq \omega_ {\beta , a} ^ {\star} + \tilde {\gamma} \quad \Longrightarrow \quad C _ {n} \neq a.
$$
For all  $a \neq a^{\star}$ , let us define  $t_{n,a}(\tilde{\gamma}) = \max \left\{t \mid M \leq t \leq n, N_{t,a} / (n - 1) < \omega_{\beta,a}^{\star} + \tilde{\gamma}\right\}$ . Since  $N_{t,a} / (n - 1) \leq N_{t,a} / (t - 1)$  for  $t \leq n$ , we have
$$
\frac {N _ {n , a}}{n - 1} \leq \frac {M - 1}{n - 1} + \frac {1}{n - 1} \sum_ {t = M} ^ {n} \mathbb {1} (I _ {t} = C _ {t} = a)
$$
$$
\begin{array}{l} \leq \frac {M - 1}{n - 1} + \frac {1}{n - 1} \sum_ {t = M} ^ {n} \mathbb {1} \left(\frac {N _ {t , a}}{n - 1} <   \omega_ {\beta , a} ^ {\star} + \tilde {\gamma}, I _ {t} = C _ {t} = a\right) \\ \leq \frac {M - 1}{n - 1} + \frac {N _ {t _ {n , a} (\tilde {\gamma}) , a}}{n - 1} <   \frac {M - 1}{n - 1} + \omega_ {\beta , a} ^ {\star} + \tilde {\gamma}. \\ \end{array}
$$
The second inequality uses Lemma 17, and the two last inequalities use the definition of  $t_{n,a}(\widetilde{\gamma})$ . Using that  $\sum_{a\in [K]}\frac{N_{n,a}}{n - 1} = \sum_{a\in [K]}\omega_{\beta ,a}^{\star} = 1$ , we obtain
$$
\frac {N _ {n , a}}{n - 1} = 1 - \sum_ {b \neq a} \frac {N _ {n , a}}{n - 1} \geq 1 - \sum_ {b \neq a} \left(\omega_ {\beta , b} ^ {\star} + \tilde {\gamma} + \frac {M - 1}{n - 1}\right) = \omega_ {\beta , a} ^ {\star} - (K - 1) \left(\tilde {\gamma} + \frac {M - 1}{n - 1}\right).
$$
Taking  $\tilde{\gamma} \leq \gamma / (2(K - 1))$  and  $n \geq \max \{M, 2(K - 1)(M - 1) / \gamma + 1\}$  yields that
$$
\left\| \frac {N _ {n}}{n - 1} - \omega_ {\beta} ^ {\star} \right\| _ {\infty} \leq \gamma .
$$
Let  $T_{\mu, \gamma}$  as in 26. Then, we showed that  $T_{\mu, \gamma} \leq \max \{M, 2(K - 1)(M - 1) / \gamma + 1\}$ . Therefore, we have
$$
\mathbb {E} _ {\nu} \left[ T _ {\mu , \gamma} \right] \leq \mathbb {E} _ {\nu} \left[ \max  \{M, 2 (K - 1) (M - 1) / \gamma + 1 \} \right] <   + \infty ,
$$
E.4 Cost of doubling and forgetting