SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \operatorname {P o s t - C V} _ {\text {I M E D}} \leqslant \mathbb {E} \left[ \sum_ {t = 1} ^ {T - 1} \mathbb {I} \left(I _ {k} (t) \leqslant I _ {k ^ {\star} (t)} (t), N _ {k} (t) > u _ {\varepsilon} (T), \mathcal {G} _ {k} (t), \mathcal {J} _ {k} (t)\right) \right] \\ \leqslant \mathbb {E} \left[ \sum_ {t = 1} ^ {T - 1} \mathbb {I} \left(N _ {k} (t) \mathcal {K} _ {k} (t) \leqslant \log \left(N _ {k ^ {\star} (t)} (t)\right) <   \log (T), N _ {k} (t) > u _ {\varepsilon} (T), \mathcal {G} _ {k} (t), \mathcal {J} _ {k} (t)\right) \right] \\ \leqslant \mathbb {E} \left[ \sum_ {t = 1} ^ {T - 1} \mathbb {I} \left(u _ {\varepsilon} (T) \left(\mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k}, \mu_ {1}\right) - \varepsilon\right) <   \log (T)\right) \right] = 0, \\ \end{array}
$$
so arm  $k$  is never pulled in the post-convergence regime with IMED. Again, these arguments directly translate to Post-CV<sub>FIMED</sub>.
Upper bounding pre-convergence term Interestingly, the design of  $\mathcal{K}_k(t)$ , satisfying  $\mathcal{K}_k(t) \geqslant \mathcal{K}_{\inf}^{\mathcal{F}}(F_k(t), \mu^\star(t))$ , makes FMED and FIMED more exploratory than MED and IMED. Hence, we can unify the proof for the vanilla algorithms and their fast update in this part.
We start with MED, following Baudry et al. [2023] we first use that for any sampling probability  $p_k(t)$  it holds that  $p_k(t) \leqslant \frac{p_k(t)}{p_1(t)} p_1(t) \leqslant \frac{1 - p_1(t)}{p_1(t)} p_1(t)$ , and obtain that
$$
\begin{array}{l} \operatorname {P r e - C V} _ {\text {M E D}} := \mathbb {E} \left[ \sum_ {t = u} ^ {T - 1} \mathbb {I} \left(N _ {k} (t) > u, A _ {t + 1} = k, \mu^ {\star} (t) \leqslant \mu_ {1} - \varepsilon_ {0}\right) \right] \\ = \mathbb {E} \left[ \sum_ {t = u} ^ {T - 1} \mathbb {I} \left(N _ {k} (t) > u, \mu^ {\star} (t) \leqslant \mu_ {1} - \varepsilon_ {0}\right) p _ {k} (t) \right] \\ \leqslant \mathbb {E} \left[ \sum_ {t = u} ^ {T - 1} \mathbb {I} \left(\mu_ {1} (t) \leqslant \mu_ {1} - \varepsilon_ {0}\right) \left(\frac {1}{p _ {1} (t)} - 1\right) p _ {1} (t) \right] \\ = \mathbb {E} \left[ \sum_ {t = u} ^ {T - 1} \mathbb {I} \left(A _ {t + 1} = 1, \mu_ {1} (t) \leqslant \mu_ {1} - \varepsilon_ {0}\right) \left(\frac {1}{p _ {1} (t)} - 1\right) \right]. \\ \end{array}
$$
Let us denote by  $p_1^{\mathrm{FMED}}(t)$  the sampling probability of arm 1 under FMED. Using that  $\sum_{k=1}^{K} e^{N_k(t) \mathcal{K}_k(t)}$  we obtain that
$$
p _ {1} ^ {\text {F M E D}} (t) \geqslant \frac {1}{K} \exp (- N _ {1} (t) \mathcal {K} _ {1} (t)) \geqslant \frac {1}{K} \exp (- N _ {1} (t) \mathcal {K} _ {\inf } ^ {\mathcal {F}} (F _ {1} (t), \mu^ {\star} (t))) := p _ {1} ^ {\text {M E D}} (t).
$$
where  $p_1^{\mathrm{MED}}(t)$  is itself a lower bound on the sampling probability of arm 1 under MED (we use this notation with a slight abuse). If follows that  $\mathrm{Pre - CV}_{\mathrm{FMED}}$  admits the same upper bound as  $\mathrm{Pre - CV}_{\mathrm{MED}}$ . The rest of the proof can be found in Baudry et al. [2023], we briefly recall it from completeness. We first obtain that
$$
\begin{array}{l} \operatorname {P r e - C V} _ {\text {M E D}} \leqslant K \sum_ {n = 1} ^ {T - 1} \mathbb {E} \left[ e ^ {n \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {1, n}, \mu_ {1} - \varepsilon_ {0}\right)} - 1 \right] + (K - 1) \sum_ {n = 1} ^ {T - 1} \mathbb {P} \left(\mu_ {1, n} \leqslant \mu_ {1} - \varepsilon_ {0}\right) \\ \leqslant K \sum_ {n = 1} ^ {T - 1} \mathbb {E} \left[ e ^ {n \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {1, n}, \mu_ {1} - \varepsilon_ {0}\right)} - 1 \right] + \mathcal {O} _ {\varepsilon_ {0}} (1). \\ \end{array}
$$
Then using properties 3. and 4. from Lemma 2 and  $\mathcal{K}_{\inf}^{\mathcal{F}}(F_{1,n},\mu_1)\leqslant \mathcal{K}^{+}\coloneqq \log \left(\frac{B - b}{B - \mu_1}\right)$  we conclude that
$$
\begin{array}{l} \operatorname {P r e - C V} _ {\text {M E D}} \leqslant K \sum_ {n = 1} ^ {T - 1} \int_ {0} ^ {e ^ {n \mathcal {K} ^ {+}} - 1} \mathbb {P} \left(\mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {1, n}, \mu_ {1}\right) \geqslant \delta_ {\varepsilon_ {0}} + \frac {\log (1 + x)}{n}\right) d x \\ \leqslant K \sum_ {n = 1} ^ {T - 1} e (n + 2) \int_ {0} ^ {e ^ {n \mathcal {K} ^ {+}} - 1} \frac {e ^ {- n \delta_ {\varepsilon_ {0}}}}{1 + x} d x \\ = K e \log \left(\frac {B - b}{B - \mu_ {1}}\right) \times \sum_ {n = 1} ^ {T - 1} n (n + 2) e ^ {- n \delta_ {\varepsilon_ {0}}} \\ = \mathcal {O} _ {\varepsilon_ {0}} (1), \\ \end{array}
$$
with  $\delta_{\varepsilon_0} = \frac{2\varepsilon_0^2}{(B - b)^2}$ . As a consequence,  $\mathrm{Pre - CV}_{\mathrm{FMED}} = \mathcal{O}_{\varepsilon_0}(1)$  too. For FIMED the relationship with the pre-convergence term of IMED is even more straightforward,
$$
\begin{array}{l} \operatorname {P r e - C V} _ {\text {F I M E D}} \leqslant \mathbb {E} \left[ \sum_ {t = u _ {\varepsilon} (T)} ^ {T - 1} \mathbb {I} \left(N _ {1} (t) \mathcal {K} _ {1} (t) + \log \left(N _ {1} (t)\right) \geqslant \log \left(N _ {k} (t)\right), N _ {k} (t) > u _ {\varepsilon} (T), \mu^ {\star} (t) \leqslant \mu_ {1} - \varepsilon_ {0}\right) \right] \\ \leqslant \mathbb {E} \left[ \sum_ {t = u _ {\varepsilon} (T)} ^ {T - 1} \mathbb {I} \left(N _ {1} (t) \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {1} (t), \mu^ {\star} (t)\right) + \log \left(N _ {1} (t)\right) \geqslant \log \left(N _ {k} (t)\right), N _ {k} (t) > u _ {\varepsilon} (T), \mu^ {\star} (t) \leqslant \mu_ {1} - \varepsilon_ {0}\right) \right] \\ = \operatorname {P r e - C V} _ {\text {I M E D}} = \mathcal {O} _ {\varepsilon_ {0}} (1). \\ \end{array}
$$
The last statement can be deduced from the regret analysis presented in [Honda and Takemura, 2015], but we now propose a simpler proof of independent interest. We use that if  $A_{t + 1} = k$ , then the index of arm  $k$  is smaller than the index of arm 1, and that  $N_{k}(t) > u_{\varepsilon}(T)$  to first write that
$$
\begin{array}{l} \operatorname {P r e - C V} _ {\text {I M E D}} \leqslant \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {I} \left(A _ {t + 1} = k, N _ {k} (t) \geqslant u _ {\varepsilon} (T), I _ {1} (t) \geqslant \log \left(N _ {k} (t)\right)\right) \right] \\ \leqslant \mathbb {E} \left[ \sum_ {t = u _ {\varepsilon} (T)} ^ {T} \sum_ {n _ {1} = 1} ^ {T} \sum_ {n _ {k} = u _ {\varepsilon} (T)} ^ {T} \mathbb {I} \left(A _ {t + 1} = k, N _ {k} (t) = n _ {k}, N _ {1} (t) = n _ {1}, I _ {1} (t) \geqslant \log (n _ {k})\right) \right] \\ \leqslant \mathbb {E} \left[ \sum_ {n _ {1} = 1} ^ {T} \sum_ {n _ {k} = u _ {\varepsilon} (T)} ^ {T} \mathbb {I} \left(n _ {1} \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {1, n}, \mu_ {1} - \varepsilon_ {0}\right) + \log \left(n _ {1}\right) \geqslant \log \left(n _ {k}\right)\right) \right]. \\ \end{array}
$$
To further upper bound this term, we observe that since  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}(F_{1,n},\mu_1 - \varepsilon_0)\leqslant \log \left(\frac{B - b}{B - \mu_1 + \varepsilon_0}\right)\coloneqq \mathcal{K}_{\varepsilon_0}^+$ , the events considered above cannot happen if  $n_1\mathcal{K}_{\varepsilon_0}^+ +\log (n_1)\leqslant \log (u_\varepsilon (T))\leqslant \log (n_k)$ . Hence, there exists a function  $g:T\mapsto \mathbb{N}$  satisfying  $g(T)\to +\infty$  ( $g(T)$  would be of order  $\log \log (T)$  without the additional  $\log (n_1)$ ) such that
$$
\operatorname {P r e - C V} _ {\text {I M E D}} \leqslant \mathbb {E} \left[ \sum_ {n _ {1} = g (T)} ^ {T} \sum_ {n _ {k} = u _ {\varepsilon} (T)} ^ {T} \mathbb {I} \left(n _ {1} \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {1, n}, \mu_ {1} - \varepsilon_ {0}\right) + \log \left(n _ {1}\right) \geqslant \log \left(n _ {k}\right)\right) \right].
$$
Then, using again properties 3. and 4. of Lemma 2 we obtain that
$$
\begin{array}{l} \operatorname {P r e - C V} _ {\text {I M E D}} \leqslant \sum_ {n _ {1} = g (u)} ^ {T} \sum_ {n _ {k} = u} ^ {T} e \left(n _ {1} + 2\right) e ^ {- \left(n _ {1} \delta_ {\varepsilon_ {0}} + \log \left(n _ {k}\right) - \log \left(n _ {1}\right)\right)} \\ \leqslant e \left(\sum_ {n _ {1} = g (T)} ^ {T} n _ {1} \left(n _ {1} + 2\right) e ^ {- n _ {1} \delta_ {\varepsilon_ {0}}}\right) \left(\sum_ {n _ {k} = u} ^ {T} \frac {1}{n _ {k}}\right) \\ \leqslant e \left(\sum_ {n _ {1} = g (T)} ^ {+ \infty} n _ {1} (n _ {1} + 2) e ^ {- n _ {1} \delta_ {\varepsilon_ {0}}}\right) \log \left(\frac {T}{u _ {\varepsilon} (T)}\right) \\ = o _ {\varepsilon_ {0}} (\log (T)), \\ \end{array}
$$
where the conclusion comes from the fact that the remaining sum is the remainder of a convergent series, or in other words  $\operatorname{Pre-CV}_{\text{IMED}} \leqslant a_T \log(T)$  where  $a_T \to 0$ . This is sufficient to conclude the proof, by noting that a careful tuning of  $\varepsilon_0$  (for instance as  $\log \log(T)$  allows to obtain that  $\lim \sup \frac{\mathbb{E}[N_k(T)]}{\log(T)} \leqslant \frac{1}{\mathcal{K}_{\inf}^{\mathcal{F}}(F_k, \mu_1)}$ ).
C Proof of Theorem 2: regret analysis of OMED and OIMED
We first recall the theorem.
Theorem 2 (Regret bound for online MED algorithms). Consider  $(F_{1},\ldots ,F_{K})\in \mathcal{F}^{K}$ , and  $\mu^{\star} = \max_{k\in [K]}\mu_k < \mu_{max}$ , for a known  $\mu_{\mathrm{max}} < B$ . Assume that OMED and OIMED use a portfolio selection algorithm ALG that satisfies for any  $(k,\ell)\in [K]^2$  and  $n\in \mathbb{N}$  the deterministic guarantee
Proof. As for the proof of Theorem 1 we start by proving a first regret upper bound that holds for both algorithms.
Lemma 4 (Generic upper bound). For any sub-optimal arm  $k$ , for any  $\varepsilon > 0$  it holds that
$$
\begin{array}{l} \mathbb {E} [ N _ {k} (t) ] \leqslant u _ {\varepsilon} (T) + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {I} (k \in \mathcal {A} _ {t + 1} , \ell_ {t} = 1 , \widetilde {N} _ {k , 1} (t) > u _ {\varepsilon} (T) , Z _ {k} (t) = 1 , \mathcal {G} _ {k} (t)) \right]} _ {P o s t - C V} \\ + 4 \log (4) \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {I} \left(1 \notin \mathcal {A} _ {t + 1} , \ell_ {t} \neq 1 , Z _ {1} (t) = 1\right) \right]} _ {P r e - C V} + \mathcal {O} _ {\varepsilon} (1) , \\ \end{array}
$$
$$
w h e r e u _ {\varepsilon} (T) = \frac {\log (T)}{\mathcal {K} _ {i n f} ^ {\mathcal {F}} \left(F _ {n} , \mu_ {1}\right) - \varepsilon}, a n d \mathcal {G} _ {k} (t) = \Big \{L _ {k, 1} (t) \geqslant \widetilde {N} _ {k, 1} (t) \big (\mathcal {K} _ {i n f} ^ {\mathcal {F}} \left(F _ {k}, \mu_ {1}\right) - \varepsilon \big) \Big \}.
$$
Before proving this result, let us detail some intuitions. First, we remark that the two expectations need to be second order terms to make the algorithms asymptotically optimal. The first expectation corresponds to pulls of arm  $k$  in a post-convergence regime, where arm 1 is the leader and the empirical distribution of arm  $k$  is close to  $F_{k}$ . On the contrary, the second term is the expected number of duels lost by arm 1 as a challenger, against a sub-optimal leader. For this reason, we name these two terms respectively Post-CV and Pre-CV (where CV abbreviates convergence). Note that Lemma 4 actually holds for any bandit algorithms that would use the same duel-based structure, independently of what they do during the non-greedy duels.
Proof. We start the proof by introducing some notation. Due to the specific structure of the duel-based algorithm we define for a challenger/leader pair  $(k,\ell)$  a pseudo-count  $\widetilde{N}_{k,\ell}(t)$  and a corresponding empirical cdf  $\widetilde{F}_{k,\ell}(t)$ : they are computed by considering only the observations of arm  $k$  that have been collected when arm  $k$  was pulled after a non-greedy duel  $(Z_{k}(t) = 1)$  performed against the leader  $\ell_t = \ell$ . Hence,  $\widetilde{N}_{k,\ell}(t) \leqslant N_k(t)$ . In several parts of the proof we also use constants  $x_{k} \in (\mu_{k},\mu_{1})$ , that we arbitrarily set to  $\frac{\mu_k + \mu_1}{2}$ . Finally, we introduce two ways to denote the rewards: we continue to call  $X_{k,n}$  the  $n$ -th reward received by arm  $k$ , but we also introduce the notation  $X_{k,\ell,n}$  for the  $n$ -th reward received by arm  $k$  after a non-greedy duel against the leader  $\ell$ .
We start the analysis by considering the case when the best arm is the leader, and the alternative. For each sub-optimal arm  $k$ , it holds that
$$
\mathbb {E} [ N _ {k} (T) ] = 1 + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T - 1} \mathbb {I} (k \in \mathcal {A} _ {t + 1} , \ell_ {t} = 1) \right]} _ {A _ {1}} + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T - 1} \mathbb {I} (k \in \mathcal {A} _ {t + 1} , \ell_ {t} \neq 1 \right]} _ {A _ {2}}
$$
We then consider the favorable case where arm 1 the leader, splitting the cases according to the value of  $Z_{k}(t)$ .
$$
A _ {1} \leqslant \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T - 1} \mathbb {I} (k \in \mathcal {A} _ {t + 1} , \ell_ {t} = 1 , \mu_ {1} (t) \leqslant \mu_ {k} (t)) \right]} _ {B _ {1} (\text {g r e e d y d u e l s})} + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T - 1} \mathbb {I} (k \in \mathcal {A} _ {t + 1} , \ell_ {t} = 1 , Z _ {k} (t) = 1) \right]} _ {B _ {2} (\text {n o n - g r e e d y d u e l s})}.
$$
We now upper bound  $B_{1}$  using that if  $\ell_t = 1$  then  $N_{1}(t) \geqslant t / K$ , and that if  $\mu_k(t) \geqslant \mu_1(t)$  then either  $\mu_k(t) \geqslant x_k$  or  $\mu_1(t) \leqslant x_k$ . By starting  $N_{k}(t)$  at 1 we cover all possible scenarios for  $Z_{k}(t) = 0$  (including  $\mu_{\ell_t}(t) \geqslant \mu_{\mathrm{max}}$ ). We then obtain
$$
\begin{array}{l} B _ {1} \leqslant \mathbb {E} \left[ \sum_ {t = 1} ^ {T - 1} \mathbb {I} \left(\ell_ {t} = 1, \mu_ {1} (t) \leqslant x _ {k}\right) \right] + \mathbb {E} \left[ \sum_ {t = 1} ^ {T - 1} \mathbb {I} \left(k \in \mathcal {A} _ {t + 1}, \mu_ {k} (t) \geqslant x _ {k}\right) \right] \\ \leqslant \sum_ {t = 1} ^ {+ \infty} \sum_ {n = t / K} ^ {+ \infty} \mathbb {P} \left(\mu_ {1, n} \leqslant x _ {k}\right) + \sum_ {n = 1} ^ {+ \infty} \mathbb {P} \left(\mu_ {k, n} \geqslant x _ {k}\right) = \mathcal {O} (1). \\ \end{array}
$$
We then consider  $B_{2}$ . We use that for any  $n$ ,  $\{k \in \mathcal{A}_{t+1}, Z_{k}(t) = 1, \widetilde{N}_{k,1}(t) = n\} = \{\widetilde{N}_{k,1}(t+1) = \widetilde{N}_{k,1}(t) + 1, \widetilde{N}_{k,1}(t) = n\}$  and can happen only once. For any  $u_{\varepsilon}(T) \in \mathbb{N}$ , we hence obtain that
$$
B _ {2} \leqslant u _ {\varepsilon} (T) + \sum_ {t = 1} ^ {T - 1} \mathbb {E} \left[ \mathbb {I} \left(\widetilde {N} _ {k, 1} (t + 1) = \widetilde {N} _ {k, 1} (t) + 1, \widetilde {N} _ {k, 1} (t) > u _ {\varepsilon} (T)\right) \right].
$$
We then upper bound  $B_{2}$  by considering separately cases when  $\mathcal{G}_k(t)$  holds or not,
$$
\begin{array}{l} B _ {2} \leqslant u + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {I} \left(\widetilde {N} _ {k , 1} (t + 1) = \widetilde {N} _ {k , 1} (t) + 1 , \widetilde {N} _ {k , 1} (t) > u , \mathcal {G} _ {k} (t)\right) \right]} _ {\text {P o s t - C V}} \\ + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {I} (\widetilde {N} _ {k , 1} (t + 1) = \widetilde {N} _ {k , 1} (t) + 1 , \widetilde {N} _ {k , 1} (t) > u , \bar {\mathcal {G}} _ {k} (t)) \right]} _ {B _ {2} ^ {\prime}}. \\ \end{array}
$$
We remark the first term is exactly the post-convergence term that we introduced in the lemma. Hence, we leave this expression as it is for this first result. We then consider  $B_2'$ . We use Equation (14), for any  $\varepsilon_0 > 0$  it holds that
$$
L _ {k, 1, \widetilde {N} _ {k, 1} (t)} = \widetilde {N} _ {k, 1} (t) \mathcal {K} _ {\inf } ^ {\mathcal {F}} (\widetilde {F} _ {k} (t), \mu_ {1} - \varepsilon_ {0}) + R _ {k, 1, \widetilde {N} _ {k, 1} (t)} + B _ {k, 1, \widetilde {N} _ {k, 1} (t)} (\mu_ {1} - \varepsilon_ {0}).
$$
Hence, we obtain that
$$
\bar {\mathcal {G}} _ {k} (t) \subset \left\{\max  \left\{\frac {B _ {k , 1 , \widetilde {N} _ {k , 1} (t)} (\mu_ {1} - \varepsilon_ {0})}{\widetilde {N} _ {k , 1} (t)}, \frac {R _ {k , 1 , \widetilde {N} _ {k , 1} (t)}}{\widetilde {N} _ {k , 1} (t)}, \mathcal {K} _ {\inf } ^ {\mathcal {F}} (F _ {k}, \mu_ {1}) - \mathcal {K} _ {\inf } ^ {\mathcal {F}} (\widetilde {F} _ {k} (t), \mu_ {1} - \varepsilon_ {0}) \right\} \geqslant \frac {\varepsilon}{3} \right\}.
$$
Note that we could have stated different thresholds for each term, but we choose  $\varepsilon /3$  in each case for simplicity.
We use the first side of our assumption on the regret of the portfolio algorithm: if  $R_{n} = o(n)$ , then there exists  $n_0 \in \mathbb{N}$  large enough such that for  $n \geqslant n_{\varepsilon}$ ,  $R_{n} \leqslant \frac{n\varepsilon}{3}$ . Hence, by defining  $u_{\varepsilon}(T) \geqslant n_{\varepsilon}$  we ensure that  $\bar{\mathcal{G}}_k(t)$  is not due to a large portfolio regret.
We now consider the term involving  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}(\widetilde{F}_k(t),\mu_1 - \varepsilon_0)$ , which is analogous to the CV-Emp term of the proof of Theorem 1 (Appendix B). We hence directly write that
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {I} \left(\widetilde {N} _ {k, 1} (t + 1) = \widetilde {N} _ {k, 1} (t) + 1, \widetilde {N} _ {k, 1} (t) > u _ {\varepsilon} (T), \mathcal {K} _ {\inf } ^ {\mathcal {F}} (\widetilde {F} _ {k} (t), \mu_ {1} - \varepsilon_ {0}) \leqslant \mathcal {K} _ {\inf } ^ {\mathcal {F}} (F _ {k}, \mu_ {1}) - \frac {\varepsilon}{3}\right) \right] \\ \leqslant \sum_ {n = u _ {\varepsilon} (T)} ^ {+ \infty} \mathbb {P} \left(\mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(\widetilde {F} _ {k, n}, \mu_ {1}\right) \leqslant \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k}, \mu_ {1}\right) + \frac {\varepsilon_ {0}}{B - \mu_ {1}} - \frac {\varepsilon}{3}\right) = \mathcal {O} _ {\varepsilon} (1), \\ \end{array}
$$
by choosing  $\varepsilon_0 = (B - \mu_1)\frac{\varepsilon}{6}$ . Finally, we use that  $B_{k,1,\widetilde{N}_{k,1}(t)}(\mu_1 - \varepsilon_0) \geqslant \widetilde{N}_{k,1}(t)\varepsilon / 3$  is possible only if at least one empirical mean computed with a "reasonable" sample size deviates. More precisely,
$$
\begin{array}{l} B _ {k, 1, \widetilde {N} _ {k, 1} (t)} \left(\mu_ {1} - \varepsilon_ {0}\right) \geqslant \widetilde {N} _ {k, 1} (t) \frac {\varepsilon}{3} \Rightarrow \exists n \geqslant \widetilde {N} _ {k, 1} (t) \frac {\varepsilon}{3}: \mu_ {1} \left(t _ {k, 1, n}\right) \leqslant \mu_ {1} - \varepsilon_ {0} \\ \Rightarrow \exists n \geqslant f \left(\widetilde {N} _ {k, 1} (t) \frac {\varepsilon}{3}\right): \mu_ {1, n} \leqslant \mu_ {1} - \varepsilon_ {0}, \\ \end{array}
$$
where we used that, thanks to our algorithm, at any time  $s$  for which the estimate  $L_{k,1}(t)$  was incremented  $N_{1}(s) \geqslant f(\widetilde{N}_{k,1}(s))$  was satisfied (check to decide that the duel is non-greedy). Interestingly, we now get an event that only depends on  $\widetilde{N}_{k,1}(t)$ . Using Lemma 1 we finally get that
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {I} \left(\widetilde {N} _ {k, 1} (t + 1) = \widetilde {N} _ {k, 1} (t) + 1, \widetilde {N} _ {k, 1} (t) > u _ {\varepsilon} (T), B _ {k, 1, \widetilde {N} _ {k, 1} (t)} \left(\mu_ {1} - \varepsilon_ {0}\right) \geqslant \widetilde {N} _ {k, 1} (t) \frac {\varepsilon}{3}\right) \right] \\ \leqslant \sum_ {t = u _ {\varepsilon} (T)} ^ {+ \infty} \mathbb {P} \left(\widetilde {N} _ {k, 1} (t + 1) = \widetilde {N} _ {k, 1} (t) + 1, \widetilde {N} _ {k, 1} (t) \geqslant u _ {\varepsilon} (T), \exists s \geqslant f \left(\widetilde {N} _ {k, 1} (t) \frac {\varepsilon}{3}\right): \mu_ {1, s} \leqslant \mu_ {1} - \varepsilon_ {0}\right) \\ \leqslant \sum_ {n = u _ {\varepsilon} (T)} ^ {+ \infty} \mathbb {P} \left(\exists s \geqslant f \left(n \frac {\varepsilon}{3}\right): \mu_ {1, s} \leqslant \mu_ {1} - \varepsilon_ {0}\right) \\ \leqslant \sum_ {n = u _ {\varepsilon} (T)} ^ {+ \infty} \sum_ {s = f \left(n \frac {\varepsilon}{3}\right)} ^ {+ \infty} \mathbb {P} \left(\mu_ {1, s} \leqslant \mu_ {1} - \varepsilon_ {0}\right) = \mathcal {O} _ {\varepsilon} (1), \\ \end{array}
$$
thanks to Hoeffding's inequality. This last step allows to conclude that  $B_2' = \mathcal{O}_{\varepsilon}(1)$ , using that  $f(s) \geq s$  for any  $s \in \mathbb{N}$ .
Remark 1. We can see that at this step  $N_{1}(s) \geqslant N_{k}(s)$  was sufficient, the enforcement of  $N_{1}(s) \geqslant f(N_{k}(s))$  is necessary only in the "pre-convergence" analysis.
Upper bounding  $A_{2}, \ell_{t} \neq 1$ : For this part of the proof, we mainly use techniques from [Chan, 2020, Baudry et al., 2021b]. In particular, we use that if the current leader is a sub-optimal arm then either 1 has already been leader and has lost leadership or 1 has never been leader. Formally, we define the event
$$
\mathcal {D} _ {t} = \left\{\exists r \in \left[ \frac {t}{4}, t \right]: \ell_ {r} = 1 \right\}.
$$
Then, we first upper bound the term
$$
C _ {1} := \sum_ {t = 1} ^ {T - 1} \mathbb {E} \left[ \mathbb {I} \left(k \in \mathcal {A} _ {t + 1}, \ell_ {t} \neq 1, \mathcal {D} _ {t}\right) \right].
$$
We use that if  $\mathcal{D}_t$  holds and  $\ell_t \neq 1$  then arm 1 was the leader at some point between  $t/4$  and  $t$  and lost its leadership. Furthermore, a change of leadership from 1 to  $j$  at time  $s$  can only happen if (1) arm  $j$  has been pulled at the previous round, (2) the two arms now satisfy  $N_1(s) = N_j(s) = n$  for some  $n \geqslant s/K \geqslant t/(4K)$ , and (3)  $\mu_{1,n} \leqslant \mu_{j,n}$ . Thanks to these properties, we obtain with some union bounds that
$$
\begin{array}{l} C _ {1} \leqslant \sum_ {j = 2} ^ {K} \sum_ {t = 1} ^ {T - 1} \sum_ {s = \lceil t / 4 \rceil} ^ {t} \sum_ {n = \lceil s / K \rceil} ^ {t} \mathbb {E} \left[ \mathbb {I} \left(j \in \mathcal {A} _ {s}, N _ {1} (s) = N _ {j} (s) = n, \mu_ {j, n} \geqslant \mu_ {1, n}\right) \right] \\ \leqslant \sum_ {j = 2} ^ {K} \sum_ {t = 1} ^ {T - 1} \sum_ {n = \lceil t / K \rceil} ^ {t} \mathbb {E} \left[ \mathbb {I} \left(\mu_ {j, n} \geqslant \mu_ {1, n}\right) \sum_ {s = \lceil t / 4 \rceil} ^ {t} \mathbb {I} (j \in \mathcal {A} _ {s}, N _ {j} (s) = n) \right] \\ \leqslant \sum_ {j = 2} ^ {K} \sum_ {t = 1} ^ {T - 1} \sum_ {n = \lceil t / K \rceil} ^ {t} \mathbb {P} \left(\mu_ {j, n} \geqslant \mu_ {1, n}\right) \\ = \sum_ {j = 2} ^ {K} \sum_ {t = 1} ^ {T - 1} \sum_ {n = \lceil t / K \rceil} ^ {t} \left(\mathbb {P} \left(\mu_ {j, n} \geqslant x _ {j}\right) + \mathbb {P} \left(\mu_ {1, n} \leqslant x _ {j}\right)\right) = \mathcal {O} (1). \\ \end{array}
$$
We now upper bound the term
$$
C _ {2} := \sum_ {t = 1} ^ {T - 1} \mathbb {E} \left[ \mathbb {I} \left(k \in \mathcal {A} _ {t + 1}, \ell_ {t} \neq 1, \bar {\mathcal {D}} _ {t}\right) \right].
$$
Following Chan [2020], we use that if arm 1 has never been leader between  $t/4$  and  $t$ , then it has necessarily lost at least  $t/4$  duels in that time interval. Using Markov inequality, we hence obtain that
$$
\begin{array}{l} C _ {2} \leqslant \sum_ {t = 1} ^ {T - 1} \mathbb {P} \left(\mathbb {I} \left(\sum_ {s = t / 4} ^ {t} \mathbb {I} (1 \notin \mathcal {A} _ {s + 1}, \ell_ {s} \neq 1) \geqslant \frac {t}{4}\right)\right) \\ \leqslant \sum_ {t = 1} ^ {T - 1} \frac {4}{t} \sum_ {s = t / 4} ^ {t} \mathbb {E} [ \mathbb {I} (1 \notin \mathcal {A} _ {s + 1}, \ell_ {s} \neq 1) ] \\ \leqslant \sum_ {s = 1} ^ {T - 1} \left(\sum_ {t = 1} ^ {T - 1} \frac {4}{t} \mathbb {I} (t \in [ s, 4 s ])\right) \mathbb {E} \left[ \mathbb {I} (1 \notin \mathcal {A} _ {s + 1}, \ell_ {s} \neq 1) \right] \\ \leqslant 4 \log (4) \sum_ {t = 1} ^ {T - 1} \mathbb {E} \left[ \mathbb {I} \left(1 \notin \mathcal {A} _ {t + 1}, \ell_ {t} \neq 1\right) \right]. \\ \end{array}
$$
Thank to these simple tricks, we are back to upper bounding the total number of duels lost by arm 1 while not being the leader at the cost of a multiplicative constant, which is close to the remaining