SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
B _ {k, \ell , n} (\mu) := \max  _ {\lambda \in [ 0, 1 ]} \sum_ {i = 1} ^ {n} \log \left(1 - \lambda \left(\frac {X _ {k , \ell , i} - \mu^ {\star} (t _ {k , \ell , i})}{B - \mu^ {\star} (t _ {k , \ell , i})}\right)\right) - n \mathcal {K} _ {\inf } ^ {\mathcal {F}} (\widetilde {F} _ {n}, \mu)
$$
is the bias of the portfolio selection sub-routine with respect to  $n\mathcal{K}_{\inf}^{\mathcal{F}}(\widetilde{F}_n,\mu)$
We also use  $L_{k,\ell_t}(t) = L_{k,\ell_t,\widetilde{N}_k(t)},B_k(t,\mu) = B_{k,\ell_t,\widetilde{N}_k(t)}(\mu).$
A.2 Detailed implementations
In this section we detail the implementations of the algorithms presented in the paper, along with their main competitors. We start with the implementation of IMED and MED, presented respectively in Algorithms 2 and 3. We then present FIMED and FMED, respectively in Algorithms 4 and 5. Finally, we detail OMED in Algorithm 6, and recall that the implementation of OIMED is presented in Section 2 (Algorithm 1).
In Algorithm 7, we also detail the implementation of the discretization process proposed by Riou and Honda [2020], that allows to map the rewards in a grid  $\mathcal{X} = (x_1,\dots,x_M)$  without changing the means of the arms.
Finally, for comparison we also detail the implementation of the two other existing optimal algorithms for general bounded rewards: KL-UCB (Algorithm 8), implemented with parameter  $f: t \mapsto \log(t) + \log \log(t)$ , and NPTS (Algorithm 9).
Algorithm 2 Indexed Minimized Empirical Divergence (IMED)
Input:  $K$  arms, support upper bound  $B$   
1 Initialization:  $\forall k\in [K]$ :  $N_{k} = 0$  (nb. of pulls),  $\widehat{\mu}_k = 0$  (emp. mean),  $\mathcal{K}_k = 0$  (emp.  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$ ).  
2 for  $t\in \mathbb{N}$  do  
3 if  $t < K$  then  
4 Set  $a = t$   
5 else  
6 Set  $a = \operatorname*{argmin}_{k\in [K]}N_k\mathcal{K}_k + \log (N_k)$ ;  $\triangleright$  Index comparison, break ties at random  
7 Pull arm  $a$ , observe reward  $X_{a,N_a + 1}$ , update (or define)  $\widehat{\mu}_a$ ,  $N_a$ ,  $\widehat{F}_a$  (emp. distrib.). If  $t > K$ :  $\forall k\in [K]$ , set  $\mathcal{K}_k = \mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}(\widehat{F}_k,\max_{k\in [K]}\widehat{\mu}_k)$ .
Algorithm 3 Minimized Empirical Divergence (MED)
Input:  $K$  arms, support upper bound  $B$   
8 Initialization:  $\forall k\in [K]$ :  $N_{k} = 0$  (nb. of pulls),  $\widehat{\mu}_k = 0$  (emp. mean),  $\mathcal{K}_k = 0$  (emp.  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$ ).  
9 for  $t\in \mathbb{N}$  do  
10 if  $t < K$  then  
11 Set  $a = t$   
12 else  
13 Sample  $a\sim \mathrm{Mult}\left(\frac{e^{-N_1\mathcal{K}_1}}{S},\dots ,\frac{e^{-N_K\mathcal{K}_K}}{S}\right)$ , with  $S = \sum_{j = 1}^{K}e^{-N_k\mathcal{K}_k}$ ;  
14 Pull arm  $a$ , observe reward  $X_{a,N_a + 1}$ , update (or define)  $\widehat{\mu}_a$ ,  $N_{a}$ ,  $\widehat{F}_a$  (emp. distrib.).  
If  $t > K$ :  $\forall k\in [K]$ , set  $\mathcal{K}_k = \mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}(\widehat{F}_k,\max_{k\in [K]}\widehat{\mu}_k)$ .  
 $\triangleright \mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  update
Algorithm 4 Fast Indexed Minimized Empirical Divergence (FIMED)
Input:  $K$  arms, support upper bound  $B$    
15 Initialization:  $\forall k\in [K]$  ..  $N_{k} = 0$  (nb. of pulls),  $\widehat{\mu}_k = 0$  (emp. mean),  $\mathcal{K}_k = 0$  (emp.  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  -   
16 for  $t\in \mathbb{N}$  do   
17 if  $t <   K$  then   
18 Set  $a = t$    
19 else   
20 Set  $a = \operatorname *{argmin}_{k\in [K]}N_k\mathcal{K}_k + \log (N_k)$  . Index comparison, break ties at random   
21 Pull arm  $a$  , observe reward  $X_{a,N_a + 1}$  , update (or define)  $\widehat{\mu}_a$ $N_{a}$ $\widehat{F}_a$  (emp. distrib.). for  $k\in [K]$  do   
22 if  $a = k$  then   
23 If  $N_{k}\geqslant 1$  , set  $\mathcal{K}_k = \mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}(\widehat{F}_k,\max_{k\in [K]}\widehat{\mu}_k)$  and  $\widetilde{\mathcal{K}}_k = \mathcal{K}_k$  where  $\widehat{F}_k$  is the empirical distribution of  $(X_{k1},\ldots ,X_{kN_k})$  .  $\triangleright$ $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  computation   
24 Keep in memory  $\widetilde{\mu}_k^\star = \max \widehat{\mu}_k$  , and  $\widetilde{\lambda}_k$  the maximizer of (4) when computing  $\mathcal{K}_k$  else   
25 Set  $\mathcal{K}_k = \widetilde{\mathcal{K}}_k + \widetilde{\lambda}_k\times (\max_{k\in [K]}\widehat{\mu}_k - \widetilde{\mu}_k^\star)$  . Fast update if  $k$  not pulled
Algorithm 5 Fast Minimized Empirical Divergence (FMED)
Input:  $K$  arms, support upper bound  $B$    
26 Initialization:  $\forall k\in [K]$  ..  $N_{k} = 0$  (nb. of pulls),  $\widehat{\mu}_k = 0$  (emp. mean),  $\mathcal{K}_k = 0$  (emp.  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$    
27 for  $t\in \mathbb{N}$  do   
28 if  $t <   K$  then   
29 Set  $a = t$    
30 else   
31 Sample  $a\sim \mathrm{Mult}\left(\frac{e^{-N_1\mathcal{K}_1}}{S},\dots ,\frac{e^{-N_K\mathcal{K}_K}}{S}\right)$  with  $S = \sum_{j = 1}^{K}e^{-N_k\mathcal{K}_k}$  ; ▷ Sampling   
32 Pull arm  $a$  , observe reward  $X_{a,N_a + 1}$  , update (or define)  $\widehat{\mu}_a$ $N_{a}$ $\widehat{F}_a$  (emp. distrib.). for  $k\in [K]$  do   
33 if  $a = k$  then   
34 If  $N_{k}\geqslant 1$  , set  $\mathcal{K}_k = \mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}(\widehat{F}_k,\max_{k\in [K]}\widehat{\mu}_k)$  and  $\widetilde{\mathcal{K}}_k = \mathcal{K}_k$  where  $\widehat{F}_k$  is the empirical distribution of  $(X_{k1},\ldots ,X_{kN_k})$  .  $\triangleright \mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  computation   
35 Keep in memory  $\widetilde{\mu}_k^\star = \max \widehat{\mu}_k$  , and  $\widetilde{\lambda}_k$  the maximizer of (4) when computing  $\mathcal{K}_k$  else   
36 Set  $\mathcal{K}_k = \widetilde{\mathcal{K}}_k + \widetilde{\lambda}_k\times (\max_{k\in [K]}\widehat{\mu}_k - \widetilde{\mu}_k^\star)$  . ▷ Fast update if  $k$  not pulled
Algorithm 6 Online Minimum Empirical Divergence (OMED)
Input:  $K$  arms, upper bound  $B$ , portfolio selection algorithm ALG (and its parameters), function  $f$ . 37 Initialization:  $\forall k \in [K]$ :  $N_{k} = 0$  (total nb. of pulls),  $\widetilde{N}_{k\ell} = 0$  (nb of pulls as a challenger after a non-greedy duel against leader  $\ell$ ),  $\widehat{\mu}_{k} = 0$  (emp. mean),  $L_{k\ell} = 0$  (emp.  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  as a challenger against leader  $\ell$ ),  $\widehat{\lambda}_{k\ell} = 1/2$  (initialization for ALG).
38 for  $t\in \mathbb{N}$  do   
39 if  $t = 1$  then   
40 Set  $\mathcal{A} = \{1,\dots ,K\}$  . Pull each arm once   
41 else   
42 Set  $\mathcal{A} = \{\}$  ; Initialize the set of arms to pull   
43 Set  $\mathcal{S} = \mathrm{argmax}_{j\in [K]}N_{j}$  ; Candidates for leadership   
44 If  $|S| > 1$  , set  $\mathcal{S}\gets \mathrm{argmax}_{j\in S}\widehat{\mu}_j$  ; Tie breaking   
45 Choose leader  $\ell$  uniformly at random in  $\mathcal{S}$    
for  $k\neq \ell$  ; Organize duels between  $\ell$  and all challengers   
46 do   
47 if  $\widetilde{N}_{k,\ell}\geqslant f(N_\ell)$  or  $\widehat{\mu}_{\ell}\geqslant \mu_{\mathrm{max}}$  then   
48 If  $\widehat{\mu}_k\geqslant \widehat{\mu}_\ell$  : add  $k$  to  $\mathcal{A}$  , set  $Z_{k} = 0$  ; Greedy dual if  $N_{\ell}$  is too small   
49 else   
50 Draw  $W\sim \mathrm{Ber}(e^{-L_{k,\ell}})$  , if  $W_{k} = 1$  then   
51 Add  $k$  to  $\mathcal{A}$  , set  $Z_{k} = 1$  ; MED dual if  $N_{\ell}$  large enough   
52 for  $k\in \mathcal{A}$  do   
53 Pull arm  $k$  , observe a reward  $X$  , update  $\widehat{\mu}_k$ $N_{k}$  .   
if  $Z_{k} = 1$  then   
54  $\widetilde{N}_{k,\ell}\gets \widetilde{N}_{k,\ell} + 1$  Update  $\widehat{\lambda}_{k,\ell}$  with ALG, knowing  $\widetilde{N}_{k,\ell}$  and  $\frac{B - X}{B - \widehat{\mu}_\ell}$  ; Portfolio update   
55  $L_{k,\ell}\gets L_{k,\ell} + \log \left(1 - \widehat{\lambda}_{k,\ell}\frac{X - \widehat{\mu}_\ell}{B - \widehat{\mu}_\ell}\right)$
Algorithm 7 Discretization algorithm from [Riou and Honda, 2020]
Input: Reward  $X$ , grid  $\mathcal{X} = (x_1, \ldots, x_M)$   
56 Set  $m = \max \{j \in [M] : x_j < X\}$  if  $X > x_1$ ,  $m = 1$  otherwise. Sample  $Z \sim \operatorname{Ber}\left(\frac{X - x_j}{x_{j+1} - x_j}\right)$   
Return:  $Y = x_j \mathbb{I}(Z = 0) + x_{j+1} \mathbb{I}(Z = 1)$
Algorithm 8 KL-UCB [Cappé et al., 2013]
Input:  $K$  arms, support upper bound  $B$    
57 Initialization:  $\forall k\in [K]$  ..  $N_{k} = 0$  (nb. of pulls), function  $f$    
58 for  $t\in \mathbb{N}$  do   
59 if  $t <   K$  then   
60 Set  $a = t$    
61 else   
62 for  $k\in [K]$  do   
63 Set  $\widehat{\mu}_k = \max \{\mu \in [b,B]:N_k\mathcal{K}_{\mathrm{inf}}^F (\widehat{F}_k,\mu)\leqslant f(t)\}$    
64 Set  $a = \operatorname {argmax}_{k\in [K]}\widehat{\mu}_k$  Index comparison, break ties at random   
65 Pull arm  $a$  , observe reward  $X_{a,N_a + 1}$  , update (or define)  $\mu_{a}$ $N_{a}$ $\widehat{F}_a$  (emp. distrib.).
Algorithm 9 Non-Parametric Thompson Sampling [Riou and Honda, 2020]  
Input:  $K$  arms, support upper bound  $B$   
Initialization:  $\forall k\in [K]$ :  $N_{k} = 0$  (nb. of pulls),  $\mathcal{H}_k = \{\}$  (history).  
for  $t\in \mathbb{N}$  do  
if  $t < K$  then  
 $\begin{array}{l}\text{Set } a = t\\ \end{array}$  else  
for  $k\in [K]$  do  
Sample weights  $w\sim \mathrm{Dir}(1,\ldots ,1)$  ( $N_{k} + 1$  ones)  
 $\begin{array}{r}\text{Set } I_k = \sum_{i = 1}^{N_k}w_iX_{ki} + w_{N_k + 1}B; \end{array}$ $\begin{array}{r}\triangleright \text{Re - weighted mean}\\ \triangleright \text{Index comparison, break ties at random}\\ \triangleright \text{Pull arm } a,\text{ observe reward } X_{a,N_a + 1},\text{ set } \mathcal{H}_a = \mathcal{H}_a\cup \{X_{a,N_a + 1}\} \end{array}$
B Proof of Theorem 1: regret analysis of the index policies
Before proving the theorem, we recall some useful properties from previous works.
B.1 Toolbox for bounded distributions.
In the proofs we use the following results, that come from several works on bounded distributions in bandits (e.g. [Honda and Takemura, 2015, Cappé et al., 2013]), and are detailed in Appendix D.3 of Baudry et al. [2023].
Lemma 2 (Useful properties of  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$ ). Let  $F \in \mathcal{F}$  be a distribution of mean  $\mu_F$ . We denote by  $F_n$  and  $\mu_n$  respectively the empirical cdf and empirical mean corresponding to  $n$  i.i.d. observations collected from  $F$ . It holds that
1.  $\mathcal{K}_{inf}^{\mathcal{F}}$  is continuous, non-decreasing and convex in its second argument (Theorem 7 of [Honda and Takemura, 2010]).
2. There exists constants  $c > 0$ ,  $\delta_0 > 0$  such that  $\forall \mu > \mu_F$ , and  $\delta \in (0, \delta_0]$  it holds that
$$
\mathbb {P} \left(\mathcal {K} _ {i n f} ^ {\mathcal {F}} \left(F _ {n}, \mu\right) \leqslant \mathcal {K} _ {i n f} ^ {\mathcal {F}} (F, \mu) - \delta\right) \leqslant e ^ {- n c \delta^ {2}}.
$$
3. For any  $\varepsilon > 0$  and  $\mu > \mu_n + \varepsilon$ , it holds that
$$
\mathcal {K} _ {i n f} ^ {\mathcal {F}} \left(F _ {n}, \mu\right) - \mathcal {K} _ {i n f} ^ {\mathcal {F}} \left(F _ {n}, \mu - \varepsilon\right) \geqslant \delta_ {\varepsilon} := \frac {2 \varepsilon^ {2}}{(B - b) ^ {2}}.
$$
4. For any  $x > 0$ , it holds that (Lemma 6 from Cappé et al. [2013])
$$
\mathbb {P} \left(\mathcal {K} _ {i n f} ^ {\mathcal {F}} \left(F _ {n}, \mu_ {1}\right) > x\right) \leqslant e (n + 2) e ^ {- n x}.
$$
5. For any  $\varepsilon > 0$  (Hoeffding's inequality),
$$
\mathbb {P} \left(\mu_ {n} \leqslant \mu_ {F} - \varepsilon\right) \leqslant e ^ {- 2 n \varepsilon^ {2}} \quad a n d \quad \mathbb {P} \left(\mu_ {n} \geqslant \mu_ {F} + \varepsilon\right) \leqslant e ^ {- 2 n \varepsilon^ {2}}.
$$
Proof. Property 3. can be deduced from Lemma 13 of Honda and Takemura [2010]. Using the convexity of  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  and Pinsker inequality we obtain that
$$
\mathcal {K} _ {\inf } ^ {\mathcal {F}} (F _ {n}, \mu) - \mathcal {K} _ {\inf } ^ {\mathcal {F}} (F _ {n}, \mu - \varepsilon) \geqslant \mathcal {K} _ {\inf } ^ {\mathcal {F}} (F _ {n}, \mu_ {n} + \varepsilon) \geqslant \mathrm {k l} (\mu_ {n}, \mu_ {n} + \varepsilon) \geqslant 2 \varepsilon^ {2},
$$
if the range is  $[0,1]$ . The factor  $\frac{1}{(B - b)^2}$  comes from using this result on rescaled distributions for general ranges.
Then, property 2. can be obtained with a straightforward adaptation of the proof of Lemma 6 of Agrawal et al. [2021] (in their Appendix B.1). Indeed, in this paper the authors consider a non-parametric family of distributions for which  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  can be expressed in a very analogous way to Equation (4).
We introduce the notation  $s_k(t) = \inf \{s \leqslant t : N_k(s) = N_k(t)\}$ , which is simply the time step corresponding to the last pull of arm  $k$  before time  $t$ , and  $k^\star(t) = \operatorname*{argmax}_{k \in [K]} \mu_k(t)$ . We recall Theorem 1.
Proof. Equation (12) can be proved for all the algorithms with the same general proof scheme. Hence, we propose a proof that tackle them all at once, explicitly mentioning which parts are specific to a given algorithm. We start by proving a first general upper bound.
Lemma 3 (Generic regret upper bound). For each sub-optimal arm  $k$  and any  $\varepsilon > 0$ , FMED and FIMED both satisfy
$$
\begin{array}{l} \mathbb {E} \left[ N _ {k} (T) \right] \leqslant u _ {\varepsilon} (T) + \underbrace {\mathbb {E} \left[ \sum_ {t = u} ^ {T - 1} \mathbb {I} \left(A _ {t + 1} = k , N _ {k} (t) > u _ {\varepsilon} (T) , \mathcal {G} _ {k} (t) , \mathcal {J} _ {k} (t)\right) \right]} _ {\text {P o s t - C V}} \\ + \underbrace {\mathbb {E} \left[ \sum_ {t = u} ^ {T - 1} \mathbb {I} (A _ {t + 1} = k , N _ {k} (t) > u , \mu^ {\star} (t) \leqslant \mu_ {1} - \varepsilon_ {0}) \right]} _ {P r e - C V} + \mathcal {O} _ {\varepsilon} (1)  , \\ \end{array}
$$
where  $u_{\varepsilon}(T) = \left\lceil \frac{\log(T)}{\mathcal{K}_{inf}^{\mathcal{F}}(F_k,\mu_1) - \varepsilon}\right\rceil$  and using the notation  $\varepsilon_0 = \frac{B - \mu_1}{4}\varepsilon$  and  $\varepsilon_{1} = \varepsilon_{2} = \frac{\varepsilon}{2}$  we define
$$
\mathcal {J} _ {k} (t) = \{\mathcal {K} _ {i n f} ^ {\mathcal {F}} (F _ {k} (t), \mu_ {1} - \varepsilon_ {0}) \geqslant \mathcal {K} _ {i n f} ^ {\mathcal {F}} (F _ {k}, \mu_ {1}) - \varepsilon_ {1} \},
$$
- For MED and IMED,  $\mathcal{G}_k(t) = \{\mu^*(t) \geqslant \mu_1 - \varepsilon_0\}$ .
- For FMED and FIMED,  $\mathcal{G}_k(t) = \left\{\mu^\star (t)\geqslant \mu_1 - \varepsilon_0,\mu^\star (s_k(t))\geqslant \mu_1 - \varepsilon_0,\frac{\mu^\star(t) - \mu^\star(s_k(t))}{B - \mu^\star(s_k(t))}\geqslant -\varepsilon_2\right\} .$
Proof. Following for instance the regret analysis of the vanilla MED presented in [Baudry et al., 2023], we upper bound the number of pulls by distinguishing several cases,
$$
\begin{array}{l} \mathbb {E} \left[ N _ {k} (T) \right] \leqslant u _ {\varepsilon} (T) + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T - 1} \mathbb {I} \left(A _ {t + 1} = k , N _ {k} (t) > u _ {\varepsilon} (T) , \mathcal {G} (t) , \mathcal {J} _ {k} (t)\right) \right]} _ {\text {P o s t - C V}} \\ + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T - 1} \mathbb {I} \left(A _ {t + 1} = k , N _ {k} (t) > u _ {\varepsilon} (T) , \mu^ {\star} (t) \leqslant \mu_ {1} - \varepsilon_ {0}\right) \right]} _ {\text {P r e - C V}} \\ + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T - 1} \mathbb {I} \left(A _ {t + 1} = k , N _ {k} (t) > u _ {\varepsilon} (T) , \overline {{\mathcal {J}}} _ {k} (t)\right) \right]} _ {\text {C V - E m p}} \\ + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T - 1} \mathbb {I} \left(A _ {t + 1} = k , \mu^ {\star} (t) \geqslant \mu_ {1} - \varepsilon_ {0} , \frac {\mu^ {\star} (t) - \mu^ {\star} \left(s _ {k} (t)\right)}{B - \mu^ {\star} \left(s _ {k} (t)\right)} \leqslant - \varepsilon_ {2}\right) \right]} _ {\text {V a r - B e s t (F M E D a n d F I M E D o n l y)}} \\ + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T - 1} \mathbb {I} \left(A _ {t + 1} = k , \mu^ {\star} (t) \geqslant \mu_ {1} - \varepsilon_ {0} , \mu^ {\star} \left(s _ {k} (t)\right) \leqslant \mu_ {1} - \varepsilon_ {0}\right) \right]} _ {\text {T r a n s i t i o n - B e s t (F M E D a n d F I M E D o n l y)}}. \\ \end{array}
$$
This upper bound already exhibits  $u_{\varepsilon}(T)$ , Post-CV and Pre-CV. It remains to prove that the additional terms are upper bounded by constants depending on  $\varepsilon$ . The term CV-Emp is necessary for all algorithms, while the two last terms are specific to FMED and FIMED. Indeed, they tackle the cases where the sub-optimal arm  $k$  is pulled due to a value of  $\mu^{\star}(s_k(t))$  that deviates from  $\mu_1$  and/or  $\mu^{\star}(t)$ . We start by upper bounding CV-Emp, writing
$$
\begin{array}{l} \operatorname {C V - E m p} \leqslant \sum_ {t = 1} ^ {T - 1} \sum_ {n = u _ {\varepsilon} (T)} ^ {T - 1} \mathbb {E} \left[ \mathbb {I} \left(A _ {t + 1} = k, N _ {k} (t) = n, \overline {{\mathcal {J}}} _ {k, n}\right) \right] \\ \leqslant \sum_ {n = u _ {\varepsilon} (T)} ^ {T - 1} \mathbb {P} \left(\mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k, n}, \mu_ {1} - \varepsilon_ {0}\right) \leqslant \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k}, \mu_ {1}\right) - \varepsilon_ {1}\right) \\ \leqslant \sum_ {n = u _ {\varepsilon} (T)} ^ {T - 1} \mathbb {P} \left(\mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k, n}, \mu_ {1}\right) \leqslant \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k}, \mu_ {1}\right) + \frac {\varepsilon_ {0}}{B - \mu_ {1}} - \varepsilon_ {1}\right), \\ \end{array}
$$
where the last line comes from Lemma 4 of Cappé et al. [2013]. We choose  $\varepsilon_{1} = 2\frac{\varepsilon_{0}}{B - \mu_{1}}$ , and obtain that CV-Emp=  $\mathcal{O}_{\varepsilon_0}(1)$  thanks to property 2. of Lemma 2. This concludes the proof of Lemma 3 regarding MED and IMED. We now consider the terms specific to FMED and FIMED.
We start with Var-Best, and analyse more precisely the implications of the event
$$
\mathcal {B} _ {k} (t) = \left\{\mu^ {\star} (t) \geqslant \mu_ {1} - \varepsilon_ {0}, \frac {\mu^ {\star} (t) - \mu^ {\star} \left(s _ {k} (t)\right)}{B - \mu^ {\star} \left(s _ {k} (t)\right)} \leqslant - \varepsilon_ {2} \right\}.
$$
We first prove that the combination of these two events ensures that  $\mu^{\star}(s_k(t)) > \mu_1$  with a proper tuning of  $\varepsilon_0$  and  $\varepsilon_{2}$ , since
$$
\begin{array}{l} \frac {\mu^ {\star} (t) - \mu^ {\star} \left(s _ {k} (t)\right)}{B - \mu^ {\star} \left(s _ {k} (t)\right)} \leqslant - \varepsilon_ {2} \Rightarrow \mu^ {\star} (t) - \mu^ {\star} \left(s _ {k} (t)\right) \leqslant - \varepsilon_ {2} \left(B - \mu^ {\star} \left(s _ {k} (t)\right)\right) \\ \Rightarrow \mu^ {\star} \left(s _ {k} (t)\right) \geqslant \frac {\mu^ {\star} (t) + \varepsilon_ {2} B}{1 + \varepsilon_ {2}} \geqslant \frac {\mu_ {1} - \varepsilon_ {0} + \varepsilon_ {2} B}{1 + \varepsilon_ {2}} \\ \Rightarrow \mu^ {\star} (s _ {k} (t)) \geqslant \mu_ {1} + \frac {\varepsilon_ {2} (B - \mu_ {1}) - \varepsilon_ {0}}{1 + \varepsilon_ {2}}. \\ \end{array}
$$
Hence, we choose  $\varepsilon_2 = 2\frac{\varepsilon_0}{B - \mu_1}$  to obtain  $\mu^{\star}(s_k(t)) \geqslant \mu_1 + \frac{\varepsilon_0}{3}$ , if it holds that  $\varepsilon_0 \leqslant B - \mu_1$ . Then, we also remark that this event also implies some variation of the best empirical mean between  $s_k(t)$  and  $t$ . Hence, only two scenarios can make  $\mathcal{B}_k(t)$  hold:
-  $k^{\star}(s_k(t))$  is pulled between  $s_k(t)$  and  $t$ , and  $\mu^{\star}(s_k(t)) \geqslant \mu_1 + \frac{\varepsilon_0}{3}$ .
-  $k^{\star}(s_k(t))$  is not pulled between  $s_k(t)$  and  $t$ ,  $k^{\star}(t) = j$  for some  $j \neq k^{\star}(s_k(t))$ . In that case,  $j$  has been pulled between  $s_k(t)$  and  $t$  and it holds that  $\mu_j(t) = \mu^{\star}(t) \geqslant \mu^{\star}(s_k(t)) \geqslant \mu_1 + \frac{\varepsilon_0}{3}$ .
As each of these scenario can cause at most one pull of arm  $k$ , we can upper bound Var-Best by simply counting the number of times an arm  $j \in [K]$  is pulled (at time  $t$ ) and either  $\mu_j(t) \geqslant \mu_1 + \varepsilon_0$  or  $\mu_j(t + 1) \geqslant \mu_1 + \varepsilon_0$ . We hence obtain that
$$
\begin{array}{l} \operatorname {V a r - B e s t} \leqslant \sum_ {j = 1} ^ {K} \sum_ {t = 1} ^ {T - 1} \mathbb {E} \left[ \mathbb {I} \left(A _ {t + 1} = j, \left\{\mu_ {j} (t) \geqslant \mu_ {1} + \frac {\varepsilon_ {0}}{3} \right\} \cup \left\{\mu_ {j} (t + 1) \geqslant \mu_ {1} + \frac {\varepsilon_ {0}}{3} \right\}\right) \right] \\ \leqslant \sum_ {j = 1} ^ {K} \sum_ {t = 1} ^ {T - 1} \sum_ {n = 1} ^ {T - 1} \mathbb {E} \left[ \mathbb {I} \left(A _ {t + 1} = j, N _ {j} (s) = n, \left\{\mu_ {j, n} \geqslant \mu_ {1} + \frac {\varepsilon_ {0}}{3} \right\} \cup \left\{\mu_ {j, n + 1} \geqslant \mu_ {1} + \frac {\varepsilon_ {0}}{3} \right\}\right) \right] \\ \leqslant 2 \sum_ {j = 1} ^ {K} \sum_ {n = 1} ^ {+ \infty} \mathbb {P} \left(\mu_ {j, n} \geqslant \mu_ {1} + \frac {\varepsilon_ {0}}{3}\right) \\ = \mathcal {O} _ {\varepsilon_ {0}} (1). \\ \end{array}
$$
We can apply the exact same arguments for transition-best, except that this time we deduce that  $k^{\star}(t)$  was pulled between  $s_k(t)$  and  $t$ , and that (if  $N_{k^{\star}(t)} = n$ )  $\mu_{j,n} \leqslant \mu_1 - \varepsilon_0$  and  $\mu_{j,n+1} \geqslant \mu_1 + \varepsilon_0$ . Depending on the identity of  $k^{\star}(t)$  one of these two events has exponentially decreasing probability. Formally, we obtain that
$$
\begin{array}{l} \text {T r a n s i t i o n - b e s t} \leqslant \sum_ {j: \mu_ {j} <   \mu_ {1}} \sum_ {n = 1} ^ {+ \infty} \mathbb {P} \left(\mu_ {j, n} \geqslant \mu_ {1} - \varepsilon_ {0}\right) + \sum_ {j: \mu_ {j} = \mu_ {1}} \sum_ {n = 1} ^ {+ \infty} \mathbb {P} \left(\mu_ {j, n} \leqslant \mu_ {1} - \varepsilon_ {0}\right) \\ = \mathcal {O} _ {\varepsilon_ {0}} (1). \\ \end{array}
$$
This concludes the proof of Lemma 3 by choosing  $\varepsilon_0 = \frac{B - \mu_1}{4}\varepsilon$ .
Building on Lemma 3, we finish to prove Theorem 1 by upper bounding the post-convergence and pre-convergence terms for all algorithms.
Upper bounding the post-convergence terms Under  $\mathcal{G}_k(t)$  and  $\mathcal{J}_k(t)$ , it holds thanks to property 1. of Lemma 2 that  $\mathcal{K}_{\inf}^{\mathcal{F}}(F_k(t),\mu^\star (t))\geqslant \mathcal{K}_{\inf}^{\mathcal{F}}(F_k,\mu_1) - \varepsilon$ . Hence, MED satisfies
$$
\begin{array}{l} \operatorname {P o s t - C V} _ {\text {M E D}} \leqslant \mathbb {E} \left[ \sum_ {t = 1} ^ {T - 1} \mathbb {I} (N _ {k} (t) > u _ {\varepsilon} (T), \mathcal {G} _ {k} (t), \mathcal {J} _ {k} (t)) \times p _ {k} (t) \right] \\ \leqslant \mathbb {E} \left[ \sum_ {t = 1} ^ {T - 1} \mathbb {I} \left(N _ {k} (t) > u _ {\varepsilon} (T), \mathcal {G} _ {k} (t), \mathcal {J} _ {k} (t)\right) \exp \left(- N _ {k} (t) \mathcal {K} _ {k} (t)\right) \right] \\ \leqslant T \exp \left(- u _ {\varepsilon} (T) \left(\mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k}, \mu_ {1}\right) - \varepsilon\right)\right) \leqslant 1, \\ \end{array}
$$
by definition of  $u_{\varepsilon}(T)\left[\frac{\log(T)}{\mathcal{K}_{\inf}^{\mathcal{F}}(F_k,\mu_1) - \varepsilon}\right]$ . Similarly, the design of  $\mathcal{G}_k(t)$  for FMED also ensure that  $\mathcal{K}_k(t)\geqslant \mathcal{K}_{\inf}^{\mathcal{F}}(F_k,\mu_1) - \varepsilon$  and Post-CVFMED  $\leqslant 1$  by the same arguments. For IMED, we obtain that