SlowGuess/NeurIPS_2023_OCR huggingface dataset:
term in our statement. The last step of this first generic analysis consists in upper bounding the regret caused by  $Z_{1}(t) = 0$  by a constant. We denote by  $f^{-1}$  the function satisfying  $f^{-1}(f(s)) = s$  for any  $s \in \mathbb{N}$ . We use that the greedy dual is caused by either  $\mu_{\ell_t}(t) \geqslant \mu_{\max}$  or  $\widetilde{N}_{1,\ell_t}(t) \geqslant f^{-1}(t / K)$ ,
$$
\begin{array}{l} D _ {1} := \sum_ {t = 1} ^ {T - 1} \mathbb {E} \left[ \mathbb {I} \left(1 \notin \mathcal {A} _ {t + 1}, \ell_ {t} \neq 1, Z _ {1} (t) = 0\right) \right] \\ \leqslant \sum_ {t = 1} ^ {T - 1} \mathbb {E} \left[ \mathbb {I} \left(\cup_ {j = 2} ^ {K} \left\{\mu_ {1} (t) \leqslant x _ {j} \cup \mu_ {j} (t) \geqslant x _ {j}, \widetilde {N} _ {1, j} (t) \geqslant f ^ {- 1} (t / K), N _ {j} (t) \geqslant t / K \right\}\right) \right] \\ + \sum_ {t = 1} ^ {T - 1} \mathbb {E} \left[ \mathbb {I} \left(\cup_ {j = 2} ^ {K} \left\{\mu_ {j} (t) \geqslant \mu_ {\max } \right\}, \ell_ {t} = j\right) \right] \\ \leqslant \sum_ {t = 1} ^ {T - 1} \mathbb {E} \left[ \mathbb {I} \left(\mu_ {1} (t) \leqslant x _ {j}, N _ {1} (t) \geqslant f ^ {- 1} (t / K)\right) \right] + 2 \sum_ {j = 2} ^ {K} \mathbb {E} \left[ \mathbb {I} \left(\mu_ {j} (t) \geqslant x _ {j}, N _ {j} (t) \geqslant t / K\right) \right] \\ \leqslant \sum_ {t = 1} ^ {T - 1} \sum_ {n = f ^ {- 1} (t / K)} ^ {t} \mathbb {P} \left(\mu_ {1, n} \leqslant x _ {j}\right) + 2 \sum_ {j = 2} ^ {K} \sum_ {t = 1} ^ {T - 1} \sum_ {n _ {j} = t / K} ^ {t} \mathbb {P} \left(\mu_ {j, n} \geqslant x _ {j}\right) = \mathcal {O} (1), \\ \end{array}
$$
where we grouped upper bounded the terms corresponding to  $\mu_j(t) \geqslant \mu_{\mathrm{max}}$  by the terms corresponding to  $\mu_j(t) \geqslant x_j$  for simplicity. Finally, the remaining term of our upper bound is exactly
$$
\operatorname {P r e - C V} := 4 \log (4) \sum_ {t = 1} ^ {T - 1} \mathbb {E} \left[ \mathbb {I} \left(1 \notin \mathcal {A} _ {t + 1}, \ell_ {t} \neq 1, Z _ {1} (t) = 1\right) \right],
$$
as stated in the Lemma 4. This concludes the proof of the lemma,
$$
\mathbb {E} \left[ N _ {k} (T) \right] \leqslant u _ {\varepsilon} (T) + \text {P o s t - C V} + \text {P r e - C V} + \mathcal {O} _ {\varepsilon} (1).
$$
We now prove the following lemma, that concludes the proof of Theorem 2.
Lemma 5. OMED and OIMED both satisfy
$$
\text {P o s t - C V} = \mathcal {O} _ {\varepsilon} (1) \quad \text {a n d} \quad \text {P r e - C V} = \mathcal {O} (1).
$$
Proof. We need to upper bound four terms. We start with the upper bounds of the two post-convergence terms, that are straightforward thanks to the tuning of  $u_{\varepsilon}(T)$  and the definition of  $\mathcal{G}_k(t)$ .
Upper bounding Post-CV We start with OMED,
$$
\begin{array}{l} \operatorname {P o s t - C V} _ {\text {O M E D}} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {I} (k \in \mathcal {A} _ {t + 1}, \ell_ {t} = 1, \widetilde {N} _ {k, 1} (t) > u _ {\varepsilon} (T), Z _ {k} (t) = 1, \mathcal {G} _ {k} (t)) \right] \\ \leqslant \left[ \sum_ {t = 1} ^ {T} e ^ {- \widetilde {N} _ {k, 1} (t) \left(\mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k}, \mu_ {1}\right) - \varepsilon\right)} \mathbb {I} \left(\widetilde {N} _ {k, 1} (t) > u _ {\varepsilon} (T), Z _ {k} (t) = 1, \mathcal {G} _ {k} (t)\right) \right] \\ \leqslant T e ^ {- u _ {\varepsilon} (T) \left(\mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k}, \mu_ {1}\right) - \varepsilon\right)} \\ \leqslant 1, \\ \end{array}
$$
thanks to the definition of  $u_{\varepsilon}(T)$ . For 0IMED, we obtain
$$
\begin{array}{l} \operatorname {P o s t - C V} _ {\text {O I M E D}} \leqslant \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \mathbb {I} (k \in \mathcal {A} _ {t + 1}, \ell_ {t} = 1, \widetilde {N} _ {k, 1} (t) > u _ {\varepsilon} (T), Z _ {k} (t) = 1, \mathcal {G} _ {k} (t)) \right] \\ \leqslant \left[ \sum_ {t = 1} ^ {T} \mathbb {I} \left(\widetilde {N} _ {k, 1} (t) \left(\mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k}, \mu_ {1}\right) - \varepsilon\right) <   \log (T), \widetilde {N} _ {k, 1} (t) > u _ {\varepsilon} (T)\right) \right] \\ = 0, \\ \end{array}
$$
again thanks to the definition of  $u_{\varepsilon}(T)$ .
Upper bounding Pre-CV, OMED We recall that
$$
\operatorname {P r e - C V} _ {\text {O M E D}} := \sum_ {t = 1} ^ {T - 1} \mathbb {E} \left[ \mathbb {I} \left(1 \notin \mathcal {A} _ {t + 1}, \ell_ {t} \neq 1, \widetilde {N} _ {1, \ell_ {t}} (t) \leqslant f ^ {- 1} \left(N _ {j} (t)\right)\right) \right].
$$
We use the notation  $p_1(t) = p_{1,j,\widetilde{N}_{1,\ell_t}(t)}$ , and  $p_{1,j,n} = e^{-L_{1,j,n}}$ . Then, with the same arguments as for the regret analysis of the vanilla MED (Baudry et al. [2023], Appendix B) we obtain that
$$
\begin{array}{l} \operatorname {P r e - C V} _ {\text {O M E D}} = \sum_ {t = 1} ^ {T - 1} \mathbb {E} \left[ \mathbb {I} \left(\ell_ {t} \neq 1, \widetilde {N} _ {1, \ell_ {t}} (t) \leqslant f ^ {- 1} \left(N _ {j} (t)\right)\right) (1 - p _ {1} (t)) \right] \\ = \sum_ {t = 1} ^ {T - 1} \mathbb {E} \left[ \mathbb {I} \left(1 \in \mathcal {A} _ {t + 1}, \ell_ {t} \neq 1, \widetilde {N} _ {1, \ell_ {t}} (t) \leqslant f ^ {- 1} \left(N _ {j} (t)\right)\right) \frac {1 - p _ {1} (t)}{p _ {1} (t)} \right] \\ \leqslant \sum_ {j = 2} ^ {K} \sum_ {n = 1} ^ {T} \mathbb {E} \left[ \frac {1}{p _ {1 , j , n}} - 1 \right] \\ := \sum_ {j = 2} ^ {K} \sum_ {n = 1} ^ {T} \mathbb {E} \left[ e ^ {L _ {1, j, n}} - 1 \right]. \\ \end{array}
$$
Now, thanks to Equation (14) we can relate  $L_{1,j,n}$  to  $\mathcal{K}_{\inf}^{\mathcal{F}}(F_n,x)$  for any  $x\in \mathbb{R}$ . We again choose  $x_{j}$  for convenience, and obtain that
$$
\operatorname {P r e - C V} _ {\mathbf {0 M E D}} \leqslant \sum_ {j = 2} ^ {K} \sum_ {n = 1} ^ {T} \mathbb {E} \left[ e ^ {(n \mathcal {K} _ {\inf } ^ {\mathcal {F}} (\widetilde {F} _ {1, j, n}, x _ {j}) - R _ {1, j, n} - B _ {1, j, n} (x _ {j})) _ {+}} - 1 \right].
$$
To relate to the proof for MED we need the portfolio regret and bias to be small enough. We use the second side of our assumption on the regret, which is that  $-R_{1,j,n} = o(n)$  for any  $j,n$ . Hence, for  $n$  large enough it holds for instance that  $-R_{1,j,n} \leqslant \frac{\delta_j}{3} n$ , with
$$
\delta_ {j} = \inf _ {F \in \mathcal {F}} \mathcal {K} _ {\inf} ^ {\mathcal {F}} (F, \mu_ {1}) - \mathcal {K} _ {\inf} ^ {\mathcal {F}} (F, x _ {j}) > 0,
$$
where  $\delta_j > 0$  is ensured by property 3. of Lemma 2. For the bias, we first obtain with Lemma 1 that
$$
- B _ {1, j, n} \left(x _ {j}\right) \leqslant \sum_ {i = 1} ^ {n} \mathbb {I} \left(\mu_ {j} \left(t _ {1, j, n}\right) \geqslant x _ {j}\right).
$$
We then define a "good event" under which the bias is controlled,
$$
\mathcal {B} _ {j, n} = \left\{\sum_ {i = 1} ^ {n} \mathbb {I} \left(\mu_ {j} \left(t _ {1, j, n}\right) \geqslant x _ {j}\right) \leqslant n \frac {\delta_ {j}}{3} \right\}.
$$
We then use a similar proof as for the post-convergence term. We first consider
$$
P _ {1} := \sum_ {j = 2} ^ {K} \sum_ {n = 1} ^ {T} \mathbb {E} \left[ \left(e ^ {L _ {1, j, n}} - 1\right) \mathbb {I} (\mathcal {B} _ {j, n}) \right] = \sum_ {j = 2} ^ {K} \sum_ {n = 1} ^ {T} \mathbb {E} \left[ e ^ {n \mathcal {K} _ {\inf} ^ {\mathcal {F}} (\widetilde {F} _ {1, j, n}, x _ {j}) + n \frac {2 \delta_ {j}}{3}} - 1 \right],
$$
that can be upper bounded with the same proof as the vanilla MED from [Baudry et al., 2023]. Using properties 3. and 4. of Lemma 2, we obtain that
$$
\begin{array}{l} P _ {1} \leqslant \sum_ {j = 2} ^ {K} \sum_ {n = 1} ^ {T} \int_ {0} ^ {e ^ {n \mathcal {K} _ {j} ^ {+} + \frac {2 \delta_ {j}}{3}} - 1} \mathbb {P} \left(\mathcal {K} _ {\inf } ^ {\mathcal {F}} (\widetilde {F} _ {1, j, n}, x _ {j}) > \frac {\delta_ {j}}{3} + \frac {1}{n} \log (1 + x)\right) \mathrm {d} x \\ \leqslant \sum_ {j = 2} ^ {K} \sum_ {n = 1} ^ {T} e n (n + 1) \left(\mathcal {K} _ {j} ^ {+} + \frac {2 \delta_ {j}}{3}\right) e ^ {- n \frac {\delta_ {j}}{3}} \\ = \mathcal {O} (1), \\ \end{array}
$$
with  $\mathcal{K}_j^+ = \log \left(\frac{B - b}{B - x_j}\right)$ . We then tackle the case where  $\mathcal{B}_{j,n}$  does not hold, which is possible only if  $\mu_{j,s} \geqslant x_{j}$  for some  $s \geqslant (n\delta_j)^2$ . Furthermore, we also use the trivial bound  $-B_{1,j,n}(x_j) \leqslant n$  thanks to Lemma 1. We then obtain
$$
\begin{array}{l} P _ {2} := \sum_ {j = 2} ^ {K} \sum_ {n = 1} ^ {T} \mathbb {E} \left[ \left(e ^ {\left(n K _ {\inf } ^ {\mathcal {F}} \left(\widetilde {F} _ {1, j, n}, x _ {j}\right) - R _ {1, j, n} - B _ {1, j, n} \left(x _ {j}\right)\right) _ {+}} - 1\right) \mathbb {I} (\bar {\mathcal {B}} _ {j, n}) \right] \\ \leqslant \sum_ {j = 2} ^ {K} \sum_ {n = 1} ^ {T} \mathbb {E} \left[ \left(e ^ {n \mathcal {K} _ {\inf } ^ {\mathcal {F}} (\widetilde {F} _ {1, j, n}, x _ {j}) + n (1 + \delta_ {j})} - 1\right) \mathbb {I} (\exists s \geqslant f (n \delta_ {j}): \mu_ {j, s} \geqslant x _ {j}) \right] \\ \leqslant \sum_ {j = 2} ^ {K} \sum_ {n = 1} ^ {T} \mathbb {E} \left[ \left(e ^ {n \mathcal {K} _ {\inf } ^ {\mathcal {F}} (\tilde {F} _ {1, j, n}, x _ {j}) + n (1 + \delta_ {j})} - 1\right) \frac {e ^ {- f (n \delta_ {j}) I _ {1} (x _ {j})}}{1 - e ^ {- I _ {1} (x _ {j})}} \right], \\ \end{array}
$$
We can conclude at this step using that  $e^{n(1 + \delta_j) - f(n\delta_j)I_1(x_j)} \to 0$ , as we ensured that  $n = o(f(n))$ , so it simply holds that
$$
P _ {1} = \mathcal {O} (1) \Rightarrow P _ {2} = \mathcal {O} (1).
$$
Upper bounding Pre-CV, 0IMED We again consider the two cases, depending on if  $\mathcal{B}(t)$  holds or not. When this is true, we can use a similar proof to the one of vanilla IMED that we propose in Appendix B,
$$
\begin{array}{l} Q _ {1} := \sum_ {j = 2} ^ {K} \sum_ {t = 1} ^ {T - 1} \mathbb {E} \left[ \mathbb {I} \left(\widetilde {N} _ {1, j} (t) \mathcal {K} _ {\inf } ^ {\mathcal {F}} (\widetilde {F} _ {1, j, n}, x _ {j}) + \log (\widetilde {N} _ {1, j} (t)) \geqslant \log (N _ {j} (t)) - \widetilde {N} _ {1, j} (t) \frac {2 \delta_ {j}}{3}, N _ {j} (t) \geqslant t / K\right) \right] \\ \leqslant \sum_ {j = 2} ^ {K} \sum_ {t = 1} ^ {T - 1} \sum_ {n = \log (t / K) / \mathcal {K} ^ {+}} ^ {T - 1} \mathbb {P} \left(n \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(\widetilde {F} _ {1, j, n}, x _ {j}\right) \geqslant \log (t / K) - n \frac {2 \delta_ {j}}{3} - \log (n)\right) \\ \leqslant \sum_ {j = 2} ^ {K} \sum_ {t = 1} ^ {T - 1} \sum_ {n = \log (t / K) / \mathcal {K} ^ {+}} ^ {T - 1} \frac {\operatorname {K e n} (n + 2)}{t} e ^ {- n \frac {\delta_ {j}}{3}} \\ \leqslant K ^ {2} e \sum_ {t = 1} ^ {T - 1} \sum_ {n = \log (t / K) / \mathcal {K} ^ {+}} ^ {T - 1} \frac {n (n + 2)}{t} e ^ {- n \frac {\delta_ {j}}{3}}, \\ \end{array}
$$
where we again used properties 3. and 4. of Lemma 2 to upper bound the probability. The convergence of this term is ensured by the fact that the sum on  $n$  starts at  $\frac{\log(t / K)}{K^+}$ . To formally prove this we can for instance use that there exists a constant  $C_j$  (that typically scales in  $\mathcal{O}(\delta_j^{-2})$  up to logarithm terms) such that  $\forall n \in \mathbb{N}, n(n + 2) \leqslant C_j e^{n\frac{\delta_j}{6}}$  for any  $n \in \mathbb{N}$ , so
$$
\begin{array}{l} Q _ {1} \leqslant C _ {j} K ^ {2} \sum_ {t = 1} ^ {T - 1} \sum_ {n = \log (t / K) / \mathcal {K} ^ {+}} ^ {T - 1} \frac {e ^ {- n \frac {\delta_ {j}}{6}}}{t} \\ \leqslant C _ {j} K ^ {2 + \frac {\delta_ {j}}{3 \kappa^ {+}}} \frac {1}{1 - e ^ {- \frac {\delta_ {j}}{6}}} \sum_ {t = 1} ^ {T - 1} \frac {1}{t ^ {1 + \frac {\delta_ {j}}{6 \kappa^ {+}}}} \\ = \mathcal {O} (1). \\ \end{array}
$$
We then consider  $\vec{B}(t)$ , under which the bias can be up to  $\widetilde{N}_{1,j}(t)$ . In that case, we use the same proof techniques as for OMED, with
$$
\begin{array}{l} Q _ {2} := \sum_ {j = 2} ^ {K} \sum_ {t = 1} ^ {T - 1} \mathbb {E} \left[ \mathbb {I} (\widetilde {N} _ {1, j} (t) \mathcal {K} _ {\inf } ^ {\mathcal {F}} (F _ {1, j, n}, x _ {j}) \geqslant \log (N _ {j} (T)) - \widetilde {N} _ {1, j} (t) (1 + \delta_ {j}), N _ {j} (t) \geqslant t / K, \bar {\mathcal {B}} (t)) \right] \\ \leqslant \sum_ {j = 2} ^ {K} \sum_ {t = 1} ^ {T - 1} \sum_ {n = \log (t / K) / \mathcal {K} ^ {+}} ^ {T - 1} \frac {K}{t} e ^ {- n \delta_ {j}} \times \frac {e ^ {n \left(1 + \frac {\delta_ {j}}{3}\right) - (n \delta_ {j}) ^ {2} I _ {1} (x _ {j})}}{1 - e ^ {- I _ {1} (x _ {j})}}. \\ \end{array}
$$
We then use that the right-hand term converges to 0, so  $Q_{1} = \mathcal{O}(1) \Rightarrow Q_{2} = (1)$ .
Remark 2. The enforcement of a sufficient sample size for the leader is justified by the analysis of the pre-convergence term. In this regime a linear bias may cost up to  $e^n$  for each sample size  $n$ . Hence, a concentration of the mean with a rate  $e^{-cn}$  is not sufficient, especially if  $c$  is small (which happens if the gaps are small). Furthermore, this kind of rate cannot be avoided without actively controlling the sample size of the leader at each update.
D Accuracy of  $\kappa_{\mathrm{inf}}^{\mathcal{F}}$  approximation and discretization trick
In this section we provide more details on the discussion of Section 1. In particular, we elaborate on the discretization trick proposed in Riou and Honda [2020] and on the sub-optimality of the algorithms that do not target  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  specifically.
For simplicity we assume that  $b = 0$  and  $B = 1$ , and all results hold for general ranges  $[b, B] \subset \mathbb{R}$  up to a rescaling of the rewards:  $X \in [b, B] \Rightarrow \frac{X - b}{B - b} \in [0, 1]$ .
Sub-optimality of  $2\Delta^2$  and  $\mathrm{kl}$  We denote by  $\mathrm{kl}:[0,1]\times [0,1)\to \mathbb{R}^{+}$  the KL divergence between two Bernoulli distributions, expressed in terms of their expectations by
$$
\forall \mu^ {\star} \geqslant \mu : \quad \operatorname {k l} (\mu , \mu^ {\star}) := \mu \log \left(\frac {\mu}{\mu^ {\star}}\right) + (1 - \mu) \log \left(\frac {1 - \mu}{1 - \mu^ {\star}}\right). \tag {16}
$$
Consider  $F \in \mathcal{F}$  of mean  $\mu \in [0,1)$  and  $\mu^{\star} \in [\mu,1)$ , and let us define  $\Delta = \mu^{\star} - \mu$ . First, it is known (see e.g. Cappé et al. [2013]) that
$$
\mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F \mu^ {\star}\right) \geqslant \mathrm {k l} \left(\mu , \mu^ {\star}\right) \geqslant 2 \Delta^ {2}. \tag {17}
$$
This result (partially) explains why bandit algorithms adapted for 1/4-sub-gaussian or Bernoulli distributions work for general bounded distributions (UCB1 [Auer et al., 2002], kl-UCB [Cappé et al., 2013]), since a smaller constant means a more exploratory algorithm. A natural question is to determine how sub-optimal these algorithms can be in the general case. In the following we assume that  $\Delta$  is small to perform some first-order approximations.
First, a simple computation shows that  $\mathrm{kl}(\mu ,\mu^{\star})\approx \frac{\Delta^2}{\mu(1 - \mu)}$  : kl adapts to the true variance of the Bernoulli distributions, exploiting better the geometry of  $\mathcal{F}$ . The sub-gaussian approximation is tight
only if  $\mu$  is close to  $1/2$  but not near the boundaries of the support, which is not surprising. We now investigate the gap between  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  and  $\mathrm{kl}$ , and recall that the two quantities match for Bernoulli distributions. The following result demonstrates that the ratio between the two can be arbitrarily large if the distributions are in fact "far" from being Bernoulli.
Lemma 6 (kl vs  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$ ). For  $\mu \in (0,1)$  and  $F \in \mathcal{F}_{[0,1]}$  of mean  $\mu - \Delta$ , the scaling of  $\frac{\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}(F,\mu)}{\mathrm{kl}(\mu_F,\mu)}$  can be as large as  $\frac{\mu}{\Delta}$ , with the maximum achieved when  $F$  is the Dirac distribution with support  $\mu - \Delta$ .
Proof. First, Jensen inequality and equation (4) provide that
$$
\mathcal {K} _ {\inf } ^ {\mathcal {F}} (F, \mu) \leqslant \log \left(1 + \frac {\Delta}{1 - \mu}\right) = \mathcal {K} _ {\inf } ^ {\mathcal {F}} (\delta_ {\mu - \Delta}, \mu),
$$
where  $\delta_x$  is the Dirac distribution at  $x$ . Then, approximating  $\mathcal{K}_{\inf}^{\mathcal{F}}(F,\mu)$  by  $\frac{\Delta}{1 - \mu}$  we obtain that
$$
\text {f o r s m a l l} \Delta > 0, \frac {\mathcal {K} _ {\inf } ^ {\mathcal {F}} (\delta_ {\mu - \Delta} , \mu)}{\mathrm {k l} (\mu - \Delta , \mu)} \approx \left(\frac {\Delta}{1 - \mu}\right) \times \left(\frac {\mu (1 - \mu)}{\Delta^ {2}}\right) = \frac {\mu}{\Delta} \underset {\Delta \rightarrow 0} {\rightarrow} + \infty .
$$
We could argue that Dirac (or highly concentrated) distributions may be unlikely in most applications, however the use-case presented in Section 1 (Figure 1) shows that large ratios  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}} / \mathrm{kl}$  may already be observed with more natural examples. Finally, to compare the impact of the approximation on the asymptotic guarantees of an algorithm on a  $K$ -armed bandits we can compare  $\sum_{k:\Delta_k > 0} \frac{\Delta_k}{\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}(F_k,\mu^\star)}$  and  $\sum_{k:\Delta_k > 0} \frac{1}{\Delta_k}$ .
Discretization trick We now detail the trick proposed by Riou and Honda [2020], and inspired for instance by the binarization trick of Agrawal and Goyal [2012]. For any  $M \in \mathbb{N}$ , we fix an arbitrary grid  $\mathcal{X}_M = (x_1 = 0 < x_2 < \dots < x_M = 1) \in [0,1]^M$ . Then, the trick consists in transforming any incoming reward  $X \notin \mathcal{X}_M$  in a discrete reward  $Y \in \mathcal{X}_M$ , as follows:
1. Let  $m \in \mathbb{N}$  satisfy  $x_{m} < X < x_{m+1}$ , and define  $p = \frac{X - x_{m}}{x_{m+1} - x_{m}}$ .
2. Sample  $Z \sim \operatorname{Ber}(p)$  and define  $Y = x_{m}\mathbb{I}(Z = 0) + x_{m + 1}\mathbb{I}(Z = 1)$ .
If this procedure is used with all rewards collected, one can use a bandit algorithm calibrated for multinomial distributions with support  $\mathcal{X}_m$ . It is easy to see that  $\mathbb{E}[Y] = X$ , so the order of the arms do not change with this discretization. Furthermore, the bandit problem becomes equivalent to a problem where the distributions would be exactly  $(F_1^{\mathcal{X}_M},\dots ,F_K^{\mathcal{X}_M})\in \mathcal{F}^K$  where, for each  $k\in [K]$ ,  $F_{k}^{\mathcal{X}_{M}}$  is the distribution of the discretized rewards (with reference grid  $\mathcal{X}_M$ ) generated with i.i.d. samples from  $F_{k}$ . It is hence clear that an asymptotically optimal bandit algorithm for (general bounded distributions) is guaranteed an asymptotic regret scaling in  $\sum_{k:\Delta_k > 0}\frac{\Delta_k}{\mathcal{K}_{\inf}^{\mathcal{F}}(F_k^{\mathcal{X}_M})}\log (T)$ , and be optimal if the distributions  $F_{1},\ldots ,F_{K}$  are already supported on  $\mathcal{X}_M$  only.
We now discuss the performance that can be achieved with such discretization. Let us assume that a constant step  $x_{m + 1} - x_m = \frac{B - b}{M - 1}$  is chosen for all  $m\in [M - 1]$ . First, due to the compactness of  $\mathcal{F}$  we can state that there exists a distribution  $G_{\star}$  satisfying  $\mathcal{K}_{\inf}^{\mathcal{F}}(F_k,\mu^\star) = \mathrm{KL}(F_k,G_\star)$ . We then denote by  $G_{\star}^{\mathcal{X}_M}$  its corresponding discretized distribution on the grid  $\mathcal{X}_M$ . Thanks to the data processing inequality, we obtain that
$$
\mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k}, \mu^ {\star}\right) = \operatorname {K L} \left(F _ {k}, G _ {\star}\right) \geqslant \operatorname {K L} \left(F _ {k} ^ {\mathcal {X} _ {M}}, G _ {\star} ^ {\mathcal {X} _ {M}}\right),
$$
which is itself larger than  $\mathcal{K}_{\inf}^{\mathcal{F}}(F_k^{\mathcal{X}_M},\mu^\star)$  by definition, since  $\mathbb{E}_{X\sim G_{\star}^{\mathcal{X}_M}}[X] = \mathbb{E}_{X\sim G_{\star}}[X]\geqslant \mu^{\star}$ . Finally, using that  $F_{k}^{\mathcal{X}_{M}}$  cannot be "closer" to  $\mu^{\star}$  than a translation of  $F_{k}$  by  $\eta$ , and Lemma 4 of Cappé et al. [2013], we obtain that
$$
\begin{array}{l} \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k}, \mu^ {\star}\right) \geqslant \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k} ^ {\mathcal {X} _ {M}}, \mu^ {\star}\right) \\ \geqslant \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k}, \mu^ {\star} - \eta\right) \geqslant \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k}, \mu^ {\star}\right) - \frac {\eta}{B - \mu^ {\star}} \\ = \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k}, \mu^ {\star}\right) - \frac {1}{M - 1} \times \frac {B - b}{B - \mu^ {\star}}. \\ \end{array}
$$
Hence, we see that the approximation can be good for large  $M$ . However, if  $\mathcal{K}_{\inf}^{\mathcal{F}}(F_k,\mu^\star) \leqslant \frac{1}{M - 1} \times \frac{B - b}{B - \mu^\star}$  (which is possible since  $M$  is fixed without knowing the distributions) then the lower bound is vacuous. In fact, we can extend Lemma 6 to any discretization process: small gaps (compared to  $M^{-1}$ ) can make the ratio  $\mathcal{K}_{\inf}^{\mathcal{F}}(F_k,\mu^\star) / \mathcal{K}_{\inf}^{\mathcal{F}}(F_k^{\mathcal{X}_m},\mu)$  arbitrarily large. Hence, even if this technique has some convenient properties, it is not entirely satisfying. This justifies our quest for alternatives that would preserve the asymptotic optimality of the algorithms.
E Online  $\kappa_{\mathrm{inf}}^{\mathcal{F}}$  computation with portfolio optimization algorithms
In this section we provide additional details regarding OMED and OIMED. We first prove Lemma 1, that is essential to their regret analysis, then we detail the Soft-Bayes algorithm [Orseau et al., 2017] that we choose in our practical implementation of the algorithms.
E.1 Proof of Lemma 1
Proof. We introduce some notation for convenience. First, we denote  $X_{k,\ell,i}$  by  $X_i$  and  $\mu^{\star}(t_{k,\ell,i})$  by  $\mu^{\star}(i)$ . Then, we define  $Y_i = \frac{X_i - \mu^{\star}(i)}{B - \mu^{\star}(i)}$ ,  $Z_i = \frac{X_i - \mu}{B - \mu}$ , and  $\lambda = \operatorname*{argmax}_{\lambda \in [0,1]} \sum_{i=1}^{n} \log(1 - \lambda Z_i)$ .
We use two elementary analysis properties. First,  $\frac{1 - \lambda Z_i}{1 - \lambda Y_i}$  is positive and non-decreasing if  $Y_{i} \geqslant Z_{i}$ . Then,  $Y_{i} \geqslant Z_{i}$  only if  $\mu^{\star}(i) \leqslant \mu$ . Otherwise,  $\log (1 - \lambda Z_i) - \log (1 - \lambda Y_i) \leqslant 0$ . Hence, it holds that
$$
\begin{array}{l} B _ {k, \ell , n} (\mu) := \sum_ {i = 1} ^ {n} \left(\log \left(1 - \lambda Z _ {i}\right) - \log \left(1 - \lambda Y _ {i}\right)\right) = \sum_ {i = 1} ^ {n} \log \left(\frac {1 - \lambda Z _ {i}}{1 - \lambda Y _ {i}}\right) \\ \leqslant \sum_ {i = 1} ^ {n} \log \left(\frac {1 - Z _ {i}}{1 - Y _ {i}}\right) \mathbb {I} (\mu \geqslant \mu^ {\star} (i)). \\ \end{array}
$$
We then plug the expression of  $Y_{i}$  and  $Z_{i}$  in this bound, obtaining that
$$
B _ {k, \ell , n} (\mu) \leqslant \sum_ {i = 1} ^ {n} \log \left(\frac {\frac {B - X _ {i}}{B - \mu}}{\frac {B - X _ {i}}{B - \mu^ {\star} (i)}}\right) \mathbb {I} (\mu \geqslant \mu^ {\star} (i)) = \sum_ {i = 1} ^ {n} \log \left(\frac {B - \mu^ {\star} (i)}{B - \mu}\right) \mathbb {I} (\mu \geqslant \mu^ {\star} (i)),
$$
which gives the result. Applying the exact same steps provides the other direction,
$$
- B _ {k, \ell , n} \leqslant \sum_ {i = 1} ^ {n} \log \left(\frac {B - \mu}{B - \mu^ {\star} (i)}\right) \mathbb {I} (\mu^ {\star} (i) \geqslant \mu)
$$
E.2 Soft-Bayes
In this section we detail the Soft-Bayes algorithm, proposed in Orseau et al. [2017]. We recall that the  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  estimation is a portfolio optimization problem of dimension 2. At each step  $n$ , the portfolio algorithm decides an allocation  $(1 - \lambda_n, \lambda_n)$  between two assets, that provide a payoff  $(1, Y_n)$ . We use in this section  $Y_n$  to denote for simplification  $\frac{B - X_{k,\ell,n}}{B - \mu^*(t_{k,\ell,n})}$ , that is used in our implementation of the estimate  $L_{k,\ell,n}$  for the challenger/leader pair  $(k,\ell)$ .
For the anytime version of the algorithm, a sequence of learning rates  $(\eta_{n})_{n\in \mathbb{N}}$  is provided as an input of the algorithm. We then define the update rule of the anytime Soft-Bayes in Algorithm 10 below.