SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dorian Baudry
Ecole Polytechnique, CREST
Palaiseau, France
dorian.baudry@ensae.fr
Fabien Pesquerel
Univ. Lille, Inria, CNRS, Centrale Lille,
UMR 9189-CRISTAL, F-59000 Lille, France
fabien.pesquerel@inria.fr
Odalric-Ambrym Maillard
odalric.maillard@inria.fr
We consider the problem of regret minimization in non-parametric stochastic bandits. When the rewards are known to be bounded from above, there exists asymptotically optimal algorithms, with asymptotic regret depending on an infimum of Kullback-Leibler divergences (KL). These algorithms are computationally expensive and require storing all past rewards, thus simpler but non-optimal algorithms are often used instead. We introduce several methods to approximate the infimum KL which reduce drastically the computational and memory costs of existing optimal algorithms, while keeping their regret guaranties. We apply our findings to design new variants of the MED and IMED algorithms, and demonstrate their interest with extensive numerical simulations.
A Multi-Armed Bandit (MAB) is a sequential decision-making problem where at each time step  $t \in \mathbb{N}$  a learner collects a reward from an arm  $A_{t}$  chosen among  $K \in \mathbb{N}$  alternatives. We consider the stochastic case, in which all rewards collected from an arm  $k \in [K]$  are i.i.d. and drawn from a fixed distribution  $F_{k}$ , of expectation  $\mu_{k}$ . For a time horizon  $T$  we define the number of pulls of an arm  $k$  by  $N_{k}(T) = \sum_{t=1}^{T} \mathbb{I}(A_{t} = k)$ , and their sub-optimality gap by  $\Delta_{k} = \mu^{\star} - \mu_{k}$ , with  $\mu^{\star} = \max_{k \in [K]} \mu_{k}$ . The goal of the learner is to maximize their expected sum of rewards, or equivalently to minimize the regret, defined as
$$
\mathcal {R} _ {T} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left(\mu^ {\star} - \mu_ {A _ {t}}\right) \right] = \sum_ {k = 1} ^ {K} \Delta_ {k} \mathbb {E} \left[ N _ {k} (T) \right]. \tag {1}
$$
To achieve this goal, the learner can leverage the rewards collected up to time  $t$  and their knowledge on the family of distributions  $\mathcal{F}$  to which  $F_{1},\ldots ,F_{K}$  belong. The definition of  $\mathcal{F}$  determines the complexity of the bandit problem, since any uniformly efficient algorithm satisfies the lower bound [Lai and Robbins, 1985, Burnetas and Katehakis, 1996]: for all  $k\in [K]$  with  $\Delta_k > 0$ ,
$$
\lim  _ {T \rightarrow \infty} \inf  _ {\log (T)} \frac {\mathbb {E} [ N _ {k} (T) ]}{\log (T)} \geqslant \frac {1}{\mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k} , \mu^ {\star}\right)}, \quad \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k}, \mu^ {\star}\right) = \inf  _ {G \in \mathcal {F}} \left\{\mathrm {K L} \left(F _ {k}, G\right): \mathbb {E} _ {G} [ X ] > \mu^ {\star} \right\}, \tag {2}
$$
$\forall (F_{1},\ldots ,F_{K})\in \mathcal{F}^{K}\colon \forall \alpha >0,\mathbb{E}[N_{k}(T)] = o(T^{\alpha})$  for all  $k$  satisfying  $\Delta_k > 0$
where  $\mathrm{KL}(.,.)$  denotes the Kullback-Leibler divergence between two distributions. We call an algorithm asymptotically optimal (sometimes shortened to optimal) if its regret upper bound matches this lower bound. That is, if for all  $k$ ,  $\lim \sup_{T\to \infty}\mathbb{E}[N_k(T)] / \log (T)\leq 1 / \mathcal{K}_{\inf}^{\mathcal{F}}(F_k,\mu^\star)$ .
For some parametric families (e.g. Gaussian),  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  has a convenient closed-form expression. However, for more general non-parametric families of distributions this may not be the case. In this work we consider the most famous example of such model, where the learner only knows that the distributions are bounded. Given a range  $[b,B]$ , we formally define
$$
\mathcal {F} _ {[ b, B ]} = \{F \in \mathcal {P} (\mathbb {R}): \operatorname {s u p p} (F) \subset [ b, B ] \subset \mathbb {R} \}, \tag {3}
$$
but in the rest of the paper we keep the notation  $\mathcal{F}$  (for  $\mathcal{F}_{[b,B]}$ ) for simplicity. We always assume that  $B$  is known, but in some cases knowing that  $b$  is finite will be sufficient. By analyzing a dual optimization problem derived from (2), Honda and Takemura [2010] obtained that
$$
\forall F \in \mathcal {F}, \mu \leqslant B: \quad \mathcal {K} _ {\inf } ^ {\mathcal {F}} (F, \mu) = \max  _ {\lambda \in \left[ 0, \frac {1}{B - \mu} \right]} \mathbb {E} _ {X \sim F} [ \log (1 - \lambda (X - \mu)) ]. \tag {4}
$$
Optimal algorithms for bounded distributions The literature on MABs has become diverse in the past years, so in the following we present a non-exhaustive selection of works that focus on asymptotically optimal algorithms for (bounded) stochastic bandits. We refer the reader to [Lattimore and Szepesvári, 2020, Bubeck et al., 2012] for broader surveys.
The most famous optimal algorithm is certainly KL-UCB [Cappé et al., 2013, Agrawal et al., 2021]. Based on the principle of optimism in face of uncertainty [Auer et al., 2002], it uses confidence intervals on empirical  $\kappa_{\mathrm{inf}}^{\mathcal{F}}$  to choose the arm with the largest plausible mean. The optimality of KL-UCB was only proved recently [Agrawal et al., 2021], as the seminal work of Cappé et al. [2013] only proved it for Multinomial distributions. A second family of algorithms, Minimized Empirical Divergence (MED) [Honda and Takemura, 2010, 2011, 2015, Baudry et al., 2023], aims at exploiting the lower bound (2): these algorithms are based on the computation of  $\kappa_{\mathrm{inf}}^{\mathcal{F}}$  for the empirical distributions and the current best empirical mean, and hence differ from KL-UCB (they perform only one computation per arm/step). More recently, an optimal algorithm based on Thompson Sampling (TS) [Thompson, 1933] was proposed for general bounded distributions, under the name of Non-Parametric Thompson Sampling (NPTS) [Riou and Honda, 2020]. In this work, the authors generalize the TS algorithms for Bernoulli distributions [Agrawal and Goyal, 2012, Kaufmann et al., 2012] by using an improper Dirac prior and a Dirichlet posterior.
Throughout the paper, we use "MED algorithms" (or simply MED) to refer to the family of algorithms. We denote by MED the randomized algorithm presented in [Honda and Takemura, 2010], that we detail in Algorithm 3, and IMED its deterministic counterpart ([Honda and Takemura, 2015], Algorithm 2).
Cost/performance trade-off In practice, the choice of a bandit algorithm may be motivated by its theoretical guarantees, but also by its computation and memory costs. Unfortunately, optimal algorithms are quite costly: MED and KL-UCB need to compute an empirical  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  (costing  $\mathcal{O}(n\log(n))$  for  $n$  samples, solving (4) with precision  $1/n$ ) for each arm/round, KL-UCB being more costly (several  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  per step), while the cost of sampling in NPTS is linear in the number of samples of each arm. For that reason, cheaper sub-optimal alternatives are often considered in place of optimal algorithms. For instance, UCB [Auer et al., 2002] has constant run time and memory, and achieves a logarithmic regret with multiplicative constant  $\mathcal{O}\left(\sum_{k:\Delta_k > 0} \Delta_k^{-1}\right)$ . More generally, all the algorithms designed for  $1/4$ -sub-gaussian distributions can be used on  $\mathcal{F}$  if the rewards are rescaled in  $[0,1]$ . A finer approximation consists in using the KL divergence of Bernoulli distributions, that lower bounds  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  [Cappé et al., 2013]. However, it is important to note that these approximations are sensitive to the value (and knowledge) of the lower bound of the support  $b$  (as a rescaling in  $[0,1]$  is necessary), contrarily to asymptotically optimal algorithms. Pushing further this idea, a procedure proposed by Riou and Honda [2020], inspired by Agrawal and Goyal [2012], consists in discretizing the rewards: a reward  $X \in [b,B]$  is transformed in  $Y \in \mathcal{X} := \{x_1 = b,\dots,x_M = B\}$  for some finite grid  $\mathcal{X}$ , such that  $\mathbb{E}[Y|X] = X$ . Hence, the expectations of the arms are unchanged while the memory is reduced to  $\mathcal{O}(MK)$  and the computation time of  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  for discretized distributions is proportional to  $M$ . Unfortunately, even if all these techniques lead to logarithmic regret, the multiplicative constant before  $\log(T)$  can be arbitrarily large compared with the optimal one. More precisely, we show in Lemma 6 that for a small gap  $\Delta$  the ratio  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}/\Delta^2$  can be of order  $\Delta^{-1}$ . We detail this result and the description of the discretization procedure in Appendix D.
We illustrate this gap with an example from [Baudry et al., 2021a], that consider a problem of crop-management optimization in agriculture. In Figure 1, we represent four distributions of crop yields generated from the DSSAT $^2$  simulator [Hoogenboom et al., 2019], corresponding to different crop-management policies. The distributions are naturally bounded due to the physical constraints of the problem and are non-parametric. In Table 1 we compare  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  to the Bernoulli KL-divergence, denoted by  $\mathrm{kl}$  (see (16) for a definition), and to  $\Delta^2$  for each arm, and we obtain on the 4-armed bandit that a regret bound scaling with  $\mathrm{kl}$  or  $2\Delta_k^2$  is almost 10 times larger than the asymptotically optimal regret defined by (2). We include this problem in our experiments of Section 4 to check the practical consequences of this asymptotic property.
Figure 1: Four yield distributions from DSSAT, and comparison of  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$ , kl, and  $2\Delta^{2}$ .
Dist.	Kinf(Fk,μ*) / kl(μk,μ*)	Kinf(Fk,μ*) / 2Δk2
F2	4.4	4.72
F3	11.73	12.44
F4	12.04	12.73
Table 1: Comparison of  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$ , kl and  $2\Delta_{k}^{2}$  for each sub-optimal arm.
Algorithm	Run time	Memory	Optimality
KL-UCB [Cappé et al., 2013]	O(n log(n)2)	n	Opt.
kl-UCB	O(log(n))	O(1)	Sub-opt. (kl)
UCB [Auer et al., 2002]	O(1)	O(1)	Sub-opt. (2Δk2)
NPTS [Riou and Honda, 2020]	O(n)	n	Opt.
MED/IMED [Honda and Takemura, 2015]	O(n log(n))	n	Opt.
Mult. MED/IMED (M items)	O(M log(n))	O(M)	Sub-opt. (KinfFmult.)
FMED/FIMED (this paper)	O(n log(n)) if pulled, O(1) otherwise.	n	Opt. (Theorem 1)
OMED/OIMED (this paper)	O(1)	O(K)	Opt. under assumptions of Theorem 2
Table 2: Comparison of memory and run time needed per step and for an arm  $k$  with  $n$  observations
Outline and contributions The results presented in previous paragraphs motivate the search for novel cheap and asymptotically optimal non-parametric bandit algorithms. We build on MED to propose two novel approaches that achieve this goal. We first propose FMED (resp. FIMED) as a fast variant of MED (resp. IMED), that computes  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  only for the arm that is pulled while for the other arms a first-order Taylor expansion is used. This simple change translates to a considerable speed-up of the algorithms, as for large enough horizons the best empirical arm (for which  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  is 0) is pulled most of the time, and preserves the theoretical guarantees of the two algorithms. However, FMED and FIMED still require to store all rewards and to fully compute  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  (sometimes). Hence we propose another approach, in which estimates of  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  are computed using an online portfolio selection algorithm. We highlight a property that would ensure that such an algorithm also keeps the guarantees of MED and IMED, while having much faster computation time and at most  $\mathcal{O}(K^2)$  memory.
$^{2}$ Decision Support System for Agrotechnology Transfer
The novel algorithms are presented in Section 2, while we discuss their guarantees in Section 3. Finally, in Section 4 we perform numerical simulations that confirm the benefits of our novel algorithms in terms of computation time, and show their strong empirical performance. Table 2 summarizes our results. We detail all computations in Appendix F.1.
2 Fast MED algorithms
In the following, we denote by  $F_{k}(t)$ ,  $\mu_{k}(t)$  and  $\mu^{\star}(t) = \max_{k} \mu_{k}(t)$  respectively the empirical distribution and mean of arm  $k$  and the best empirical mean at time  $t$ . We propose fast variants of MED and IMED, that avoid computing  $\mathcal{K}_{\inf}^{\mathcal{F}}(F_k(t), \mu^{\star}(t))$  for each arm and at each time.
2.1 On-access update and linearization: FMED and FIMED
We use that  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  is non-decreasing and convex in its second argument (Lemma 2, prop. 1). Given  $F \in \mathcal{F}$  and two thresholds  $(\mu', \mu) \in [b, B]^2$ , and assuming that  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}(F, \mu)$  and its derivative (w.r.t  $\mu$ ) have already been computed and are available in memory, we can use that
$$
\mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F, \mu^ {\prime}\right) \geq \mathcal {K} _ {\inf } ^ {\mathcal {F}} (F, \mu) + \frac {\partial \mathcal {K} _ {\inf } ^ {\mathcal {F}}}{\partial \mu} (F, \mu) \left(\mu^ {\prime} - \mu\right). \tag {5}
$$
This inequality is a cheap approximation of  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  from below when only the second argument changes: as long as  $k \in [K]$  is not pulled its empirical distribution  $F_{k}(t)$  is constant. Furthermore, the derivative is equal to the maximizer of (4) [Honda and Takemura, 2015], which is provided at no additional cost when computing  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$ . Using this result, we propose FMED and FIMED as fast variants of MED and IMED respectively, for which at each time step  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}(F_{k}(t), \mu^{\star}(t))$  is replaced by
$$
\mathcal {K} _ {k} (t) = \max  \left\{0, \mathcal {K} _ {\inf } ^ {\mathcal {F}} \left(F _ {k} (t), \mu^ {\star} \left(s _ {k} (t)\right)\right) + \lambda_ {k} \left(s _ {k} (t)\right) \times \left(\mu^ {\star} (t) - \mu^ {\star} \left(s _ {k} (t)\right)\right) \right\}, \tag {6}
$$
where  $s_k(t) = \sup \{s \leqslant t : A_s = k\}$  is the time of last pull of arm  $k$  before  $t$ , and for  $s \in \mathbb{N}$ ,
$$
\lambda_ {k} (s) = \operatorname * {a r g m a x} _ {\lambda \in [ 0, (B - \mu^ {\star} (s)) ^ {- 1} ]} \mathbb {E} _ {F _ {k} (s)} \left[ \log \left(1 - \lambda (X - \mu^ {\star} (s))\right) \right].
$$
We detail their implementation in Appendix A.2. Despite its simplicity, this method already permits a huge gain of computation time. Indeed, in Theorem 1 (see Section 3) we prove that each suboptimal arm is pulled only  $\mathcal{O}(\log T)$  in expectation, which is thus the expected number of full  $\kappa_{\mathrm{inf}}^{\mathcal{F}}$  computations. Additionally, at most one  $\kappa_{\mathrm{inf}}^{\mathcal{F}}$  is computed at each stage, while for MED/IMED  $K - 1$  computations are required. However, FMED/FIMED still require to keep every sample in memory. This motivates us to investigate an alternative approach in the next section.
2.2 Sequential update:OMED and OIMED
We now propose bandit algorithms that update  $\mathcal{K}_{\inf}^{\mathcal{F}}(F_k(t),\mu^\star (t))$  in a purely sequential fashion, for any arm  $k$ . We use that its formulation as a maximization problem (4) can be reformulated as an online portfolio selection problem, a class of online convex optimization problems. In the following we drop the index  $k$  to simplify the presentation, and thus use the notation  $N(t)$  and  $F(t)$  for the sample size and empirical cdf of the selected arm at time  $t$ , and by  $X_{n}$  the  $n$ -th observation collected by this arm.
**Formulation** Online portfolio selection is a sequential interaction between a learner and an adversary. Before the interaction, the adversary selects  $(x_{n})_{n\in [N]}\in \mathbb{R}^{N\times d}$  for a dimension  $d\geq 2$ . At iteration  $n\in \mathbb{N}$ , the learner chooses  $\lambda_{n}\in \triangle_{d}\subseteq \mathbb{R}^{d}$  (the simplex of dimension  $d - 1$ ) and receives the reward  $\log (\lambda_n^\top x_n)$ . The regret of the learner after  $N$  iterations is defined by
$$
R _ {N} \left(x _ {1}, \dots , x _ {n}\right) = \max  _ {\lambda \in \triangle_ {d}} \sum_ {n = 1} ^ {N} \log \left(\lambda^ {\top} x _ {n}\right) - \sum_ {n = 1} ^ {N} \log \left(\lambda_ {n} ^ {\top} x _ {n}\right). \tag {7}
$$
The objective of the learner is then to find a sequence  $(\lambda_{n})_{n\in \mathbb{N}}$ , where  $\lambda_{n}$  depends only on  $(\lambda_1,x_1,\ldots ,\lambda_{n - 1},x_{n - 1})$ , that minimizes  $R_{N}$ . We now express the computation of  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}(F(t),\mu^{\star}(t))$
as a portfolio selection algorithm in dimension 2,
$$
N (t) \mathcal {K} _ {\inf } ^ {\mathcal {F}} (F (t), \mu^ {\star} (t)) = \max  _ {\lambda \in \Delta_ {2}} \sum_ {n = 1} ^ {N (t)} \log \left(\lambda^ {\top} x _ {n} (t)\right), \quad \text {w i t h} x _ {n} (t) = \left(1, \frac {B - X _ {n}}{B - \mu^ {\star} (t)}\right).
$$
However, this expression is unpractical since  $x_{1}(t), \ldots, x_{n}(t)$  all depend on  $\mu^{\star}(t)$ , that is not revealed before time  $t$ . At the  $n$ -th iteration, we can only use the current best empirical mean, denoted by  $\mu_{n}^{\star}$  for simplicity: a portfolio algorithm can only try to minimize  $R_{N}(x_{1}, \ldots, x_{N})$ , with  $x_{n} = (1, (B - X_{n}) / (B - \mu_{n}^{\star}))$ . We call that quantity portfolio regret in the rest of the paper.
On the other hand, if  $\mu_n^\star$  diverges too often from  $\mu^{\star}(t)$  the estimation of  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  will not be accurate. We define the bias of the portfolio algorithm (or portfolio bias) by
$$
B _ {N (t)} \left(\mu^ {\star} (t)\right) = \max  _ {\lambda \in \triangle_ {2}} \sum_ {n = 1} ^ {N _ {k} (t)} \log \left(\lambda^ {\top} x _ {n}\right) - \max  _ {\lambda \in \Delta_ {2}} \sum_ {n = 1} ^ {N (t)} \log \left(\lambda^ {\top} x _ {n} (t)\right). \tag {8}
$$
This term can be studied independently of the portfolio regret and only depends on the variations of the best empirical mean. It is not negligible: we will have to modify the structure of the MED algorithms and assume that the best mean is not too close to  $B$  to control its magnitude.
In summary, and using all the previously introduced notation, at each stage  $t$  we propose to approximate  $N(t)\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}(F(t),\mu^{\star}(t))$  by  $L_{N(t)}\coloneqq \sum_{n = 1}^{N(t)}\log (\lambda_n^\top x_n)$ , where  $\lambda_{n}$  is updated sequentially by a portfolio algorithm. Furthermore, the accuracy of the estimate can be expressed in terms of the portfolio bias and regret as follows,
$$
N (t) \mathcal {K} _ {\inf } ^ {\mathcal {F}} (F (t), \mu^ {\star} (t)) = L _ {N (t)} + R _ {N (t)} + B _ {N (t)} \left(\mu^ {\star} (t)\right), \tag {9}
$$
the theoretical results presented in Section 3 are hence obtained by bounding these two quantities.
Portfolio algorithms See [Tsai et al., 2023] for a recent review of portfolio selection algorithms. Different methods vary based on their computational complexities and their regret. Some have regret upper bounds as low as  $\mathcal{O}(d\log T)$  but are computationally expensive, like UPS [Cover, 1991, Cover and Ordentlich, 1996, Kalai and Vempala, 2002]. Our goal is to use portfolio methods to obtain a computationally efficient  $\kappa_{\mathrm{inf}}^{\mathcal{F}}$ , hence we cannot use those. On the other hand, we can afford a looser regret upper bound since we will use it on  $\mathcal{O}(\log T)$  samples from the sub-optimal arms. Other algorithms have  $\mathcal{O}(d)$  computational complexity per round, which is cheap in our case where  $d = 2$ . This is, for example, the case for Soft-Bayes [Orseau et al., 2017], which has regret  $\mathcal{O}(\sqrt{dT})$ . Other algorithms achieve intermediate trade-offs: see [Zimmert et al., 2022, Tsai et al., 2023]. Among the computationally cheap algorithms, we chose Soft-Bayes for its simple implementation and the fact that Orseau et al. [2017] provide bounds on the regret valid for an adaptive step-size parameter, which we need since we don't know the total number of samples we will see for the arms.
While we are the first to use a portfolio algorithm to estimate  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  in a bandit algorithm, the link between  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  and portfolio selection was previously used in [Agrawal et al., 2021, Lemma E.1], where the authors use the existence of a portfolio algorithm with logarithmic regret to obtain a concentration inequality on  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$ . However, they only use that observation to obtain a bound for the analysis and they don't explore any algorithmic use of the portfolio formulation.
Furthermore, our theoretical analysis is based on a non-standard assumption, that the regret of the portfolio algorithm admits sub-linear upper and lower bounds. We discuss this assumption in the next section.
Algorithm structure of OMED and OIMED We introduce the OMED and OIMED algorithms for a generic portfolio algorithm. As discussed above, using an online estimate of empirical  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  makes the MED algorithms highly sensitive to variations of the best empirical mean. For this reason, we apply several structural changes to the algorithms in order to prevent the portfolio bias to be large, and assume the knowledge of an upper bound  $\mu_{\mathrm{max}}$  on  $\mu^{\star}$  with  $\mu_{\mathrm{max}} < B$ .
First, we use a *duel-based algorithm*: inspired by Chan [2020], at each round  $t$  we define a leader as  $\ell_t \in \mathrm{argmax} N_k(t)^3$ . Then, the other arms (called challengers) compete against the leader in pairwise
${}^{3}$  ties are broken in favor of the arm with best empirical mean,then at random if several candidates remain
Algorithm 1 Online Indexed Minimum Empirical Divergence 0IMED
Input:  $K$  arms,  $B$ , portfolio selection algorithm ALG (and its parameters), function  $f$ ,  $\mu_{\mathrm{max}}$ .  
Initialization: Pull each arm once.  $\forall (k,\ell)\in [K]^2$  set  $N_{k} = 1$ ,  $\widetilde{N}_{k,\ell} = 0$ ,  $\widehat{\mu}_k = X_{k,1}$ ,  $L_{k,\ell} = 0$ ,  $\widehat{\lambda}_{k\ell} = 1 / 2$  (init. for ALG).  
for  $t\geq K$  do  
Set  $\mathcal{A} = \{\}$ , choose leader  $\ell \in \arg \max_{k\in [K]}N_k$ ; Break ties comparing  $(\widehat{\mu}_k)$   
for  $k\neq \ell$  do  
if  $N_{\ell}\leqslant f(\widetilde{N}_{k,\ell})$  or  $\widehat{\mu}_{\ell}\geqslant \mu_{\mathrm{max}}$  then (if  $\widehat{\mu}_k\geqslant \widehat{\mu}_\ell$  then add  $k$  to  $\mathcal{A}$ , set  $Z_{k} = 0$ ); Greedy else (if  $L_{k,\ell} + \log (\widetilde{N}_{k,\ell})\leqslant \log (N_{\ell})$  then add  $k$  to  $\mathcal{A}$ , set  $Z_{k} = 1$ ); IMED duel  
if  $\mathcal{A} = \{\}$  then add  $l$  to  $\mathcal{A}$ , set  $Z_{l} = 0$ ; Pull leader if no challenger  
for  $k\in \mathcal{A}$  do  
Pull  $k$ , observe reward  $X$ , update  $\widehat{\mu}_k$ ,  $N_{k}$ .; General update (even if  $Z_{k} = 0$ )  
if  $Z_{k} = 1$  then  
Update  $\widehat{\lambda}_{k,\ell}$  with ALG, knowing  $\widetilde{N}_{k,\ell}$  and  $\frac{B - X}{B - \widehat{\mu}_{\ell}}$ ; Portfolio update  
 $\widetilde{N}_{k,\ell}\gets \widetilde{N}_{k,\ell} + 1$ ,  $L_{k,\ell}\gets L_{k,\ell} + \log \left(1 - \widehat{\lambda}_{k,\ell}\frac{X - \widehat{\mu}_{\ell}}{B - \widehat{\mu}_{\ell}}\right)$ .
comparisons called duels. At the end of the round, all winning challengers (if any) are pulled. If there are none, the leader is pulled. Hence several arms can be pulled per round. The main ideas are to replace  $\mu^{\star}(t)$  by  $\mu_{\ell_t}(t)$  for the reference value used by the challengers, and to implement a different  $\mathcal{K}_{\mathrm{inf}}^{\mathcal{F}}$  estimate  $L_{k,\ell_t}(t)$  for each possible pair  $(k,\ell_t)$ .