Title: Fast Asymptotically Optimal Algorithms for Non-Parametric Stochastic Bandits
Venue: NeurIPS 2023
Accepted: yes
Keywords: Multi-Armed Bandits
Abstract: We consider the problem of regret minimization in non-parametric stochastic bandits. When the rewards are known to be bounded from above, there exists asymptotically optimal algorithms, with asymptotic regret depending on an infimum of Kullback-Leibler divergences (KL). These algorithms are computationally expensive and require storing all past rewards, thus simpler but non-optimal algorithms are often used instead. We introduce several methods to approximate the infimum KL which reduce drastically the computational and memory costs of existing optimal algorithms, while keeping their regret guaranties. We apply our findings to design new variants of the MED and IMED algorithms, and demonstrate their interest with extensive numerical simulations.

Fast Asymptotically Optimal Algorithms for Non-Parametric Stochastic Bandits

Abstract

We consider the problem of regret minimization in non-parametric stochastic bandits. When the rewards are known to be bounded from above, there exists asymptotically optimal algorithms, with asymptotic regret depending on an infimum of Kullback-Leibler divergences (KL). These algorithms are computationally expensive and require storing all past rewards, thus simpler but non-optimal algorithms are often used instead. We introduce several methods to approximate the infimum KL which reduce drastically the computational and memory costs of existing optimal algorithms, while keeping their regret guaranties. We apply our findings to design new variants of the MED and IMED algorithms, and demonstrate their interest with extensive numerical simulations.

1 Introduction

A Multi-Armed Bandit (MAB) is a sequential decision-making problem where at each time step t ∈ N a learner collects a reward from an arm A t chosen among K ∈ N alternatives. We consider the stochastic case, in which all rewards collected from an arm k ∈ [ K ] are i.i.d. and drawn from a fixed distribution F k , of expectation µ k . For a time horizon T we define the number of pulls of an arm k by N k ( T ) = ∑ T t =1 I ( A t = k ) , and their sub-optimality gap by ∆ k = µ glyph[star] -µ k , with µ glyph[star] = max k ∈ [ K ] µ k . The goal of the learner is to maximize their expected sum of rewards, or equivalently to minimize the regret , defined as

To achieve this goal, the learner can leverage the rewards collected up to time t and their knowledge on the family of distributions F to which F 1 , . . . , F K belong. The definition of F determines the complexity of the bandit problem, since any uniformly efficient algorithm 1 satisfies the lower bound [Lai and Robbins, 1985, Burnetas and Katehakis, 1996]: for all k ∈ [ K ] with ∆ k > 0 ,

where KL ( ., . ) denotes the Kullback-Leibler divergence between two distributions. We call an algorithm asymptotically optimal (sometimes shortened to optimal) if its regret upper bound matches this lower bound. That is, if for all k , lim sup T →∞ E [ N k ( T )] / log( T ) ≤ 1 / K F inf ( F k , µ glyph[star] ) .

For some parametric families (e.g. Gaussian), K F inf has a convenient closed-form expression. However, for more general non-parametric families of distributions this may not be the case. In this work we consider the most famous example of such model, where the learner only knows that the distributions are bounded. Given a range [ b, B ] , we formally define

but in the rest of the paper we keep the notation F (for F [ b,B ] ) for simplicity. We always assume that B is known, but in some cases knowing that b is finite will be sufficient. By analyzing a dual optimization problem derived from (2), Honda and Takemura [2010] obtained that

Optimal algorithms for bounded distributions The literature on MABs has become diverse in the past years, so in the following we present a non-exhaustive selection of works that focus on asymptotically optimal algorithms for (bounded) stochastic bandits. We refer the reader to [Lattimore and Szepesvári, 2020, Bubeck et al., 2012] for broader surveys.

The most famous optimal algorithm is certainly KL-UCB [Cappé et al., 2013, Agrawal et al., 2021]. Based on the principle of optimism in face of uncertainty [Auer et al., 2002], it uses confidence intervals on empirical K F inf to choose the arm with the largest plausible mean. The optimality of KL-UCB was only proved recently [Agrawal et al., 2021], as the seminal work of Cappé et al. [2013] only proved it for Multinomial distributions. A second family of algorithms, Minimized Empirical Divergence (MED) [Honda and Takemura, 2010, 2011, 2015, Baudry et al., 2023], aims at exploiting the lower bound (2): these algorithms are based on the computation of K F inf for the empirical distributions and the current best empirical mean, and hence differ from KL-UCB (they perform only one computation per arm/step). More recently, an optimal algorithm based on Thompson Sampling (TS) [Thompson, 1933] was proposed for general bounded distributions, under the name of Non-Parametric Thompson Sampling (NPTS) [Riou and Honda, 2020]. In this work, the authors generalize the TS algorithms for Bernoulli distributions [Agrawal and Goyal, 2012, Kaufmann et al., 2012] by using an improper Dirac prior and a Dirichlet posterior.

Throughout the paper, we use 'MED algorithms' (or simply MED) to refer to the family of algorithms. We denote by MED the randomized algorithm presented in [Honda and Takemura, 2010], that we detail in Algorithm 3, and IMED its deterministic counterpart ([Honda and Takemura, 2015], Algorithm 2).

Cost/performance trade-off In practice, the choice of a bandit algorithm may be motivated by its theoretical guarantees, but also by its computation and memory costs. Unfortunately, optimal algorithms are quite costly: MED and KL-UCB need to compute an empirical K F inf (costing O ( n log( n )) for n samples, solving (4) with precision 1 /n ) for each arm/round, KL-UCB being more costly (several K F inf per step), while the cost of sampling in NPTS is linear in the number of samples of each arm. For that reason, cheaper sub-optimal alternatives are often considered in place of optimal algorithms. For instance, UCB [Auer et al., 2002] has constant run time and memory, and achieves a logarithmic regret with multiplicative constant O (∑ k :∆ k > 0 ∆ -1 k ) . More generally, all the algorithms designed for 1 / 4 -sub-gaussian distributions can be used on F if the rewards are rescaled in [0 , 1] . A finer approximation consists in using the KL divergence of Bernoulli distributions, that lower bounds K F inf [Cappé et al., 2013]. However, it is important to note that these approximations are sensitive to the value (and knowledge) of the lower bound of the support b (as a rescaling in [0 , 1] is necessary), contrarily to asymptotically optimal algorithms. Pushing further this idea, a procedure proposed by Riou and Honda [2020], inspired by Agrawal and Goyal [2012], consists in discretizing the rewards: a reward X ∈ [ b, B ] is transformed in Y ∈ X := { x 1 = b, . . . , x M = B } for some finite grid X , such that E [ Y | X ] = X . Hence, the expectations of the arms are unchanged while the memory is reduced to O ( MK ) and the computation time of K F inf for discretized distributions is proportional to M . Unfortunately, even if all these techniques lead to logarithmic regret, the multiplicative constant before log( T ) can be arbitrarily large compared with the optimal one. More precisely, we show in Lemma 6 that for a small gap ∆ the ratio K F inf / ∆ 2 can be of order ∆ -1 . We detail this result and the description of the discretization procedure in Appendix D.

We illustrate this gap with an example from [Baudry et al., 2021a], that consider a problem of crop-management optimization in agriculture. In Figure 1, we represent four distributions of crop yields generated from the DSSAT 2 simulator [Hoogenboom et al., 2019], corresponding to different crop-management policies. The distributions are naturally bounded due to the physical constraints of the problem and are non-parametric. In Table 1 we compare K F inf to the Bernoulli KL-divergence, denoted by kl (see (16) for a definition), and to ∆ 2 for each arm, and we obtain on the 4-armed bandit that a regret bound scaling with kl or 2∆ 2 k is almost 10 times larger than the asymptotically optimal regret defined by (2). We include this problem in our experiments of Section 4 to check the practical consequences of this asymptotic property.

0

2000

4000

6000

8000

10000



0

2

4

6

8



1e

5

F1 (Opt.)

= 3630

F2

= 3013

F3

= 3317

F4

= 3396

Dist.

K

F

inf

(

F

k

,µ

glyph[star]

)

kl

(

µ

k

,µ

glyph[star]

)

K

F

inf

(

F

k

,µ

glyph[star]

)

2∆

2

k

F

2

4

.

4

4

.

72

F

3

11

.

73

12

.

44

F

4

12

.

04

12

.

73

Outline and contributions The results presented in previous paragraphs motivate the search for novel cheap and asymptotically optimal non-parametric bandit algorithms. We build on MED to propose two novel approaches that achieve this goal. We first propose FMED (resp. FIMED ) as a fast variant of MED (resp. IMED ), that computes K F inf only for the arm that is pulled while for the other arms a first-order Taylor expansion is used. This simple change translates to a considerable speed-up of the algorithms, as for large enough horizons the best empirical arm (for which K F inf is 0 ) is pulled most of the time, and preserves the theoretical guarantees of the two algorithms. However, FMED and FIMED still require to store all rewards and to fully compute K F inf (sometimes). Hence we propose another approach, in which estimates of K F inf are computed using an online portfolio selection algorithm. We highlight a property that would ensure that such an algorithm also keeps the guarantees of MED and IMED , while having much faster computation time and at most O ( K 2 ) memory.

The novel algorithms are presented in Section 2, while we discuss their guarantees in Section 3. Finally, in Section 4 we perform numerical simulations that confirm the benefits of our novel algorithms in terms of computation time, and show their strong empirical performance. Table 2 summarizes our results. We detail all computations in Appendix F.1.

2 Fast MED algorithms

In the following, we denote by F k ( t ) , µ k ( t ) and µ glyph[star] ( t ) = max k µ k ( t ) respectively the empirical distribution and mean of arm k and the best empirical mean at time t . We propose fast variants of MED and IMED , that avoid computing K F inf ( F k ( t ) , µ glyph[star] ( t )) for each arm and at each time.

2.1 On-access update and linearization: FMED and FIMED

We use that K F inf is non-decreasing and convex in its second argument (Lemma 2, prop. 1). Given F ∈ F and two thresholds ( µ ′ , µ ) ∈ [ b, B ] 2 , and assuming that K F inf ( F, µ ) and its derivative (w.r.t µ ) have already been computed and are available in memory, we can use that

This inequality is a cheap approximation of K F inf from below when only the second argument changes: as long as k ∈ [ K ] is not pulled its empirical distribution F k ( t ) is constant. Furthermore, the derivative is equal to the maximizer of (4) [Honda and Takemura, 2015], which is provided at no additional cost when computing K F inf . Using this result, we propose FMED and FIMED as fast variants of MED and IMED respectively, for which at each time step K F inf ( F k ( t ) , µ glyph[star] ( t )) is replaced by

where s k ( t ) = sup { s glyph[lessorequalslant] t : A s = k } is the time of last pull of arm k before t , and for s ∈ N ,

We detail their implementation in Appendix A.2. Despite its simplicity, this method already permits a huge gain of computation time. Indeed, in Theorem 1 (see Section 3) we prove that each suboptimal arm is pulled only O (log T ) in expectation, which is thus the expected number of full K F inf computations. Additionally, at most one K F inf is computed at each stage, while for MED / IMED K -1 computations are required. However, FMED / FIMED still require to keep every sample in memory. This motivates us to investigate an alternative approach in the next section.

2.2 Sequential update: OMED and OIMED

We now propose bandit algorithms that update K F inf ( F k ( t ) , µ glyph[star] ( t )) in a purely sequential fashion, for any arm k . We use that its formulation as a maximization problem (4) can be reformulated as an online portfolio selection problem, a class of online convex optimization problems. In the following we drop the index k to simplify the presentation, and thus use the notation N ( t ) and F ( t ) for the sample size and empirical cdf of the selected arm at time t , and by X n the n -th observation collected by this arm.

Formulation Online portfolio selection is a sequential interaction between a learner and an adversary. Before the interaction, the adversary selects ( x n ) n ∈ [ N ] ∈ R N × d for a dimension d ≥ 2 . At iteration n ∈ N , the learner chooses λ n ∈ glyph[triangle] d ⊆ R d (the simplex of dimension d -1 ) and receives the reward log( λ glyph[latticetop] n x n ) . The regret of the learner after N iterations is defined by

The objective of the learner is then to find a sequence ( λ n ) n ∈ N , where λ n depends only on ( λ 1 , x 1 , . . . , λ n -1 , x n -1 ) , that minimizes R N . We now express the computation of K F inf ( F ( t ) , µ glyph[star] ( t ))

as a portfolio selection algorithm in dimension 2,

However, this expression is unpractical since x 1 ( t ) , . . . , x n ( t ) all depend on µ glyph[star] ( t ) , that is not revealed before time t . At the n -th iteration, we can only use the current best empirical mean, denoted by µ glyph[star] n for simplicity: a portfolio algorithm can only try to minimize R N ( x 1 , . . . , x N ) , with x n = (1 , ( B -X n ) / ( B -µ glyph[star] n )) . We call that quantity portfolio regret in the rest of the paper.

On the other hand, if µ glyph[star] n diverges too often from µ glyph[star] ( t ) the estimation of K F inf will not be accurate. We define the bias of the portfolio algorithm (or portfolio bias) by

This term can be studied independently of the portfolio regret and only depends on the variations of the best empirical mean. It is not negligible: we will have to modify the structure of the MED algorithms and assume that the best mean is not too close to B to control its magnitude.

In summary, and using all the previously introduced notation, at each stage t we propose to approximate N ( t ) K F inf ( F ( t ) , µ glyph[star] ( t )) by L N ( t ) := ∑ N ( t ) n =1 log( λ glyph[latticetop] n x n ) , where λ n is updated sequentially by a portfolio algorithm. Furthermore, the accuracy of the estimate can be expressed in terms of the portfolio bias and regret as follows,

the theoretical results presented in Section 3 are hence obtained by bounding these two quantities.

Portfolio algorithms See [Tsai et al., 2023] for a recent review of portfolio selection algorithms. Different methods vary based on their computational complexities and their regret. Some have regret upper bounds as low as O ( d log T ) but are computationally expensive, like UPS [Cover, 1991, Cover and Ordentlich, 1996, Kalai and Vempala, 2002]. Our goal is to use portfolio methods to obtain a computationally efficient K F inf , hence we cannot use those. On the other hand, we can afford a looser regret upper bound since we will use it on O (log T ) samples from the sub-optimal arms. Other algorithms have O ( d ) computational complexity per round, which is cheap in our case where d = 2 . This is, for example, the case for Soft-Bayes [Orseau et al., 2017], which has regret O ( √ dT ) . Other algorithms achieve intermediate trade-offs: see [Zimmert et al., 2022, Tsai et al., 2023]. Among the computationally cheap algorithms, we chose Soft-Bayes for its simple implementation and the fact that Orseau et al. [2017] provide bounds on the regret valid for an adaptive step-size parameter, which we need since we don't known the total number of samples we will see for the arms.

While we are the first to use a portfolio algorithm to estimate K F inf in a bandit algorithm, the link between K F inf and portfolio selection was previously used in [Agrawal et al., 2021, Lemma E.1], where the authors use the existence of a portfolio algorithm with logarithmic regret to obtain a concentration inequality on K F inf . However, they only use that observation to obtain a bound for the analysis and they don't explore any algorithmic use of the portfolio formulation.

Furthermore, our theoretical analysis is based on a non-standard assumption, that the regret of the portfolio algorithm admits sub-linear upper and lower bounds. We discuss this assumption in the next section.

Algorithm structure of OMED and OIMED We introduce the OMED and OIMED algorithms for a generic portfolio algorithm. As discussed above, using an online estimate of empirical K F inf makes the MED algorithms highly sensitive to variations of the best empirical mean. For this reason, we apply several structural changes to the algorithms in order to prevent the portfolio bias to be large, and assume the knowledge of an upper bound µ max on µ glyph[star] with µ max < B .

First, we use a duel-based algorithm: inspired by Chan [2020], at each round t we define a leader as glyph[lscript] t ∈ argmax N k ( t ) 3 . Then, the other arms (called challengers) compete against the leader in pairwise

Algorithm 1 Online Indexed Minimum Empirical Divergence OIMED

glyph[negationslash]

comparisons called duels . At the end of the round, all winning challengers (if any) are pulled. If there are none, the leader is pulled. Hence several arms can be pulled per round. The main ideas are to replace µ glyph[star] ( t ) by µ glyph[lscript] t ( t ) for the reference value used by the challengers, and to implement a different K F inf estimate L k,glyph[lscript] t ( t ) for each possible pair ( k, glyph[lscript] t ) .

We now introduce some notation and terminology to describe the duel between a challenger k and a leader glyph[lscript] . In some cases (described below) we perform a greedy duel , for which the winner is argmax { µ k ( t ) , µ glyph[lscript] ( t ) } . We introduce a variable Z k ( t ) , that indicates if the duel played by arm k was greedy ( Z k ( t ) = 0 ) or not ( Z k ( t ) = 1 ), and then ˜ N k,glyph[lscript] ( t ) = ∑ t -1 s =1 I ( k ∈ A s +1 , glyph[lscript] s = glyph[lscript],Z k ( s ) = 1) , the number of observations of a challenger k collected against the leader glyph[lscript] after a non-greedy duel .

Using this notation and considering a function f : N ↦→ R + satisfying n = o ( f ( n )) , the duels of OMED / OIMED are implemented as follow: if N glyph[lscript] ( t ) glyph[lessorequalslant] f ( ˜ N k,glyph[lscript] ( t )) or µ glyph[lscript] ( t ) glyph[greaterorequalslant] µ max the duel is greedy, otherwise MED and IMED are respectively adapted as follows.

Then, we update L k,glyph[lscript] ( t ) using the portfolio selection algorithm only if the duel was non-greedy, i.e. Z k ( t ) = 1 , providing µ glyph[lscript] ( t ) and the reward collected from k : this is the main ingredient to control the portfolio bias in our analysis. We provide a condensed implementation of OIMED in Algorithm 1, and the detailed implementation of OMED in Appendix A.2 (Algorithm 6).

3 Theoretical guarantees

We present our theoretical results on the MED algorithms introduced in Section 2, and some insights from their analysis. We start with the theoretical guarantees of FMED and FIMED .

Theorem 1 (Regret bound for fast MED) . Consider ( F 1 , . . . , F K ) ∈ F K and µ glyph[star] = max k ∈ [ K ] µ k . Then, for any time horizon T ∈ N and ε > 0 , MED , IMED , FMED and FIMED all satisfy

where o ε (log( T )) denotes a term that is asymptotically dominated by log( T ) for a fixed ε , but with a polynomial dependency in ε -1 . Furthermore, all the algorithms are asymptotically optimal .

We prove Theorem 1 in Appendix B. The main result is that FMED and FIMED both preserve the theoretical guarantees of their original algorithm. We apply our analysis to MED and IMED too, in

order to exhibit more precisely the new terms induced by the approximation, and their scaling in ε . Note that the components of the o ε (log( T )) terms are explicit in the proof. When ε is small, we obtain more precisely a scaling in ε -6 , that allows to obtain a sub-linear problem-independent bound of order T 5 / 6 , though not the optimal √ T bound. Note however that the same scaling can be obtained for MED with the proof techniques presented in [Baudry et al., 2023], so the fast implementation do not deteriorate that bound. The strong empirical performance of MED / IMED hints that this result is likely to be not tight, but to the best of our knowledge it remains open to prove that an optimal 4 (instance-dependent) algorithm achieves the O ( √ T ) problem-independent bound for general bounded distributions. Finally, we emphasize that the analysis of IMED presented in Appendix B is drastically simplified compared to Honda and Takemura [2015], although their result is more general (they allow b = -∞ ).

Proof sketch. First, the inequality K k ( t ) glyph[greaterorequalslant] K F inf ( F k ( t ) , µ glyph[star] ( t )) (Eq. (5) and (6)) makes FMED and FIMED at least as exploratory as the vanilla algorithms. Their regret is thus smaller in a preconvergence regime, defined by the time steps for which µ glyph[star] ( t ) glyph[lessorequalslant] µ glyph[star] -ε . The main challenge is then to prove that using ( K k ( t )) k ∈ [ K ] do not lead to over-exploration of the sub-optimal arms in a post-convergence regime, where µ glyph[star] ( t ) glyph[greaterorequalslant] µ glyph[star] -ε . The main ingredient of the proof consists in showing that this kind of scenario can only be caused by events of the form

Each of them can cause at most 1 pull of a sub-optimal arm. With union bounds and Hoeffding's inequality we obtain an additional O ( ε -2 ) term in the regret bound compared to MED and IMED .

We now provide a very similar result on OMED and OIMED , under some additional assumptions: first, we need the largest mean to be well-separated from B to control the portfolio bias. This is a mild assumption, since the decision-maker can also choose to slightly increase the value of B used by the algorithm if it does not clearly hold. Then, we require strong guarantees on the portfolio regret, that we discuss at the end of this section.

Theorem 2 (Regret bound for online MED algorithms) . Consider ( F 1 , . . . , F K ) ∈ F K , and µ glyph[star] = max k ∈ [ K ] µ k < µ max, for a known µ max < B . Assume that OMED and OIMED use a portfolio selection algorithm ALG that satisfies for any ( k, glyph[lscript] ) ∈ [ K ] 2 and n ∈ N the deterministic guarantee

Then, both OMED and OIMED satisfy Equation (12) (with a different second-order term) and are asymptotically optimal : they enjoy the same asymptotic guarantees as MED and IMED .

Strikingly, the performance of OMED / OIMED only depend on the regret bounds of the portfolio regret. As detailed below, this is largely due to our modified algorithm's structure.

Proof sketch. The general proof scheme is inspired by [Chan, 2020], that proposed a similar duel-based approach. The main difficulty consists in controlling the deviation of L k,glyph[lscript] t ( t ) from K F inf ( F k,glyph[lscript] t ( t ) , µ glyph[lscript] t ( t )) , denoting by F k,glyph[lscript] t ( t ) the empirical distribution of the ˜ N k,glyph[lscript] t ( t ) observations used to update L k,glyph[lscript] t ( t ) . Both over and under-estimation are bad: one may prevent sufficient exploration of the best arm, while the other may lead to over-exploration of sub-optimal arms. Building on the discussion of Section 2, our proof relies on the following crucial result: for fixed arms ( k, glyph[lscript] ) , a sample size ˜ N k,glyph[lscript] ( t ) = n , and any threshold µ it holds that

where B k,glyph[lscript],n ( µ ) is the portfolio bias with respect to a fixed threshold µ , defined in (8), and where L k,glyph[lscript],n and F k,glyph[lscript],n are used to denote L k,glyph[lscript] ( t ) and F k,glyph[lscript] ( t ) when ˜ N k,glyph[lscript] ( t ) = n , for simplicity. In the proofs, the portfolio regret is controlled by assumption, but the bias requires more careful examination. To analyze it, we prove the following result (details in Appendix E). For i glyph[lessorequalslant] n , we denote by t k,glyph[lscript],i the time when the i -th iteration of L k,glyph[lscript],n occurred.

Lemma 1 (Deviations of B k,glyph[lscript],n ( µ ) ) . Let µ ∈ (0 , 1) , and set C k,glyph[lscript],i = B -µ glyph[lscript] ( t k,glyph[lscript],i ) B -µ it holds that

glyph[negationslash]

We always use this result for µ ∈ (max k : µ k <µ glyph[star] µ k , µ glyph[star] ) , and µ glyph[lscript] ( t k,glyph[lscript],i ) < µ max. Hence, the upper bound (resp. lower bound) of (15) is expected to be sub-linear when glyph[lscript] = 1 (resp. glyph[lscript] = 1 ), thanks to the use of different estimates according to the leader's identity. The rate of convergence of these sums is also important: our proof techniques work if n = o ( N glyph[lscript] ( t k,glyph[lscript],n )) , which justifies the greedy duels. Indeed, we need e an P ( B k,glyph[lscript],n < -nx ) to be small enough (for some a, x > 0 and large n ), and with this trick we obtain e an -f ( n ) y x → 0 (for some y x depending on x ).

The modifications introduced in OMED and OIMED guarantee a small absolute bias with large probability. It is natural to ask whether they are necessary. We cannot formally prove it, but our experiments in Section 4 suggest that the regret of OIMED / OMED may be linear without them.

Assumption on the portfolio regret As detailed in Section 2 there are several candidates for the implementation of OIMED / IMED . The literature on regret lower bounds is however scarce. In [Gofer and Mansour, 2016], the authors characterize algorithms with non-negative regret, but for linear losses. Guzmán et al. [2021] showed that the regret of FTRL algorithms is bounded as O ( √ T ) both from above and below, in a setting that encompasses portfolio selection. Unfortunately, FTRL is computationally inefficient, which makes it unsuitable for our application. We used Soft-Bayes for its simplicity and computational efficiency, but it might not have the desired lower bound. Our problem seems to be among the first where a large negative regret is detrimental, and obtaining a portfolio algorithm with both small computational complexity and a regret lower bound is an open question. On the other hand, it may be that this lower bound requirement could be relaxed in the analysis, and that for example a high-probability lower bound could be enough. Our experimental study in the next section supports the hypothesis that using Soft-Bayes in OIMED gives good regret bounds for the bandit problem.

4 Experiments

We numerically assess the regret and run time of the two novel approaches presented in this paper. We benchmark our approaches with the known asymptotically optimal algorithms: IMED [Honda and Takemura, 2015], KL-UCB [Cappé et al., 2013] and NPTS [Riou and Honda, 2020]; and the more efficient but sub-optimal UCB [Auer et al., 2002], kl-UCB [Cappé et al., 2013] and kl-IMED (that denotes the binarized version of IMED ). We provide the pseudo-code of KL-UCB and NPTS in Appendix A.2 for completeness. Our code is available in the supplementary material of the paper.

We showcase our main findings by presenting a selection of experiments, focusing on FIMED and OIMED , while we present a broader set of experiments and more detailed results in Appendix F. In each case, we plot the average run time and regret of each algorithm, along with their quantiles 10% -90% . The run times are dependent of the Python implementation of our algorithms, and are thus only indicative. We refer to Appendix F.1 for discussions on their theoretical complexity.

Distributions from DSSAT We first consider the crop-management optimization problem introduced in Section 1 and detailed for instance in [Baudry et al., 2021a], as it illustrates the theoretical advantage of asymptotically optimal algorithms (Figure 1). A learner needs to select a planting date for maize grains among seven possible options. For each distribution, the rewards are drawn uniformly at random among 10 4 points sampled from the DSSAT simulator, in order to emulate the distributions at a reduced cost. 5 . Our results, displayed in Figure 2, show that on this problem optimal algorithms also achieve better empirical performance. IMED , NPTS and KL-UCB perform similarly, while kl-IMED and UCB achieve significantly larger regret. Furthermore, FIMED is 10 times faster than IMED for T = 10 4 (and as fast as kl-UCB ), with almost no difference in terms of regret. As expected, OIMED is as fast as UCB and kl-IMED . It achieves better regret than those algorithms, but its performance is deteriorated compare to IMED . This can be seen as the cost of the ability to forget

past observations. In summary, this experiment illustrates the respective benefits of our two novel approaches: FIMED is the fastest algorithm among those with the smallest regret, while OIMED has the smallest regret among the fastest algorithm.

0

2000

4000

6000

8000

10000



0

50

100

150

200

250



UCB

NPTS

IMED

IMED-kl

OIMED

FIMED

kl-UCB

KL-UCB

0

2000

4000

6000

8000

10000



0

50

100

150

200

250



Bernoulli bandit For our second experiment we consider a Bernoulli bandit with several means close to 0 . 5 , for which all optimal algorithms match their implementation with the Bernoulli KLdivergence. Intuitively, sequences of 0 and 1 with high variance may lead to the most potentially confusing inputs for portfolio algorithms, so our objective is to check the performance of OIMED in that case. Our results, summarized in Figure 3, are promising: the average regret of OIMED is on par with other algorithms, while it still is among the fastest. In that experiment only UCB is sub-optimal, and performs much worse than the other algorithms.

0

2000

4000

6000

8000

10000



0

50

100

150

200

250

300

350



UCB

NPTS

IMED

OIMED

FIMED

KL-UCB

0

2000

4000

6000

8000

10000



0

5

10

15

20

25

30



Importance of the duel-based structure In Section 3 we detailed our theoretical motivations for introducing a duel-based approach when approximating K F inf with a portfolio algorithm. We now provide empirical evidence that this change may prevent OIMED from suffering a linear regret. In Figure 4, we compare the regret of OIMED and a variant of OIMED that does not use the duel-based structure of Algorithm 1 for the two experiments introduced in previous paragraphs. In that algorithm, the online estimates of K F inf are updated at each pull of an arm with the current best empirical mean. We observe that the regret of the 'no duel' variant of OIMED may be better in some cases: for the DSSAT experiment, its average regret matches the best algorithms of Figure 2. However, on the Bernoulli experiment its regret is linear. Overall, OIMED needs a modification like the duels to ensure that it has sub-linear regret.

Further experiments In Appendix F we show that FMED and OMED exhibit the same characteristics as FIMED and OIMED , as expected. We also consider other Bernoulli bandits, for instance with means close to the boundaries of the support. We then show that the conclusions of this section still hold on problems with distributions of various shapes, by testing our algorithms on several examples of Beta distributions. We also investigate the sensitivity of OIMED to the learning rate of Soft-Bayes, which is the only hyper-parameter of the algorithm. Finally, we perform experiments with the discretized IMED algorithms that is briefly presented in the introduction and discussed more thoroughly in Appendix D.

0

2000

4000

6000

8000

10000

0

20

40

60

80

100

120

140



0

2000

4000

6000

8000

10000



0

200

400

600

800

1000

1200

1400



OIMED

OIMED no duels



5 Conclusion

We introduced methods to compute efficiently approximations of K F inf and demonstrated their use in algorithms for regret minimization in stochastic bandits. The FMED and FIMED variants have the same asymptotic optimality properties as the base MED algorithms, but have a much reduced computational complexity. OMED and OIMED push the computational gains further and are also memory efficient. They can also be asymptotically optimal, under the hypothesis that the portfolio algorithm they use satisfies a deterministic regret lower bound.

While our experiments show the good practical performance of OIMED and OMED with the Soft-Bayes portfolio algorithm, this question is however still open: can we have a portfolio algorithm which is computationally efficient, does not store all past gains in memory and has sub-linear regret upper and lower bounds?

Finally, our work towards enabling the use of K F inf in a computationally efficient way has potential applications beyond regret minimization in stochastic bandits. First, similar quantities are used in RL algorithms like IMED-RL [Pesquerel and Maillard, 2022]. Second, other bandit tasks like best arm identification also have complexities that depend on a K F inf quantity, and could benefit from faster variants of the algorithms that need to compute it [Jourdan et al., 2022].

References

Agrawal and N. Goyal. Analysis of Thompson sampling for the multi-armed bandit problem. In Proceedings of the 25th Annual Conference on Learning Theory , 2012.

Agrawal, S. Juneja, and W. M. Koolen. Regret minimization in heavy-tailed bandits. In Conference on Learning Theory, COLT 2021, 15-19 August 2021, Boulder, Colorado, USA , 2021.

Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47(2):235-256, 2002.

Baudry, R. Gautron, E. Kaufmann, and O. Maillard. Optimal thompson sampling strategies for support-aware cvar bandits. Proceedings of Machine Learning Research. PMLR, 2021a.

Baudry, P. Saux, and O.-A. Maillard. From optimality to robustness: Adaptive re-sampling strategies in stochastic bandits. In Advances in Neural Information Processing Systems , 2021b.

Baudry, K. Suzuki, and J. Honda. A general recipe for the analysis of randomized multi-armed bandit algorithms, 2023.

Bubeck, N. Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends R © in Machine Learning , 5(1):1-122, 2012.

N. Burnetas and M. N. Katehakis. Optimal adaptive policies for sequential allocation problems. Advances in Applied Mathematics , 17(2):122-142, 1996.

Cappé, A. Garivier, O.-A. Maillard, R. Munos, and G. Stoltz. Kullback-Leibler upper confidence bounds for optimal sequential allocation. Annals of Statistics , 41(3):1516-1541, 2013.

P. Chan. The multi-armed bandit problem: An efficient nonparametric solution. The Annals of Statistics , 48(1):346-373, 2020.

M. Cover. Universal portfolios. Mathematical finance , 1(1):1-29, 1991.

M. Cover and E. Ordentlich. Universal portfolios with side information. IEEE Transactions on Information Theory , 42(2):348-363, 1996.

Gofer and Y. Mansour. Lower bounds on individual sequence regret. Machine Learning , 103: 1-26, 2016.

Guzmán, N. Mehta, and A. Mortazavi. Best-case lower bounds in online learning. Advances in Neural Information Processing Systems , 34:21923-21934, 2021.

Honda and A. Takemura. An asymptotically optimal bandit algorithm for bounded support models. In COLT , pages 67-79, 2010.

Honda and A. Takemura. An asymptotically optimal policy for finite support models in the multiarmed bandit problem. Mach. Learn. , 2011.

Honda and A. Takemura. Non-asymptotic analysis of a new bandit algorithm for semi-bounded rewards. J. Mach. Learn. Res. , 16:3721-3756, 2015.

Hoogenboom, C. Porter, K. Boote, V. Shelia, P. Wilkens, U. Singh, J. White, S. Asseng, J. Lizaso, L. Moreno, et al. The dssat crop modeling ecosystem. Advances in crop modelling for a sustainable agriculture , pages 173-216, 2019.

Jourdan, R. Degenne, D. Baudry, R. de Heide, and E. Kaufmann. Top two algorithms revisited. In NeurIPS 2022-36th Conference on Neural Information Processing System , 2022.

T. Kalai and S. Vempala. Efficient algorithms for universal portfolios. Journal of Machine Learning Research , pages 423-440, 2002.

Kaufmann, N. Korda, and R. Munos. Thompson sampling: An asymptotically optimal finite-time analysis. In Algorithmic Learning Theory - 23rd International Conference, ALT 2012 , 2012.

L. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1):4-22, 1985.

Lattimore and C. Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Orseau, T. Lattimore, and S. Legg. Soft-bayes: Prod for mixtures of experts with log-loss. In International Conference on Algorithmic Learning Theory , pages 372-399. PMLR, 2017.

Pesquerel and O.-A. Maillard. Imed-rl: Regret optimal learning of ergodic markov decision processes. In NeurIPS 2022-Thirty-sixth Conference on Neural Information Processing Systems , 2022.

Riou and J. Honda. Bandit algorithms based on Thompson sampling for bounded reward distributions. In Algorithmic Learning Theory , pages 777-826. PMLR, 2020.

R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3/4):285-294, 1933.

C.-E. Tsai, H.-C. Cheng, and Y.-H. Li. Online self-concordant and relatively smooth minimization, with applications to online portfolio selection and learning quantum states. In International Conference on Algorithmic Learning Theory , pages 1481-1483. PMLR, 2023.

Zimmert, N. Agarwal, and S. Kale. Pushing the efficiency-regret pareto frontier for online learning of portfolios and quantum states. In Conference on Learning Theory , pages 182-226. PMLR, 2022.

A Index, notation and algorithms

A.1 Index and notation

In this section we summarize the notation used in the proofs of Theorem 1 and 2, and recall some of the definitions presented in the paper.

[ K ] = { 1 , . . . , K } .

F the family of distributions supported on [ b, B ] ⊂ R .

( F 1 , . . . , F K ) ∈ F K denote the distribution of the K arms of a bandit problem. Their expectations are denoted by µ 1 , . . . , µ K . Without loss of generality we assume in the proofs that µ 1 = max k ∈ [ K ] µ k , and also define x k = µ 1 + µ k 2 .

For any F ∈ F , µ ∈ [ b, B ] , K F inf ( F, µ ) = max λ ∈ [0 , 1] E F [ log ( 1 -λ X -µ B -µ )] .

A t +1 : arm selected by the learner at the beginning of round t +1 .

N k ( t ) : number of pulls of arm k up to (including) time step t .

( X k,n ) k ∈ [ K ] ,n ∈ N : sequences of rewards: X k,n is the n -th reward collected from arm k .

F k ( t ) : empirical distribution of arm k at time t . We also denote for convenience by F k,n the empirical distribution of arm k after collecting n observations : F k,N k ( t ) = F k ( t ) .

µ k ( t ) and µ k,n denote the empirical mean of arm k respectively at time t and after collecting n observations.

µ glyph[star] ( t ) = max k ∈ [ K ] µ k ( t ) , k glyph[star] ( t ) = argmax k ∈ [ K ] µ k ( t ) .

For a randomized algorithm, we use the shorthand notation p k ( t ) := P ( A t +1 = k |F t ) , where F t denotes the σ -algebra generated by the rewards collected by the learner up to (and including) time t . We call p k ( t ) the sampling probability of arm k at time t .

We use 'MED' a general terminology to call some existing bandit algorithms based on the computation of K F inf ( F k ( t ) , µ glyph[star] ( t )) . Some of these algorithms are DMED [Honda and Takemura, 2010], MED [Honda and Takemura, 2011] and IMED [Honda and Takemura, 2015]. Hence, MED denotes a specific algorithm and MED a general family of algorithms in this paper.

FMED and FIMED :

s k ( t ) : time of the last pull of arm k at time t , formally s k ( t ) = inf { s glyph[lessorequalslant] t : N k ( s ) = N k ( t ) } .

•

is a first order approximation of K F inf ( F k ( t ) , µ glyph[star] ( t )) satisfying K k ( t ) glyph[lessorequalslant] K F inf ( F k ( t ) , µ glyph[star] ( t )) , where for any s ∈ N

OMED and OIMED :

glyph[lscript] t : leader at round t , defined as glyph[lscript] t ∈ argmax k ∈ [ K ] N k ( t ) . Ties are broken in favor of the arm with

the best empirical mean. If there are still several candidates choose one of them randomly.

Challengers: all arms except the leader glyph[lscript] t

A t +1 : set of arms to pull at the beginning of round t +1

Duel: comparison between a challenger arm k and the leader glyph[lscript] t at a time t .

Z k ( t ) = 0 if the duel for challenger k is a greedy comparison, Z k ( t ) = 1 otherwise (e.g. if the index comparison of IMED is used). We use the expressions 'greedy duel' and 'non-greedy duel' to distinguish the two cases.

˜ N k ( t ) = ∑ t -1 s =1 I ( k ∈ A t +1 , glyph[lscript] s = glyph[lscript] t , Z k ( s ) = 1) : number of observations collected by arm k as a challenger against the current leader glyph[lscript] ( t ) and thanks to a non-greedy duel. We denote by ˜ F k ( t ) the empirical distribution associated with these observations, and use the shorthand notation ˜ F k,n for ˜ N k ( t ) = n .

For any k ∈ [ K ] , glyph[lscript] ∈ [ K ] , n ∈ N , X k,glyph[lscript],n is the n -th observation collected from arm k against the leader glyph[lscript] after a non-greedy duel

The online estimate used by arm k against leader glyph[lscript] with ˜ N k ( t ) = n is

where t k,glyph[lscript],i is the time step satisfying { ˜ N k ( t +1) = ˜ N k ( t ) + 1 , ˜ N k ( t ) = n } , and ( λ k,i ) i ∈ N is provided by a portfolio selection algorithm.

is the portfolio regret.

For any µ ∈ [ b, B ) ,

is the bias of the portfolio selection sub-routine with respect to n K F inf ( ˜ F n , µ ) .

We also use L k,glyph[lscript] t ( t ) = L k,glyph[lscript] t , ˜ N k ( t ) , B k ( t, µ ) = B k,glyph[lscript] t , ˜ N k ( t ) ( µ ) .

A.2 Detailed implementations

In this section we detail the implementations of the algorithms presented in the paper, along with their main competitors. We start with the implementation of IMED and MED , presented respectively in Algorithms 2 and 3. We then present FIMED and FMED , respectively in Algorithms 4 and 5. Finally, we detail OMED in Algorithm 6, and recall that the implementation of OIMED is presented in Section 2 (Algorithm 1).

In Algorithm 7, we also detail the implementation of the discretization process proposed by Riou and Honda [2020], that allows to map the rewards in a grid X = ( x 1 , . . . , x M ) without changing the means of the arms.

Finally, for comparison we also detail the implementation of the two other existing optimal algorithms for general bounded rewards: KL-UCB (Algorithm 8), implemented with parameter f : t ↦→ log( t ) + log log( t ) , and NPTS (Algorithm 9).

•

Algorithm 3 Minimized Empirical Divergence ( MED )

Algorithm 4 Fast Indexed Minimized Empirical Divergence ( FIMED )

Algorithm 5 Fast Minimized Empirical Divergence ( FMED )

Algorithm 6 Online Minimum Empirical Divergence (

Input: K arms, upper bound B , portfolio selection algorithm ALG (and its parameters), function f . 37 Initialization: ∀ k ∈ [ K ] : N k = 0 (total nb. of pulls), ˜ N kglyph[lscript] = 0 (nb of pulls as a challenger after a non-greedy duel against leader glyph[lscript] ), ̂ µ k = 0 (emp. mean), L kglyph[lscript] = 0 (emp. K F inf as a glyph[lscript] λ = 1 / 2

glyph[negationslash]

Algorithm 7

Algorithm 8

B Proof of Theorem 1: regret analysis of the index policies

Before proving the theorem, we recall some useful properties from previous works.

B.1 Toolbox for bounded distributions.

In the proofs we use the following results, that come from several works on bounded distributions in bandits (e.g. [Honda and Takemura, 2015, Cappé et al., 2013]), and are detailed in Appendix D.3 of Baudry et al. [2023].

Lemma 2 (Useful properties of K F inf ) . Let F ∈ F be a distribution of mean µ F . We denote by F n and µ n respectively the empirical cdf and empirical mean corresponding to n i.i.d. observations collected from F . It holds that

K F inf is continuous, non-decreasing and convex in its second argument (Theorem 7 of [Honda and Takemura, 2010]).

There exists constants c > 0 , δ 0 > 0 such that ∀ µ > µ F , and δ ∈ (0 , δ 0 ] it holds that

For any ε > 0 and µ > µ n + ε , it holds that

For any x > 0 , it holds that (Lemma 6 from Cappé et al. [2013])

For any ε > 0 (Hoeffding's inequality),

Proof. Property 3. can be deduced from Lemma 13 of Honda and Takemura [2010]. Using the convexity of K F inf and Pinsker inequality we obtain that

if the range is [0 , 1] . The factor 1 ( B -b ) 2 comes from using this result on rescaled distributions for general ranges.

Then, property 2. can be obtained with a straightforward adaptation of the proof of Lemma 6 of Agrawal et al. [2021] (in their Appendix B.1). Indeed, in this paper the authors consider a non-parametric family of distributions for which K F inf can be expressed in a very analogous way to Equation (4).

B.2 Proof of Theorem 1

We introduce the notation s k ( t ) = inf { s glyph[lessorequalslant] t : N k ( s ) = N k ( t ) } , which is simply the time step corresponding to the last pull of arm k before time t , and k glyph[star] ( t ) = argmax k ∈ [ K ] µ k ( t ) . We recall Theorem 1.

Theorem 1 (Regret bound for fast MED) . Consider ( F 1 , . . . , F K ) ∈ F K and µ glyph[star] = max k ∈ [ K ] µ k . Then, for any time horizon T ∈ N and ε > 0 , MED , IMED , FMED and FIMED all satisfy

where o ε (log( T )) denotes a term that is asymptotically dominated by log( T ) for a fixed ε , but with a polynomial dependency in ε -1 . Furthermore, all the algorithms are asymptotically optimal .

Proof. Equation (12) can be proved for all the algorithms with the same general proof scheme. Hence, we propose a proof that tackle them all at once, explicitly mentioning which parts are specific to a given algorithm. We start by proving a first general upper bound.

Lemma 3 (Generic regret upper bound) . For each sub-optimal arm k and any ε > 0 , FMED and FIMED both satisfy

where u ε ( T ) = ⌈ log( T ) K F inf ( F k ,µ 1 ) -ε ⌉ , and using the notation ε 0 = B -µ 1 4 ε and ε 1 = ε 2 = ε 2 we define

and

For MED and IMED , G k ( t ) = { µ glyph[star] ( t ) glyph[greaterorequalslant] µ 1 -ε 0 } .

For FMED and FIMED , G k ( t ) = { µ glyph[star] ( t ) glyph[greaterorequalslant] µ 1 -ε 0 , µ glyph[star] ( s k ( t )) glyph[greaterorequalslant] µ 1 -ε 0 , µ glyph[star] ( t ) -µ glyph[star] ( s k ( t )) B -µ glyph[star] ( s k ( t )) glyph[greaterorequalslant] -ε 2 } .

Proof. Following for instance the regret analysis of the vanilla MED presented in [Baudry et al., 2023], we upper bound the number of pulls by distinguishing several cases,

This upper bound already exhibits u ε ( T ) , Post-CV and Pre-CV. It remains to prove that the additional terms are upper bounded by constants depending on ε . The term CV-Emp is necessary for all algorithms, while the two last terms are specific to FMED and FIMED . Indeed, they tackle the cases where the sub-optimal arm k is pulled due to a value of µ glyph[star] ( s k ( t )) that deviates from µ 1 and/or µ glyph[star] ( t ) . We start by upper bounding CV-Emp, writing

where the last line comes from Lemma 4 of Cappé et al. [2013]. We choose ε 1 = 2 ε 0 B -µ 1 , and obtain that CV-Emp = O ε 0 (1) thanks to property 2. of Lemma 2. This concludes the proof of Lemma 3 regarding MED and IMED . We now consider the terms specific to FMED and FIMED .

We start with Var-Best, and analyse more precisely the implications of the event

We first prove that the combination of these two events ensure that µ glyph[star] ( s k ( t )) > µ 1 with a proper tuning of ε 0 and ε 2 , since

Hence, we choose ε 2 = 2 ε 0 B -µ 1 to obtain µ glyph[star] ( s k ( t )) glyph[greaterorequalslant] µ 1 + ε 0 3 , if it holds that ε 0 glyph[lessorequalslant] B -µ 1 . Then, we also remark that this event also implies some variation of the best empirical mean between s k ( t ) and t . Hence, only two scenarios can make B k ( t ) hold:

k glyph[star] ( s k ( t )) is pulled between s k ( t ) and t , and µ glyph[star] ( s k ( t )) glyph[greaterorequalslant] µ 1 + ε 0 3 .

k glyph[star] ( s k ( t )) is not pulled between s k ( t ) and t , k glyph[star] ( t ) = j for some j = k glyph[star] ( s k ( t )) . In that case, j has been pulled between s k ( t ) and t and it holds that µ j ( t ) = µ glyph[star] ( t ) glyph[greaterorequalslant] µ glyph[star] ( s k ( t )) glyph[greaterorequalslant] µ 1 + ε 0 3 .

glyph[negationslash]

As each of these scenario can cause at most one pull of arm k , we can upper bound Var-Best by simply counting the number of times an arm j ∈ [ K ] is pulled (at time t ) and either µ j ( t ) glyph[greaterorequalslant] µ 1 + ε 0 or µ j ( t +1) glyph[greaterorequalslant] µ 1 + ε 0 . We hence obtain that

We can apply the exact same arguments for transition-best, except that this time we deduce that k glyph[star] ( t ) was pulled between s k ( t ) and t , and that (if N k glyph[star] ( t ) = n ) µ j,n glyph[lessorequalslant] µ 1 -ε 0 and µ j,n +1 glyph[greaterorequalslant] µ 1 + ε 0 . Depending on the identity of k glyph[star] ( t ) one of these two events has exponentially decreasing probability. Formally, we obtain that

This concludes the proof of Lemma 3 by choosing ε 0 = B -µ 1 4 ε .

Building on Lemma 3, we finish to prove Theorem 1 by upper bounding the post-convergence and pre-convergence terms for all algorithms.

Upper bounding the post-convergence terms Under G k ( t ) and J k ( t ) , it holds thanks to property 1. of Lemma 2 that K F inf ( F k ( t ) , µ glyph[star] ( t )) glyph[greaterorequalslant] K F inf ( F k , µ 1 ) -ε . Hence, MED satisfies

by definition of u ε ( T ) ⌈ log( T ) K F inf ( F k ,µ 1 ) -ε ⌉ . Similarly, the design of G k ( t ) for FMED also ensure that K k ( t ) glyph[greaterorequalslant] K F inf ( F k , µ 1 ) -ε and Post-CV FMED glyph[lessorequalslant] 1 by the same arguments. For IMED , we obtain that

so arm k is never pulled in the post-convergence regime with IMED . Again, these arguments directly translate to Post-CV FIMED .

Upper bounding pre-convergence term Interestingly, the design of K k ( t ) , satisfying K k ( t ) glyph[greaterorequalslant] K F inf ( F k ( t ) , µ glyph[star] ( t )) , makes FMED and FIMED more exploratory than MED and IMED . Hence, we can unify the proof for the vanilla algorithms and their fast update in this part.

We start with MED , following Baudry et al. [2023] we first use that for any sampling probability p k ( t ) it holds that p k ( t ) glyph[lessorequalslant] p k ( t ) p 1 ( t ) p 1 ( t ) glyph[lessorequalslant] 1 -p 1 ( t ) p 1 ( t ) p 1 ( t ) , and obtain that

Let us denote by p FMED 1 ( t ) the sampling probability of arm 1 under FMED . Using that ∑ K k =1 e N k ( t ) K k ( t ) we obtain that

where p MED 1 ( t ) is itself a lower bound on the sampling probability of arm 1 under MED (we use this notation with a slight abuse). If follows that Pre-CV FMED admits the same upper bound as Pre-CV MED . The rest of the proof can be found in Baudry et al. [2023], we briefly recall it from completeness. We first obtain that

Then using properties 3. and 4. from Lemma 2 and K F inf ( F 1 ,n , µ 1 ) glyph[lessorequalslant] K + := log ( B -b B -µ 1 ) we conclude that

with δ ε 0 = 2 ε 2 0 ( B -b ) 2 . As a consequence, Pre-CV FMED = O ε 0 (1) too. For FIMED the relationship with the pre-convergence term of IMED is even more straightforward,

The last statement can be deduced from the regret analysis presented in [Honda and Takemura, 2015], but we now propose a simpler proof of independent interest. We use that if A t +1 = k , then the index of arm k is smaller than the index of arm 1 , and that N k ( t ) > u ε ( T ) to first write that

To further upper bound this term, we observe that since K F inf ( F 1 ,n , µ 1 -ε 0 ) glyph[lessorequalslant] log ( B -b B -µ 1 + ε 0 ) := K + ε 0 , the events considered above cannot happen if n 1 K + ε 0 +log( n 1 ) glyph[lessorequalslant] log( u ε ( T )) glyph[lessorequalslant] log( n k ) . Hence, there exists a function g : T ↦→ N satisfying g ( T ) → + ∞ ( g ( T ) would be of order log log( T ) without the additional log( n 1 ) ) such that

Then, using again properties 3. and 4. of Lemma 2 we obtain that

where the conclusion comes from the fact that the remaining sum is the remainder of a convergent series, or in other words Pre-CV IMED glyph[lessorequalslant] a T log( T ) where a T → 0 . This is sufficient to conclude the proof, by noting that a careful tuning of ε 0 (for instance as log log( T ) allows to obtain that lim sup E [ N k ( T )] log( T ) glyph[lessorequalslant] 1 K F inf ( F k ,µ 1 ) .

C Proof of Theorem 2: regret analysis of OMED and OIMED

We first recall the theorem.

Theorem 2 (Regret bound for online MED algorithms) . Consider ( F 1 , . . . , F K ) ∈ F K , and µ glyph[star] = max k ∈ [ K ] µ k < µ max, for a known µ max < B . Assume that OMED and OIMED use a portfolio selection algorithm ALG that satisfies for any ( k, glyph[lscript] ) ∈ [ K ] 2 and n ∈ N the deterministic guarantee

Then, both OMED and OIMED satisfy Equation (12) (with a different second-order term) and are asymptotically optimal : they enjoy the same asymptotic guarantees as MED and IMED .

Proof. As for the proof of Theorem 1 we start by proving a first regret upper bound that holds for both algorithms.

Lemma 4 (Generic upper bound) . For any sub-optimal arm k , for any ε > 0 it holds that glyph[negationslash]

Before proving this result, let us detail some intuitions. First, we remark that the two expectations need to be second order terms to make the algorithms asymptotically optimal. The first expectation corresponds to pulls of arm k in a post-convergence regime , where arm 1 is the leader and the empirical distribution of arm k is close to F k . On the contrary, the second term is the expected number of duels lost by arm 1 as a challenger, against a sub-optimal leader. For this reason, we name these two terms respectively Post-CV and Pre-CV (where CV abbreviates convergence). Note that Lemma 4 actually holds for any bandit algorithms that would use the same duel-based structure, independently of what they do during the non-greedy duels.

Proof. Westart the proof by introducing some notation. Due to the specific structure of the duel-based algorithm we define for a challenger/leader pair ( k, glyph[lscript] ) a pseudo-count ˜ N k,glyph[lscript] ( t ) and a corresponding empirical cdf ˜ F k,glyph[lscript] ( t ) : they are computed by considering only the observations of arm k that have been collected when arm k was pulled after a non-greedy duel ( Z k ( t ) = 1 ) performed against the leader glyph[lscript] t = glyph[lscript] . Hence, ˜ N k,glyph[lscript] ( t ) glyph[lessorequalslant] N k ( t ) . In several parts of the proof we also use constants x k ∈ ( µ k , µ 1 ) , that we arbitrarily set to µ k + µ 1 2 . Finally, we introduce two ways to denote the rewards: we continue to call X k,n the n -th reward received by arm k , but we also introduce the notation X k,glyph[lscript],n for the n -th reward received by arm k after a non-greedy duel against the leader glyph[lscript] .

We start the analysis by considering the case when the best arm is the leader, and the alternative. For each sub-optimal arm k , it holds that glyph[negationslash]

We then consider the favorable case where arm 1 the leader, splitting the cases according to the value of Z k ( t ) .

We now upper bound B 1 using that if glyph[lscript] t = 1 then N 1 ( t ) glyph[greaterorequalslant] t/K , and that if µ k ( t ) glyph[greaterorequalslant] µ 1 ( t ) then either µ k ( t ) glyph[greaterorequalslant] x k or µ 1 ( t ) glyph[lessorequalslant] x k . By starting N k ( t ) at 1 we cover all possible scenarios for Z k ( t ) = 0 (including µ glyph[lscript] t ( t ) glyph[greaterorequalslant] µ max). We then obtain

We then consider B 2 . We use that for any n , { k ∈ A t +1 , Z k ( t ) = 1 , ˜ N k, 1 ( t ) = n } = { ˜ N k, 1 ( t +1) = ˜ N k, 1 ( t ) + 1 , ˜ N k, 1 ( t ) = n } and can happen only once. For any u ε ( T ) ∈ N , we hence obtain that

We then upper bound B 2 by considering separately cases when G k ( t ) holds or not,

We remark the first term is exactly the post-convergence term that we introduced in the lemma. Hence, we leave this expression as it is for this first result. We then consider B ′ 2 . We use Equation (14), for any ε 0 > 0 it holds that

Hence, we obtain that

Note that we could have stated different thresholds for each term, but we choose ε/ 3 in each case for simplicity.

We use the first side of our assumption on the regret of the portfolio algorithm: if R n = o ( n ) , then there exists n 0 ∈ N large enough such that for n glyph[greaterorequalslant] n ε , R n glyph[lessorequalslant] nε 3 . Hence, by defining u ε ( T ) glyph[greaterorequalslant] n ε we ensure that ¯ G k ( t ) is not due to a large portfolio regret.

We now consider the term involving K F inf ( ˜ F k ( t ) , µ 1 -ε 0 ) , which is analogous to the CV-Emp term of the proof of Theorem 1 (Appendix B). We hence directly write that

by choosing ε 0 = ( B -µ 1 ) ε 6 . Finally, we use that B k, 1 , ˜ N k, 1 ( t ) ( µ 1 -ε 0 ) glyph[greaterorequalslant] ˜ N k, 1 ( t ) ε/ 3 is possible only if at least one empirical mean computed with a 'reasonable' sample size deviates. More precisely,

where we used that, thanks to our algorithm, at any time s for which the estimate L k, 1 ( t ) was incremented N 1 ( s ) glyph[greaterorequalslant] f ( ˜ N k, 1 ( s )) was satisfied (check to decide that the duel is non-greedy). Interestingly, we now get an event that only depends on ˜ N k, 1 ( t ) . Using Lemma 1 we finally get that

thanks to Hoeffding's inequality. This last step allows to conclude that B ′ 2 = O ε (1) , using that f ( s ) glyph[greaterorequalslant] s for any s ∈ N .

Remark 1. We can see that at this step N 1 ( s ) glyph[greaterorequalslant] N k ( s ) was sufficient, the enforcement of N 1 ( s ) glyph[greaterorequalslant] f ( N k ( s )) is necessary only in the 'pre-convergence' analysis.

glyph[negationslash]

Upper bounding A 2 , glyph[lscript] t = 1 : For this part of the proof, we mainly use techniques from [Chan, 2020, Baudry et al., 2021b]. In particular, we use that if the current leader is a sub-optimal arm then either 1 has already been leader and has lost leadership or 1 has never been leader. Formally, we define the event

Then, we first upper bound the term

glyph[negationslash]

glyph[negationslash]

We use that if D t holds and glyph[lscript] t = 1 then arm 1 was the leader at some point between t/ 4 and t and lost its leadership. Furthermore, a change of leadership from 1 to j at time s can only happen if (1) arm j has been pulled at the previous round, (2) the two arms now satisfy N 1 ( s ) = N j ( s ) = n for some n glyph[greaterorequalslant] s/K glyph[greaterorequalslant] t/ (4 K ) , and (3) µ 1 ,n glyph[lessorequalslant] µ j,n . Thanks to these properties, we obtain with some union bounds that

We now upper bound the term

glyph[negationslash]

Following Chan [2020], we use that if arm 1 has never been leader between t/ 4 and t , then it has necessarily lost at least t/ 4 duels in that time interval. Using Markov inequality, we hence obtain that glyph[negationslash]

glyph[negationslash]

Thank to these simple tricks, we are back to upper bounding the total number of duels lost by arm 1 while not being the leader at the cost of a multiplicative constant, which is close to the remaining glyph[negationslash]

glyph[negationslash]

term in our statement. The last step of this first generic analysis consists in upper bounding the regret caused by Z 1 ( t ) = 0 by a constant. We denote by f -1 the function satisfying f -1 ( f ( s )) = s for any s ∈ N . We use that the greedy duel is caused by either µ glyph[lscript] t ( t ) glyph[greaterorequalslant] µ max or ˜ N 1 ,glyph[lscript] t ( t ) glyph[greaterorequalslant] f -1 ( t/K ) , glyph[negationslash]

where we grouped upper bounded the terms corresponding to µ j ( t ) glyph[greaterorequalslant] µ max by the terms corresponding to µ j ( t ) glyph[greaterorequalslant] x j for simplicity. Finally, the remaining term of our upper bound is exactly glyph[negationslash]

as stated in the Lemma 4. This concludes the proof of the lemma,

We now prove the following lemma, that concludes the proof of Theorem 2.

Lemma 5. OMED and OIMED both satisfy

Proof. We need to upper bound four terms. We start with the upper bounds of the two postconvergence terms, that are straightforward thanks to the tuning of u ε ( T ) and the definition of G k ( t ) .

Upper bounding Post-CV We start with OMED ,

thanks to the definition of u ε ( T ) . For OIMED , we obtain

again thanks to the definition of u ε ( T ) .

Upper bounding Pre-CV, OMED We recall that

glyph[negationslash]

We use the notation p 1 ( t ) = p 1 ,j, ˜ N 1 ,glyph[lscript] t ( t ) , and p 1 ,j,n = e -L 1 ,j,n . Then, with the same arguments as for the regret analysis of the vanilla MED (Baudry et al. [2023], Appendix B) we obtain that glyph[negationslash]

glyph[negationslash]

Now, thanks to Equation (14) we can relate L 1 ,j,n to K F inf ( F n , x ) for any x ∈ R . We again choose x j for convenience, and obtain that

To relate to the proof for MED we need the portfolio regret and bias to be small enough. We use the second side of our assumption on the regret, which is that -R 1 ,j,n = o ( n ) for any j, n . Hence, for n large enough it holds for instance that -R 1 ,j,n glyph[lessorequalslant] δ j 3 n , with

where δ j > 0 is ensured by property 3. of Lemma 2. For the bias, we first obtain with Lemma 1 that

We then define a 'good event' under which the bias is controlled,

We then use a similar proof as for the post-convergence term. We first consider

that can be upper bounded with the same proof as the vanilla MED from [Baudry et al., 2023]. Using properties 3. and 4. of Lemma 2, we obtain that

with K + j = log ( B -b B -x j ) . We then tackle the case where B j,n does not hold, which is possible only if µ j,s glyph[greaterorequalslant] x j for some s glyph[greaterorequalslant] ( nδ j ) 2 . Furthermore, we also use the trivial bound -B 1 ,j,n ( x j ) glyph[lessorequalslant] n thanks to Lemma 1. We then obtain

We can conclude at this step using that e n (1+ δ j ) -f ( nδ j ) I 1 ( x j ) → 0 , as we ensured that n = o ( f ( n )) , so it simply holds that

Upper bounding Pre-CV, OIMED We again consider the two cases, depending on if B ( t ) holds or not. When this is true, we can use a similar proof to the one of vanilla IMED that we propose in Appendix B,

where we again used properties 3. and 4. of Lemma 2 to upper bound the probability. The convergence of this term is ensured by the fact that the sum on n starts at log( t/K ) K + . To formally prove this we can for instance use that there exists a constant C j (that typically scales in O ( δ -2 j ) up to logarithm terms)

such that

∀

n

∈

, n

(

n

+2)

C

e

n

δ j

6

for any

n

∈

, so

N

glyph[lessorequalslant]

j

N

We then consider ¯ B ( t ) , under which the bias can be up to ˜ N 1 ,j ( t ) . In that case, we use the same proof techniques as for OMED , with

We then use that the right-hand term converges to 0 , so Q 1 = O (1) ⇒ Q 2 = (1) .

Remark 2. The enforcement of a sufficient sample size for the leader is justified by the analysis of the pre-convergence term. In this regime a linear bias may cost up to e n for each sample size n . Hence, a concentration of the mean with a rate e -cn is not sufficient, especially if c is small (which happens if the gaps are small). Furthermore, this kind of rate cannot be avoided without actively controlling the sample size of the leader at each update.

This concludes the proof of Theorem 2.

D Accuracy of K F inf approximation and discretization trick

In this section we provide more details on the discussion of Section 1. In particular, we elaborate on the discretization trick proposed in Riou and Honda [2020] and on the sub-optimality of the algorithms that do not target K F inf specifically.

For simplicity we assume that b = 0 and B = 1 , and all results hold for general ranges [ b, B ] ⊂ R up to a rescaling of the rewards: X ∈ [ b, B ] ⇒ X -b B -b ∈ [0 , 1] .

Sub-optimality of 2∆ 2 and kl We denote by kl : [0 , 1] × [0 , 1) → R + the KL divergence between two Bernoulli distributions, expressed in terms of their expectations by

Consider F ∈ F of mean µ ∈ [0 , 1) and µ glyph[star] ∈ [ µ, 1) , and let us define ∆ = µ glyph[star] -µ . First, it is known (see e.g. Cappé et al. [2013]) that

This result (partially) explains why bandit algorithms adapted for 1 / 4 -sub-gaussian or Bernoulli distributions work for general bounded distributions (UCB1 [Auer et al., 2002], kl-UCB [Cappé et al., 2013]), since a smaller constant means a more exploratory algorithm. A natural question is to determine how sub-optimal these algorithms can be in the general case. In the following we assume that ∆ is small to perform some first-order approximations.

First, a simple computation shows that kl ( µ, µ glyph[star] ) ≈ ∆ 2 µ (1 -µ ) : kl adapts to the true variance of the Bernoulli distributions, exploiting better the geometry of F . The sub-gaussian approximation is tight

only if µ is close to 1 / 2 but not near the boundaries of the support, which is not surprising. We now investigate the gap between K F inf and kl, and recall that the two quantities match for Bernoulli distributions. The following result demonstrates that the ratio between the two can be arbitrarily large if the distributions are in fact 'far' from being Bernoulli.

Lemma 6 (kl vs K F inf ) . For µ ∈ (0 , 1) and F ∈ F [0 , 1] of mean µ -∆ , the scaling of K F inf ( F,µ ) kl ( µ F ,µ ) can be as large as µ ∆ , with the maximum achieved when F is the Dirac distribution with support µ -∆ .

Proof. First, Jensen inequality and equation (4) provide that

where δ x is the Dirac distribution at x . Then, approximating K F inf ( F, µ ) by ∆ 1 -µ we obtain that

We could argue that Dirac (or highly concentrated) distributions may be unlikely in most applications, however the use-case presented in Section 1 (Figure 1) shows that large ratios K F inf / kl may already be observed with more natural examples. Finally, to compare the impact of the approximation on the asymptotic guarantees of an algorithm on a K -armed bandits we can compare ∑ k :∆ k > 0 ∆ k K F inf ( F k ,µ glyph[star] ) and ∑ k :∆ k > 0 1 ∆ k .

Discretization trick We now detail the trick proposed by Riou and Honda [2020], and inspired for instance by the binarization trick of Agrawal and Goyal [2012]. For any M ∈ N , we fix an arbitrary grid X M = ( x 1 = 0 < x 2 < · · · < x M = 1) ∈ [0 , 1] M . Then, the trick consists in transforming any incoming reward X / ∈ X M in a discrete reward Y ∈ X M , as follows:

If this procedure is used with all rewards collected, one can use a bandit algorithm calibrated for multinomial distributions with support X m . It is easy to see that E [ Y ] = X , so the order of the arms do not change with this discretization. Furthermore, the bandit problem becomes equivalent to a problem where the distributions would be exactly ( F X M 1 , . . . , F X M K ) ∈ F K where, for each k ∈ [ K ] , F X M k is the distribution of the discretized rewards (with reference grid X M ) generated with i.i.d. samples from F k . It is hence clear that an asymptotically optimal bandit algorithm for (general bounded distributions) is guaranteed an asymptotic regret scaling in ∑ k :∆ k > 0 ∆ k K F inf ( F X k M ) log( T ) , and be optimal if the distributions F 1 , . . . , F K are already supported on X M only.

We now discuss the performance that can be achieved with such discretization. Let us assume that a constant step x m +1 -x m = B -b M -1 is chosen for all m ∈ [ M -1] . First, due to the compactness of F we can state that there exists a distribution G glyph[star] satisfying K F inf ( F k , µ glyph[star] ) = KL ( F k , G glyph[star] ) . We then denote by G X M glyph[star] its corresponding discretized distribution on the grid X M . Thanks to the data processing inequality, we obtain that

which is itself larger than K F inf ( F X M k , µ glyph[star] ) by definition, since E X ∼ G X glyph[star] M [ X ] = E X ∼ G glyph[star] [ X ] glyph[greaterorequalslant] µ glyph[star] . Finally, using that F X M k cannot be 'closer' to µ glyph[star] than a translation of F k by η , and Lemma 4 of Cappé et al. [2013], we obtain that

Hence, we see that the approximation can be good for large M . However, if K F inf ( F k , µ glyph[star] ) glyph[lessorequalslant] 1 M -1 × B -b B -µ glyph[star] (which is possible since M is fixed without knowing the distributions) then the lower bound is vacuous. In fact, we can extend Lemma 6 to any discretization process: small gaps (compared to M -1 ) can make the ratio K F inf ( F k , µ glyph[star] ) / K F inf ( F X m k , µ ) arbitrarily large. Hence, even if this technique has some convenient properties, it is not entirely satisfying. This justifies our quest for alternatives that would preserve the asymptotic optimality of the algorithms.

E Online K F inf computation with portfolio optimization algorithms

In this section we provide additional details regarding OMED and OIMED . We first prove Lemma 1, that is essential to their regret analysis, then we detail the Soft-Bayes algorithm [Orseau et al., 2017] that we choose in our practical implementation of the algorithms.

E.1 Proof of Lemma 1

Lemma 1 (Deviations of B k,glyph[lscript],n ( µ ) ) . Let µ ∈ (0 , 1) , and set C k,glyph[lscript],i = B -µ glyph[lscript] ( t k,glyph[lscript],i ) B -µ it holds that

Proof. We introduce some notation for convenience. First, we denote X k,glyph[lscript],i by X i and µ glyph[star] ( t k,glyph[lscript],i ) by µ glyph[star] ( i ) . Then, we define Y i = X i -µ glyph[star] ( i ) B -µ glyph[star] ( i ) , Z i = X i -µ B -µ , and λ = argmax λ ∈ [0 , 1] ∑ n i =1 log(1 -λZ i ) .

We use two elementary analysis properties. First, 1 -λZ i 1 -λY i is positive and non-decreasing if Y i glyph[greaterorequalslant] Z i . Then, Y i glyph[greaterorequalslant] Z i only if µ glyph[star] ( i ) glyph[lessorequalslant] µ . Otherwise, log(1 -λZ i ) -log(1 -λY i ) glyph[lessorequalslant] 0 . Hence, it holds that

We then plug the expression of Y i and Z i in this bound, obtaining that

which gives the result. Applying the exact same steps provides the other direction,

E.2 Soft-Bayes

In this section we detail the Soft-Bayes algorithm, proposed in Orseau et al. [2017]. We recall that the K F inf estimation is a portfolio optimization problem of dimension 2 . At each step n , the portfolio algorithm decides an allocation (1 -λ n , λ n ) between two assets, that provide a payoff (1 , Y n ) . We use in this section Y n to denote for simplification B -X k,glyph[lscript],n B -µ glyph[star] ( t k,glyph[lscript],n ) , that is used in our implementation of the estimate L k,glyph[lscript],n for the challenger/leader pair ( k, glyph[lscript] ) .

For the anytime version of the algorithm, a sequence of learning rates ( η n ) n ∈ N is provided as an input of the algorithm. We then define the update rule of the anytime Soft-Bayes in Algorithm 10 below.

Algorithm 10 Anytime Soft-Bayes [Orseau et al., 2017]

Input: Parameter λ n , sequence of learning rates ( η n ) n ∈ N , initial parameter λ 1 = 1 / 2

Orseau et al. [2017] prove that defining the learning rate as η n = √ log(2) 4 n ensures an upper bound on the portfolio regret

However, they do not provide a lower bound on the regret.

F Supplementary experimental results

F.1 Detail of the costs of each algorithm

We start this section by detailing the computation of the costs of the algorithms presented in Table 2. We first recall the results that we want to prove.

In the table, n is a number of observations for 1 arm, and the costs are computed for this arm only for simplicity.

Memory cost We start with the memory, which is easier to estimate. Most optimal algorithms (KL-UCB, NPTS, MED / IMED , FMED / FIMED ) require to store all the observations, for a cost of n . On the contrary, UCB and kl-UCB require constant memory, since they just need to store the number of pulls and the empirical means of each arm. Finally, the multinomial version of the algorithms have a memory cost of M : they need to store the total number of pulls, the frequency of each item ( x 1 , . . . , x M ) on the chosen grid, and the empirical means.

Finally, K appears in the memory cost of OIMED because for each arm k we need to store (at most) K online estimates of K F inf and pseudo-counts, in addition to the total number of pulls and empirical mean of the arm. This is a worst-case memory, in case each arm have been leader at least once. The total memory cost of OIMED (summing over all arms) is then at most O ( K 2 ) , while it is O ( T ) for the optimal algorithms that store all observations.

Computation costs First, we assume that any algorithm that use only quantities that can be updated sequentially has a cost of O (1) . We include in this category updates of means, number of pulls, or the output of a portfolio algorithm with a running time independent of n . We insist that in practice the cost of these algorithms is not exactly the same, but it can be verified that they remain constant over time.

We now consider NPTS: the main cost comes from the sampling of n +1 weights from a Dirichlet distribution of parameter (1 , . . . , 1) ( n ones). This sampling is linear in n , since it can be done by sampling n +1 i.i.d. samples R 1 , . . . , R n +1 from the exponential distribution E (1) , and then defining w i = R i ∑ n +1 j =1 R j for any i ∈ [ n +1] . This justifies the cost in O ( n ) in the table.

The remaining algorithms (KL-UCB and MED algorithms) require to solve the optimization problem defined by (4) to compute empirical K F inf . This can be done with any search algorithm, which costs n log(1 /ε ) where ε is the precision asked for the solution. In the implementations we always choose ε = 1 /n , so that the precision is enough to preserve the theoretical guarantees of the algorithms. This already provides the cost of the MED algorithms, that require to compute exactly one K F inf for each arm at each time step. For FMED / FIMED , we precise that this computation happens only if the arm wad pulled. Finally, for the discretized version of the algorithms the computation time becomes linear in M instead of n , but the log( n ) is kept since the precision is still 1 /n .

For KL-UCB and kl-UCB, another optimization procedure is needed: for a given function f we solve max { µ ∈ [ b, B ] : N k ( t ) K F inf ( F k ( t ) , µ ) glyph[lessorequalslant] f ( t ) } . We also set the precision of this search to 1 /n , which gives the supplementary multiplicative log( n ) factor compared with MED algorithms: each K F inf computation costs O ( n log( n )) , and we perform O (log( n )) such computations for a step of KL-UCB. This is also the reason why the cost of kl-UCB is not constant: we perform O (log( n )) computations of kl.

Remark 3 (Total computation time) . We detailed the cost for one arm and one step of the algorithms only for simplicity, however for those depending on n there are some differences when considering the total computation time. Indeed, for MED algorithms we do not compute K F inf for the best empirical arm. However, this arm is expected to converge to the true best arm and to be sampled linearly. Hence, if there is only one best arm the costs presented in Table 3 will be paid only for n = O (log( T )) . On the contrary, KL-UCB and NPTS will pay their cost for all arms, including the best empirical arm, so one of them satisfy n = Ω( T ) . The analysis of these algorithms suggest that this cost may not be necessary (see Remark 3 of [Riou and Honda, 2020]), nonetheless this is how they are typically implemented in the literature.

F.2 DSSAT distributions

The DSSAT problem was presented in Section 1 and the experimental results in the last Section 4. To give the reader the full picture of the problem, we plot in Figure 5 all the densities of the seven distributions.

0

2000

4000

6000

8000

10000



0

1

2

3

4

5



1e

5

This plot visually confirms the multimodality of all distributions and the difficulty to fit this problem into a parametric framework. We complete the Table 1 presented in the introduction by listing in Table 4 all ratios of the form K F inf kl and K F inf 2∆ 2 , thus confirming the theoretical importance of using the true K F inf quantities over their usual relaxed versions.

To ground this theoretical analysis, we plot again the regret curves and average run time of tested algorithms (Figure 6) as well as Table 5 that compiles the average regret and average run time on the DSSAT experiment at the horizon 10 000. Reading this table, it is clear that FIMED improves the run time of IMED without compromising the regret and that OIMED improves the run time even further, close to the one of UCB , but at the cost of loosing a bit in the regret term. However, of the fastest methods (those in O (1) run time), OIMED clearly has the smallest regret. If the practitioner is willing to pay a bit more time complexity and space complexity, then our proposed FIMED seems to be the best algorithmic method.

0

2000

4000

6000

8000

10000



0

50

100

150

200

250



UCB

NPTS

IMED

IMED-kl

OIMED

FIMED

kl-UCB

KL-UCB

0

2000

4000

6000

8000

10000



0

50

100

150

200

250



F.3 Additional bandit experiments

Here, we confirm the analysis of the previous section. First, we benchmark our FIMED and OIMED algorithms against IMED using the experimental setting of the original paper by Honda and Takemura [2015] in which the IMED algorithm is introduced. Then, we run two sets of experiments: one with Bernoulli distributions to complete our main experiment presented in Section 4, Figure 3, and another one with Beta distributions.

F.3.1 Original testbed of IMED

Negative exponential We consider the negative exponential setting in which the law of an arm k is such that X k = 1 -X ′ k where X ′ k is an exponential random variable with parameter µ ′ k . In that case, the expected reward of arm k is µ k = 1 -µ ′ k and its support is in ( -∞ , 1] . In Figure 7, we show simulation result of an experiment on a 5-armed negative exponential bandit problem

where, following the setting of Honda and Takemura [2015], { µ ′ k } k = { 1 / 5 , 1 / 4 , 1 / 3 , 1 / 2 , 1 } , i.e. { µ k } k = { 4 / 5 , 3 / 4 , 2 / 3 , 1 / 2 , 0 } .

regret_linear_quantile.pdf

0

2000

4000

6000

8000

10000

Number of samples

0

10

20

30

40

50

60

Mean regret

IMED

R=24.40

FIMED

R=24.50

OIMED

R=43.91

time.pdf

0

2000

4000

6000

8000

10000

number of samples

0

10

20

30

40

run time

IMED

T=36.0 s

FIMED

T=4.3 s

OIMED

T=1.3 s

On this setting, the benefit of the FIMED algorithm over the original IMED can easily be read from Figure 7. FIMED enjoys the same regret as IMED while having an order of magnitude smaller computational time, confirming our theoretical analysis of FIMED . OIMED further reduce the computational time (and space) complexity, however at the cost of a larger experimental regret. The regret still exhibit the logarithmic shape in a reasonable experimental time and the choice is left to the practitioner whether to trade regret against time and space complexity or not.

Truncated Gaussian We continue benchmarking our algorithms using the original testbed of Honda and Takemura [2015], we run two experiments where arms are truncated Gaussian distributions. In the first experiment, Figure 8, the distributions are truncated on [0 , 1] . In the second experiment, Figure 9, the distributions are truncated on ( -∞ , 1] . In both experiments, we use the same set of Gaussian distributions, of means (before truncation) { 0 . 6 , 0 . 5 , 0 . 5 , 0 . 4 , 0 . 4 } and variances { 0 . 4 , 0 . 2 , 0 . 4 , 0 . 2 , 0 . 4 } . After truncation on [0 , 1] , the expected values are { 0 . 519 , 0 . 5 , 0 . 5 , 0 . 465 , 0 . 481 } , and after truncation on ( -∞ , 1] , the expected values are { 0 . 319 , 0 . 390 , 0 . 265 , 0 . 320 , 0 . 206 } .

regret_linear_std.pdf

0

2000

4000

6000

8000

10000

0

10

20

30

40

50

60

Mean regret

IMED

R=31.05

FIMED

R=30.06

OIMED

R=46.60

time.pdf

0

2000

4000

6000

8000

10000

0

10

20

30

40

run time

IMED

T=40.0 s

FIMED

T=6.5 s

OIMED

T=3.1 s

Number of samples number of samples

Quite accordingly to the intuition, the regret of all algorithms are better on the [0 , 1] -truncated bandit problem than on the ( -∞ , 1] -truncated one. Indeed, the variance of the former is smaller than the variance on the latter and an increased variance, all things being equal, tend to decrease the divergence between distributions, increasing the need for exploration. Both on Figure 8 and Figure 9, the regret pf FIMED is on par with that of IMED while being much faster. Following our analysis of running times, OIMED indeed is experimentally the fastest but does seem to trade this speed for a larger regret. However, one can see that in Figure 9 that the regret gap is smaller than in Figure 9, probably because the variance is larger. This variance interpretation is confirmed by the experiment on a Bernoulli bandit that is presented in Honda and Takemura [2015].

regret_linear_std.pdf

0

2000

4000

6000

8000

10000

Number of samples

0

10

20

30

40

50

60

70

80

Mean regret

IMED

R=43.92

FIMED

R=43.50

OIMED

R=56.61

time.pdf

0

2000

4000

6000

8000

10000

number of samples

0

10

20

30

40

50

run time

IMED

T=39.9 s

FIMED

T=8.2 s

OIMED

T=3.2 s

Bernoulli In Figure 10, we show simulation results for a 10-arms bandit with Bernoulli rewards with expected values { 0 . 1 , 0 . 05 , 0 . 05 , 0 . 05 , 0 . 02 , 0 . 02 , 0 . 02 , 0 . 01 , 0 . 01 , 0 . 01 } . It can be seen that the performance gap between OIMED and IMED is larger than in any of the previous experiments. It can be attributed to the fact that all the rewards are all very close to zero and the variance of the distributions are small. However, the learning rate of the portfolio algorithm does not take those information into account, and it takes some time for the portfolio algorithm to converge, a time that can be seen in Figure 10 where the regret of OIMED is close to linear with no variance (pure exploration) for the 3 500 first time steps. In contrast, IMED and OIMED , which do not rely on a learning rate that is independent of the variance, have a much smaller regret.

regret_linear_quantile.pdf

0

2000

4000

6000

8000

10000

Number of samples

0

100

200

300

400

500

Mean regret

IMED

R=70.82

FIMED

R=69.79

OIMED

R=381.99

Reproducing the experiments of the original IMED Honda and Takemura [2015], we confirmed that our theoretical analysis is experimentally supported. FIMED always seem to be on par with the original IMED algorithm while enjoying an order of magnitude faster numerical complexity, as proven in the analysis of running times. OIMED enjoys competitive regret, albeit experimentally larger than the original IMED . However, it has the smallest computational time complexity and its space complexity also is minimal. Therefore, it seems like there is currently a trade-off between achieving the smallest empirical computational complexity (and space complexity) and achieving the smallest cumulative regret. The original IMED lies at one extreme, OIMED at the other extreme and FIMED sits in-between. The algorithms studied in this paper allows the practitioner to choose the trade-off that suit her the best. Whether one can find an algorithm as fast as OIMED with experimental regret guarantees that exactly match IMED is a question for future work.

F.3.2 Bernoulli bandits

In Section 4, we studied a Bernoulli bandit problem where all the means are close to 0.5 (Figure 3), identified as a 'difficult' problem because the variance of the distributions in maximized among those supported in [0 , 1] . Here, we explore two Bernoulli bandit problems, one where the means are located close to 1 (Figure 11, Table 6) and another hereafter where means are located close to 0 (Figure 12, Table 7). For each experiment, we plot the regret curves, the run time curves, as well as a table compiling the average regrets and run times of tested algorithms at the time horizon T = 10000 .

0

2000

4000

6000

8000

10000



0

50

100

150

200

250

300



UCB

NPTS

IMED

OIMED

FIMED

KL-UCB

0

2000

4000

6000

8000

10000

0

2

4

6

8

10

12

14

16





We recall that for Bernoulli bandit, IMED and kl-IMED are the same algorithms (we assume that IMED is implemented as kl-IMED while only 0s and 1s are observed). For FIMED we could have done the same thing to improve its run time without changing its regret in order to emphasize that FIMED is always faster (or as fast as) kl-UCB , even when using the Bernoulli kl. While the regret and time values change from one experiment to another, the conclusions that can be drawn from them do not, especially when considering the confidence intervals represented by the 10%-90% quantiles. Experimentally, OIMED (or OMED , see Section F.4) should be the preferred O (1) method and FIMED (or FMED , see Section F.4) should be preferred if we really target an empirically optimal regret without compromising too much on the running time.

0

2000

4000

6000

8000

10000



0

50

100

150

200

250

300

350



UCB

NPTS

IMED

OIMED

FIMED

KL-UCB

0

2000

4000

6000

8000

10000



0

2

4

6

8

10

12

14

16



F.3.3 Beta bandits

Beta distribution of a given mean can have different shapes. In particular, a Beta distribution can be close to a Bernoulli distribution (shape parameter close to zero) with most of the density located around 0 and 1, close to a Dirac distribution (shape parameter significantly larger than one) with most of the density located around the mean, and close to a truncated Gaussian distribution (shape parameter larger than one) with the characteristic bell shape distribution around the mean.

In Section 4, we studied a Bernoulli bandit problem where all the means are close to 0.5 (Figure 3). Here, using the same set of means, { 0 . 3 , 0 . 4 , 0 . 45 , 0 . 5 , 0 . 52 , 0 . 55 } , we run two experiments, one on Beta distributions with an intermediate shape parameter of 5 (Figure 13) and another where Beta distributions have a shape parameter of 50 (Figure 15), hence with highly piked distributions. We do so to illustrate the effect of changing the shape of distributions without changing the set of means.

0.0

0.2

0.4

0.6

0.8

1.0



0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00



0

2000

4000

6000

8000

10000



0

50

100

150

200

250

300

350



UCB

NPTS

IMED

IMED-kl

OIMED

FIMED

kl-UCB

KL-UCB

0

2000

4000

6000

8000

10000



0

50

100

150

200

250

300

350



Comparing the regret curves of Figure 3, Figure 14, and Figure 16, it is clear that the Bernoulli distributions does induce the larger variance on the regret curves as we identified while the Beta distributions with the largest shape parameter (more concentrated around their means) induce the smallest variance on the regret curve. This, once again, confirms our theoretical findings. While the order of regret curves is globally preserved, one can see on Figure 3 that, except for UCB, the Bernoulli experiment makes all curves to be very similar, which is normal since, except for UCB, all algorithms are roughly solving the same problem (all sample are 0s and 1s). When dealing with Beta distributions, IMED is different than kl-IMED and KL-UCB is different than kl-UCB .

0.0

0.2

0.4

0.6

0.8

1.0



0

1

2

3

4

5

6



0

2000

4000

6000

8000

10000



0

50

100

150

200

250

300



UCB

NPTS

IMED

IMED-kl

OIMED

FIMED

kl-UCB

KL-UCB

0

2000

4000

6000

8000

10000



0

50

100

150

200

250

300



Those experiments confirm our numerical findings that OIMED (or OMED , see Section F.4) should be the preferred method with O (1) computation time, and FIMED (or FMED , see Section F.4), should be preferred if we really target an empirically optimal regret.

F.4 Comparison of MED and IMED versions

In this paper, we derived algorithmic and theoretical results for modification of both IMED and MED algorithms. In this section, we compare IMED , its modifications FIMED and OIMED , MED , and its modifications FMED and OMED . The comparison is made on three experimental setting, the DSSAT bandit problem and Bernoulli bandit problem presented in the main experimental Section 4, as well as a Beta bandit problem with an intermediate shape parameter of 5 and centered means equal to the Bernoulli setting.

0

2000

4000

6000

8000

10000



0

20

40

60

80

100

120



IMED

FIMED

OIMED

MED

FMED

OMED

0

2000

4000

6000

8000

10000



0

10

20

30

40

50

60

70



In Figure 17 and Table 10, we observe that IMED slightly overperforms MED . Consequently, FIMED also slightly overperforms FMED for the regret metric. Regarding the running time, the MED versions of algorithms are a bit slower than the IMED versions due to the numerical complexity of sampling, larger than that of computing the argmin of IMED indexes. Interestingly, OMED overperforms OIMED in terms of average regret performance. As we can see in Figure 18 and Table 11 presenting the results of the Bernoulli experiment, this is not a general rule.

0

2000

4000

6000

8000

10000



0

50

100

150

200



IMED

FIMED

OIMED

MED

FMED

OMED

0

2000

4000

6000

8000

10000



0

10

20

30

40



0

2000

4000

6000

8000

10000



0

20

40

60

80



IMED

FIMED

OIMED

MED

FMED

OMED

0

2000

4000

6000

8000

10000



0

10

20

30

40

50

60



The experiment on Beta distribution, Figure 19 and Table 12, confirms that the regret loss incurred by the transition from MED to OMED is smaller than the one of transitioning from IMED to OIMED . It may

be because the sampling strategy of MED is better to handle the statistical approximation of portfolio algorithms.

F.5 Stability of OIMED with respect to the learning rate

The portfolio algorithm, Soft Bayes, behind OIMED and OMED depends on a hyperparameter, η , the learning rate. A learning rate scheme is prescribed in Theorem 3 of [Orseau et al., 2017] with

where n is the number of collected samples. In section, we test the numerical stability of our OIMED algorithm with respect to the learning rate. To do so, we will replace this original η by

where r will range from 0.01 to 100. We illustrate the stability of OIMED on three bandit settings: the DSSAT bandit problem and Bernoulli problem that were introduced in the main Section 4 and a Beta bandit problem where all the means are centered around 0.5 and the same as in the Bernoulli bandit.

First, we test the stability of our OIMED algorithm on the DSSAT bandit problem that we already used in the main experimental Section 4 and Appendix F. On Figure F.5, the average regret ranges from 103.5 for the smallest values of the parameter r to 88.5 for the largest value of r with an intermediate regret of 99.5 for our original value r = 1 . On this same plot, it is interesting to see that despite the large range of tested parameter size, the quantile tubes remain of roughly the same size with no evident bad behavior. While the effect of changing r may seem important, in the range of tested parameter, the effect of changing r would not have affected the ranking of algorithms in the original experiment, Figure 2 and Table 5. This experiment shows the stability of OIMED with respect to the learning rate, in the sense that a multiplicative constant does not seem to be able to dramatically deteriorate its performance.

0

2000

4000

6000

8000

10000



0

20

40

60

80

100

120

140



OIMED

r = 0.01

r = 0.05

r = 0.1

r = 0.5

r = 1

r = 5

r = 10

r = 50

r = 100

Those findings are confirmed on a second experiment performed on a Bernoulli bandit problem with means { 0 . 3 , 0 . 4 , 0 . 45 , 0 . 5 , 0 . 52 , 0 . 55 } . The regret changes from 107 to 124 when varying the parameter r from 0 . 01 to 100 , and, while the upper quantile is larger for small value of r , we still observe a logarithmic curve which confirms the stability of OIMED .

0

2000

4000

6000

8000

10000



0

50

100

150

200

250



OIMED

r = 0.01

r = 0.05

r = 0.1

r = 0.5

r = 1

r = 5

r = 10

r = 50

r = 100

Finally, we check the effect of the learning rate on two Beta bandit problems where means are { 0 . 3 , 0 . 4 , 0 . 45 , 0 . 5 , 0 . 52 , 0 . 55 } and one problem has Beta distributions with shape parameter 5 and the other with shape parameter 50. Those two Beta bandit problems are presented in Appendix F.3.3.

0

2000

4000

6000

8000

10000



0

20

40

60

80



OIMED

r = 0.01

r = 0.05

r = 0.1

r = 0.5

r = 1

r = 5

r = 10

r = 50

r = 100

0

2000

4000

6000

8000

10000



0

10

20

30

40

50

60



OIMED

r = 0.01

r = 0.05

r = 0.1

r = 0.5

r = 1

r = 5

r = 10

r = 50

r = 100

When the variance of the distributions is smaller (large shape parameter), we can see that regret curves are better separated and the the bonus of choosing a larger learning rate parameter r is amplified. When the variance is larger (small shape parameter), regret curves are closer to each other and more within each others quantile tubes. This last experiment confirms all our previous findings about the numerical stability of OIMED .

F.6 IMED with discretized rewards

In Section 1 of this paper we introduced a known trick to reduce the time and memory complexities of MED / IMED . It consists in using algorithms designed for multinomial rewards, by using a discretization procedure on the collected rewards. We furthermore detailed this technique in Appendix F.1, and proved its inevitable sub-optimality for some problems. We end this section by experimenting with multinomial IMED , comparing the empirical performance and computation time of several instances using different grids. In each case, we fix a number of ticks M , and use the grid { 0 , 1 M -1 , . . . , M -2 M -1 , 1 } ( { 0 , 1 } if M = 2 ). Our objective is to identify the number of ticks needed

to get a regret close to IMED , and the evolution of the computation time with the number of ticks. We first run an experiment on the DSSAT bandit problem, and then on a Beta bandit problem where means are close to zero.

0

2000

4000

6000

8000

10000

0

50

100

150

200



Mult.  IMED - 2 ticks

Mult.  IMED - 3 ticks

Mult.  IMED - 5 ticks

Mult.  IMED - 7 ticks

Mult.  IMED - 10 ticks

Mult.  IMED - 20 ticks

IMED

0

2000

4000

6000

8000

10000

0

10

20

30

40

50

60

70







Unsurprisingly, the more ticks, the better the regret and the larger the running time, with IMED and kl-IMED providing the range for both metrics. For this problem, M = 20 (intervals of length ≈ 0 . 05 between each tick) seems to be enough to make the algorithm using discretized rewards almost match the true IMED in terms of regret, while being 25% faster.

We remark that the evolution of the time complexity per time step is not linear in the number of ticks. This is likely because it is actually dependent of the 'effective' number of ticks used in memory, i.e. those that contain at least 1 samples. Getting at least 1 sample for each tick/arm takes some time, that depends on the shape of the distributions on the [0 , 1] support.

We run a final experiments on a Beta bandit problem that confirms our findings. IMED and kl-IMED again provide the range for both computation time and regret, and for this problem M = 20 is also the value from which IMED and discretized IMED start to match in terms of regret, while the latter is approximately 20% faster.

0

2000

4000

6000

8000

10000

0

20

40

60

80

100

120



Mult.  IMED - 2 ticks

Mult.  IMED - 3 ticks

Mult.  IMED - 5 ticks

Mult.  IMED - 7 ticks

Mult.  IMED - 10 ticks

Mult.  IMED - 20 ticks

IMED

0

2000

4000

6000

8000

10000

0

10

20

30

40

50

60

70







Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the classical regret minimization problem in the stochastic multi-armed bandit setting with arms having distributions with support bounded from above by a known constant B, and sometimes also lower bounded by b. Typically, all the known asymptotically optimal algorithms for regret-minimization problem in the bandit setup depend on computing an optimization problem involving the KL divergences. This is known to be computationally expensive in practice. The main object of study of this paper is K-inf. In this paper, authors develop approximations for empirical version of K-inf and study the performance of algorithms using these approximations, instead of exactly computing K-inf at each step. \n\nThe first of these approximations simply uses the convexity of K-inf in the second argument, while the second one derives from exploiting a connection between K-inf and online no-regret learning.\n\nWhile using the first approximation in IMED and MED still leads to asymp. optimal algorithms, the second approximation requires assumptions on the no-regret algorithm used, and finding a no-regret algo. satisfying those assumptions remains open.\n\nNumerical evidence supporting the theory is also provided.\n\nSTRENGTHS:\nI enjoyed reading the paper. It is generally well-written. While theory for asymptotically optimal algorithms for bandits have lately been developed in great generality, these optimal algorithms can be computationally demanding. These optimal algorithms all mostly depend on empirical K-inf. While the dual of this problem is a convex optimization problem, it is well known that its computation cost increases with number of samples (Cappé et al., 2013, Honda et al., 2015, Agrawal et al., 2021). \n\nThe paper studies an important practical problem of addressing the computational aspect of these optimal algorithms trying to avoid compromizing their optimality. \n\n\nIt is interesting to see the use of ideas from online portfolio selection algorithms in the bandit setting. As pointed by the authors, while this connection was previously observed in developing concentration inequality for K-inf by Agrawal et al. 2021, exploiting it to get fast algorithms is an interesting step in solving the computational aspect of K-inf, which, in my opinion, can be useful in other bandit settings as well.\n\nWEAKNESSES:\n1. The assumption on the no-regret algorithm in the second approach is a bit strange.\n\n2. The paper can be a bit hard to read for someone new in the community. \n    - Before introducing the fast MED algorithms in Section 2, it would be good to introduce the basic idea of \n    MED algorithms (both IMED and MED) or how K-inf appears in them (or point the reader to relevant \n    appendix where it is discussed, if space is a concern).\n    - 1 line justification for the re-formulation of K-inf below line 126.\n    - Details of why K-inf computations only happen for O(log T) times in line 112 of the paper. Though the \n    reader is referred to Section 3 for discussion, I couldn't find a discussion on this there either. \n\n3. In the introduction, the authors mention that KL-UCB requires several computations of K-inf per step, which I believe is not necessary - see, for example, Agrawal et. al, 2021 (Appendix D) where the index is expressed as a single 2D opt. problem. Similar approach can be used for KL-UCB algorithm in the setting considered here.\n\n4. Minor comments:\n    - Line 142: is --> are\n    - Line 143: This is, for example, ...\n\nQUESTIONS:\n1. Can the ideas be extended beyond bounded distribution class? For example, to the moment-bounded class considered in Agrawal et al., 2021 or even the centered-moment bound class considered in Baudry et al., 2023? What would the challenges be in extending?\n\n2. Could you clarify how you obtained the inequality after the data-processing inequality on Page 30, line 703?\n\n3. In Figure 3, since we are looking at performance of OIMED and FIMED, it will be good to include IMED as well.\n\n4. Also see weakness above.\n\nLIMITATIONS:\nnone", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

comments: SUMMARY:
In this paper they provide fast optimal algorithms for (non-parametric) stochastic bandits.
Specifically, they consider the MED family of algorithms that require the computation of $\mathcal{K}_{inf}$.
Based on this family of algorithms they construct new variants that require the computation of $\mathcal{K}_{inf}$ for one arm and approximate $\mathcal{K}_{inf}$ for other arms. Although, these variants (FMED, FIMED) achieve computational speedups they store all the rewards in memory. To address this issue they use portfolio algorithms.

STRENGTHS:
- This work addresses the well known issue of optimal bandit algorithms being impractical due to high computational complexity. 
I think this is an important step towards practical optimal bandit algorithms.
- Novel (algorithmic) use of portfolio algorithms to estimate $\mathcal{K}_{inf}$.
- Well written
- Experiments prove the applicability of the proposed algorithms.

WEAKNESSES:
Maybe you have to mention the regret assumption for the portfolio algorithms upfront, i.e. in section 2 (instead of the end of section 3).

QUESTIONS:
- What is kl in line 78? Please clarify.
- How coupled is your approach with the knowledge of $b$ and $B$ the lower and upper bounds for the distribution support?
In case we don't exactly know these bounds, does under- or overestimating this parameters lead to poor perfomance? Specifically how is perfomance affects?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This paper consideres classic multi armed bandit problem. The focus is on developing asymptotically optimal algorithms with efficient computational and memory complexity. The results for the proposed algorithms are reported in Table 2. FMED and FIMED compute KL divergence for the armed that is pulled while using first order Taylor expansions  for the other arms. In addition an online portfolio selection algorithm is used to estimate the KL divergences that further improves the computational and memory performance.

STRENGTHS:
The paper contributes to the bandit literature by improving the computational and memory requirement of asymptotically optimal algorithms with non parameteric distributions (where it is assumed that an upper bound on the rewards is known).

WEAKNESSES:
Although the results are interesting, they rely on known results from Honda and Takemura [2010] on approximating KL diveregnce and results on online portfolio selection. This limits the significance of the contribution of the paper.

QUESTIONS:
Could authors provide more details on analytical challenges beyond using the results from Honda and Takemura [2010]  and online portfolio selection. This seems important as this is mainly a theoretical paper.

LIMITATIONS:
The paper is mainly a theoretical paper, targeting the computational and memory requiremets of exisiting algorithmis.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-09

comments: SUMMARY:
This paper studies non-parametric stochastic bandits. In particular, the author proposes algorithms named Fast (Indexed) Minimum Empirical Divergence (FMED, FIMED), and Online (Indexed) Minimum Empirical Divergence (OMED, OIMED). These algorithms are designed based on Minimum Empirical Divergence (MED). Regret guarantees comparable to that of MED is established for FMED, FIMED, OMED, and OIMED. In addition, these new algorithms have much better computational complexities. Numerical experiments have been conducted to support these claims.

STRENGTHS:
This problem is well-motivated and the paper is clearly written and easy to follow. The paper also presents a complete story from motivation to algorithmic design to theoretical guarantees and empirical justifications.

WEAKNESSES:
I have listed a couple questions in the “Questions” part of this review.

QUESTIONS:
1. It would be great if the authors could elaborate on the result established by Theorem 1. For example, the bound is dependent on $\epsilon$—can $o_{\epsilon}$ be enumerated and this bound be optimized over $\epsilon$ to get a more interpretable bound? 
2. In addition, it would be helpful to include a more detailed comparison with Honda and Takemura 2015 on both the proof techniques and the regret bound and computational complexity of the algorithms. 
3. Section 4 provides nice empirical justifications on the superior performance and computational complexity of the proposed algorithms. Is it possible to replicate the experiments conducted in Honda and Takemura and compare those algorithms there?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis papers studies asymptotically optimal algorithms for regret minimization in the multi-armed bandit setting with bounded reward distributions. The main contribution is an efficient approximation of the KL constraint of the lower bound, leading to faster algorithms with (asymptotic) regret guarantees. The proposed approach extends the MED / IMED by optimizing a lower bound on the KL, and using online learning algorithms to estimate the KL constraint. The claims are corroborated with numerical experiments on real-world data for crop management.\n\nSTRENGTHS:\nAsymptotically instance-optimal bandit algorithms are well-studied, however existing approaches are expensive to compute because any optimal approach needs to (approximately) solve a linear program with KL constraints. Developing more efficient and practical algorithms is an import problem and this paper makes significant progress towards this direction. Both the relaxation of the KL and the use of online algorithms to compute K_inf are interesting novel ideas.\n\nThe paper is generally well-written, however fairly technical and could use some more guidance/intuition for the reader in a few places (e.g. how is the loss of the online algorithm connect to K_inf? how is the bias of the portfolio algorithm controlled?)\n\nRelated work is adequately discussed. Note that the use of online algorithms for approximating/minmizing KL objectives in bandit setting also appears in related context, e.g. [1-3].\n\n[1] Foster, Dylan, and Alexander Rakhlin. \"Beyond ucb: Optimal and efficient contextual bandits with regression oracles.\" In International Conference on Machine Learning, pp. 3199-3210. PMLR, 2020.\n\n[2] Foster, Dylan J., Sham M. Kakade, Jian Qian, and Alexander Rakhlin. \"The statistical complexity of interactive decision making.\" arXiv preprint arXiv:2112.13487 (2021).\n\n[3] Kirschner, Johannes, Tor Lattimore, Claire Vernade, and Csaba Szepesvári. \"Asymptotically optimal information-directed sampling.\" In Conference on Learning Theory, pp. 2777-2821. PMLR, 2021.\n\nWEAKNESSES:\nThe approach appears to be tailored to bound reward distributions, whereas in general, asymptotically optimal algorithms are known for a much more general class of algorithms and reward distributions. It is unclear if the proposed techniques can be extend beyond the current setting. It would be great if the authors could comment on this point.\n\nAs a minor comment, while I appreciate the use of real-world data, the simulation of the crop-management tasks is run over 10000 rounds. Is this realistic in this type of applications? Do the algorithms perform well over much smaller horizons?\n\nCan anything be said about finite-time or worst-case performance of the algorithms?\n\nQUESTIONS:\n* The main result (Theorem 1) uses small-o-notation to suppress the lower-order terms. Are these terms controlled in the analysis? Could you provide a more detailed discussion of the finite-time performance of the algorithm, both from a theoretical and empirical point of view?\n\nLIMITATIONS:\n* The online algorithm used to approximate the KL is assumed to have a bound on the absolute value of the regret. This is in rather unusual requirement and strongly limits the choice of online learning algorithms, as the authors discuss at the end of section 3. \n* The approach requires an upper bound on the best reward, mu_max < B. It looks like this can easily be satisfied by replacing B by B + eps for small eps > 0. Does the bound depend on the gap B - mu_max? Is knowing B/mu_max a strong assumption?\n\nI also suggest to add a more detailed explanation/clarify for the following points:\n\n* eq below line 126: why do we get this equality? (How) Is N_k(t) defined?\n* line 164-168: The dueling procedure is hard to understand, i.e. what is a \"greedy duel\"?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}