Title: Kullback-Leibler Maillard Sampling for Multi-armed Bandits with Bounded Rewards
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi-armed bandits

bounded rewards
Abstract: We study $K$-armed bandit problems where the reward distributions of the arms are all supported on the $[0,1]$ interval. Maillard sampling\cite{maillard13apprentissage}, an attractive alternative to Thompson sampling, has recently been shown to achieve competitive regret guarantees in the sub-Gaussian reward setting\cite{bian2022maillard} while maintaining closed-form action probabilities, which is useful for offline policy evaluation. In this work, we analyze the Kullback-Leibler Maillard Sampling (KL-MS) algorithm, a natural extension of Maillard sampling {and a special case of Minimum Empirical Divergence (MED)~\cite{honda2011asymptotically}} for achieving a KL-style finite-time gap-dependent regret bound. We show that KL-MS enjoys the asymptotic optimality when the rewards are Bernoulli and has an {adaptive} worst-case regret bound of the form $O(\sqrt{\mu^*(1-\mu^*) K T \ln K} + K \ln T)$, where $\mu^*$ is the expected reward of the optimal arm, and $T$ is the time horizon length; {this is the first time such adaptivity is reported in the literature for an algorithm with asymptotic optimality guarantees.}

Kullback-Leibler Maillard Sampling for Multi-armed Bandits with Bounded Rewards

Abstract

We study K -armed bandit problems where the reward distributions of the arms are all supported on the [0 , 1] interval. Maillard sampling [30], an attractive alternative to Thompson sampling, has recently been shown to achieve competitive regret guarantees in the sub-Gaussian reward setting [11] while maintaining closed-form action probabilities, which is useful for offline policy evaluation. In this work, we analyze the Kullback-Leibler Maillard Sampling (KL-MS) algorithm, a natural extension of Maillard sampling and a special case of Minimum Empirical Divergence (MED) [19] for achieving a KL-style finite-time gap-dependent regret bound. We show that KL-MS enjoys the asymptotic optimality when the rewards are Bernoulli and has an adaptive worst-case regret bound of the form O ( √ µ ∗ (1 -µ ∗ ) KT ln K + K ln T ) , where µ ∗ is the expected reward of the optimal arm, and T is the time horizon length; this is the first time such adaptivity is reported in the literature for an algorithm with asymptotic optimality guarantees.

1 Introduction

The multi-armed bandit (abbrev. MAB) problem [41, 27, 29], a stateless version of the reinforcement learning problem, has received much attention by the research community, due to its relevance in may applications such as online advertising, recommendation, and clinical trials. In a multi-armed bandit problem, a learning agent has access to a set of K arms (also known as actions), where for each i ∈ [ K ] := { 1 , . . . , K } , arm i is associated with a distribution ν i with mean µ i ; at each time step t , the agent adaptively chooses an arm I t ∈ [ K ] by sampling from a probability distribution p t ∈ ∆ K -1 and receives reward y t ∼ ν I t , based on the information the agent has so far. The goal of the agent is to minimize its pseudo-regret over T time steps: Reg( T ) = Tµ ∗ -E ∑ T t =1 y t , where µ ∗ = max i µ i is the optimal expected reward.

In this paper, we study the multi-armed bandit setting where reward distributions of all arms are supported on [0 , 1] . 1 An important special case is Bernoulli bandits, where for each arm i , ν i = Bernoulli( µ i ) for some µ i ∈ [0 , 1] . It has practical relevance in settings such as computational advertising, where the reward feedback is oftentimes binary (click vs. not-click, buy vs. not-buy).

Broadly speaking, there are two popular families of provably regret-efficient algorithms for boundedreward bandit problems: deterministic exploration algorithms (such as KL-UCB [17, 13, 31]) and randomized exploration algorithms (such as Thompson sampling (TS) [41]). Randomized exploration algorithms such as TS have been very popular, perhaps due to its excellent empirical performance and the ability to cope with delayed rewards better than deterministic counterparts [15]. In addition, the logged data collected from randomized exploration, of the form ( I t , p t,I t , y t ) T t =1 , where p t,I t is the probability with which arm I t was chosen, are useful for offline evaluation purposes by

100

BernoulliTS

KL-MS

80

Oracle

uency

BernoulliTS

60

KL-MS

freque

40

20

0

0.75

1.00

1.25

1.50

1.75

2.00

averaaereward

employing the inverse propensity weighting (IPW) estimator [22] or the doubly robust estimator [38]. However, calculating the arm sampling probability distribution p t for Thompson sampling is nontrivial. Specifically, there is no known closed-form 2 , and generic numerical integration methods and Monte-Carlo approximations suffer from instability issues: the time complexity for obtaining a numerical precision of ϵ is Ω(poly(1 /ϵ )) [36]. This is too slow to be useful especially for web-scale deployments; e.g., Google AdWords receives ∼ 237M clicks per day. Furthermore, the computed probability will be used after taking the inversion, which means that even moderate amount of errors are intolerable. Indeed, Figure 1 shows that the offline evaluation with Thompson sampling as the behavioral policy will be largely biased and inaccurate due to the errors from the Monte Carlo approximation.

Recently, many studies have introduced alternative randomized algorithms that allow an efficient computation of p t [19, 30, 14, 43]. Of these, Maillard sampling (MS) [30, 11], a Gaussian adaptation of the Minimum Empirical Divergence (MED) algorithm [19] originally designed for finite-support reward distributions, provides a simple algorithm for the sub-Gaussian bandit setting that computes p t in a closed form:

where at time step t , N t,a is the number of pulling arm a . We define the estimator of µ a as ˆ µ t,a := ∑ t s =1 1 { I t = a } y t N t,a and the best performed mean value as ˆ µ t, max := max a ∈ [ K ] µ t,a . ˆ ∆ t -1 ,a = max a ′ ˆ µ t -1 ,a ′ -ˆ µ t -1 ,a is the empirical suboptimality gap of arm a , and σ is the subgaussian parameter of the reward distribution of all arms. For sub-Gaussian reward distributions, MS enjoys the asymptotic optimality under the special case of Gaussian rewards and a near-minimax optimality [11], making it an attractive alternative to Thompson sampling. Also, MS satisfies the sub-UCB criterion (see Section 2 for a precise definition) to help establish sharp finite-time instance-dependent regret guarantees. Can we adapt MS to the bounded reward setting and achieve the asymptotic, minimax optimality and sub-UCB criterion while computing the sampling probability in a closed-form? In this paper, we make significant progress on this question.

Our contributions. We focus on a Bernoulli adaptation of MS that we call Kullback-Leibler Maillard Sampling (abbrev. KL-MS) and perform a finite-time analysis of it in the bounded-reward bandit problem. KL-MS uses a sampling probability similar to MS but tailored to the [0 , 1] -bounded reward setting:

where kl ( µ, µ ′ ) := µ ln µ µ ′ +(1 -µ ) ln 1 -µ 1 -µ ′ is the binary Kullback-Leibler (KL) divergence. We can also view KL-MS as an instantiation of MED [19] for Bernoulli rewards; See Section 3 for a

̸

Table 1: Comparison of regret bounds for bounded reward distributions; for space constraints we only include those that achieves the asymptotic optimality for the special case of Bernoulli distributions (this excludes, e.g., Maillard Sampling [30, 11], Tsallis-INF [43] and UCB-V [8]). ' -'indicates that the corresponding analysis is not reported. 'N/A'indicates that the algorithm does have closed-form, but it is deterministic. ' ⋆ 'indicates that its computational complexity for calculating the action probability is ln(1 / precision ) . ' ⋆⋆ 'indicates that we conjecture that the algorithm is not sub-UCB. The results on TS are reported by Agrawal and Goyal [3, 4], Korda et al. [26].

detailed comparison. KL-MS performs an efficient exploration for bounded rewards since one can use kl ( a, b ) ≥ 2( a -b ) 2 to verify that the probability being assigned to each empirical non-best arm by KL-MS is never larger than that of MS with σ 2 = 1 / 4 , the best sub-Gaussian parameter for the bounded rewards in [0 , 1] . We show that KL-MS achieves a sharp finite-time regret guarantee (Theorem 1) that can be simultaneously converted to:

an asymptotic regret upper bound (Theorem 4), which is asymptotically optimal when specialized to the Bernoulli bandit setting;

a T -style regret guarantee of O ( √ µ ∗ (1 -µ ∗ ) KT ln K + K ln( T )) (Theorem 3) where µ ∗ is the mean reward of the best arm. This bound has two salient features. First, in the worst case, it is at most a √ ln K factor suboptimal than the minimax optimal regret of Θ( √ KT ) [5, 10]. Second, its ˜ O ( √ µ ∗ (1 -µ ∗ ) coefficient adapts to the variance of the optimal arm reward; this is the first time such adaptivity is reported in the literature for an algorithm with asymptotical optimality guarantees. 3

√

a sub-UCB regret guarantee, which many existing minimax optimal algorithms [33, 18] have not been proven to satisfy.

We also conduct experiments that show that thanks to its closed-form action probabilities, KL-MS generates much more reliable logged data than Bernoulli TS with Monte Carlo estimation of action probabilities; this is reflected in their offline evaluation performance using the IPW estimator; see Figure 1 and Appendix H for more details.

2 Preliminaries

Let N t,a be the number of times arm a has been pulled until time step t (inclusively). Denote the suboptimality gap of arm a by ∆ a := µ ∗ -µ a , where µ ∗ = max i ∈ [ K ] µ i is the optimal expected reward. Denote the empirical suboptimality gap of arm a by ˆ ∆ t,a := ˆ µ t, max -ˆ µ t,a ; here, ˆ µ t,a is the empirical estimation to µ a up to time step t , i.e., ˆ µ t,a := 1 N t,a ∑ t s =1 y s 1 { I s = a } , and ˆ µ t, max = max a ∈ [ K ] ˆ µ t,a is the best empirical reward at time step t . For arm a , define τ a ( s ) := min { t ≥ 1 : N t,a = s } at the time step when arm a is pulled for the s -th time, which is a stopping time; we also use ˆ µ ( s ) ,a := ˆ µ τ a ( s ) ,a to denote empirical mean of the first s reward values received from pulling arm a .

We define the Kullback-Leibler divergence between two distributions ν and ρ as KL ( ν, ρ ) = E X ∼ ν [ ln d ν d ρ ( X ) ] if ν is absolutely continuous w.r.t. ρ , and = + ∞ otherwise. Recall that we define the binary Kullback-Leibler divergence between two numbers µ, µ ′ in [0 , 1] as kl ( µ, µ ′ ) := µ ln µ µ ′ + (1 -µ ) ln 1 -µ 1 -µ ′ , which is also the KL divergence between two Bernoulli distributions with mean parameters µ and µ ′ respectively. We define ˙ µ = µ (1 -µ ) , which is the variance of Bernoulli( µ ) but otherwise an upper bound on any distribution supported on [0 , 1] with mean µ ; see Lemma 16 for a formal justification.

In the regret analysis, we will oftentimes use the following notation for comparison up to constant factors: define f ≲ g (resp. f ≳ g ) to denote that f ≤ Cg (resp. f ≥ Cg ) for some numerical constant C > 0 . We define a ∨ b and a ∧ b as max( a, b ) and min( a, b ) , respectively. For an event E , we use E c to denote its complement.

Below, we define some useful criteria for measuring the performance of bandit algorithms, specialized to the [0 , 1] bounded reward setting.

Asymptotic optimality in the Bernoulli reward setting An algorithm is asymptotically optimal in the Bernoulli reward setting [27, 12] if for any Bernoulli bandit instance ( ν a = Bernoulli( µ a )) a ∈ [ K ] , lim sup T →∞ Reg( T ) ln T = ∑ a :∆ a > 0 ∆ a kl ( µ a ,µ ∗ ) .

Minimax ratio The minimax optimal regret of the [0 , 1] bounded reward bandit problem is Θ ( √ KT ) [5, 10]. Given a K -armed bandit problem with time horizon T , an algorithm has a minimax ratio of f ( T, K ) if its has a worst-case regret bound of O ( √ KTf ( T, K )) .

Sub-UCB Sub-UCB is originally defined in the context of sub-Gaussian bandits [29]: given a bandit problem with K arms whose reward distributions are all sub-Gaussian, an algorithm is said to be sub-UCB if there exists some positive constants C 1 and C 2 , such that for all σ 2 -sub-Gaussian bandit instances, Reg( T ) ≤ C 1 ∑ a :∆ a > 0 ∆ a + C 2 ∑ a :∆ a > 0 σ 2 ∆ a ln T . Specialized to our setting, as any distribution supported on [0 , 1] is also 1 4 -sub-Gaussian, and all suboptimal arm gaps ∆ a ∈ (0 , 1] are such that ∆ a < 1 ∆ a , the above sub-UCB criterion simplifies to: there exists some positive constant C , such that for all [0 , 1] -bounded reward bandit instances, Reg( T ) ≤ C ∑ a :∆ a > 0 ln T ∆ a .

3 Related Work

Bandits with bounded rewards. Early works of Lai et al. [27], Burnetas and Katehakis [12] show that in the bounded reward setting, for any consistent stochastic bandit algorithm, the regret is lower bounded by (1 + o (1)) ∑ a :∆ a > 0 ∆ a ln T KL inf ( ν a ,µ ∗ ) and KL inf ( ν a , µ ∗ ) is defined as

where the random variable follows a distribution ρ bounded in [0 , 1] . Therefore, any algorithm whose regret upper bound matches the lower bound is said to achieve asymptotic optimality. Cappé et al. [13] propose the KL-UCB algorithm and provide a finite time regret analysis, which is further refined by Lattimore and Szepesvári [29, Chapter 10]. Another line of work establishes asymptotic and finite-time regret guarantees for Thompson sampling algorithms and its variants [2, 4, 25, 24], which, when specialized to the Bernoulli bandit setting, can be combined with Beta priors for the Bernoulli parameters to design efficient algorithms.

A number of studies even go beyond the Bernoulli-KL-type regret bound and adapt to the variance of each arm in the bounded reward setting. UCB-V [7] achieves a regret bound that adapts to the variance. Efficient-UCBV [35] achieves a variance-adaptive regret bound and also an optimal minimax regret bound O ( √ KT ) , but it is not sub-UCB. Honda and Takemura [19] propose the MED algorithm that is asymptotically optimal for bounded rewards, but it only works for rewards that with finite supports. Honda and Takemura [21] propose the Indexed MED (IMED) algorithm that can handle a more challenging case where the reward distributions are supported in ( -∞ , 1] .

As with worst-case regret bounds, first, it is well-known that for Bernoulli bandits as well as bandits with [0 , 1] bounded rewards, the minimax optimal regrets are of order Θ( √ KT ) [10, 5]. Of the algorithms that enjoy asymptotic optimality under the Bernoulli reward setting described above, KL-UCB [13] has a worst-case regret bound of O ( √ KT ln T ) , which is refined by the KL-UCB++

√

algorithm [33] that has a worst-case regret bound of O ( KT ) . We also show in Appendix F.1 and F.2 that with some modifications of existing analysis, KL-UCB and KL-UCB++ enjoy a regret bound of O ( √ µ ∗ (1 -µ ∗ ) KT ln T ) and O ( √ µ ∗ (1 -µ ∗ ) K 3 T ln T ) respectively. Although the regret is worse in the order of K , it adapts to µ ∗ and will have a better regret when µ ∗ is small (say, µ ∗ ≤ 1 /K 2 ). KL-UCB++[34] and KL-UCB-Switch[18] achieves O ( √ KT ) regret in the finite-time regime and asymptotic optimality, while the sub-UCB criterion has not been satisfied. However, Lattimore [28, §3] shows that MOSS [6] suffers a sub-optimal regret worse than UCB-like algorithms because of not satisfying sub-UCB criteria, and we suspect that KL-UCB-switch experience the same issue as MOSS. For Thompson Sampling style algorithms, Agrawal and Goyal [3] shows that the original Thompson Sampling algorithm has a worst-case regret of O ( √ KT ln K ) , and the ExpTS+ algorithm [24] has a worst-case regret of O ( √ KT ) .

Randomized exploration for bandits. Many randomized exploration methods have been proposed for multi-armed bandits. Perhaps the most well-known is Thompson sampling [41], which is shown to achieve Bayesian and frequentist-style regret bounds in a broad range of settings [39, 2, 25, 26, 23, 24]. A drawback of Thompson sampling, as mentioned above, is that the action probabilities cannot be obtained easily and robustly. To cope with this, a line of works design randomized exploration algorithms with action probabilities in closed forms. For sub-Gaussian bandits, Cesa-Bianchi et al. [14] propose a variant of the Boltzmann exploration rule (that is, the action probabilities are proportional to exponential to empirical rewards, scaled by some positive numbers), and show that it has O ( K ln 2 T ∆ ) instance-dependent and O ( √ KT ln K ) worst-case regret bounds respectively, where ∆ = min a :∆ a > 0 ∆ a is the minimum suboptimalty gap. Maillard sampling (MS; Eq. (1)) is an algorithm proposed by the thesis of Maillard [30] where the author reports that MS achieves the asymptotic optimality and has a finite-time regret of order ∑ a :∆ a > 0 ( ln T ∆ a + 1 ∆ 3 a ) from which a worst-case regret bound of O ( √ KT 3 / 4 ) can be derived. MED [19], albeit achieves asymptotic optimality for a broad family of bandits with finitely supported reward distributions, also has a high finite-time regret bound of at least ∑ a :∆ a > 0 ( ln T ∆ a + 1 ∆ 2 | supp( ν 1 ) |-1 a ) . 4 Recently, Bian and Jun [11] report a refined analysis of Maillard [30]'s sampling rule, showing that it has a finite time regret of order ∑ a :∆ a > 0 ln( T ∆ 2 a ) ∆ a + O ( ∑ a :∆ a > 0 1 ∆ a ln( 1 ∆ a ) ) , and additionally enjoys a O ( √ KT ln T ) worst-case regret, and by inflating the exploration slightly (called MS + ), the bound can be improved and enjoy the minimax regret of O ( √ KT ln K ) , which matches the best-known regret bound among those that satisfy sub-UCB criterion, except for AdaUCB. In fact, it is easy to adapt our proof technique in this paper to show that MS, without any further modification, achieves a O ( √ KT ln K ) worst-case regret.

Randomized exploration has also been studied from a nonstochastic bandit perspective [10, 5], where randomization serves both as a tool for exploration and a way to hedge bets against the nonstationarity of the arm rewards. Many recent efforts focus on designing randomized exploration bandit algorithms that achieve 'best of both worlds' adaptive guarantees, i.e., achieving logarithmic regret for stochastic environments while achieving √ T regret for adversarial environments [e.g. 43, 42].

Binarization trick. It is a folklore result that bandits with [0 , 1] bounded reward distributions can be reduced to Bernoulli bandits via a simple binarization trick: at each time step t , the learner sees reward r t ∈ [0 , 1] , draws ˜ r t ∼ Bernoulli( r t ) and feeds it to a Bernoulli bandit algorithm. However, this reduction does not result in asymptotic optimality for the general bounded reward setting, where the asymptotic optimal regret is of the form (1 + o (1)) ∑ a :∆ a > 0 ∆ a ln T KL inf ( ν a ,µ ∗ ) with KL inf ( ν a , µ ∗ ) defined in the Eq (2). If we combine the binarization trick and the MED algorithm in the bounded reward setting, the size of the support set is viewed as 2 , the finite-time regret bound is at best as O ( K 1 / 4 T 3 / 4 ) (ignoring logarithmic factors), which is much higher than O ( √ KT ) .

Algorithm 1 KL Maillard Sampling (KL-MS)

1: Input: K ≥ 2

2: for t = 1 , 2 , · · · , T do

3: if t ≤ K then

4: Pull the arm I t = t and observe reward y t ∼ ν i .

5: else

6: For every a ∈ [ K ] , compute

where M t = ∑ K a =1 exp( -N t -1 ,a kl (ˆ µ t -1 ,a , ˆ µ t -1 , max )) 7: Pull the arm I t ∼ p t .

8: Observe reward y t ∼ ν I t .

9: end if

10: end for

Bandit algorithms with worst-case regrets that depend on the optimal reward. Recent linear logistic bandit works have shown worst-case regret bounds that depend on the variance of the best arm [32, 1]. When the arms are standard basis vectors, logistic bandits are equivalent to Bernoulli bandits, and the bounds of Abeille et al. [1] become ˜ O ( K √ ˙ µ ∗ T + K 2 ˙ µ min ∧ ( K 2 + A ) ) where ˙ µ min = min i ∈ [ K ] ˙ µ i and A is an instance dependent quantity that can be as large as T . This bound, compared to ours, has an extra factor of √ K in the leading term and the lower order term has an extra factor of K . Even worse, it has the term ˙ µ -1 min in the lower order term, which can be arbitrarily large. The bound in Mason et al. [32] becomes ˜ O ( √ ˙ µ ∗ KT + ˙ µ -1 min K 2 ) , which matches our bound in the leading term up to logarithmic factors yet still have extra factors of K and ˙ µ -1 min in the lower order term.

4 Main Result

The KL Maillard Sampling Algorithm. We propose an algorithm called KL Maillard sampling (KL-MS) for bounded reward distributions (Algorithm 1). For the first K times steps, the algorithm pulls each arm once (steps 3 to 4); this ensures that starting from time step K +1 , the estimates of the reward distribution of all arms are well-defined. From time step t = K +1 on, the learner computes the empirical mean ˆ µ t -1 ,a of all arms a . For each arm a , the learner computes the binary KL divergence between ˆ µ t -1 ,a and ˆ µ t -1 , max , kl (ˆ µ t -1 ,a , ˆ µ t -1 , max ) , as a measure of empirical suboptimality of that arm. The sampling probability of arm a , denoted by p t,a , is proportional to the exponential of negative product between N t -1 ,a and kl (ˆ µ t -1 ,a , ˆ µ t -1 , max ) (Eq. (3) of step 6). This policy naturally trades off between exploration and exploitation: arm a is sampled with higher probability, if either it has not been pulled many times ( N t -1 ,a is small) or it appears to be close to optimal empirically ( kl (ˆ µ t -1 ,a , ˆ µ t -1 , max )) is small). The algorithm samples an arm I t from p t , and observe a reward y t of the arm chosen.

We remark that if the reward distributions ν i 's are Bernoulli, KL-MS is equivalent to the MED algorithm [19] since in this case, all reward distributions have a binary support of { 0 , 1 } . However, KL-MS is different from MED in general: MED computes the empirical distributions of arm rewards ˆ F t -1 ,a , and chooses action according to probabilities p t,a ∝ exp( -N t -1 ,a D t -1 ,a ) ; here, D t -1 ,a := KL ( ˆ F t -1 ,a , ˆ µ t -1 , max ) (recall its definition in Section 3) is the 'minimum empirical divergence' between arm a and the highest empirical mean reward, which is different from the binary KL divergence of the mean rewards used in KL-MS.

4.1 Main Regret Theorem

Our main result of this paper is the following theorem on the regret guarantee of KL-MS (Algorithm 1). Without loss of generality, throughout the rest of the paper, we assume µ 1 ≥ µ 2 ≥ · · · ≥ µ K .

is the normalizer.

Theorem 1. For any K -arm bandit problem with reward distribution supported on [0 , 1] , KL-MS has regret bounded as follows. For any ∆ ≥ 0 and c ∈ (0 , 1 4 ] :

The regret bound of Theorem 1 is composed of three terms. The first term is T ∆ , which controls the contribution of regret from all ∆ -near-optimal arms. The second term is asymptotically (1 + o (1)) ∑ a :∆ a > 0 ∆ a kl ( µ a ,µ 1 ) ln( T ) with an appropriate choice of c , which is a term that grows in T in a logarithmic rate. The third term is simultaneously upper bounded by two expressions. One is ∑ a :∆ a > 0 ( ˙ µ 1 +∆ a c 2 ∆ a ) ln ( c 2 T ∆ 2 a ˙ µ 1 +∆ a ∨ e 2 ) , which is of order ln T and helps establish a tight worst-case regret bound (Theorem 3); the other is ∑ a :∆ a > 0 ( ˙ µ 1 +∆ a c 2 ∆ a ) ln ( ( ˙ µ 1 +∆ a c 2 ∆ 2 a ) ∨ e 2 ) , which does not grow in T and helps establish a tight asymptotic upper bound on the regret (Theorem 4).

To the best of our knowledge, existing regret analysis on Bernoulli bandits or bandits with bounded support have regret bounds of the form

for some c > 0 , where the third term is much larger than its counterpart given by Theorem 1 when ∆ a and ˙ µ 1 are small. As we will see shortly, as a consequence of its tighter bounds, our regret theorem yields a superior worst-case regret guarantee over previous works.

Theorem 2 (Sub-UCB) . KL-MS's regret is bounded by Reg( T ) ≲ ∑ a :∆ a > 0 ln T ∆ a . Therefore, KL-MS is sub-UCB.

Sub-UCB criterion is important for measuring a bandit algorithm's finite-time instance-dependent performance. Indeed, Lattimore [28, §3] points out that MOSS [6] does not satisfy sub-UCB and that it leads to a strictly suboptimal regret in a specific instance compared to the standard UCB algorithm [9]. A close inspection of the finite-time regret bounds of existing asymptotically optimal and minimax optimal algorithms for the [0 , 1] -reward setting, such as KL-UCB++ [33] and KLUCB-switch [18], reveals that they are not sub-UCB. Thus, we speculate that they would also have a suboptimal performance in the aforementioned instance.

In light of Theorem 1, our first corollary is that KL Maillard sampling achieves the following adaptive worst-case regret guarantee.

Theorem 3 (Adaptive worst-case regret) . For any K -arm bandit problem with reward distribution supported on [0 , 1] , KL-MS has regret bounded as: Reg( T ) ≲ √ ˙ µ 1 KT ln K + K ln T .

√

An immediate corollary is that KL Maillard sampling has a regret of order O ( KT ln K ) , which is a factor of O ( √ ln K ) within the minimax optimal regret Θ( √ KT ) [33, 5]. This also matches the worst-case regret bound O ( √ V KT ln( K )) of Jin et al. [24] where V = 1 4 is the worst-case variance for Bernoulli bandits using a Thompson sampling-style algorithm. Another main feature of this regret bound is its adaptivity to ˙ µ 1 , the variance of the reward of the optimal arm for the Bernoulli bandit setting, or its upper bound in the general bounded reward setting (see Lemma 16). Specifically, if µ 1 is close to 0 or 1, ˙ µ 1 is very small, which results in the regret being much smaller than O ( √ KT ln K ) .

Note that UCB-V [7] and KL-UCB/KL-UCB++, while not reported, enjoy a worst-case regret bound of O ( √ ˙ µ 1 KT ln T ) , which is worse than our bound in its logarithmic factor; see Appendix F.3 and F.1 for the proofs. Among these, UCB-V does not achieve the asymptotic optimality for the Bernoulli case. While logistic linear bandits [1, 32] can be applied to Bernoulli K -armed bandits and achieve similar worst-case regret bounds involving ˙ µ 1 , their lower order term can be much worse as discussed in Section 3.

Our second corollary is that KL Maillard sampling achieves a tight asymptotic regret guarantee for the special case of Bernoulli rewards:

Theorem 4. (Asymptotic Optimality) For any K -arm bandit problem with reward distribution supported on [0 , 1] , KL-MS satisfies the following asymptotic regret upper bound:

Specialized to the Bernoulli bandit setting, in light of the asymptotic lower bounds [27, 12], the above asymptotic regret upper bound implies that KL-MS is asymptotically optimal.

While the regret guarantee of KL-MS is not asymptotically optimal for the general [0 , 1] bounded reward setting, it nevertheless is a better regret guarantee than naively viewing this problem as a sub-Gaussian bandit problem and applying sub-Gaussian bandit algorithms on it. To see this, note that any reward distribution supported on [0 , 1] is 1 4 -sub-Gaussian; therefore, standard sub-Gaussian bandit algorithms will yield an asymptotic regret (1 + o (1)) ∑ a ∈ [ K ]:∆ a > 0 ln T 2∆ a . This is always no better than the asymptotic regret provided by Eq. (5), in view of Pinsker's inequality that kl ( µ a , µ 1 ) ≥ 2∆ 2 a .

5 Proof Sketch of Theorem 1

We provide an outline of our proof of Theorem 1, with full proof details deferred to Appendix C. Our approach is akin to the recent analysis of the sub-Gaussian Maillard Sampling algorithm in Bian and Jun [11] with several refinements tailored to the bounded reward setting and achieving √ ln K minimax ratio. First, for any time horizon length T , Reg( T ) can be bounded by:

i.e., the total regret can be decomposed to a T ∆ term and the sum of regret ∆ a E [ N T,a ] from pulling ∆ -suboptimal arms a . Therefore, in subsequent analysis, we focus on bounding E [ N T,a ] . To this end, we show the following lemma.

Lemma 5. For any suboptimal arm a , let ε 1 , ε 2 > 0 be such that ε 1 + ε 2 < ∆ a . Then its expected number of pulls is bounded as:

where H := 1 (1 -µ 1 + ε 2 )( µ 1 -ε 2 ) h 2 ( µ 1 ,ε 2 ) ≲ 2 ˙ µ 1 + ε 2 ε 2 2 and h ( µ 1 , ε 2 ) := ln ( (1 -µ 1 + ε 2 ) µ 1 (1 -µ 1 )( µ 1 -ε 2 ) ) .

Theorem 1 follows immediately from Lemma 5. See section C for details; we show a sketch here.

Proof sketch of Theorem 1. Fix any c ∈ (0 , 1 4 ] . Let ε 1 = ε 2 = c ∆ a ; by the choice of c , ε 1 + ε 2 < ∆ a . From Lemma 5, E [ N T,a ] is bounded by Eq. (7). Plugging in the values of ε 1 = ε 2 , and using Lemma 26 that lower bounds the binary KL divergence, along with Lemma 22 that gives H ≲ 2 ˙ µ 1 + ε 2 ε 2 2 , and algebra, all terms except the second term on the right hand side of Eq. (7) are bounded by

As a result, KL-MS satisfies that, for any arm a , for any c ∈ (0 , 1 4 ] :

Theorem 1 follows by plugging the above bound to Eq. (6) for arms a s.t. ∆ a > ∆ with c = 1 4 .

5.1 Proof sketch of Lemma 5

We sketch the proof of Lemma 5 in this subsection. For full details of the proof, please refer to

Appendix C.2. We first set up some useful notations that will be used throughout the proof. Let 2

By algebra, one has the following elementary upper bound on E [ N T,a ] : E [ N T,a ] ≤ u + E [ ∑ T t = K +1 1 { A t , B c t -1 } ] . Intuitively, the u term serves to control the length of a "burn-in" phase when the number of pulls to arm a is at most u . It now remains to control the second term, the number of pulls to arm a after it is large enough, i.e., N t -1 ,a ≥ u . We decompose it to F 1 , F 2 , and F 3 , resulting in the following inequality:

Here:

F 1 corresponds to the 'steady state' when the empirical means of arm a and the optimal arm are both estimated accurately, i.e., ˆ µ t -1 , max ≥ µ 1 -ε 2 and ˆ µ t -1 ,a ≤ µ a + ε 1 . It can be straightforwardly bounded by 1 kl ( µ a + ε 1 ,µ 1 -ε 2 ) , as we show in Lemma 10 (section D.1).

F 2 corresponds to the case when the empirical mean of arm a is abnormally high, i.e., ˆ µ t -1 ,a > µ a + ε 1 . It can be straightforwardly bounded by 1 kl ( µ a + ε 1 ,µ a ) , as we show in Lemma 11 (section D.2).

F 3 corresponds to the case when the empirical mean of the optimal arm is abnormally low, i.e., ˆ µ t -1 , max ≤ µ 1 -ε 2 ; it is the most challenging term and we discuss our techniques in bounding it in detail below (section D.3).

We provide an outline of our analysis of F 3 in Appendix D.3.1 and sketch its main ideas and technical challenges here.

We follow the derivation from Bian and Jun [11] by first using a probability transferring argument (Lemma 23) to bound the expected counts of pulling suboptimal arm a by the expectation of indicators of pulling the optimal arm with a multiplicative factor and then change the counting from global time step t to local count of pulling the optimal arm. Then, F 3 is bounded by,

Intuitively, each M k should be controlled: when exp( k · kl (ˆ µ ( k ) , 1 , µ 1 -ε 2 )) is large, ˆ µ ( k ) , 1 must significantly negatively deviate from µ 1 -ε 2 , which happens with low probability by Chernoff bound (Lemma 25). Using a double integration argument, we can bound each M k by

Summing over all k , we can bound F 3 1 by O ( H ln ( H ∨ e 2 ) + 1 kl ( µ 1 -ε 2 ,µ 1 ) ) . Combining the bounds on F 1 and F 2 , we can show a bound on E [ N T,a ] similar to Eq. (7) without the ' T H ∧ ' term in the logarithmic factor. This yields a regret bound of KL-MS, in the form of Eq. (4) without the ' ∧ c 2 T ∆ 2 a ˙ µ 1 +∆ a ' term in the logarithmic factor. Such a regret bound can be readily used to show KL-MS's

Bernoulli asympototic optimality and sub-UCB property. An adaptive worst-case regret bound of √ ˙ µ 1 KT ln ( T ) also follows immediately.

To show that MS has a tighter adaptive worst-case regret bound of √ ˙ µ 1 KT ln ( K ) , we adopt a technique in [33, 24]. First, we observe that the looseness of the above bound on F 3 comes from small k (denoted as F 3 1 := ∑ k ≤ H M k ), as the summation of M k for large k (denoted as F 3 2 := ∑ k>H M k ) is well-controlled. The key challenge in a better control of F 3 1 comes from the difficulty in bounding the tail probability of ˆ µ ( k ) , 1 for k < H beyond Chernoff bound. To cope with this, we observe that a modified version of F 3 1 that contains an extra favorable indicator of kl (ˆ µ ( k ) , 1 , µ 1 ) ≤ 2 ln ( T/k ) , denoted as:

k

C

can be well-controlled. Utilizing this introduces another term in the regret analysis, T · P ( E ) , where E = { ∀ k ∈ [1 , H ] , kl (ˆ µ ( k ) , 1 , µ 1 ) ≤ 2 ln ( T/k ) k } , which we bound by O ( H ) via a timeuniform version of Chernoff bound. Putting everything together, we prove a bound of F 3 of O ( H ln ( ( T H ∧ H ) ∨ e 2 ) + 1 kl ( µ 1 -ε 2 ,µ 1 ) ) , which yields our final regret bound of KL-MS in Theorem 1 and the refined minimax ratio.

Remark 6. Although our technique is inspired by [24, 33], we carefully set the case splitting threshold for N t -1 , 1 (to obtain F 3 1 and F 3 2 ) to be H = O ( ˙ µ 1 + ϵ 2 ϵ 2 2 ) , which is significantly different from prior works ( ˜ O ( 1 2 ) ).

ϵ 2

Remark 7. One can port our proof strategy back to sub-Gaussian MS and show that it achieves a minimax ratio of √ ln K as opposed to √ ln T reported in Bian and Jun [11]; a sketch of the proof is in Appendix G. Recall that Bian and Jun [11] proposed another algorithm MS + that achieved the minimax ratio of √ ln K at the price of extra exploration. Our result makes MS + obsolete; MS should be preferred over MS + at all times.

6 Conclusion

We have proposed KL-MS, a KL version of Maillard sampling for stochastic multi-armed bandits in the [0 , 1] -bounded reward setting, with a closed-form probability computation, which is highly amenable to off-policy evaluation. Our algorithm requires constant time complexity with respect to the target numerical precision in computing the action probabilities, and our regret analysis shows that KL-MS achieves the best regret bound among those in the literature that allows computing the action probabilities with O ( polylog (1 / precision )) time complexity, for example, Tsallis-INF [43], EXP3++ [40], in the stochastic setting.

Our study opens up numerous open problems. One immediate open problem is to generalize KL-MS to handle exponential family reward distributions. Another exciting direction is to design randomized and off-policy-amenable algorithms that achieve the asymptotic optimality for bounded rewards (i.e., as good as IMED [21]).

One possible avenue is to extend MED [19] and remove the restriction that the reward distribution must have bounded support. Furthermore, it would be interesting to extend MS to structured bandits and find connections to the Decision-Estimation Coefficient [16], which have recently been reported to characterize the optimal minimax regret rate for structured bandits. Finally, we believe MS is practical by incorporating the booster hyperparameter introduced in Bian and Jun [11]. Extensive empirical evaluations on real-world problems would be an interesting future research direction.

References

M. Abeille, L. Faury, and C. Calauzenes. Instance-Wise Minimax-Optimal Algorithms for Logistic Bandits. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 3691-3699, 2021.

S. Agrawal and N. Goyal. Analysis of Thompson Sampling for the Multi-armed Bandit Problem. In Proceedings of the Conference on Learning Theory (COLT) , volume 23, pages 39.1-39.26, 2012.

S. Agrawal and N. Goyal. Further optimal regret bounds for thompson sampling. In Artificial intelligence and statistics , pages 99-107, 2013.

S. Agrawal and N. Goyal. Near-optimal regret bounds for thompson sampling. Journal of the ACM (JACM) , 64(5):1-24, 2017.

J.-Y. Audibert, S. Bubeck, and Others. Minimax Policies for Adversarial and Stochastic Bandits. In Proceedings of the Conference on Learning Theory (COLT) , 2009.

J.-Y. Audibert, S. Bubeck, et al. Minimax policies for adversarial and stochastic bandits. In COLT , volume 7, pages 1-122, 2009.

J.-Y. Audibert, R. Munos, and C. Szepesvári. Exploration-exploitation tradeoff using variance estimates in multi-armed bandits. Theoretical Computer Science , 410(19):1876-1902, 2009.

J.-Y. Audibert, R. Munos, and C. Szepesvári. Exploration-exploitation tradeoff using variance estimates in multi-armed bandits. Theoretical Computer Science , 410(19):1876-1902, 2009.

P. Auer. Using Confidence Bounds for Exploitation-Exploration Trade-offs. Journal of Machine Learning Research , 3:397-422, 2002.

P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The Nonstochastic Multiarmed Bandit Problem. SIAM J. Comput. , 32(1):48-77, jan 2003. ISSN 0097-5397. doi: 10.1137/ S0097539701398375.

J. Bian and K.-S. Jun. Maillard Sampling: Boltzmann Exploration Done Optimally. In International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 54-72, 2022.

A. N. Burnetas and M. N. Katehakis. Optimal adaptive policies for sequential allocation problems. Advances in Applied Mathematics , 17(2):122-142, 1996.

O. Cappé, A. Garivier, O.-A. Maillard, R. Munos, and G. Stoltz. Kullback-leibler upper confidence bounds for optimal sequential allocation. The Annals of Statistics , pages 1516-1541, 2013.

N. Cesa-Bianchi, C. Gentile, G. Lugosi, and G. Neu. Boltzmann exploration done right. Advances in Neural Information Processing Systems (NeurIPS) , 2017.

O. Chapelle and L. Li. An Empirical Evaluation of Thompson Sampling. In Advances in Neural Information Processing Systems (NIPS) , pages 2249-2257, 2011.

D. J. Foster, S. M. Kakade, J. Qian, and A. Rakhlin. The Statistical Complexity of Interactive Decision Making. CoRR , abs/2112.1, 2021.

A. Garivier and O. Cappé. The kl-ucb algorithm for bounded stochastic bandits and beyond. In Proceedings of the 24th annual conference on learning theory , pages 359-376. JMLR Workshop and Conference Proceedings, 2011.

A. Garivier, H. Hadiji, P. Menard, and G. Stoltz. Kl-ucb-switch: optimal regret bounds for stochastic bandits from both a distribution-dependent and a distribution-free viewpoints. The Journal of Machine Learning Research , 23(1):8049-8114, 2022.

J. Honda and A. Takemura. An asymptotically optimal policy for finite support models in the multiarmed bandit problem. Machine Learning , 85(3), 2011.

J. Honda and A. Takemura. Finite-time regret bound of a bandit algorithm for the semi-bounded support model. arXiv preprint arXiv:1202.2277 , 2012.

J. Honda and A. Takemura. Non-asymptotic analysis of a new bandit algorithm for semibounded rewards. J. Mach. Learn. Res. , 16:3721-3756, 2015.

D. G. Horvitz and D. J. Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American statistical Association , 47(260):663-685, 1952.

T. Jin, P. Xu, J. Shi, X. Xiao, and Q. Gu. Mots: Minimax optimal thompson sampling. In International Conference on Machine Learning , pages 5074-5083. PMLR, 2021.

T. Jin, P. Xu, X. Xiao, and A. Anandkumar. Finite-time regret of thompson sampling algorithms for exponential family multi-armed bandits. In Advances in Neural Information Processing Systems , 2022.

E. Kaufmann, N. Korda, and R. Munos. Thompson sampling: An asymptotically optimal finite-time analysis. In Proceedings of the international conference on Algorithmic Learning Theory (ALT) , pages 199-213, 2012.

N. Korda, E. Kaufmann, and R. Munos. Thompson sampling for 1-dimensional exponential family bandits. Advances in neural information processing systems , 26, 2013.

T. L. Lai, H. Robbins, et al. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1):4-22, 1985.

T. Lattimore. Refining the confidence level for optimistic bandit strategies. The Journal of Machine Learning Research , 19(1):765-796, 2018.

T. Lattimore and C. Szepesvári. Bandit Algorithms . Cambridge University Press, 2020. URL https://tor-lattimore.com/downloads/book/book.pdf .

O.-A. Maillard. APPRENTISSAGE SÉQUENTIEL: Bandits, Statistique et Renforcement. PhD thesis, Université des Sciences et Technologie de Lille-Lille I, 2013.

O.-A. Maillard, R. Munos, and G. Stoltz. A finite-time analysis of multi-armed bandits problems with kullback-leibler divergences. In Proceedings of the Conference On Learning Theory (COLT) , pages 497-514, 2011.

B. Mason, K.-S. Jun, and L. Jain. An experimental design approach for regret minimization in logistic bandits. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7736-7743, 2022.

P. Ménard and A. Garivier. A minimax and asymptotically optimal algorithm for stochastic bandits. In Proceedings of the international conference on Algorithmic Learning Theory (ALT) , pages 223-237, 2017.

P. Ménard and A. Garivier. A minimax and asymptotically optimal algorithm for stochastic bandits. In International Conference on Algorithmic Learning Theory , pages 223-237. PMLR, 2017.

S. Mukherjee, K. P. Naveen, N. Sudarsanam, and B. Ravindran. Efficient-ucbv: An almost optimal algorithm using variance estimates. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) , 2018.

E. Novak. Some results on the complexity of numerical integration. Monte Carlo and QuasiMonte Carlo Methods: MCQMC, Leuven, Belgium, April 2014 , pages 161-183, 2016.

F. Orabona. A modern introduction to online learning. arXiv preprint arXiv:1912.13213 , 2019.

J. M. Robins and A. Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association , 90(429):122-129, 1995.

D. Russo and B. V. Roy. Learning to Optimize via Posterior Sampling. Mathematics of Operations Research , 39(4):1221-1243, 2014.

Y. Seldin and A. Slivkins. One practical algorithm for both stochastic and adversarial bandits. In International Conference on Machine Learning , pages 1287-1295. PMLR, 2014.

W. R. Thompson. On the Likelihood that One Unknown Probability Exceeds Another in View of the Evidence of Two Samples. Biometrika , 25(3/4):285, 1933.

C.-Y. Wei and H. Luo. More Adaptive Algorithms for Adversarial Bandits. In Proceedings of the Conference on Learning Theory (COLT) , pages 1263-1291, 2018. URL http://arxiv. org/abs/1801.03265 .

J. Zimmert and Y. Seldin. Tsallis-inf: An optimal algorithm for stochastic and adversarial bandits. The Journal of Machine Learning Research , 22(1):1310-1358, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the author analyzes the MED algorithm proposed by Honda \\& Takemura (2011) for Bernoulli distributions in the context of general bounded distributions, and under the name KL-Maillard Sampling. This work is a follow-up of a previous work that proposed Maillard Sampling for sub-gaussian distributions. \n\nKL-MS is a bandit algorithm that samples an arm at each time step with probability $p(t) \\propto \\exp(-N_a(t) \\text{kl}(\\mu_{t-1,a}, \\mu_{t-1, \\text{max}}))$, where kl is the KL divergence corresponding to Bernoulli distributions. The interest of this simple strategy is that one can explicitly compute the probability to pull each arm, which is useful for instance in the context of off-policy evaluation.\n\nContrarily to the initial work of Honda &  Takemura that focused on instance-dependent bounds, and in the spirit of the previous MS paper, the authors provide both optimal instance-dependent and minimax bounds for the algorithm for Bernoulli distributions. The same guarantees naturally hold for general bounded distributions, losing the optimality for problem-dependent guarantees compared to the original MED algorithm using the ``tight'' divergence. It is also proved that the worst-case guarantees scale in the standard deviation of the best arm, which is on par with what is known from the sub-gaussian case.\n\nSTRENGTHS:\n* The paper is well-written, clear, and easy to follow. Furthermore, it seems technically sound, and the proofs are carefully detailed. The literature review is well-covered regarding the scope of the paper.\n \n*  The analysis of asymptotic optimality of MED for the Bernoulli case is largely simplified compared to the original proof of Honda \\& Takemura (2011). Furthermore, the worst case optimality is a novel result compared to this work. Compared to the previous MS paper, it is also interesting that the optimal minimax ratio is achieved without tweaking the algorithm. To prove these results, the the authors perfectly used the analysis tricks introduced recently in the TS literature (e.g all the cited works from Jin et al.).\n  \n* The main novel element/insight of the paper compared to its two major inspirations is the refined analysis of the \"under-exploration\" term (F3) in the regret analysis, that lead to the $\\log(K)$ vs. $\\log(T)$ improvement of the minimax ratio of MS without having to change the algorithm, making MS+ obsolete. The changes in the proof for this are rather substantial so the contribution is valuable. \n\n* Maybe the most surprising result in the paper is the minimax bound scaling with the variance term $\\mu_1(1-\\mu_1)$. Therefore I would have appreciated some explanation in the main text as to where Theorem 4 comes from. If I understand correctly, it follows from a tighter version of Pinsker's inequality (Lemma 28) which is worth highlighting. While interesting, this trick could certainly be applied to the analysis of other bandit algorithms (as done for KL-UCB in an Appendix), so the result cannot really be interpreted as an indicator of the superior performance of KL-MS.\n\nWEAKNESSES:\n* I am a bit uncomfortable with the re-branding of MED as KL-MS. This was understandable for the initial MS paper since Honda & Takemura tackled bounded distributions, but here the algorithm exactly matches MED for Bernoulli distributions. Furthermore, it is folklore that for algorithms using divergences the Bernoulli divergence can be used for general bounded distributions. Hence, there is no real reason for this re-branding in my opinion. However, I insist on the fact that I find the novel elements of analysis intereting.\n\n* Regarding the analysis, it seems that it differs from the original paper only from term (F3). The way the authors handle this term is very interesting and a valuable contribution in itself, but this should be better highlighted in the paper. However, even this part seems largely inspired by recent papers from Jian et al. for the analysis of Thompson Sampling, so I wonder if there is a really novel theoretical contribution in the paper. \n\n* Minor: the authors should cite a recent follow-up of Honda and co-authors on the MED algorithm: https://arxiv.org/abs/2303.06058 \nThis do not alter the contribution of this paper since the authors focus on problem-dependent guarantees of MED for a broader class of distributions, but answer some of the questions presented in the conclusion of the paper.\n\nQUESTIONS:\nI don't have specific questions for this work, everything seems clear to me.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a classic bandit problem, where the algorithm should explicitly output the random distribution of the next pulling arm (as a comparason, in classic case, the algortihm only needs to generate one arm from this random distribution and outputs that arm). Existing results only work on the case that the random rewards are unbounded and subgaussian. In this paper, the authors extend the existing works to the case that the rewards are bounded in $[0,1]$, design a KL-MS algorithm (using a KL-divergence approach). They show that the regret upper bound of KL-MS is near optimal. They also use some experiments to show that the performance of KL-MS (in outputing the random distribution precisely) outperforms existing baselines.\n\nSTRENGTHS:\nThe regret bound in this paper is nearly tight.\nThe writting is clear for me to understand.\n\nWEAKNESSES:\nMy first concern is about the model setting, i.e., why we require to know the exact random distribution of pulling the arm? Though there is an example of estimating the average reward, I do not think this is a well-motivated one. Can you provide more examples about the why we require that distribution in reality? Besides, why not just use a UCB-type algorithm, which can easily give you the exact probability distribution, as long as tight analysis? I know that there are works, e.g., MOSS, to achieve the tight $O(\\sqrt{KT})$ regret upper bound, but not very sure whether there are UCB algorithms that achieve $O(\\sqrt{\\mu(1-\\mu)KT})$ regret upper bound (though I think after using some variance-based concentration, the steps are straightforward). \n\nMy second question is about the experiments of this paper (in appendix H). I do not see any comparason about the regrets between different algorithms, and I am wondering the regret performances of KL-MS.\n\nFinally, do you think the idea (of either MS or KL-MS) could be applied to infinite-arm case (e.g., linear bandits), for example, return a distribution supported on an infinite set?\n\nQUESTIONS:\nPlease see the above \"Weaknesses\"\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission considers the vanilla setting of stochastic K-armed bandits and studies a strategy introduced by Maillard (2013), which relies on exponential weights and outputs at each round probabilities of taking each action. This is often convenient in offline policy evaluation, when estimates based on inverse propensity weighting [IPW] are constructed. Distribution-dependent and distribution-free regret bounds are provided, either in a general non-parametric model of all probability distributions over [0,1], or in the much specific model of Bernoulli distributions. The general distribution-dependent regret bounds asymptotically match the gap-based bounds of UCB (Theorem 1 and Remark 2) and is actually optimal in the Bernoulli model (Theorem 5). The general distribution-free bound improves on the one for UCB by featuring a \\sqrt{\\mu^\\star (1-\\mu^\\star)} term (Theorem 3). Another main result is formed by Figure 1 and Table 1: there are actually few randomized strategies (Thompson sampling, MED) and none of them exhibits closed-form expressions for probabilities of plays. \n\nThis shows that the core result of this article is: a strategy for the vanilla case of stochastic K-armed bandits, with decent (though not optimal) distribution-dependent and distribution-free regret bounds, and based on determining actual probabilities of playing arms, which is useful for offline policy evaluation.\n\nSTRENGTHS:\nThe idea of constraining the strategy to output probability distributions while getting decent bounds is nice and may turn useful---I have witnessed several recent articles critically using IPW in bandit contexts. By 'decent bounds', I mean bounds that are as good as, or slightly better than, UCB, but not optimal (as IMED and recent versions of KL-UCB achieve).\n\nThe exposition is clear and I enjoyed reading the main body of the submission.\n\nWEAKNESSES:\n1.\nExponential weights are actually difficult to compute in practice with a good accuracy, at least for suboptimal arms that are played often. Perhaps this case does not arise (suboptimal arms are played only logarithmically many times and the probabilities are easy to compute with a good accuracy), but the accuracy in the computation should be commented, especially given the critiques against Thompson sampling on these issues on page 2. \n\n2. \nThe comparison to previous works could be clarified and reorganized in pages 4--5. In particular, it would have been better to recall first the typical (e.g., for UCB) as well as the optimal distribution-dependent and distribution-free regret bounds, in the Bernoulli model and in the model P(0,1) of all distributions over [0,1], when no constraint of outputting probability distributions is imposed. The sub-UCB criterion could be omitted, I don't think it adds anything. The literature review seems a bit outdated. In particular, a new reference is critically missing: https://www.jmlr.org/papers/volume23/20-717/20-717.pdf / it shows that in the model P(0,1) there exists a strategy called KL-UCB-switch that achieves simultaneously the optimal distribution-dependent and the optimal distribution-free bounds. \n\n3.\nThe regret bounds are unprecise: (i) they involve O(...) terms and (ii) even the main terms are difficult to read because of +/- c \\Delta_a terms in the kl, (iii) not mentioning the additive T \\Delta term, where \\Delta is a parameter that must then be << \\ln T / T and therefore should vanish. Even worse, Lemma 9 and Theorem 5 are proved by taking \\Delta = 0 in the bound of Theorem 1, while Theorem 1 assumes \\Delta > 0.\n\n4.\nThe proof sketches are too vague: pages 8-9 merely indicate a proof structure in terms of decompositions of events and other immediate considerations, but the actual boundings of the probabilities of interest is not explained in the main body (but is detailed in the extremely long appendix). At least two or three salient (new?) ingredients of these proofs should have been given in the main body. What I read on pages 8-9 is too high level and actually takes almost 2 pages without learning anything specific to the reader. I regretfully couldn't check the proofs and get a sense of their correctness, but I don't feel guilty for this, as nothing or almost helped me doing this in the main body. Better editorial choices could have been made as far as proof sketches are concerned. \n\n\nOther comments / remarks / typos along the text:\n- Lines 9-10: I wouldn't insist on the distribution-dependent optimality for Bernoulli distributions (which is a minor point) but rather on getting decent (better than UCB) bounds \n- Line 28: also Garivier and Cappé 2011 \n- Footnote 1: yes, but for known L and U \n- Caption of Figure 1: difficult to understand in itself, I have to read lines 770--773 in appendix to understand (these explanations should thus be moved in the main body)\n- Line 48: the empirical averages \\mu_{t,a} were not formally defined \n- Lines 54-55: sounds like an overstatement; the submission proposes decent but not optimal distribution-dependent and distribution-free regret bounds \n- Table 1: Tsallis-INF is among the few strategies that output distributions, so it should not have been excluded on the ground of a minor issue given its non-optimal bound for Bernoulli distribution; this Table 1 is generally not helpful given that it contains too many strategies that do not output distributions\n- Lines 65-67: I would be less enthusiastic; getting rid of the \\sqrt{\\ln K} is uneasy but was achieved for some algorithms that are optimal from the distribution-dependent viewpoint (see https://www.jmlr.org/papers/volume23/20-717/20-717.pdf) though I appreciate as well the \\sqrt{\\mu^*(1-\\mu^*)} term, which indeed may be smaller than \\sqrt{\\ln K} in some situations \n- Line 69: drop 'with'\n- Line 82: OK for this definition in case of absolute continuity, otherwise, = +\\infty\n- Lines 108-109: syntax issue\n- Line 119: > b, not \\geq b; this quantity is called K_inf \n- Lines 154-155: some strategies like KL-UCB-switch (see https://www.jmlr.org/papers/volume23/20-717/20-717.pdf) would be sub-UCB and enjoy a \\sqrt{KT} distribution-free regret bound---I thus disagree with the statement made here \n- Lines 205-206: same kind of comments \n- Lines 239-241: there is no such issue if the Bernoullization trick of lines 115-119 is implemented \n- Line 258: rather (6) instead of (8)\n\nQUESTIONS:\nI would like to read authors' opinion on the four main weaknesses that I raised.\n\nLIMITATIONS:\n(They are well-addressed in the conclusion, Section 6, and include extensions of the optimality results of Maillard sampling to general exponential families and even to the non-parametric setting of all distributions over [0,1].)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the classical regret-minimization problem in the stochastic multi-armed bandit framework. In particular, the manuscript's focus is on randomized algorithms with an aim to develop one with closed-form arm-selection probabilities at each step. Data collected by such algorithms can be used for offline policy evaluation. The manuscript proposes an algorithm called KL-MS for bounded-support distributions, that achieves KL-style regret guarantees. It is asymptotically optimal (in the instance-dependent stochastic sense) for bernoulli bandits, and order optimal for more general bounded-support distributions. It also enjoys an optimal worst-case regret guarantee. The paper also presents numerical study comparing the offline evaluation when the data is collected using the proposed KL-MS and Thompson Sampling with Monte-carlo on bernoulli bandits.\n\nSTRENGTHS:\nThe paper is written well and easy-to-read. I particularly enjoyed the various remarks and discussions after the results, providing insights into the results and comparing the analysis to existing ones. While in some settings randomness should also be treated as a resource (and hence be used with care), there are indeed benefits to using randomized algorithms in other settings. For example, as highlighted in the paper, the data collected by randomized algorithms can be used for offline evaluation. The paper develops an algorithm that simultaneously satisfies different desirable properties, while being optimal (or close-to-optimal) in a non-parametric setting of bounded-support distributions.\n\nWEAKNESSES:\n1. The plots in the appendix are not very clear. The text along the vertical lines is overlapping and unreadable. It would be good to spread-out the fugures and probably figures in a vector form for clearer display. \n\n2. In view of the recent results on fragility of optimized bandit algorithm, I believe that the MAB results should be studied and stated beyond the expected regret. See for instance \"Fan, Lin, and Peter W. Glynn. \"The fragility of optimized bandit algorithms.\" arXiv preprint arXiv:2109.13595 (2021).\"\n\nQUESTIONS:\n1. In line 21, the Reg(T) is referred to as pseudo-regret. How is this different from the usual expected regret? Why \"pseudo\" in the regret?\n\n2. How would the analysis change (or challenges in extending) if instead of Bernoulli-kl in the exponent, one uses KL (the lower bound opt.)? Could the current results and analysis be already extended to that algorithm? I believe that is then the MED algorithm? It would then suggest a natural way to extend the algorithm and analysis for exponential families and more general distributions, like the heavy-tailed ones considered in Agrawal, Juneja, Koolen, 2021. A discussion along these lines would be interesting to see. \n\n3. How does MS compare with KL-MS numerically? \n\n4. How is the performance of the algorithm affected if the bounds [0,1] are not exactly known. For example, if the samples are from some misspecified setting, i.e., distribution with support in  [-0.5, 0.5] but the algorithms is guaranteed only [0,1]-supported distributions?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}