Title: PRODuctive bandits: Importance Weighting No More
Venue: NeurIPS 2024
Accepted: yes
Keywords: Bandit Algorithms

Online Learning
Abstract: Prod is a seminal algorithm in full-information online learning, which has been conjectured to be fundamentally sub-optimal for multi-armed bandits.
By leveraging the interpretation of Prod as a first-order OMD approximation, we present the following surprising results:
1. Variants of Prod can obtain optimal regret for adversarial multi-armed bandits. 2. There exists a simple and (arguably) importance-weighting free variant with optimal rate. 
3. One can even achieve best-both-worlds guarantees with logarithmic regret in the stochastic regime.

The bandit algorithms in this work use simple arithmetic update rules without the need of solving optimization problems typical in prior work. Finally, the results directly improve the state of the art of incentive-compatible bandits.

PRODuctive bandits: Importance Weighting No More

Abstract

Prod is a seminal algorithm in full-information online learning, which has been conjectured to be fundamentally sub-optimal for multi-armed bandits. By leveraging the interpretation of Prod as a first-order OMD approximation, we present the following surprising results: 1. Variants of Prod can obtain optimal regret for adversarial multi-armed bandits. 2. There exists a simple and (arguably) importance-weighting free variant with optimal rate. 3. One can even achieve best-both-worlds guarantees with logarithmic regret in the stochastic regime.

The bandit algorithms in this work use simple arithmetic update rules without the need of solving optimization problems typical in prior work. Finally, the results directly improve the state of the art of incentive-compatible bandits.

1 Introduction

The adversarial multi-armed bandit (MAB) problem is a seminal online learning problem with applications in experimental design, online advertisement and more [Thompson, 1933, Lai and Robbins, 1985, Auer et al., 2002a,b]. MABs are characterized by the limited feedback given to the learner in every round, the so-called bandit feedback, in which the learner only observes the loss of their selected action, unlike in the full information, also known as the experts, setting where the loss of all actions are provided as feedback.

The first nearly optimal algorithm for the adversarial MAB problem is EXP3 [Auer et al., 2002b]. EXP3 is a direct adaptation of the Hedge algorithm [Littlestone and Warmuth, 1994, Cesa-Bianchi et al., 1997, Freund and Schapire, 1997] with importance weighting to handle partial bandit feedback. Hedge and EXP3 are special versions of online mirror descent (OMD), where the Bregman divergence is the KL divergence induced by the Negative entropy potential. The OMD view of online learning [Abernethy et al., 2008] has lead to a wide range of MAB algorithms such as TsallisINF [Audibert and Bubeck, 2009] and Logbarrier [Agarwal et al., 2017], which enjoy improved regret guarantees. These guarantees can be attributed to regularizers more suited to the bandit feedback setting, compared to the negative entropy regularizer. A downside of OMD is that usually the mirror descent update is not closed form and requires (approximately) solving optimization problems at every iteration.

Alternative full-information algorithms with simple arithmetic updates are Prod [Cesa-Bianchi et al., 2007, Even-Dar et al., 2008, Gaillard et al., 2014], which enjoys second order regret bounds and Multiplicative Weights Update (MWU) [Arora et al., 2012]. Prod is known to be closely related to Hedge, both of which are different generalizations of the weighted majority algorithm [Littlestone and Warmuth, 1994] to non-binary feedback.

More recently, Freeman et al. [2020] studied incentive-compatible online learning, a setting where experts are not necessarily truthful but make predictions strategically with regard to the agent's algorithm. Motivated by deriving an algorithm where the incentives of experts align with the agent,

they propose WSU which can be seen as an instance of Prod. Freeman et al. [2020] further introduce a bandit adaptation WSU-UX, the first Prod algorithm for bandits. Unfortunately, WSU-UX only enjoys T 2 3 regret guarantees. This is not merely an issue with their analysis as has been shown via lower bounds [Mortazavi et al., 2024] and leads to the conjecture that this might be a fundamental separation between full-information and bandits.

We make the following contributions for understanding Prod under bandit feedback:

We disprove the separation conjecture by providing a simple modification of WSU-UX with nearly optimal O ( √ KT log( K )) regret guarantees.

We present a Prod variant that does not require importance weighting and yet enjoys O ( √ KT log( T )) regret bounds.

We present a Prod variant that achieves best of both worlds regret guarantees, i.e. it enjoys improved O (log( T )) regret bounds when the losses are stochastic, while maintaining worstcase O ( √ KT ) regret bounds.

Notation: For N ∈ N , let [ N ] = { 1 , . . . , N } . We use 〈· , ·〉 to denote the regular Euclidean scalar product and ∆( A ) to denote the probability simplex over a finite set A . O is the standard Landau notation hiding numerical constants, while ˜ O omits polylogaritmic factors as well. The expectation E is always taken over all randomness of the algorithm, losses and experts, while E t [ · ] = E [ · |F t ] , where F t is the filtration over all randomness up to step t . I ( E ) denotes the indicator function function for the event E . For a convex differentiable function F , the Bregman divergence is defined by D F ( y, x ) = F ( y ) -F ( x ) -〈 y -x, ∇ F ( x ) 〉 .

2 Problem setting and related work

The adversarial bandit problem is formally defined as follows. In every round t = 1 , . . . , T , an (oblivious) adversary selects a loss glyph[lscript] t ∈ [0 , 1] K (it is possible to extend the loss range to [ -1 , 1] K ) unknown to the agent. The agent simultaneously selects an expert A t ∼ π t , π t ∈ ∆([ K ]) . The agent incurs and observes the loss glyph[lscript] t,A t , but does not see the losses of other experts. The goal is to minimize the pseudo-regret 1

Popular families of algorithms for this problem setting include online mirror descent (OMD) and follow the regularized leader (FTRL). Typically the algorithms use unbiased loss estimates of the loss vector via importance weighting: ˆ glyph[lscript] t,i = glyph[lscript] t,i π t,i I ( A t = i ) . We note that other types of importance weighted estimators have been used in literature such as the implicit exploration estimator Kocák et al. [2014], which has improved variance properties. The algorithms are defined by a twice-differentiable convex potential function F : R K → R and a learning rate schedule η t , the agent maintains a distribution via

OMD optimizes locally given the last loss and, as we will show, is most closely related to Prod. FTRL on the other hand performs a global optimization and is generally considered superior for adaptive bounds with time-dependent learning rates. In some special cases, such as time-independent learning rate with potentials that satisfy ‖∇ F ( x ) ‖ → ∞ on the border of the optimization set, both algorithms are equivalent. Common potentials in the bandit literature are given in Table 1 and we refer to their respective Bregman divergences as D KL , D TS and D LB respectively. The negative entropy is the potential which defines Hedge and Exp-3 (and derivatives) [Littlestone and

Warmuth, 1994, Vovk, 1995, Freund and Schapire, 1997, Auer et al., 2002b, Kocák et al., 2014]. The 1 / 2 -Tsallis Entropy is the key to achieving optimal best-of-both-worlds regret guarantees as was first demonstrated by Zimmert and Seldin [2021]. The Logbarrier potential was used by Agarwal et al. [2017] to first solve the corralling of bandits problem and has found many applications in model-selection problems [Foster et al., 2020], regret bounds which depend on the properties of the loss sequence [Wei and Luo, 2018, Lee et al., 2020b,a] and various other bandit problems.

2.1 Prod family of algorithms

The original version of Prod [Cesa-Bianchi et al., 2007] maintains weights w t,i for each experts which are updated via w t +1 ,i = w t,i (1 -ηglyph[lscript] t,i ) and the agent plays the policy π t,i ∝ w t,i . This framework has been extended to D-Prod [Even-Dar et al., 2008], which shifts the losses in the weight update by the loss of a fixed policy, and ML-Prod [Gaillard et al., 2014] that shifts losses by the mean of the current policy (among other modifications). With a suitable shift in losses, one can ensure that the weights sum up to 1 and hence operate directly on the policy space. In its simplest form, this is

We refer to this update as Vanilla-Prod to emphasize its connection to the Prod literature, however this algorithm is exactly WSU [Freeman et al., 2020] derived for incentive-compatible online learning. We consider any algorithm a variant of Prod if it performs product updates of the form π t +1 ,i = π t,i (1 -ηL t,i ( glyph[lscript] t ; A t )) , where L t,i are linear affine functions of the loss. From now on we always assume the initial policy is π t,i = 1 /K , and this holds for all algorithms presented in the paper. The appeal of Prod algorithms lies in their simple arithmetic update rule. A second motivation for using Prod updates is the mentioned incentive-compatibility.

2.2 Incentive-compatible online learning

In the incentive-compatible online learning setting, introduced by Freeman et al. [2020], experts provide recommendations, for example a prediction of whether it will rain on the next day. Each expert has an internal belief and the agent would like to receive each expert's true beliefs in order to learn to follow the best expert. In the simplest setting the experts make predictions about binary outcomes, with the i -th expert having (private) belief b t,i ∈ [0 , 1] about the t -th round outcome. The expert's belief is unknown to the agent and the expert only reports a prediction p t,i ∈ [0 , 1] about the outcome. Based on the expert predictions, { p t,i } i ∈ [ K ] the agent makes a prediction based on ¯ p t = ∑ K i =1 π t,i p t,i and incurs a loss L (¯ p t , r t ) ∈ [0 , 1] based on the realized outcome r t ∈ { 0 , 1 } . In the weather forecasting example the outcome is the indicator if it rains the next day and the loss is L (¯ p t , r t ) = ( r t -¯ p t ) 2 . In Freeman et al. [2020] the experts only care about maximizing the probability that they are selected which does not necessarily result in truthful reporting, that is b t,i may differ from p t,i . The agent's goal of receiving the true beliefs, { b t,i } i ∈ [ K ] , can be achieved by playing an incentive compatible strategy which will always prefer selecting an truthful expert, that is the probability of π t +1 ,i of selecting expert i when the expert reports b t,i may only decrease if the expert reports any other p t,i instead, no matter how the remaining experts act throughout the game. This is made precise in Definition 2.1 of Freeman et al. [2020].

glyph[negationslash]

Freeman et al. [2020] show that standard OMD and FTRL algorithms are in fact not incentive compatible even when the loss L is restricted to be proper that is E r ∼ Bern ( b ) [ L ( p, r )] ≥ E r ∼ Bern ( b ) [ L ( b, r )] for all p = b . It turns out that any update for π t +1 which is linear affine in the proper loss function will lead to incentive compatibility and so the Prod family will ensures that experts report their true believes in this setting, i.e. they are incentive-compatible. The state of the art for incentive-compatible bandits is T 2 3 regret and any improvement for Prod directly transfers to better rates for this setting as well.

3 Modifying WSU-UX for nearly optimal regret guarantees

We begin by presenting a minimal modification of Algorithm WSU-UX which is sufficient for a regret guarantee of the order O ( √ KT log( K )) .

WSU-UX uses importance-weighted updates and injects a small uniform exploration.

where γ is the mixture coefficient. The role of uniform exploration is to ensure that the policy updates are proper i.e. π t +1 ,i ∈ (0 , 1) . Freeman et al. [2020] uses the following key lemmas in their analysis, which hold for any sequence of losses glyph[lscript] t ∈ [0 , 1] K . For completeness we restate the results we use below.

Lemma 1 (Lemma 4.1 [Freeman et al., 2020]) . If ηK/γ ≤ 1 2 , the WSU-UX weights π t and ˜ π t are valid probability distributions for all t ∈ [ T ] .

Lemma 2 (Lemma 4.3 [Freeman et al., 2020]) . For WSU-UX, the probability vectors { π t } t ∈ [ T ] and loss estimators ˆ glyph[lscript] t satisfy the following second order-bound

where i glyph[star] is the optimal expert/arm.

The bound in Lemma 2 is almost the standard regret bound that appears in the analysis of Hedge, except for the term η ∑ T t =1 ˆ glyph[lscript] 2 t,i glyph[star] . This term is the reason why prior work can not show regret bounds smaller than T 2 3 . Even after taking the expectation over the randomness of the agents actions, this term scales with with 1 /π t,i glyph[star] , which is potentially unbounded. Alternatively this term can be written as E j ∼ π glyph[star] t [ η ˆ glyph[lscript] 2 tj ] (where π glyph[star] is the policy picking i glyph[star] with probability 1) and if one could perform a change of measure to E j ∼ ˜ π t t [ η ˆ glyph[lscript] 2 tj ] , this term is immediately controllable.

In fact, change of measure techniques for bandits are now well established [Foster et al., 2020, Luo et al., 2021] by introducing biases to the losses. Assume we construct a bias to the losses ˜ glyph[lscript] t = glyph[lscript] t + δ t , which satisfies the same regret guarantee, Reg , as the original loss sequence, then running an algorithm over the biased loss sequence ˜ glyph[lscript] t which selects A t ∼ ˜ π t ensures

We introduce now the following modification to the losses

which corresponds to δ ti = ηglyph[lscript] ti ˜ π t,i . This yields the change of measure term

which is sufficient for controlling the term η ∑ T t =1 ˆ glyph[lscript] 2 t,i glyph[star] in Lemma 2.

Theorem 1. Running WSU-UX with the loss estimators in Equation 1 and γ = ηK 2 , η = Θ( √ log( K ) KT ) guarantees the following regret bound

The proof of Theorem 1 is deferred to Appendix B.

3.1 Intuition on biasing the update and the Prod family of algorithms

We provide an intuition in this section about why WSU-UX is not tight and why the bias we choose is able to correct the regret. The modern analysis of OMD (or FTRL) with a divergence function D is follows the template 2

where the inequality crucially relies on π OMD t +1 being the 1-step OMD update to the previous policy. OMD algorithms like Hedge choose the policy that minimizes the per-step stability term in every round, which is what allows for the stability term to be bounded appropriately. If instead of playing π OMD t an approximate policy π t ≈ π OMD t is played, the template regret analysis can be changed by adding the terms

where π OMD t +1 is now the 1-step OMD update from π t . When D is the KL divergence and the approximate policy π t is coming from the WSU-UX, this term contributes the undesirable η ˆ glyph[lscript] 2 t,i glyph[star] .

We now explain how our loss biasing solves this issue. The Vanilla-Prod/WSU update can be seen as a first order approximation to the Hedge update, that is

where λ t is a normalization factor. Tuning λ t such that ∑ K i =1 π t +1 ,i = 1 recovers Vanilla-Prod/WSU. To control the undesirable terms, we have to make the approximation tighter. The loss-biasing introduced in the previous section acts as a correction which brings the Vanilla-Prod/WSU update closer to the second order approximation of the Hedge update. Indeed, we have

and so our loss adjustment in Equation 1, ˜ glyph[lscript] t,i = glyph[lscript] t,i (1 -η/ ˜ π t,i ) , can be seen as a second order correction to the term η 2 2 ( ˆ glyph[lscript] t,i -λ t ) 2 . We cannot exactly correct the second order difference with linear update rules, which we address by slightly overcorrecting, i.e. biasing by a larger amount than the second order adjustments implies as necessary. That is, the correction term we use is of the order η/ ˜ π t,i instead of ηglyph[lscript] t,i / ˜ π t,i . Fortunately, the regret analysis is not sensitive towards this as we have shown in Theorem 1.

4 Importance weighting free adversarial MAB with LB-Prod

While our biased WSU-UX obtains optimal regret, it still has to go through the extra complexity of injecting additional uniform exploration at a rate of γ to ensure proper updates and add bias to the losses. As mentioned in the introduction, prior work proposed other potential functions that have favourable properties for bandit feedback. Using the same linearization argument to derive a Prod version based on the Logbarrier leads to a surprisingly simple algorithm without loss biasing that is arguably importance weighting free. LB-Prod differs from WSU-UX by using the masked loss ˜ glyph[lscript] t,i = glyph[lscript] t,i I ( A t = i ) instead of the importance weighted loss and a non-symmetric normalization λ t,i :

It is easy to confirm ˜ glyph[lscript] t,i -λ t,i ∈ [ -1 , 1] via π t,i π t,A t ≤ ( π 2 t,i + π 2 t,A t ) / 2 yielding proper updates for η < 1 . The following theorem shows that this simple algorithm is rate optimal under the right tuning. Theorem 2. For any sequence of losses glyph[lscript] t ∈ [ -1 , 1] K and any η < 1 , LB-Prod produces valid distributions π t ∈ ∆([ K ]) and its regret is bounded by

Tuning η = √ log( T ) K 2 T results in a regret bound of O ( √ KT log( T )) for any T > K log( T ) 2 . The proof of Theorem 2 is deferred to the end of the section.

4.1 Intuition of LB-Prod

As mentioned before, LB-Prod is the linear approximation of OMD with Bregman divergence induced by the Logbarrier potential, that is D LB induced by the potential fuction F ( x ) = -∑ K i =1 log( x i ) . The one-step Logbarrier OMD update of a policy π t with importance-weighting is known (see e.g. [Zimmert and Seldin, 2021]) to take the form

where λ t is a normalization constant that ensures π OMD t is a probability distribution. If instead λ t is tuned so that ∑ K i =1 π t +1 ,i = 1 , the LB-Prod update is recovered. This can be be verified by setting ˜ glyph[lscript] t,i = π t,i ˆ glyph[lscript] t,i and λ t,i = π t,i λ t . The curvature of the Logbarrier regularization is what ensures that the importance weighted loss ˆ glyph[lscript] t,i is always multiplied with its probability π t,i , allowing to run the algorithm on the masked non-weighted loss sequence directly.

Additionally, the second order approximation is

The additional undesirable terms, unlike in the case for WSU-UX, will only contribute ηT regret if not adjusted for as we show next.

4.2 Analysis of LB-Prod

The following technical lemma is proven in the appendix.

Lemma 3. For any timestep t and arm i , it holds

where c t ∈ [ -1 , 1] is an arm independent constant.

In Section 3.1 we argued that one needs to bound the additional term η -1 ( D LB ( π glyph[star] , π t +1 ) -D LB ( π glyph[star] , π OMD t +1 )) to reduce the analysis to standard OMD. While this term is nicely bounded for LBProd, it turns out that it is easier to directly bound the 'prototype' of the stability term in Equation (2) due to the fact that we have a closed form expression of π t +1 .

Lemma 4. For any time t ∈ [ T ] and any u ∈ ∆([ K ]) , it holds

The proof is an algebraic exercise and deferred to the supplementary material. Finally, we can prove the main regret guarantee.

Proof of Theorem 2. To show that this algorithm outputs proper probability distributions, note that

Additionally we have seen that | ˜ glyph[lscript] t,i -λ t,i | ≤ 1 , hence for any η < 1 , the probability of any arm is strictly positive. For any comparator u glyph[star] , we define u = u glyph[star] + 1 T ( π 1 -u glyph[star] ) , which satisfies ∑ T t =1 〈 u -u glyph[star] , glyph[lscript] t 〉 ≤ 2 . Using Lemma 4 and Equation (2), we obtain by the telescoping sum of Bregman terms

4.3 The perturbation analysis

We outline an alternative analysis that reuses established machinery and might be more accessible for some readers. Our analysis begins by viewing the Prod update as an exact OMD update over a perturbed loss sequence. Indeed, there is a sequence of perturbations { glyph[epsilon1] t } t ∈ [ T ] , glyph[epsilon1] t ∈ R K , such that

The exact form of glyph[epsilon1] t,i satisfies the following

Since Prod is exactly OMD over the sequence ˆ glyph[lscript] t -glyph[epsilon1] t , we can decompose the regret as follows

The analysis is not entirely straightforward as the loss range for the OMD update becomes [ -1 -O ( η ) , 1 + O ( η )] because of the shift introduced by the perturbation of the losses, and this posses some additional difficulties.

5 Best of both worlds algorithms

In applications where the loss is potentially more benign, for example sampled i.i.d. from a a fixed distribution over [0 , 1] K , it is desirable to obtain faster rates in nice environments while preserving worst-case guarantees. Probably the simplest algorithm with this property is Tsallis-INF [Zimmert and Seldin, 2021], which is FTRL with 1/2-Tsallis entropy and η t ∝ 1 / √ t learning rate.

5.1 TS-Prod

Recall the 1/2-Tsallis regularizer is F ( x ) = -∑ K i =1 2 √ x i . Unlike OMD, FTRL is not canonically expressed as a 1-step update of the previous policy. Instead, the 1/2-Tsallis-INF policy is given with a normalization constant Λ t (see [Zimmert and Seldin, 2021])

Recursively using this expression yields

where ξ t = 1 η 2 t -1 η t η t -1 and λ t = Λ t +1 -η t η t -1 Λ t . The first order approximation is

We ensure following the approximation in expectation by directly biasing the losses with η t ξ t / √ π t,i . Additionally, one needs to perform a second-order correction as discussed in Section 3.1. We omit a formal derivation, but notice that WSU-UX required η/π t,i correction, while LB-Prod works without correction because the error is of order η . As the intermediate potential between KL and Logbarrier, Tsallis-INF turns out to require a correction of order η/ √ π t,i , which we tighten by an additional factor of (1 -π t,i ) necessary to ensure stochastic bounds.

With this, we are ready to present

glyph[negationslash]

Theorem 3. The regret of TS-Prod with η t = 1 √ K +26 t , γ = 13 2 is bounded by O ( √ KT + K log( T )) in the adversarial setting and by O ( ∑ i = i glyph[star] log( T ) ∆ i ) in the stochastic setting.

5.2 Analysis of TS-Prod

We first show that the loss biasing is sufficient to ensure that the distribution is well defined.

Lemma 5. If ξ t is a non-increasing sequence, η t < 2 √ K ( ξ t + γ ) 2 and η 2 t +1 ≤ η 2 t (1 -2 γη 2 t ) for all t , then the update rule of TS-Prod is proper and satisfies π t,i > ( ξ t + γ ) 2 η 2 t for any arm and loss sequence at all time steps.

Next we present the moving parts of the analysis.

The change of measure is by construction

We now bound the stability and penalty.

Lemma 6. For any time t such that π t,i > ( ξ t + γ ) 2 η 2 t , it holds

The tuning of Theorem 3 satisfies the conditions.

Lemma 7. The proto-penalty is bounded by

We are ready to prove the main regret guarantee.

Proof of Theorem 3. By Lemma 5 we have a proper update rule. Using Lemma 6, equation (3), where we tuned γ = 13 2 and Lemma 7 yields glyph[negationslash]

The adversarial regret follows from ∑ K i =1 √ π t,i ≤ √ K , ξ t ≤ 4 , ∑ T t =1 η t = O ( √ T ) . For the stochastic regret, the proof follows standard arguments using the self-bounding trick as in Zimmert and Seldin [2021]. For details on the self-bounding trick see the supplementary material.

5.3 TS-Prod and stabilized OMD

Even though we derived TS-Prod from FTRL, it turns out that one can also interpret the update as an approximation of stabilized OMDproposed by Fang et al. [2022]. In Appendix E, we formalize this connection and present a slight variation of TS-Prod. We then analyse this variant via the perturbation technique described in Section 4.3. Our analysis also shows that the stabilized OMD algorithm induced by the 1 / 2 -Tsallis entropy enjoys best-of-both worlds regret guarantees which to the best of our knowledge is novel.

6 Discussion

We have provided an extensive study of incentive-compatible bandits. We have negatively resolved an open question of whether incentive-compatibility as defined in Freeman et al. [2020] is harder than regular bandits. Using linear approximations, partly with second order corrections, allows to recover results from well studied algorithms in the literature. We even obtain an algorithm with bestof-both-world guarantees. Our algorithms are conceptually simpler than existing bandit algorithms, they update the probability distributions with basic arithmetic operations without the need to solve optimization problems.

Our successes make it likely that one can transfer even more sophisticated methods, such as first-order, second-order, path-norm bounds and online learning with graph feedback to this framework. We leave this investigation to future work.

References

Jacob D Abernethy, Elad Hazan, and Alexander Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. In COLT , pages 263-274. Citeseer, 2008.

Alekh Agarwal, Haipeng Luo, Behnam Neyshabur, and Robert E Schapire. Corralling a band of bandit algorithms. In Conference on Learning Theory , pages 12-38. PMLR, 2017.

Sanjeev Arora, Elad Hazan, and Satyen Kale. The multiplicative weights update method: a metaalgorithm and applications. Theory of computing , 8(1):121-164, 2012.

Jean-Yves Audibert and Sébastien Bubeck. Minimax policies for adversarial and stochastic bandits. In Conference on Learning Theory , 2009.

Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47(2-3), 2002a.

Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1), 2002b.

glyph[negationslash]

Nicolo Cesa-Bianchi, Yoav Freund, David Haussler, David P Helmbold, Robert E Schapire, and Manfred K Warmuth. How to use expert advice. Journal of the ACM (JACM) , 44(3):427-485, 1997.

Nicolo Cesa-Bianchi, Yishay Mansour, and Gilles Stoltz. Improved second-order bounds for prediction with expert advice. Machine Learning , 66(2):321-352, 2007.

Eyal Even-Dar, Michael Kearns, Yishay Mansour, and Jennifer Wortman. Regret to the best vs. regret to the average. Machine Learning , 72(1):21-37, 2008.

Huang Fang, Nicholas JA Harvey, Victor S Portella, and Michael P Friedlander. Online mirror descent and dual averaging: keeping pace in the dynamic case. The Journal of Machine Learning Research , 23(1):5271-5308, 2022.

Dylan J Foster, Claudio Gentile, Mehryar Mohri, and Julian Zimmert. Adapting to misspecification in contextual bandits. Advances in Neural Information Processing Systems , 33:11478-11489, 2020.

Rupert Freeman, David Pennock, Chara Podimata, and Jennifer Wortman Vaughan. No-regret and incentive-compatible online learning. In International Conference on Machine Learning , pages 3270-3279. PMLR, 2020.

Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences , 55(1):119-139, 1997.

Pierre Gaillard, Gilles Stoltz, and Tim Van Erven. A second-order bound with excess losses. In Conference on Learning Theory , pages 176-196. PMLR, 2014.

Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends R © in Optimization , 2(3-4):157-325, 2016.

Tomáš Kocák, Gergely Neu, Michal Valko, and Rémi Munos. Efficient learning by implicit exploration in bandit problems with side observations. Advances in Neural Information Processing Systems , 27, 2014.

Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1), 1985.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Chung-Wei Lee, Haipeng Luo, Chen-Yu Wei, and Mengxiao Zhang. Bias no more: high-probability data-dependent regret bounds for adversarial bandits and mdps. Advances in neural information processing systems , 33:15522-15533, 2020a.

Chung-Wei Lee, Haipeng Luo, and Mengxiao Zhang. A closer look at small-loss bounds for bandits with graph feedback. In Conference on Learning Theory , pages 2516-2564. PMLR, 2020b.

Nick Littlestone and Manfred K Warmuth. The weighted majority algorithm. Information and computation , 108(2):212-261, 1994.

Haipeng Luo, Chen-Yu Wei, and Chung-Wei Lee. Policy optimization in adversarial mdps: Improved exploration via dilated bonuses. Advances in Neural Information Processing Systems , 34:2293122942, 2021.

Ali Mortazavi, Junhao Lin, and Nishant Mehta. On the price of exact truthfulness in incentivecompatible online learning with bandit feedback: a regret lower bound for wsu-ux. In International Conference on Artificial Intelligence and Statistics , pages 4681-4689. PMLR, 2024.

Shai Shalev-Shwartz. Online learning and online convex optimization. Foundations and Trends R © in Machine Learning , 4(2), 2012.

William R Thompson. On the likelihood that one unknown probability exceeds another in view ooster the evidence of two samples. Biometrika , 25(3/4), 1933.

Vladimir G Vovk. A game of prediction with expert advice. In Proceedings of the eighth annual conference on Computational learning theory , pages 51-60, 1995.

Chen-Yu Wei and Haipeng Luo. More adaptive algorithms for adversarial bandits. In Conference on Learning Theory , 2018.

Julian Zimmert and Yevgeny Seldin. Tsallis-inf: An optimal algorithm for stochastic and adversarial bandits. Journal of Machine Learning Research , 22(28):1-49, 2021.

Contents

A Background on FTRL and OMD

Wegive a brief overview of the template analysis for FTRL and OMD. For an extensive discussion and regret analysis of these two frameworks we refer the interested readers to Chapter 2 of Shalev-Shwartz [2012], Chapter 5 of Hazan et al. [2016] or Chapter 28 of Lattimore and Szepesvári [2020].

A.1 OMDanalysis overview

The OMD update

can be written in two steps as

where the first step is an unconstrained optimization over the linear loss at time t together with a regularization term given by the Bregman divergence induced by F , and the second step is the Bregman projection onto the probability simplex. Thie first step of the OMD update can be re-written as

Let F t = 1 η t . Since π t +1 is the minimizer of the OMD update, and D F w is convex we have that

Further, it holds that

Combining the two inequalities together we have that one step of the regret to any u ∈ ∆([ K ]) is bounded as

For a fixed step-size η t = η the above telescopes to bound the regret as

As long as F is twice differentiable, each of the terms can be bounded as D F ( π t , ˜ π t +1 ) ≤ O ( η 2 ‖ ˆ glyph[lscript] t ‖ 2 ∇ 2 ( F ∗ )( w t ) ) , where F ∗ is the Fenchel conjugate of F . Controlling ‖ ˆ glyph[lscript] t ‖ 2 ∇ 2 ( F ∗ )( w t ) in OCO is usually done by assuming some boundedness of the losses. In bandit literature controlling this term is slightly more involved and depends on the choice of F .

When η is not constant, telescoping the above sum does not work and the analysis becomes much more involved. It is possible to construct sequences of losses for which the OMD update does not enjoy sub-linear regret for η t = 1 √ t . Fang et al. [2022] introduce a stabilization term to the OMD update which overcomes this problem and show that this new version does enjoy the standard OMD regret guarantees.

A.2 FTRL analysis overview

The FTRL analysis follows similar ideas, however, the one step regret is bounded as

where ˆ L t = ∑ t -1 s =1 ˆ glyph[lscript] t , and I ∆[ K ] is the characteristic function of ∆ [ K ] , i.e., I ∆[ K ] ( π ) = 0 if π ∈ ∆[ K ] and I ∆[ K ] ( π ) = + ∞ otherwise. The term D F ∗ ( -ˆ L t , -ˆ L t -1 ) can be thought of as the equivalent to D F ( π t , ˜ π t +1 ) in the OMD analysis. The term ( F t + I ∆([ K ]) ) ∗ ( -ˆ L t -1 ) -( F t + I ∆([ K ]) ) ∗ ( -ˆ L t ) needs to be telescoped in an appropriate way. For more details we refer the reader to the penalty term bound of Zimmert and Seldin [2021].

B Missing Proof Section 3

Proof of Theorem 1. WLOG we assume that T ≥ K . We begin with the bound from Lemma 2. The second and third term in the RHS of the inequality are bounded as is standard in the Exp3 analysis

where F t -1 is the filtration generated by the random play and randomness of the losses up to time t -1 . We now consider the expectation of the LHS which evaluates to

Thus combining the bounds on the LHS and RHS we have

To complete the proof we only note that ∑ T t =1 ∑ K i =1 E [ π t,i glyph[lscript] t,i ] -∑ T t =1 E [ glyph[lscript] t,A t ] ≤ 2 Tγ = ηKT .

C Missing Proofs Section 4

Proof of Lemma 3. The expectation of ˜ glyph[lscript] ti = glyph[lscript] ti I ( A t = i ) is obviously π ti glyph[lscript] ti , hence by the definition of λ ti , we have

For the second part, we have

The proof is completed by noting

Proof of Lemma 4.

D Missing Proofs Section 5.2

D.1 Technical Lemmas

Proof. We first show that the optimal point is smaller than 1 9 , by looking at the derivative

For the enumerator, we have for all x ≥ 1 9 :

Hence

Lemma 9. For any a, b ≥ 0 such that a + b ≥ 1 , it holds

Proof. We can assume w.l.o.g. that a = 1 -b , otherwise scale both a and b down and reduce the objective. The resulting problem is symmetric with a = 1 2 as the unique minimizer resulting in the statement.

D.2 Minimal probability

Lemma 10. Assume that π ti > ( ξ t + γ ) 2 η 2 t holds for all arms, then

Proof. Let ˜ glyph[lscript] t,i = π t,i ˆ glyph[lscript] t,i and λ t,i = π t,i λ t . Note that ˜ glyph[lscript] ti ∈ [ -1 , 1] by the condition on π ti and ˜ glyph[lscript] ti = 0 for i = I t by construction of the loss estimate. Hence glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

where ˜ π tj = π tj 1 -π ti . We bound the two terms in the bracket separately, for the first term we have glyph[negationslash]

The second term is

glyph[negationslash]

Lemma 11 (Lemma 5) . If ξ t is a non-increasing sequence, η t < 2 √ K ( ξ t + γ ) 2 and η 2 t +1 ≤ η 2 t (1 -2 γη 2 t ) for all t , then the update rule of TS-Prod is well defined and satisfies π t,i > ( ξ t + γ ) 2 η 2 t for any arm and loss sequence at all time steps.

glyph[negationslash]

Proof. The proof follows by induction. At t = 1 the statement is true by definition. Let the claim hold at time t , then the probability of an arm only decreases when ˆ glyph[lscript] t,i -λ t is positive. We look at the cases where A t = i and A t = i independently.

Case A t = i :

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

This is a quadratic function in π t,i with minimizer η 2 t (1 -2 γη 2 t ) 2 < ( ξ t + γ ) 2 η 2 t , hence the value is lower bounded by setting π t,i to ( ξ t + γ ) 2 η 2 t

glyph[negationslash]

Case A t = i :

This is a concave function (in π t,i ) so the minimizer is at either at π t,i = ( ξ t + γ ) 2 η 2 t or at π t,i = 1 . For the latter, we have have π t +1 ,i > 1 2 , so the minimum is obtained for the first case.

Lemma 12 (Lemma 6) . For any time t such that π t,i > ( ξ t + γ ) 2 η 2 t , it holds

Proof of Lemma 6.

We now show that the requirements of Lemma 5 are satisfied with the tuning of Theorem 3. With η t = 1 √ K +26 t , we have c t = ( K +26 t -√ ( K +26 t )( K +26 t -26) ) > 2 , which is monotonically decreasing. c t > 2 and γ = 13 2 ensures that η t = 1 √ K +26 t < 2 √ K ( c t + γ ) 2 . Further we have

Proof of Lemma 7. Using η 0 = ∞ and D TS ( u, Π T +1 ) ≥ 0

glyph[negationslash]

where the last inequality follows from ∑ K u i ≥ 1 π t,i glyph[star] .

D.3 Self-bounding trick

We now quickly describe how to apply the self-bounding trick from Zimmert and Seldin [2021]. Assume that we have a regret bound of the form glyph[negationslash]

glyph[negationslash]

for some positive a and b . The above inequality implies glyph[negationslash]

glyph[negationslash]

For a fixed i the term ( a √ t -∆ i 3 ) ≤ 0 if t ≥ 9 a 2 ∆ 2 i and so the maximum regret from

Further the term √ π t,i ( b √ t -∆ i √ π t,i 3 ) ≤ 2 b 2 t ∆ i for t ≥ 4 b 2 ∆ 2 i . This implies

Combining the two bounds we have glyph[negationslash]

E TS-Prod and stabilized OMD

The TS-Prod update is a linearization of the FTRL update as explained in Section 5.2. In this section we show a regret bound for the TS-Prod algorithm by linearizing the OMD update as we did for WSU-UX and LB-Prod. This comes with its own set of challenges. First, we believe that a decreasing step-size in the OMD update is important for achieving optimal regret bounds in the stochastic setting. Second, the vanilla OMD update with decreasing step-size might incur linear regret in the adversarial setting. This second issue is resolved by the stabilized OMD algorithms proposed by Fang et al. [2022]. TS-Prod turns out to be equivalent to the dual stabilized OMD algorithm of Fang et al. [2022]

glyph[negationslash]

and hence will inherit the adversarial regret guarantees. In the rest of the section we sketch the regret analysis for the stochastic setting and how we can reduce the regret analysis in the adversarial setting to that of Fang et al. [2022].

Stabilization as introduced by Fang et al. [2022] is the process of mixing the gradient mapping, ∇ F ( π t ) , of the current iterate with the gradient mapping of the first iterate, ∇ F ( π 1 ) , in the mirror descent update in the dual space, where F ( x ) = -∑ K i =1 2 √ x i . This mixing turns out to be equivalent to the negative biasing of losses in Equation TS-Prod by the ξ t dependent terms.

The regret analysis begins by defining the perturbations { glyph[epsilon1] t,i } t ∈ [ T ] ,i ∈ [ K ] so that

that is glyph[epsilon1] t,i makes the update in Equation TS-Prod equivalent to the 1 / 2 -Tsallis mirror descent update. We note that the glyph[epsilon1] t,i is only defined to assist with the regret analysis and it never needs to be computed for the actual update. The perturbations, glyph[epsilon1] t,i , are well controlled as we show next.

Lemma 13. For every t ∈ [ T ] , i ∈ [ K ] , there exists glyph[epsilon1] t,i such that glyph[epsilon1] t,i ≤ 4 η t √ π t,i ˆ L 2 t,i for all i and glyph[lscript] t,i .

Lemma 13 allows us to proceed with the analysis for the stochastic and adversarial cases by using the standard regret decomposition into a penalty and stability terms. In the stochastic case we can bound the two terms in the following way

Lemma 14. For stochastic losses the penalty term is bounded in expectation by glyph[negationslash]

glyph[negationslash]

Lemma 15. For stochastic losses the stability term is bounded by

The stochastic regret bound proof can now be completed by a careful self-bounding argument.

In the adversarial case we reduce the regret bound to that of Fang et al. [2022] in the following way. Let Φ = F + I ∆ K -1 be potential defined by mixing the 1 / 2 -Tsallis potential together with the indicator function for the probability simplex. The update of Algorithm 2 (Dual Stabilized OMD) can then be written as

It turns out that this update is equivalent to the OMD update with respect to ˆ L t,i in Equation 4. This allows us to use the regret bound in Theorem 3 [Fang et al., 2022]. Overall the regret of the perturbed OMDversion is bounded as follows.

Theorem 4. The regret of the algorithm defined by the update in Equation 4 is bounded by glyph[negationslash]

in the stochastic case, where ∆ min is the smallest gap between the expected losses. Further the regret in the adversarial setting is bounded by O ( √ KT ) .

E.1 Proof of Theorem 4

Proof of Lemma 13. We work under the following assumption which is satisfied with the choice of η t and γ by Lemma 5. Further, we are going to work with the following slight modification of the

losses ˆ glyph[lscript] t in the update of TS-Prod:

We quickly check that the variance with this definition of ˆ glyph[lscript] t is bounded by Lemma 10

Assumption 1. Assume that for all t ∈ [ T ] , i ∈ [ K ] it holds that | η t √ π t,i ˆ L t,i | ≤ 1 6 .

Lemma 13. First for non-negative glyph[lscript] we show that glyph[epsilon1] ∈ [0 , glyph[lscript] ] . This follows by observing that 1 / (1 + 3 / 2 x ) 2 ≤ 1 -2 x for x ∈ [0 , 1 / 6] and 1 / (1 + x ) 2 ≥ 1 -2 x for x ≥ 0 so by the Intermediate Value theorem there exists an glyph[epsilon1] ∈ [0 , glyph[lscript] ] such that equality is obtained. Next, for glyph[lscript] < 0 for glyph[epsilon1] = 0 we have 1 / (1 -x ) 2 ≥ 1 + 2 x for x ≥ 0 and for glyph[epsilon1] = glyph[lscript]/ 2 we have 1 / (1 -x/ 2) 2 ≤ 1 -2 x for x ∈ [0 , 1 / 4] and so we have glyph[epsilon1] = [ -| glyph[lscript] | 2 , | glyph[lscript] | 2 ] .

For the second part of the lemma using the Taylor expansion around 0 of 1 / (1 + x ) 2 implies that

Stochastic bound. Let D TS t ( u, w ) = 1 η t D TS ( u, v ) and let

where λ t = glyph[lscript] t,A t . We note that π t +1 is now the projection of ˜ π t +1 onto the simplex. Further by the 3-point rule for Bregman divergence we have that

Penalty term.

Lemma 16 (Lemma 14) . For stochastic losses the penalty term is bounded as follows glyph[negationslash]

and so

Proof. for u = e i ∗ :

From the update in Equation 4 we have

which implies

First we bound (1 - √ π t +1 ,i ∗ ) 2 :

glyph[negationslash]

In the stochastic case WLOG we can take E [ glyph[lscript] t,i ∗ ] = 0 , ∀ t ∈ [ T ] . We first control ˆ L t,i ∗ . We have

The first term, glyph[lscript] t,i ∗ I ( A t = i ∗ ) π t,i ∗ is 0 in expectation. The second term, -η t γ (1 -π t,i ∗ ) √ π t,i ∗ I ( A t = i ∗ ) π t,i ∗ , will be used to cancel out the contribution from the perturbation glyph[epsilon1] t,i ∗ . The third term is there to help with the adversarial setting analysis. Next, we decompose the fourth term as

The first part of the above has a non-positive contribution to ˆ L t,i ∗ in expectation. The only nonnegative contribution now comes from

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

and so we bound

Next we are going to bound E [ glyph[epsilon1] t,i ∗ ] using Lemma 10 together with Lemma 13:

This term is exactly canceled out by η t γ (1 -π t,i ∗ ) √ π t,i ∗ as γ = 13 2 . For the final bound we have glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

Stability term. Recall that the stability term is D TS t ( π t , ˜ π t +1 ) . This term is bounded in a standard way. We proceed to do so as follows for any t ≥ 4 √ K :

Lemma 17 (Lemma 15) . For stochastic losses the stability term is bounded as follows

where D TS t ( u, v ) = 1 η t D TS ( u, v ) .

glyph[negationslash]

Proof. We have the following

where for the second to last inequality we only need to check η t +1 √ π t,i ( glyph[epsilon1] t,i -λ t ) ≥ -1 2 . We have η t +1 √ π t,i λ t ≥ -1 √ t and Lemma 13 implies

We bound E [ π 3 / 2 t,i ( ˆ glyph[lscript] t,i + glyph[epsilon1] t,i -λ t ) 2 ] ≤ 2 E [ π 3 / 2 t,i glyph[epsilon1] 2 t,i ] + 2 E [ η 2 t +1 π 3 / 2 t,i ( ˆ glyph[lscript] t,i -λ t ) 2 ] . For the first term we have

For the second term we use Lemma 10 to get E [ η 2 t +1 π 3 / 2 t,i ( ˆ glyph[lscript] t,i -λ t ) 2 ] ≤ 13 2 E [ η 2 t +1 √ π t,i (1 -π t,i )] .

Self-bounding the regret for stochastic losses.

Theorem 4, stochastic losses. Combining the bound in Lemma 14 and Lemma 15 together with the adversarial bound we have that the total regret is bounded as follows glyph[negationslash]

In the above we bound the lower order term from the stability as ∑ T t =1 ∑ K i =1 γ 2 t √ π t,i = O ( K 3 / 2 ) and decompose the regret into four parts. The first and second line correspond to the two terms from the penalty bound. Each of the two lines are decomposed into two terms. The first term is the result of the self-bounding trick and the second term is the additional regret for the initial number of rounds before the self-bounding trick can be applied.

We repeatedly use the following inequality 2 a √ x -bx ≤ a 2 b , which holds for a, b ≥ 0 . For the first line of the decomposition we take T 0 = 8 K log 2 ( K/ ∆ min ) ∆ min , where ∆ min is the smallest non-zero expected loss. We note that

glyph[negationslash]

for any ∆ min ≤ 1 2024 . If ∆ min > 1 2024 , we take T 0 = 8 K log 2 (2024) ∆ min . The above implies that √ π t,i ( 1 √ t - √ π t,i ( ∆ i -√ K log( t ) √ t )) ≤ 2 t ∆ i and further ∑ i = i ∗ ∑ T 0 -1 t =1 π t,i √ K log( t ) √ t = O ( K log 2 (1 / ∆ min ) ∆ min ) . The final regret bound is glyph[negationslash]

Adversarial losses. We now present the argument for the regret bound in the adversarial setting.

Theorem 4, adversarial losses. Let ˜ glyph[lscript] t,i = ( glyph[lscript] t,i -η t γ (1 -π t,i ) √ π t,i ) I ( A t = i ) π t,i and χ t = η t η t -1 We recall the update for Algorithm 2 in Fang et al. [2022]

where Φ is the 1 / 2 -Tsallis potential plus the indicator function for the probability simplex ∆ K -1 . Since π 1 is uniform the second step of the update is equivalent to ˆ y t +1 = χ t ˆ w t +1 . Re-writing the first step of the update we have

Since ∇ Φ ∗ is invariant under constant vector perturbations we finally have

And so the update in Algorithm 2 of Fang et al. [2022] is equivalent to the perturbed OMD update which we have shown enjoys an optimistic regret guarantee. The regret guarantee in the adversarial setting is now recovered from Theorem 3 in Fang et al. [2022]. In particular the theorem guarantees that the regret is bounded as

Every term in the sum is bounded in the same way as the stability terms, that is D TS t ( π t , ∇ F ∗ ( ∇ F ( π t ) -η t ( ˆ glyph[lscript] t + glyph[epsilon1] t ))) ≤ ∑ K i =1 π 3 / 2 t,i ( ˆ glyph[lscript] 2 t,i + glyph[epsilon1] t,i ) . We can now use the bound in the proof of Lemma 15 to complete proof of the adversarial bound and the proof of Theorem 4.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All results are formalized in Theorems which we proof in the supplementary material additionally to selective proof outlines in the main body.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [NA]

Justification: This is purely theoretical work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are stated in the Theorem statements and the proof can be found in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: No experiments

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: No experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: No experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: No experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: No experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Theoretical work without direct societal impact.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Theoretical work without societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Theoretical work without experiments or data.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Theoretical work without existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Theoretical work without new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Theoretical work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Theoretical work.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first considers incentive-compatible online learning problem, which is first considered in [Freeman et al., 2020]. Specifically, the original WSU-UX algorithm only achieves $O(T^{2/3})$ regret shown in [Freeman et al., 2020]. By injecting a bias in the loss estimator constuction to the original WSU-UX algorithm, the authors achieve $O(\\sqrt{KT\\log K})$ regret for this problem. The added biases can be viewed as certain second-order approximation of the orginal exponential weight algorithm. Besides this algorithm, the authors also propose another algorithm which is still incentive-compatible and interestingly does not use the inverse propensity weighting. The authors show that the LB-Prod algorithm is similar to FTRL with log-barrier regularizer and also achieves $O(\\sqrt{KT\\log T})$ optimal regret. The authors also consider the Tsallis entropy variant of this Prod-type algorithm and achieve best-of-both-world guarantee for multi-armed bandits in adversarial and stochastic environments.\n\nSTRENGTHS:\n- This problem resolves an open problem proposed in [Freeman et al., 2020], achieving $O(\\sqrt{T})$-type regret guarantee for incentive-compatible online learning with bandit feedback. The proposed two Prod-type algorithms in the bandit feedback setting are very interesting to me. The relationship between LB-Prod and FTRL with log-barrier regularizer shown in the analysis is also new to me.\n\n- The proposed Tsallis variant of Prod algorithm is also interesting since most previous BOBW algorithm relies on the FTRL framework.\n\nWEAKNESSES:\nI do not find main weakness of this paper but I have some concerns on the results and some configurations of the algorithms.\n\n- While the authors argue that the added bias in the first algorithm is related to the second-order approximation of the exponential weight algorithm, I still have difficulty understanding why the bias is of this form.\n\n- In Section \"perturbation analysis\", the authors argue that there is an equivalence between LB-Prod and LB-FTRL with a shifted loss. I wonder whether the standard OMD analysis also works? This can make the relationship between LB-Prod and LB-FTRL clearer.\n\n- The BOBW regret for the stochastic environment is a bit away from the optimal due to the additional $K/\\Delta_{\\min}$ lower order term.\n\nQUESTIONS:\n- Can the authors explain more on what difficulties are if one uses OMD with Logbarrier and perturbation analysis to analyze LB-Prod?\n\n- Whether there is also an inverse-propensity-weighting-free version of Tsallis-Prod that achieves BOBW for adversarial MAB?\n\n- Can the authors explain more about how the added bias in the modification of WSU-UX algorithm is related to the second-order approximation of MWU? Does this similarity come from the algorithm dynamic or the analysis perspective?\n\n- Can the stochastic regret bound for Tsallis-Prod be improved to optimal?\n\nLIMITATIONS:\nSee \"weakness\" and \"questions\".", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper revisits the simple PROD-type algorithms, originally introduced by Cesa-Bianchi et al. in 2007 for online learning under full feedback, and does an excellent job in doing so. Specifically, the authors demonstrate that some version of these algorithms can achieve optimality even for the $K$-armed bandit problem. They show that three different types of prod algorithms can achieve optimality. Interestingly, one of these algorithms can be seen as the first multi-armed bandit algorithm that solves the problem and does not rely on importance weighting. Notably, one of these algorithms also enjoys best-of-both-worlds guarantees. As another significant corollary of their work, the paper refutes a conjecture by Freeman et al. (2020) about incentive-compatible multi-armed bandits being intrinsically more difficult than regular multi-armed bandits.\n\nSTRENGTHS:\n1) The paper provides valuable insights about how to generate PROD-type algorithms as first-order OMD approximation with biased losses of classic multi-armed bandits algorithms.\n2) The paper provides the first importance-weighting free algorithm to solve the multi-armed bandits problem.\n3) The paper solves (negatively) a conjecture about incentive-compatible multi-armed bandits being more difficult than ordinary multi-armed bandits.\n4) The paper provides a PROD-type algorithm that enjoys BOBW guarantees.\n\nWEAKNESSES:\nThe presentation targets a specialized audience, assuming the reader is well-versed in template proofs of OMD (Online Mirror Descent) and FTRL (Follow-The-Regularized-Leader) algorithms, as well as techniques such as \"change of measure\".\nTo enhance accessibility for less experienced readers, consider adding dedicated appendices or providing precise references to relevant literature where these templates/definitions/techniques can be found. These additions can help guide readers through the nuances of the paper, broadening the potential audience of the paper, which I believe deserves to be broad.\n\nQUESTIONS:\nDo you believe that your techniques can be applied to other problems, such as contextual/linear/combinatorial bandits, or to feedback graph and partial monitoring? Do you foresee any challenges in adapting these techniques beyond the domain of classic multi-armed bandits?\n\nLIMITATIONS:\nI believe that this category does not apply to a paper which is entirely theoretical.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAdversarial bandits have typically been solved through techniques such as online mirror descent or variants of multiplicative weight updates, but these might rely on some notion of importance-based weighting. To develop solutions free of such importance-weighting, the authors propose using the Prod algorithm, which serves as a simpler approximation of online mirror descent. Algorithms based on Prod can then be combined with Tsalis regularization to get a best-of-both-worlds guarantee, in both the adversarial and stochastic regimes.\n\nSTRENGTHS:\n1. **Paper resolves open question** - As noted in the discussion in Section 6, the authors resolve an open question related to incentive compatibility and bandits posed by prior work. By resolving such a conjecture, they're able to provide a bridge between full-information and bandit settings. \n2. **Algorithm provides nice best-of-both-world guarantee** - Section 5 of the paper introduces an application of the Prod-based methodology using the Tsallis regularizer, and shows how this can achieve a best-of-both-worlds guarantee. While the connection with the previous sections, and the overall story of the paper, isn't super clear, the bounds proven in Section 5 provide some nice properties when using Prod.\n\nWEAKNESSES:\n1. **Unclear rationale behind using Prod** - Throughout the paper, the Prod algorithm is introduced and detailed, yet it is unclear why such an algorithm is necessary and the types of benefits brought upon by using such an algorithm. Early in the paper, the authors suggest that the simplicity of their update rules provides motivations for using Prod, but this motivation is not expanded upon, making it difficult to understand the motivation of the paper. \n2. **Unclear why importance weighting is undesirable** - Another rationale pointed out throughout the paper is that Prod can eliminate the need for performing importance weighting. However, it's unclear why such a property is desirable. The paper never details why we should eliminate importance weighting, and what the benefits are for algorithms that lack this. \n3. **Connection between Section 4 and 5 is unclear** - Section 4 of the paper details how Prod leads to a lack of importance weighting, while Section 5 details the best-of-both-worlds property with Prod. While both sections are interesting on their own, it is unclear how they each contribute to the main idea of the story. From my understanding, it seems to be the lack of a need to use importance weighting. In this case, it is not clear how Section 5 contributes to that story, and how Section 5 builds on the ideas from Section 4.\n\nQUESTIONS:\n1. What is the benefit of using Prod/what is the rationale for studying this problem?\n\nLIMITATIONS:\nThe paper lacks any limitations section. The authors write that the paper lacks a limitations section because it is \"purely theoretical work\", though I believe that limitations in the analysis or assumptions made should be pointed out even for theoretical work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents novel algorithms for the multi-armed bandit (MAB) problem that are shown to achieve optimal regret bounds even without relying on importance weighting. The authors introduce variants of the well-known Prod algorithm that are effective for both adversarial and stochastic settings, presenting significant improvements over existing state-of-the-art methods. Main contributions include disproving the conjecture that Prod-like algorithms are sub-optimal for bandits (that is, bandit feedback is shown not to be much harder than full information for algorithms in the Prod family), introducing a variant with nearly optimal regret, and achieving best-of-both-worlds guarantees with logarithmic regret in stochastic regimes.\n\nSTRENGTHS:\nThe main results of achieving optimal regret bounds with variants of Prod and even optimal best-of-both-worlds guarantees are quite surprising and interesting. Even if the update rule of the proposed algorithm is extremely simple, this is shown to have a controlled effect on the regret via careful corrections.\nAdditionally, the paper is nicely structured and smoothly introduces the variants of Prod by providing intuitive reasoning behind their design.\n\nThe techniques adopted, together with the intuitions and the analogies provided, are also nontrivial and require a novel analysis template that has the potential to lead to simple algorithms with near-optimal guarantees for other online learning problems.\n\nWEAKNESSES:\nI have no major weaknesses to report. On a side note, having more details on the setting of incentive-compatible online learning would have been better, as it is a central motivation for this work. For instance, even just explicitly defining what a “proper scoring rule” is and more technical motivations as to why Prod-like algorithms are important in this setting.\n\nQUESTIONS:\n- For the stochastic regret guarantee of TS-Prod, do you require that the optimal arm is unique? If so, please be explicit about this assumption.\n- At lines 155-159, you state that an overcorrection is necessary. However, a correction term of order $\\\\eta/\\\\tilde{\\\\pi}\\_{t,i}$ instead of $\\\\eta\\\\ell\\_{t,i} / \\\\tilde{\\\\pi}\\_{t,i}$ could be one of the main factors that are preventing the achievement of first-order and second-order bounds. How much do you believe your current techniques for designing Prod-like algorithms prohibit achieving these guarantees?\n\nMinor comments/typos:\n- Line 7: “best-of-both-worlds” instead of “best-both-worlds”\n- Table 1: the first column is simply the negative entropy, and calling it “KL divergence” too might be confusing\n- Math display below line 130: the second equality should be an inequality if “Reg” is just an upper bound on the regret with the biased losses\n- Line 172: for example, $T > K \\\\log K$ should suffice in order to satisfy the condition $T > (K/2) \\\\log T$ (the latter is not an explicit condition on $T$).\n- Line 180: “be” instead of “be be”\n- Line 211: “poses” instead of “posses”\n- Line 214: “a fixed” instead of “a a fixed”\n- Line 227: in “intermediate potential between KL and Logbarrier”, negative entropy is the potential function (whose Bregman divergence is the KL)\n- Math display below line 360: $\\\\pi\\_{t,i}^2$ between parentheses after the first equality should be $\\\\pi\\_{t,i}$\n\nLIMITATIONS:\nThe authors addressed potential limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the multi-armed bandit (MAB) problem. The main focus is on the Prod algorithm, which is fundamentally considered to be sub-optimal for MAB settings. The authors challenge this conjecture by leveraging Prod's interpretation as a first-order Online Mirror Descent (OMD) approximation. They make following main contributions: first, they present variants of Prod that achieve optimal regret bounds for adversarial MABs. Second, they propose an algorithm that achieves best-of-both-worlds guarantees, showing logarithmic regret in the stochastic bandit setting. The authors emphasize the simplicity and efficiency of their approach, using arithmetic update rules instead of solving complex optimization problems. The paper systematically develops these ideas, starting from the problem definition and related work, moving through the theoretical analysis of the proposed modifications, and proof of their performance guarantees.\n\nSTRENGTHS:\nThe paper has following strengths:\n\n- The paper is well written and easy to understand\n- Authors provide a very good overview of the current literature related to the work presented in this paper\n- The paper challenges and disproves a long-standing conjecture about the sub-optimality of Prod in the context of MAB\n- Theoretical results appear to be correct\n\nWEAKNESSES:\n- One main weakness of the paper is that authors do not provide any experimental results. I highly encourage authors to provide experimental results to illustrate their theoretical claims\n\n- Another main weakness is that authors do not provide any comparison (discussion or results) with other algorithms that achieve optimal regret guarantees in similar bandit settings\n\n- Authors do not provide any discussion on real world applications of their algorithms\n\n- The focus of the paper is heavily theoretical, which might reduce its accessibility and usability for practitioners looking for direct application insights.\n\nQUESTIONS:\nCan you provide experimental results to illustrate the theoretical claims?\n\nHow does your algorithms compare with other algorithms in similar settings?\n\nLIMITATIONS:\nWhile the theoretical contributions are strong, the lack of empirical validation means that the practical effectiveness of the algorithms in environments remains to be seen.\n\nThe scope of the paper is restricted to multi-armed bandits, and the paper does not discuss the potential application of the proposed methods to other types of online learning problems.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}