Title: Taming Heavy-Tailed Losses in Adversarial Bandits and the Best-of-Both-Worlds Setting
Venue: NeurIPS 2024
Accepted: yes
Keywords: bandits

regret minimization

the best of both worlds

heavy tails

online learning
Abstract: In this paper, we study the multi-armed bandit problem in the best-of-both-worlds (BOBW) setting with heavy-tailed losses, where the losses can be negative and unbounded but have $(1+v)$-th raw moments bounded by $u^{1+v}$ for some known $u>0$ and $v\in(0,1]$. Specifically, we consider the BOBW setting where the underlying environment could be either (oblivious) adversarial (i.e., the loss distribution can change arbitrarily over time) or stochastic (i.e., the loss distribution is fixed over time) and is unknown to the decision-maker a prior, and propose an algorithm that achieves a $T^{\frac{1}{1+v}}$-type worst-case (pseudo-)regret in the adversarial regime and a $\log T$-type gap-dependent regret in the stochastic regime, where $T$ is the time horizon. Compared to the state-of-the-art results, our algorithm offers stronger \emph{high-probability} regret guarantees rather than expected regret guarantees, and more importantly, relaxes a strong technical assumption on the loss distribution. This assumption is needed even for the weaker expected regret obtained in the literature and is generally hard to verify in practice. As a byproduct, relaxing this assumption leads to the first near-optimal regret result for heavy-tailed bandits with Huber contamination in the adversarial regime, in contrast to all previous works focused on the (easier) stochastic regime. Our result also implies a high-probability BOBW regret guarantee when the bounded true losses are protected with pure Local Differential Privacy (LDP), while the existing work ensures the (weaker) \emph{approximate} LDP with the regret bounds in expectation only.

Taming Heavy-Tailed Losses in Adversarial Bandits and the Best-of-Both-Worlds Setting

Abstract

In this paper, we study the multi-armed bandits problem in the best-of-both-worlds (BOBW) setting with heavy-tailed losses, where the losses can be negative and unbounded but have (1 + v ) -th raw moments bounded by u 1+ v for some known u > 0 and v ∈ (0 , 1] . Specifically, we consider the BOBW setting where the underlying environment can be either (oblivious) adversarial (i.e., the loss distribution can change arbitrarily over time) or stochastic (i.e., the loss distribution is fixed over time), which is unknown to the decision-maker a prior. We propose an algorithm and prove that it achieves a T 1 1+ v -type worst-case (pseudo-)regret in the adversarial regime and a log T -type gap-dependent regret in the stochastic regime, where T is the time horizon. Compared to the state-of-the-art results, our algorithm offers stronger high-probability regret guarantees (vs. expected regret guarantees), and more importantly, relaxes a strong technical assumption on the loss distribution, which is generally hard to verify in practice. As a byproduct, relaxing this assumption leads to the first near-optimal regret result for heavy-tailed bandits with Huber contamination in the adversarial regime (vs. the easier stochastic regime studied in all previous works). Our result also implies a high-probability BOBW regret guarantee when the bounded true losses are protected with pure Local Differential Privacy (LDP), while the existing work ensures the (weaker) approximate LDP with the regret bounds in expectation only.

1 Introduction

Consider the multi-armed bandits (MAB) problem (Auer et al., 2002a,b), which is a useful framework for sequential decision-making under uncertainty and can be formulated as repeated interactions between the environment and a (learning) algorithm. In each of the total T rounds indexed by t , the algorithm plays an action a t from a fixed set of K actions (assuming K ⩽ T ). Simultaneously, the environment determines the losses of all actions ℓ t ∈ R K . The algorithm observes and suffers the loss associated with a t (denoted by ℓ t,a t ). The goal of the algorithm is to minimize the cumulative loss over T rounds, or equivalently, to minimize the regret , defined as the difference between its cumulative loss and that incurred by playing the best-fixed action (in hindsight) all the time. Without observing the losses of the other actions, the algorithm must 'infer' the optimal action through interactions on the fly, facing the well-known trade-off between exploration and exploitation.

Depending on how the losses are determined, MAB problem is typically studied in two regimes: 1) the stochastic regime, where the loss of each action is drawn from a fixed (but unknown) distribution over time; 2) the adversarial regime, where the losses can be arbitrary (within some known class). Typically, the losses are assumed to have a support on a bounded interval (e.g., [0 , 1] ), and the fundamental limits have been well understood: 1) in the stochastic regime, a number of optimism-

Bo Ji Virginia Tech

√

based algorithms achieve the Θ( KT ) 1 worst-case (pseudo-)regret and Θ( ∑ i :∆ i > 0 (1 / ∆ i ) log T ) gap-dependent regret, where ∆ i (formally defined in Section 2) is the sub-optimality gap between action i and the optimal action (Lai & Robbins, 1985; Auer et al., 2002a; Agrawal & Goyal, 2017); 2) in the adversarial regime, the Θ( √ KT ) worst-case regret can be achieved by classic Online Learning algorithms, such as Follow-the-Regularized-Leader (FTRL), Online-Mirror-Descent (OMD), and Follow-the-Perturbed-Leader (FTPL) (Audibert & Bubeck, 2009; Lee et al., 2024).

Despite these progresses, the optimal algorithms in the two regimes fall into different frameworks (i.e., optimism-based algorithms for the stochastic regime vs. Online Learning algorithms for the adversarial regime). Moreover, while the former ones enjoy a logarithmic regret (i.e., O (log T ) ) in the stochastic regime, even their ˜ O ( √ KT ) worst-case guarantees no longer hold when the environment deviates from the stochastic regime with empirical evidence provided in Zimmert & Seldin (2021). On the other hand, while Online Learning algorithms always preserve worst-case optimality, they could be too 'conservative' to enjoy logarithmic regrets in stochastic environments.

These performance discrepancies motivated the study of the Best-of-Both-Worlds (BOBW) setting. That is, one single algorithm preserves the optimal worst-case regret in the adversarial regime and adapts to the stochastic regime with a logarithmic regret, without knowing the type of the regime in advance. Bubeck & Slivkins (2012) initiated the study by proposing a detect-switch framework, which preserves the optimal ˜ O ( √ KT ) regret in the adversarial regime and enjoys O ( (log T ) 2 K/ ∆ ) regret in the stochastic regime, where ∆ := min i :∆ i > 0 ∆ i is the smallest sub-optimality gap. Under this framework, Auer & Chiang (2016) improved the gap-dependent term from K/ ∆ to ∑ i :∆ i > 0 (1 / ∆ i ) .

Another line of work showed that without explicit detect-switch, OMD, originally designed for the adversarial regime, can automatically adapt to stochastic environments with a logarithmic regret (Wei &Luo, 2018; Zimmert & Seldin, 2021). In particular, Zimmert & Seldin (2021) achieved optimal regrets in the BOBW setting. Following these works, the power of Online Learning algorithms towards BOBW has been extended to various setups (Ito, 2021; Ito et al., 2022; Kong et al., 2023; Ito &Takemura, 2023; Dann et al., 2023; Jin et al., 2024; Lee et al., 2024; Tsuchiya et al., 2024).

While the aforementioned works require bounded losses, real-world data from application domains such as finance (Cont, 2001) and imaging (Hamza & Krim, 2001) often exhibits a heavy-tailed distribution. Intuitively, heavy-tailed losses make learning problems harder (compared to bounded losses) as they are 'noisier' and 'less informative' (Zhang et al., 2020). When losses are unbounded but have (1 + v ) -th raw moment bounded by u 1+ v for some known u > 0 and v ∈ (0 , 1] , Bubeck et al. (2013) showed ˜ Θ( uK v 1+ v T 1 1+ v ) worst-case regret and Θ (∑ i :∆ i > 0 ( u 1+1 /v / ∆ i ) 1 /v log T ) gap-dependent regret 2 in the stochastic regime by integrating robust mean estimators (e.g., trimmed mean and median-of-means) into optimism-based algorithms.

To address heavy-tailed losses in the adversarial regime and BOBW setting, Huang et al. (2022) showed that with calibrated adaptive loss trimming thresholds, FTRL with Tsallis entropy regularizer (Audibert & Bubeck, 2009) enjoys the optimal BOBW expected regrets under the 'truncated nonnegative losses' assumption (see Assumption 2). Without this strong assumption, it is unclear whether the near-optimal worst-case regret can still be achieved in the adversarial regime, let alone the BOBW setting. The key technical challenge here is that heavy-tailed losses can be both negative and unbounded, which is known to break the regret guarantees of the Online Learning algorithms. Extensive discussions and insights for our solution are provided in Section 3.

Given this challenge, the gap raises an interesting question: In heavy-tailed MAB, are there any fundamental barriers to the worst-case optimality in the adversarial regime and the BOBW guarantees?

We offer a positive answer to the above question, which implies that there is no such barrier. Our main contributions are as follows:

In the adversarial regime, we propose an OMD-based algorithm achieving the (near-)optimal ˜ O ( uK v 1+ v T 1 1+ v ) pseudo-regret with high probability . Our approach relaxes the undesired 'truncated non-negative losses' assumption, which is needed in the state-of-the-art results

̸

̸

even for the weaker expected regret guarantee (Huang et al., 2022). Relaxing it also allows us to obtain the near-optimal worst-case guarantee against the Huber contamination, which, to our best knowledge, was only studied for the stochastic regime in the literature. This suggests broader implications of our approach.

On top of the above advance in the adversarial regime, by leveraging the detect-switch framework, we further extend the (near-)optimal regret guarantees to the BOBW setting. Specifically, our algorithm preserves the optimal ˜ O ( uK v 1+ v T 1 1+ v ) regret in the adversarial regime and enjoys O ( K ( u 1+1 /v / ∆) 1 /v log( K )(log T ) 4 ) gap-dependent logarithmic regret in the stochastic regime, both with high probability , which implies that there is no fundamental barrier to achieving the BOBW guarantees when the loss distributions are heavy-tailed. This result also immediately imply the first high-probability BOBW regret guarantees with pure Local Differential Privacy (LDP) protection on the true losses, while the existing result ensures the weaker approximate LDP protection with expected regret guarantees only.

Technique-wise, we leverage the inherent stronger stability of log-barrier to relax a strong technical assumption made in previous works and utilize the increasing-learning-rates trick (Lee et al., 2020) to obtain the stronger high-probability guarantee in the adversarial regime. Moreover, we adapt the detect-switch framework by Bubeck & Slivkins (2012), originally designed for BOBW in the bounded-loss case, to the heavy-tailed setup. The adaptation introduces non-trivial challenges in the analysis due to the history-dependent trimmed estimator. In particular, to obtain the desired concentration rate in the adversarial regime, the proof does not follow its existing counterpart in the stochastic regime. Beyond addressing these challenges, we identify a novel use (i.e., handling history-dependent trimming in martingale concentrations) of an adaptive variant of Freedman's inequality (originally proposed by Lee et al. (2020) and improved by Zimmert & Lattimore (2022) for high-probability regret in adversarial bandits), which may be of independent interest.

We refer the readers to Table 1 for a summary of the most relevant results, in which we also include adaptive results on the case when u, v are unknown. We also present a comprehensive discussion about related work, which is deferred to Appendix A due to the page limit.

̸

2 Problem Setup

In this section, we formally introduce the problem setup and define needed notations. To formulate heavy-tailed losses, we let ℓ t,i denote the loss of action i in round t , which is drawn from distribution P t,i satisfying the following assumption:

Assumption 1. The (1 + v ) -th (raw) moments of losses (which have potentially unbounded support in R ) are bounded by u 1+ v for some constants u > 0 and v ∈ (0 , 1] , i.e., E ℓ t,i ∼ P t,i [ | ℓ t,i | 1+ v ] ⩽ u 1+ v , ∀ t ∈ [ T ] , i ∈ [ K ] , where [ n ] denotes set { 1 , . . . , n } for any integer n ⩾ 1 .

In the heavy-tailed MAB problem, the (learning) algorithm and environment perform the following interactions repeatedly in round t = 1 , . . . , T :

The algorithm samples action a t from [ K ] via a t ∼ w t := ( w t, 1 , . . . , w t,K ) in the probability simplex Ω := { x ∈ [0 , 1] K ∣ ∣ ∑ K i =1 x ( i ) = 1 } , i.e., action i ∈ [ K ] is sampled with probability w t,i . The environment draws loss ℓ t,i ∼ P t,i for every action i ∈ [ K ] .

The algorithm observes ℓ t,a t only; the losses of all other actions are unrevealed.

The algorithm determines w t +1 based on the history ( w 1 , a 1 , ℓ 1 ,a 1 , . . . , w t , a t , ℓ t,a t ).

Remark 1 . All of our algorithms and their regret bounds allow moment order (1 + v ) ∈ (1 , 2] only. That is, one may not obtain any regret guarantee by running our algorithms with v > 1 . If the losses have higher-order moments ( v > 1 ), one can simply run our algorithms with v = 1 and obtaining the corresponding regret bounds (since bounded higher-order moments imply lower-order ones). Note that Bubeck et al. (2013) showed that for all v ⩾ 1 , the lower bounds in terms of both worst-case regret and gap-dependent regret are the same as the case of v = 1 .

We assume that heavy tail parameters u and v and time horizon T are known to the algorithm a priori. 3 The objective of the algorithm is to minimize the pseudo-regret R T , defined as

where µ t,i := E ℓ t,i ∼ P t,i [ ℓ t,i ] denotes the mean loss of action i ∈ [ K ] in round t ∈ [ T ] , and i ∗ ∈ argmin i ∈ [ K ] ∑ T t =1 µ t,i denotes any best-fixed action in hindsight.

Depending on how loss distributions are determined, we further define the following two regimes:

Stochastic regime: For every action i ∈ [ K ] , the loss distributions are identical in all rounds. That is, we have P 1 ,i = · · · = P T,i = P ( i ) , implying µ 1 ,i = · · · = µ T,i = µ ( i ) and i ∗ ∈ argmin i ∈ [ K ] µ ( i ) . We also define ∆ i := µ ( i ) -µ ( i ∗ ) and ∆ := min i :∆ i > 0 ∆ i .

(Oblivious) Adversarial regime: All loss distributions are chosen arbitrarily (by some adversary, with the full knowledge of the algorithm) before the interaction begins. Our regret definition and adversarial model are also considered in Huang et al. (2022).

Remark 2 . It is not hard to see that the stochastic regime is a special (and 'easy') case of the adversarial regime. There are two differences between our heavy-tailed setup and the bounded-loss setup in the adversarial bandits literature: 1) Typically in the (bounded-loss) adversarial regime, the losses are considered to be deterministic rather than randomized (and this difference also leads to some new challenges in the analysis when we adopt the detect-switch framework from the bounded case (Bubeck & Slivkins, 2012) as we will show later in Section 4.2), and 2) a stronger notion of regret, defined as R T := ∑ T t =1 ℓ t,a t -min i ∈ [ K ] ∑ T t =1 ℓ t,i , is considered; this is stronger than pseudo-regret since E [ R T ] ⩾ E [ R T ] . However, when adapted to the heavy-tailed case, it is natural to still consider randomized losses and pseudo-regret as in the stochastic regime: The 'easiness' of the regime boils down to how heavy-tailed distributions are chosen. Moreover, a low stronger regret depends not only on playing good actions (with low loss means), but also on the realization of potentially unbounded losses, which loses the standard meaning in evaluating the algorithm. Therefore, when switching from stochastic to adversarial regime in heavy-tailed bandits, we keep pseudo-regret as the metric. 4

Our goal is to design one single algorithm that can achieve the near-optimal ˜ O ( uK v 1+ v T 1 1+ v ) worstcase regret in the adversarial regime and enjoys log T -type regret when the regime is stochastic, without being informed of the regime type in advance.

A closely related work by Huang et al. (2022) studied the same BOBW setup (i.e., achieving BOBW guarantee when u, v are known). They proposed an FTRL-based algorithm with Tsallis entropy regularizer and carefully-chosen history-dependent trimming threshold for loss magnitude control and showed O (∑ i :∆ i > 0 ( u 1+1 /v / ∆ i ) 1 /v log T ) regret in the stochastic regime and O ( uK v 1+ v T 1 1+ v ) regret in the adversarial regime, both of which are in expectation and optimal. However, their regret guarantees rely heavily on a strong technical assumption:

Assumption 2 (Truncated non-negative losses (Huang et al., 2022)) . Given any fixed M > 0 , the loss distributions of the optimal action i ∗ satisfy E ℓ t,i ∗ ∼ P t,i ∗ [ ℓ t,i ∗ · I {| ℓ t,i ∗ | > M } ] ⩾ 0 , ∀ t ∈ [ T ] .

In the following sections, we first show that by resorting to the log-barrier regularizer, we obtain the near-optimal regret bound in the adversarial regime without Assumption 2 and naturally extend it to the stronger high-probability guarantee (Section 3). On top of that, we further adapt the detect-switch framework in Bubeck & Slivkins (2012) and obtain high-probability bounds in BOBW (Section 4).

Additional Notations. For any round t and action i , we let I t,i := I { a t = i } denote whether action i is pulled in round t and N t,i := ∑ t s =1 I s,i denote the number of times when action i is pulled before the end of round t . We use ̂ ℓ t,i := ℓ t,i I t,i I {| ℓ t,i | ⩽ M t,i } /w t,i to denote the (trimmed) IW estimate with respect to some threshold M t,i and ̂ µ t,i := ∑ t s =1 ℓ s,i I s,i I {| ℓ s,i | ⩽ B s,i } /N t,i to denote the (trimmed) empirical average with respect to some threshold B t,i . 5 We also use µ ′ t,i := E ℓ t,i ∼ P t,i [ ℓ t,i I {| ℓ t,i | ⩽ M t,i } ] to denote the mean of the trimmed loss and ̂ L t,i := ∑ t s =1 ̂ ℓ s,i to denote the cumulative IW estimate.

3 High-probability Near-optimal Regret in the Adversarial Regime

This section is dedicated to high-probability regret in the adversarial regime. We first present detailed discussions on why Assumption 2 is needed in Huang et al. (2022) and how we get rid of it (Section 3.1), followed by the description of our algorithm design and regret guarantee (Section 3.2).

3.1 Technical Challenges and Insights

The main technical challenge we encounter comes from the potentially-unbounded negative losses (even for regret bounds in expectation only). We illustrate this challenge using the previous work of Huang et al. (2022) as an example. Their algorithm is running standard FTRL with (1 + v ) -1 -Tsallis entropy regularizer over the trimmed loss estimate sequence ̂ ℓ 1 , . . . , ̂ ℓ T with respect to some trimming threshold ( M t,i ) t ∈ [ T ] ,i ∈ [ K ] to be introduced later in this subsection.

To bound the expected regret, Huang et al. (2022) first rewrite it as

where y i is the K -dim vector such that the i -th entry is one and all the others are zero.

The analyses begin with Part III. The desired upper bound on Part III holds only under the well-known 'stability condition' associated with Tsallis entropy (Jin et al., 2024, Lemma C.5.3):

where constant C ( u, v ) > 0 depends on u and v only and the learning rate η t,i is chosen as u -1 t -1 1+ v . While this condition is trivially satisfied when losses are non-negative, due to potentially-unbounded

Algorithm 1 OMDwith log-barrier and increasing learning rates for heavy-tailed MABs ( OMD-LB-HT )

1: Input: failure probability ζ , initial learning rate η , trimming threshold { M t,i } t ∈ [ T ] ,i ∈ [ K ] 2: Define: learning rate increase factor κ = e 1 / log T ; log-barrier regularizer ϕ t ( x ) = -∑ K i =1 log( x ( i )) /η t,i ; Bregman divergence D ϕ t ( x, x ′ ) = ∑ d i =1 ( x ( i ) /x ′ ( i ) -log( x ( i ) /x ′ ( i )) -1) /η t,i ; simplex truncation parameter λ = T -v 1+ v K -1 1+ v ; truncated probability simplex Ω ′ := { x ∈ Ω : x ( i ) ⩾ λ/K, ∀ i ∈ [ K ] } 3: Initialization: For every action i ∈ [ K ] , define w 1 ,i = 1 /K,ρ 1 ,i = 2 K,η 1 ,i = η 4: for t = 1 : T do 5: Take action a t sampled from w t and observe ℓ t,a t 6: Construct loss estimate ̂ ℓ t,i = I { a t = i } I {| ℓ t,i | ⩽ M t,i } ℓ t,i /w t,i , ∀ i ∈ [ K ] 7: Calculate w t +1 = argmin x ∈ Ω ′ ( ⟨ x, ̂ ℓ t ⟩ + D ψ t ( x, w t ) ) 8: for i ∈ [ K ] do 9: if 1 /w t +1 ,i > ρ t,i , then ρ t +1 ,i = 2 /w t +1 ,i , η t +1 ,i = η t,i κ 10: else ρ t +1 ,i = ρ t,i , η t +1 ,i = η t,i 11: end for 12: end for

negative losses, threshold M t,i here is chosen to be Θ(( t · w t,i ) 1 1+ v ) (in particular, to fully 'cancel' the ( w t,i ) -1 from ̂ ℓ t,i ). Otherwise, negative losses break this condition whenever w t,a t is very small.

While such a threshold suffices to bound Part I and Part III with worst-case optimality (and even in the BOBW setting), applying the analysis for Part I to Part II leads to an upper bound of form ∑ T t =1 ( w t,i ∗ ) -v 1+ v on Part II, which is potentially unbounded since w t,i ∗ can be very close to zero. However, with the help of Assumption 2, Part II itself is non-positive and hence can be ignored.

Summary. The key issue above is that, due to unbounded and negative losses, Eq. (3) results in a threshold M t,i which scales with ( w t,i ) 1 1+ v (in particular, for i = i ∗ ), rendering Part II hard to bound. Remark 3 . This issue may not be fixed by simply shifting all loss estimates to become positive and satisfy Eq. (3). Roughly speaking, the reason is that obtaining desired regret bounds relies heavily on the well-bounded (1 + v ) -th moment of the losses (before shifting). However, to ensure positive losses, the needed shift is too 'significant' and breaks the 'nice' moment conditions.

To handle this issue and relax Assumption 2, we resort to the log-barrier regularizer, which offers the standard regret guarantee with the following stability condition (Agarwal et al., 2017):

Importantly, this condition itself already provides a w t,i (rather than the previous ( w t,i ) 1 -1 1+ v in Eq. (3)) to 'cancel' the ( w t,i ) -1 from ̂ ℓ t,i , meaning that we do not need any additional w t,i contributed from threshold M t,i . As a result, we can choose a different M t,i that scales with K -1 1+ v rather than the previous ( w t,i ) 1 1+ v such that all three parts are bounded by ˜ O ( uK v 1+ v T 1 1+ v ) without Assumption 2 .

We further adopt the increasing-learning-rates trick (Lee et al., 2020) together with an adaptive variant of Freedman's inequality (stated in Lemma 12) to obtain the stronger high-probability regret.

3.2 Algorithm Overview and Regret Guarantee

We now discuss the algorithm design and present the full pseudo-code in Algorithm 1. At a high level, our algorithm is running standard OMD over loss sequence ̂ ℓ 1 , . . . , ̂ ℓ T with respect to some fixed threshold M t,i = u · ( T/K ) 1 1+ v . The key ingredients (to obtain high-probability regret) are the special learning rate schedule and probability simplex truncation (in OMD update) (Wei & Luo, 2018; Lee et al., 2020), which we briefly introduce below.

Increasing Learning Rates. Weuse vectors { ρ t } t ∈ [ T ] to keep track of smallest sampling probabilities throughout the interaction. To be more specific, for every action i , if w t +1 ,i is so small that 1 /w t +1 ,i > ρ t,i holds, we increase the learning rate by a factor of κ > 1 and set ρ t +1 ,i = 2 /w t +1 ,i (Line 9). Otherwise, we keep the learning rate unchanged and set ρ t +1 ,i = ρ t,i (Line 10). In the analysis, such

a schedule introduces some negative term in the upper bound, which cancels out a positive term that could potentially be very large due to the variance of loss estimates (Lee et al., 2020).

Probability Simplex Truncation. We perform the OMD update over the truncated probability simplex Ω ′ := { x ∈ Ω : x ( i ) ⩾ λ/K, ∀ i ∈ [ K ] } (where λ controls the degree of truncation), of which the purpose is to ensure that w t,i is always at least λ/K and hence to control the variance of the loss estimates. The value of λ here is T -v 1+ v K -1 1+ v adapted to the heavy-tailed case and differs from the original value 1 /T for the bounded-loss case in Lee et al. (2020).

The regret guarantee of Algorithm 1 is formally stated below with full proofs presented in Appendix B.

Theorem 1. In the adversarial regime, for any failure probability ζ , by choosing initial learning rate η = (40 M log( T ) log (8 KT/ζ )) -1 and trimming threshold M t,i = u · ( T/K ) 1 1+ v , Algorithm 1 ensures that with probability at least 1 -ζ , R T = O ( uK v 1+ v T 1 1+ v (log T ) 2 log( T/ζ )) . By further choosing ζ = 1 /T , Algorithm 1 ensures that E [ R T ] = O ( uK v 1+ v T 1 1+ v (log T ) 3 ) .

Remark 4 . Removing Assumption 2 is crucial to obtain the first and near-optimal worst-case regret in heavy-tailed MAB when the feedback could be contaminated by the Huber model in the adversarial regime , in contrast to all previous works that study the (easier) stochastic regime (Guan et al., 2020; Agrawal et al., 2024; Wu et al., 2024). We provide all details in Appendix D.

4 High-probability Regrets in the Best-of-Both-Worlds Setting

With Algorithm 1 achieving high-probability optimal regret in the adversarial regime, we further leverage the detect-switch framework proposed by Bubeck & Slivkins (2012) named SAO to achieve high-probability bounds in the BOBW setting. We first present the BOBW guarantee, followed by an algorithm overview and analysis sketch. The complete proofs are provided in Appendix C.

Theorem 2. In the adversarial regime, for any failure probability ζ , by choosing constant c 1 = 6 , Algorithm 2 ensures that with probability at least 1 -ζ ,

which (by further choosing ζ = 1 /T ) implies that E [ R T ] = O ( uK v 1+ v T 1 1+ v log( K )(log T ) 4 ) . In the stochastic regime, Algorithm 2 ensures that with probability at least 1 -ζ ,

which implies that E [ R T ] = O ( K log( K )( u 1+1 /v / ∆) 1 /v (log T ) 4 ) .

Remark 5 . High-probability bounds also are powerful tools to handle adaptive adversaries (who determine the current distribution P t,i based on past actions a 1 , . . . , a t -1 ). Following the literature (Audibert & Bubeck, 2010; Lee et al., 2020; Zimmert & Lattimore, 2022), based on our highprobability bounds against oblivious adversaries, one may further derive both high-probability and expected regret bounds against adaptive adversaries, which we leave as future investigation.

Remark 6 . This theorem immediately implies a (high-probability) BOBW regret guarantee when the losses are bounded (in [0 , 1] ) and protected with ε -Local Differential Privacy (LDP) via showing that the privatized losses (with Laplacian noise) have second moments bounded by O ( ε -1 ) (Agarwal & Singh, 2017; Tossou & Dimitrakakis, 2017; Zheng et al., 2020; Ren et al., 2020). To the best of our knowledge, this is the first result showing (high probability) BOBW regret guarantee with pure LDP protection, while the state-of-the-art result (Zheng et al., 2020) ensures (the weaker) approximate LDP protection, and only expected regret bounds are provided. Full details are given in Appendix E.

4.1 Algorithm Overview

Our algorithm design follows from the detect-switch framework by Bubeck & Slivkins (2012). The high-level idea is to keep performing statistical tests (to 'identify' the environment) and carefully maintain the sampling distribution over all arms. Once some certain test fails (which implies that the environment is unlikely stochastic), we switch to Algorithm 1 and run it over the remaining rounds.

In each round after playing action a t sampled from distribution w t , if any active action i ∈ A t satisfies the test in Eq. (5), it is deactivated, and we use τ i and q i to store the round and the sampling

Algorithm 2 SAO for heavy-tailed MABs ( SAO-HT )

1: Input: failure probability ζ ; constant c 1 ⩾ 6

(

t

) = 12

uK

v

1+

v

t

1

1+

v

(log(

β/ζ

))

3

3

v

v

+1

3: Initialization: Play each action once; for every i ∈ [ K ] , let w K +1 ,i = 1 K , τ i = T ; set A K = [ K ] .

4: for t = K +1 , . . . , T do

5: Sample and play action a t ∼ w t ; observe ℓ t,a t

6: for i ∈ A t -1 do

7: if

9:

10:

end if

end for

11: if any of the three conditions holds then run Algorithm 1 for the remaining rounds, let t sw = t , and let q i = w t,i , τ i = t for every i ∈ A t :

12:

13: end if

15:

end for

probability when it is deactivated, respectively (Line 8). After that, the algorithm performs tests in Eqs. (6)-(8) for environment identification. If any of them is satisfied, the procedure is terminated, and we instead run Algorithm 1 over the remaining rounds. We use t sw to denote the round when the algorithm switch happens. Otherwise, we update the distribution w t +1 for the next round as in Line 14. To make notations and analyses well-defined, we deactivate all remaining arms in A t sw when the algorithm switch happens.

4.2 Analysis Sketch

In this subsection, we present the key steps of the regret analysis in the proof of Theorem 2.

Before diving into the specific regime, we first derive a set of concentration results (good events). Informally, all of the following hold simultaneously with high probability in all rounds t ∈ [ T ] :

where Width ( t ) := 12 uK v 1+ v t 1 1+ v (log( β/ζ )) 3 v 3 v +1 . All analyses below are conditioned on these good events, and we omit 'with high probability' in the arguments.

Remark 7 . Our main non-trivial adaptation deviating from the case with bounded losses in Bubeck & Slivkins (2012) is the good event in Eq. (10) due to jointly randomized and heavy-tailed losses. In particular, we need the concentration result of trimmed mean ̂ µ t,i specified in Eq. (10) in both regimes . In the stochastic regime, it been shown in Bubeck et al. (2013). We need Eq. (10) in adversarial regime (while Bubeck & Slivkins (2012) does not) as losses are deterministic therein. However, with heavy tails, the proof of Eq. (10) in the adversarial regime does not follow straightforwardly from the stochastic regime. More technical details are presented later in Remark 8. Nonetheless, this matter is specific to the nature of the trimmed estimator we use, which essentially could be replaced with any other estimator, as long as the concentration rate is preserved. However, it is unclear whether other estimators can handle non-identical distributions.

4.2.1 Analysis Overview in the Stochastic Regime

Step 1: Showing that tests in Eqs. (6) -(8) are never satisfied. We show by good events that tests in Eqs. (6)-(8) are never satisfied, implying that we never switch to Algorithm 1.

Step 2: Building the connection between ∆ i and τ i . From tests in Eqs. (7) and (8), we can show that for every suboptimal action i with ∆ i > 0 , its sub-optimality gap ∆ i = ˜ O (( K/τ i ) v 1+ v ) . Intuitively, an action with a smaller sub-optimality gap stays active for a longer time.

Step 3: Bounding the total regret. By the definition of pseudo-regret, we now have

We complete the proof by showing ∑ i :∆ i > 0 q i ⩽ ∑ K i =1 (1 /i ) = O (log K ) .

4.2.2 Analysis Overview in the Adversarial Regime

In the adversarial regime, whenever we switch to Algorithm 1, it provides ˜ O ( u ( T -t sw ) 1 1+ v K v 1+ v ) (high-probability) regret guarantee for the remaining ( T -t sw ) rounds. Therefore, it suffices to show that the cumulative regret before the switch is ˜ O ( u ( t sw ) 1 1+ v K v 1+ v ) .

In our analysis, we trivially bound the regret in the single round t sw by 2 u , and it remains to show that the regret in the first ( t sw -1) rounds is ˜ O ( u ( t sw -1) 1 1+ v K v 1+ v ) , which is explained in the following.

Step 1: Regret decomposition. We first get an regret upper bound in terms of i ∗ t := argmin i ∈ [ K ] ∑ t s =1 µ s,i for t = t sw -1 :

and then we bound Parts A, B, and C, separately.

Step 2: Bounding Part A. We rewrite Part A as Part A = ( ∑ t sw -1 s =1 µ s,i I s,i N t sw -1 ,i -̂ µ t sw -1 ,i ) + ( ̂ µ t sw -1 ,i -̂ L t sw -1 ,i t sw -1 ) , where the first term on the right-hand side is O ( u ( log( β/ζ ) N t sw -1 ,i ) v 1+ v ) due to Eq. (10) and the second term is O ( u ( log( β/ζ ) N t sw -1 ,i ) v 1+ v + Width ( t sw -1) τ i -1 ) due to test in Eq. (6).

Remark 8 . Due to heavy tails, the losses are trimmed by some history-dependent threshold B t,i (which depends on the number of pulls N t,i ) for a rate-optimal concentration in good event (10). To

show this in the stochastic regime, one can treat the observed losses from one action as i.i.d. samples via the 'reward tape/table' argument (Slivkins, 2019) and apply Bernstein's inequality for every fixed N t,i (and set the uniform upper bound as B t,i associated with it) as in Bubeck et al. (2013). However, one cannot simply follow the same path in the adversarial regime, since the distributions are no longer identical , and has to follow the martingale-based analysis (Agarwal et al., 2021, Lemma 6.2).

Now one may readily see the issue: The desired uniform upper bound B t,i is determined on-the-fly , while the standard Freedman's inequality for martingales (e.g., Lemma 11) requires a fixed uniform upper bound. To close this gap, we again exploit an adaptive variant of Freedman's inequality by Zimmert & Lattimore (2022) (Lemma 12; which was originally proposed for a totally different use, namely, obtaining high-probability bounds in adversarial bandits), in which we can replace the fixed uniform upper bound with the largest realization , satisfying our need perfectly.

Step 3: Bounding Part B. By considering two disjoint cases (i.e., action i ∈ A t sw -1 or not), we can show Part B = O ( Width ( t sw -1) τ i -1 ) using the tests in Eqs. (5) and (7).

Step 4: Bounding Part C. The good event in Eq. (9) simply implies Part C = O ( Width ( t sw -1))

.

Step 5: Putting all pieces together. Combing Steps 1-4 and the good event in Eq. (11) yields

5 Conclusion

In this paper, we show that there is indeed no fundamental barrier to achieving the BOBW guarantee in heavy-tailed MAB by relaxing a strong, hard-to-verify technical assumption on the loss distributions of the optimal action needed for the state-of-the-art results. We further leverage the increasing-learningrates trick and the detect-switch framework to achieve the stronger high-probability guarantees. Our results also imply the first and near-optimal regret in the adversarial regime where the feedback could be contaminated in the Huber model, and the high-probability BOBW regret guarantee when losses are bounded and protected with pure LDP, while the state-of-the-art result only ensures the weaker approximate LDP protection with regret guarantees in expectation.

One follow-up question is whether the gap-dependent term K (∆) -1 /v can be improved to the refined ∑ i :∆ i > 0 (∆ i ) -1 /v , which is achieved only in the stochastic regime (Bubeck et al., 2013). Under the detect-switch framework, we tend to believe this is possible by adapting more sophisticated tests designed in Auer & Chiang (2016) for the bounded-loss case, where the gap dependency is improved from K/ ∆ to ∑ i :∆ i > 0 (1 / ∆) , although a higher computational complexity is expected.

It will also be interesting to understand whether canonical Online Learning algorithms (i.e., without explicit detection and switch) provably enjoy BOBW guarantees in heavy-tailed MAB (and if so, whether a refined gap dependency can be achieved). In other words, do heavy tails break the implicit adaption of Online Learning algorithms to the stochastic regime (in the worst case)? One promising direction is to still utilize log-barrier regularizer together with potentially more advance learning rate and/or trimming threshold design. Notably, one unique advantage of it over the detect-switch framework is that it typically directly extends the BOBW regret guarantee to the corrupted regime, which is an intermediate regime that smoothly extrapolates between the purely adversarial and stochastic regime (Zimmert & Seldin, 2021).

References

Agarwal, A., Luo, H., Neyshabur, B., and Schapire, R. E. Corralling a band of bandit algorithms. In Conference on Learning Theory , pp. 12-38. PMLR, 2017.

Agarwal, A., Jiang, N., Kakade, S. M., and Sun, W. Reinforcement learning: Theory and algorithms. 2021. URL https://rltheorybook.github.io/ .

Agarwal, N. and Singh, K. The price of differential privacy for online learning. In International Conference on Machine Learning , pp. 32-40. PMLR, 2017.

Agrawal, S. and Goyal, N. Near-optimal regret bounds for thompson sampling. Journal of the ACM (JACM) , 64(5):1-24, 2017.

Agrawal, S., Juneja, S. K., and Koolen, W. M. Regret minimization in heavy-tailed bandits. In Conference on Learning Theory , pp. 26-62. PMLR, 2021.

Agrawal, S., Mathieu, T., Basu, D., and Maillard, O.-A. Crimed: Lower and upper bounds on regret for bandits with unbounded stochastic corruption. In International Conference on Algorithmic Learning Theory , pp. 74-124. PMLR, 2024.

Ashutosh, K., Nair, J., Kagrecha, A., and Jagannathan, K. Bandit algorithms: Letting go of logarithmic regret for statistical robustness. In International Conference on Artificial Intelligence and Statistics , pp. 622-630. PMLR, 2021.

Audibert, J.-Y. and Bubeck, S. Minimax policies for adversarial and stochastic bandits. In COLT , pp. 217-226, 2009.

Audibert, J.-Y. and Bubeck, S. Regret bounds and minimax policies under partial monitoring. The Journal of Machine Learning Research , 11:2785-2836, 2010.

Auer, P. and Chiang, C.-K. An algorithm with nearly optimal pseudo-regret for both stochastic and adversarial bandits. In Conference on Learning Theory , pp. 116-120. PMLR, 2016.

Auer, P., Cesa-Bianchi, N., and Fischer, P. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47:235-256, 2002a.

Auer, P., Cesa-Bianchi, N., Freund, Y., and Schapire, R. E. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77, 2002b.

Bubeck, S. and Slivkins, A. The best of both worlds: Stochastic and adversarial bandits. In Conference on Learning Theory , pp. 42-1. JMLR Workshop and Conference Proceedings, 2012.

Bubeck, S., Cesa-Bianchi, N., and Lugosi, G. Bandits with heavy tail. IEEE Transactions on Information Theory , 59(11):7711-7717, 2013.

Chen, M., Gao, C., and Ren, Z. Robust covariance and scatter matrix estimation under huber's contamination model. The Annals of Statistics , 46(5):1932-1960, 2018.

Chen, Y., Huang, J., Dai, Y., and Huang, L. uniinf: Best-of-both-worlds algorithm for parameter-free heavy-tailed mabs. arXiv preprint arXiv:2410.03284 , 2024.

Cont, R. Empirical properties of asset returns: stylized facts and statistical issues. Quantitative finance , 1(2):223, 2001.

Dann, C., Wei, C.-Y., and Zimmert, J. A blackbox approach to best of both worlds in bandits and beyond. In The Thirty Sixth Annual Conference on Learning Theory , pp. 5503-5570. PMLR, 2023.

Dwork, C., Roth, A., et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science , 9(3-4):211-407, 2014.

Foster, D. J., Gentile, C., Mohri, M., and Zimmert, J. Adapting to misspecification in contextual bandits. arXiv preprint arXiv:2107.05745 , 2021.

Garcelon, E., Perchet, V., Pike-Burke, C., and Pirotta, M. Local differential privacy for regret minimization in reinforcement learning. Advances in Neural Information Processing Systems , 34: 10561-10573, 2021.

Genalti, G., Marsigli, L., Gatti, N., and Metelli, A. M. ( ε, u ) -adaptive regret minimization in heavy-tailed bandits, 2024.

Guan, Z., Ji, K., Bucci Jr, D. J., Hu, T. Y ., Palombo, J., Liston, M., and Liang, Y . Robust stochastic bandit algorithms under probabilistic unbounded adversarial attack. In Proceedings of the aaai conference on artificial intelligence , volume 34, pp. 4036-4043, 2020.

Hamza, A. B. and Krim, H. Image denoising: A nonlinear robust statistical approach. IEEE transactions on signal processing , 49(12):3045-3054, 2001.

Hsu, D. and Sabato, S. Heavy-tailed regression with a generalized median-of-means. In International Conference on Machine Learning , pp. 37-45. PMLR, 2014.

Huang, J., Dai, Y., and Huang, L. Adaptive best-of-both-worlds algorithm for heavy-tailed multiarmed bandits. In international conference on machine learning , pp. 9173-9200. PMLR, 2022.

Huang, J., Zhong, H., Wang, L., and Yang, L. Tackling heavy-tailed rewards in reinforcement learning with function approximation: Minimax optimal and instance-dependent regret bounds. Advances in Neural Information Processing Systems , 36, 2024.

Huber, P. J. Robust statistical procedures . SIAM, 1996.

Ito, S. Parameter-free multi-armed bandit algorithms with hybrid data-dependent regret bounds. In Conference on Learning Theory , pp. 2552-2583. PMLR, 2021.

Ito, S. and Takemura, K. Best-of-three-worlds linear bandit algorithm with variance-adaptive regret bounds. In The Thirty Sixth Annual Conference on Learning Theory , pp. 2653-2677. PMLR, 2023.

Ito, S., Tsuchiya, T., and Honda, J. Adversarially robust multi-armed bandit algorithm with variancedependent regret bounds. In Conference on Learning Theory , pp. 1421-1422. PMLR, 2022.

Jin, C., Jin, T., Luo, H., Sra, S., and Yu, T. Learning adversarial markov decision processes with bandit feedback and unknown transition. In International Conference on Machine Learning , pp. 4860-4869. PMLR, 2020.

Jin, T. and Luo, H. Simultaneously learning stochastic and adversarial episodic mdps with known transition. Advances in neural information processing systems , 33:16557-16566, 2020.

Jin, T., Huang, L., and Luo, H. The best of both worlds: stochastic and adversarial episodic mdps with unknown transition. Advances in Neural Information Processing Systems , 34:20491-20502, 2021.

Jin, T., Liu, J., and Luo, H. Improved best-of-both-worlds guarantees for multi-armed bandits: Ftrl with general regularizers and multiple optimal arms. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Jin, T., Liu, J., and Luo, H. Improved best-of-both-worlds guarantees for multi-armed bandits: Ftrl with general regularizers and multiple optimal arms. Advances in Neural Information Processing Systems , 36, 2024.

Kato, M. and Ito, S. Best-of-both-worlds linear contextual bandits. arXiv preprint arXiv:2312.16489 , 2023.

Kong, F., Zhao, C., and Li, S. Best-of-three-worlds analysis for linear bandits with follow-theregularized-leader algorithm. In The Thirty Sixth Annual Conference on Learning Theory , pp. 657-673. PMLR, 2023.

Kuroki, Y., Rumi, A., Tsuchiya, T., Vitale, F., and Cesa-Bianchi, N. Best-of-both-worlds algorithms for linear contextual bandits. arXiv preprint arXiv:2312.15433 , 2023.

Lai, T. L. and Robbins, H. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1):4-22, 1985.

Lattimore, T. and Szepesvári, C. Bandit algorithms . Cambridge University Press, 2020.

Lee, C.-W., Luo, H., Wei, C.-Y., and Zhang, M. Bias no more: high-probability data-dependent regret bounds for adversarial bandits and mdps. Advances in neural information processing systems , 33: 15522-15533, 2020.

Lee, C.-W., Luo, H., Wei, C.-Y., Zhang, M., and Zhang, X. Achieving near instance-optimality and minimax-optimality in stochastic and adversarial linear bandits simultaneously. In International Conference on Machine Learning , pp. 6142-6151. PMLR, 2021.

Lee, J., Honda, J., Ito, S., and Oh, M.-h. Follow-the-perturbed-leader with fr \ ' { e } chet-type tail distributions: Optimality in adversarial bandits and best-of-both-worlds. arXiv preprint arXiv:2403.05134 , 2024.

Lee, K. and Lim, S. Minimax optimal bandits for heavy tail rewards. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Li, L.-F., Zhao, P., and Zhou, Z.-H. Improved algorithm for adversarial linear mixture mdps with bandit feedback and unknown transition. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS) , 2024.

Lu, S., Wang, G., Hu, Y., and Zhang, L. Optimal algorithms for lipschitz bandits with heavy-tailed rewards. In international conference on machine learning , pp. 4154-4163. PMLR, 2019.

Neu, G. Explore no more: Improved high-probability regret bounds for non-stochastic bandits. Advances in Neural Information Processing Systems , 28, 2015.

Rakhlin, A., Shamir, O., and Sridharan, K. Making gradient descent optimal for strongly convex stochastic optimization. arXiv preprint arXiv:1109.5647 , 2011.

Ray Chowdhury, S. and Gopalan, A. Bayesian optimization under heavy-tailed payoffs. Advances in Neural Information Processing Systems , 32, 2019.

Ren, W., Zhou, X., Liu, J., and Shroff, N. B. Multi-armed bandits with local differential privacy. arXiv preprint arXiv:2007.03121 , 2020.

Shao, H., Yu, X., King, I., and Lyu, M. R. Almost optimal algorithms for linear stochastic bandits with heavy-tailed payoffs. Advances in Neural Information Processing Systems , 31, 2018.

Slivkins, A. Introduction to multi-armed bandits. arXiv preprint arXiv:1904.07272 , 2019.

Tao, Y., Wu, Y., Zhao, P., and Wang, D. Optimal rates of (locally) differentially private heavy-tailed multi-armed bandits. In International Conference on Artificial Intelligence and Statistics , pp. 1546-1574. PMLR, 2022.

Tossou, A. and Dimitrakakis, C. Achieving privacy in the adversarial multi-armed bandit. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 31, 2017.

Tsuchiya, T., Ito, S., and Honda, J. Stability-penalty-adaptive follow-the-regularized-leader: Sparsity, game-dependency, and best-of-both-worlds. Advances in Neural Information Processing Systems , 36, 2024.

Vershynin, R. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Wei, C.-Y. and Luo, H. More adaptive algorithms for adversarial bandits. In Conference On Learning Theory , pp. 1263-1291. PMLR, 2018.

Wu, Y., Zhou, X., Tao, Y., and Wang, D. On private and robust bandits. Advances in Neural Information Processing Systems , 36, 2024.

Xue, B., Wang, G., Wang, Y., and Zhang, L. Nearly optimal regret for stochastic linear bandits with heavy-tailed payoffs. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence , pp. 2936-2942, 2021.

Zhang, J., Karimireddy, S. P., Veit, A., Kim, S., Reddi, S., Kumar, S., and Sra, S. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems , 33: 15383-15393, 2020.

Zheng, K., Cai, T., Huang, W., Li, Z., and Wang, L. Locally differentially private (contextual) bandits learning. Advances in Neural Information Processing Systems , 33:12300-12310, 2020.

Zhong, H., Huang, J., Yang, L., and Wang, L. Breaking the moments condition barrier: No-regret algorithm for bandits with super heavy-tailed payoffs. Advances in Neural Information Processing Systems , 34:15710-15720, 2021.

Zhuang, V. and Sui, Y. No-regret reinforcement learning with heavy-tailed rewards. In International Conference on Artificial Intelligence and Statistics , pp. 3385-3393. PMLR, 2021.

Zimmert, J. and Lattimore, T. Return of the bias: Almost minimax optimal high probability bounds for adversarial linear bandits. In Conference on Learning Theory , pp. 3285-3312. PMLR, 2022.

Zimmert, J. and Seldin, Y. Tsallis-inf: An optimal algorithm for stochastic and adversarial bandits. Journal of Machine Learning Research , 22(28):1-49, 2021.

A Related Work

In this section, we give a comprehensive discussion on the related work.

Regret Minimization in the BOBW Setting (with Bounded Losses). There are mainly two different approaches towards enjoying BOBW regret guarantees. The first approach is the detectswitch framework initially proposed by Bubeck & Slivkins (2012). The framework runs in a two-stage manner: In the first stage, the algorithm performs some tests in every round and carefully maintain the sampling distribution over all actions. Once any of the tests fails, it runs any off-the-shelf algorithm purely for the adversarial regime in the remaining rounds and inherits the regret guarantee of that algorithm. Bubeck & Slivkins (2012) proposed an algorithm achieving near-optimal regret in the adversarial regime and O ( (log T ) 2 K/ ∆ ) regret in the stochastic regime, both with high-probability. Auer & Chiang (2016) further refined the tests and improved the gap-dependent term to the optimal ∑ i :∆ i > 0 (1 / ∆ i ) . This framework has been generalized to setups including linear bandits (Lee et al., 2021) and bandits with feedback graph (Kong et al., 2023).

Somewhat surprisingly, since the work by Wei & Luo (2018), there have been a large group of papers showing that Online Learning algorithms (e.g., OMD), originally for the adversarial regime, implicit adapts to stochastic environments and enjoys logarithmic regrets. BOBW guarantees of Online Learning algorithms have been established in quite broad setups including MAB (Wei & Luo, 2018; Zimmert & Seldin, 2021; Jin et al., 2023), (contextual) linear bandits (Ito & Takemura, 2023; Kong et al., 2023; Kuroki et al., 2023; Kato & Ito, 2023), and (tabular) Markov Decision Processes (MDPs) (Jin & Luo, 2020; Jin et al., 2021). One desirable advantage of this approach is that, thanks to the flexibility of Online Learning framework, it is possible to additionally enjoy some other adaptive regret bounds on top of the standard BOBW guarantee (Ito, 2021; Tsuchiya et al., 2024).

Regret Minimization in Heavy-tailed Stochastic Environments. There are a large body of works focusing on regret minimization in the stochastic environment where the losses are potentially heavytailed (Bubeck et al., 2013; Shao et al., 2018; Lu et al., 2019; Ray Chowdhury & Gopalan, 2019; Zhong et al., 2021; Agrawal et al., 2021; Xue et al., 2021; Lee & Lim, 2022; Zhuang & Sui, 2021; Huang et al., 2024). The general recipe of the algorithm design is to derive concentration results for the loss mean estimation with robust estimators (e.g., median-of-means (Hsu & Sabato, 2014), truncated/trimmed estimator (Bubeck et al., 2013), and Huber estimator (Huber, 1996)), and then integrate them with celebrated optimism-based algorithms.

High-probability Regrets in Adversarial MAB and the BOBW Setting. The central piece towards high-probability bounds in adversarial MAB lies in balancing between the variance of the loss estimator and the total regret. In general, there are three types of approaches in the literature: 1) adding negative bonus to the Importance-Weighted (IW) estimates (Auer et al., 2002b; Zimmert & Lattimore, 2022), 2) utilizing log-barrier regularizer together with increasing learning rates (Lee et al., 2020), and 3) replacing the IW estimator with the Implicit-eXploration (IX) estimator (Neu, 2015). While the IX estimator enables a simplified analysis and is thus widely-used (e.g., Jin et al. (2020); Li et al. (2024)), it heavily exploits the non-negativity of the losses, and it is unclear to us how to adapt it to the case of potentially-negative losses. For a comparison between the first two approaches, readers are referred to Foster et al. (2021) and Zimmert & Lattimore (2022). The detect-switch framework by Bubeck & Slivkins (2012) naturally provides high-probability bounds in BOBW, whenever a subroutine algorithm achieving high-probability regret in the adversarial regime (e.g., EXP3.P (Auer et al., 2002b)) is integrated in a plug-and-play manner. While the Online Learning framework naturally provides BOBW guarantee via elegant analysis in expectation (Zimmert & Seldin, 2021), it is unclear how to adapt it for high-probability guarantees.

Adaptive/Parameter-free Heavy-tailed MAB. Throughout this work, we assume that the heavy tail parameters u and v are known to the learning algorithm. In this case, Bubeck et al. (2013) showed minimax lower bound Ω( uT 1 1+ v K v 1+ v ) and gap-dependent lower bound Ω (∑ i :∆ i > 0 ( u/ ∆ i ) 1 /v (log T ) ) using stochastic environments, so these lower bounds also apply to the adversarial regime. However, in practice one may not know the values of u, v exactly in advance, therefore it is natural to ask whether it is possible to enjoy the same regret rate as if u, v were known, which we refer to as 'adaptation (to heavy tails) for free'. On the negative side, Genalti et al. (2024) showed that there is no such an algorithm that can enjoy ˜ O ( uT 1 1+ v K v 1+ v ) worst-case regret for all unknown u, v in the stochastic regime. In terms of gap-dependent regret, similar 'no adaptation for free' effect has been shown in an earlier work by Ashutosh et al. (2021). On the positive side, in the same work by

Genalti et al. (2024), it is shown that, somewhat surprisingly, with the same 'truncated non-negative losses' assumption (Assumption 2), a UCB-based algorithm proposed by them enjoys 'adaptation for free' in terms of both minimax and gap-dependent regrets in the stochastic regime. With the same assumption in the adversarial regime, Huang et al. (2022) proposed a variant of FTRL, which achieves the O ( uK v 1+ v T 1 1+ v + √ KT ) worst-case regret guarantee. In a very recent work by Chen et al. (2024), an FTRL-based algorithm named uniINF is shown to enjoy the BOBW guarantee even when u and v are both unknown, although the gap dependency of their logarithmic regret is K/ ∆ 1 /v , and still Assumption 2 is needed. Exploring broader scenarios in which we can enjoy 'adaptation for free' (and to what extent) is an interesting future direction.

B Omitted Details in Section 3

In this section, we provide complete proof for Theorem 1.

To begin with, we first define ℓ ′ t,i as the trimmed value of ℓ t,i with respect to M , i.e., ℓ ′ t,i = ℓ t,i I {| ℓ t,i | ⩽ M } . Then, we decompose the pseudo-regret by

B.1 Bounding TRIMERR

Lemma 1. For any fixed trimming threshold M t,i = M > 0 and ζ < 1 / ( e log T ) , with probability at least 1 -ζ , it holds that

Proof of Lemma 1. First note that ( µ ′ t,a t -ℓ ′ t,a t ) t =1 ,...,T form a martingale difference sequence adapted to the filtration F 1 , . . . , F T where F t := σ ( a 1 , ℓ 1 ,a 1 , . . . , a t -1 , ℓ t -1 ,a t -1 ) . Moreover, we have ∣ ∣ µ ′ t,a t -ℓ ′ t,a t ∣ ∣ ⩽ 2 M and

where in the last inequality we utilize the fact that ∑ K i =1 w t,i = 1 and Assumption 1. By Lemma 11, we have with probability at least 1 -ζ ,

Together with Lemma 8 which bounds µ t,a t -µ ′ t,a t , we arrive at

With exactly the same reasoning, we have with probability at least 1 -ζ that

Taking a union bound over the two terms completes the proof.

B.2 Bounding TRIMREG

For any i ∈ [ K ] , let y i be the K -dimensional one-hot vector such that the i -th element is 1, and all the others are zero. Define vector y ′ := (1 -λ ) y i ∗ + λw 0 , where w 0 = (1 /K,... , 1 /K ) is the uniform exploration. We can rewrite TRIMREG as

In the following subsections, we bound each of the three terms in the right-hand-sight. To give a preview:

TRIMREG I is the standard online learning regret over loss sequence ̂ ℓ 1 , · · · , ̂ ℓ T , and is handled using regret analysis of OMD.

TRIMREG II is a bias term due to that we cannot directly compete with y i ∗ and have to choose a close neighbor y ′ . It is under control since y ′ and y i ∗ are close to each other.

TRIMREG III is the main challenge in getting high-probability bounds in adversarial bandits. While it contains some terms that could be potentially large due to the high variance of ̂ ℓ t , they would be cancelled by some negative terms in TRIMREG I introduced by the increasing learning rates so that eventually the sum of the three terms can be bounded.

B.2.1 Bounding TRIMREG I

Lemma 2. Suppose η ⩽ 1 10 M and λ = T -v 1+ v K -1 1+ v , for any M t,i = M , Algorithm 1 ensures that with probability at least 1 -ζ ,

Proof of Lemma 2. The proof is a combination of Agarwal et al. (2017, Lemma 12) and Lee et al. (2020, Lemma 2.1). For the (active) OMD update rule w t +1 = argmin x ∈ Ω ′ ( ⟨ x, ̂ ℓ t ⟩ + D ψ t ( x, w t ) ) in Algorithm 1, it is equivalent to first obtaining some intermediate vector ˜ w t +1 such that ∇ ψ t ( ˜ w t +1 ) = ∇ ψ t ( w t ) -̂ ℓ t , and then obtaining w t +1 = argmin x ∈ Ω ′ D ψ t ( x, ˜ w t +1 ) .

For any competitor y ∈ Ω ′ , in each round t ∈ [ T ] , we have

where in step (a) we utilize the definition of Bregman divergence (or the so-called 'three-point property') and step (b) is due to the generalized Pythagorean theorem.

Throughout this proof, we choose λ = T -v 1+ v K -1 1+ v as in Algorithm 1. Recall that we define y ′ := (1 -λ ) y i ∗ + λw 0 . Therefore, for any i ∈ [ K ] , we have y ′ i ⩾ λ/K and hence y ′ ∈ Ω ′ .

For any scalar c > 0 , we define function h ( c ) := c -1 -log c , which is always non-negative. Taking the summation over all T rounds and letting y = y ′ , we get

where in the inequality we get rid of -D ψ T ( y ′ , w T +1 ) since Bregman divergence is always nonnegative. In the following, we bound each of these three terms in the right-hand-side.

Step 1: bounding the first term. Plugging in the exact expression of Bregman divergence concerning the log-barrier regularizer ψ t we define in Algorithm 1, we have

By choosing λ = T -v 1+ v K -1 1+ v , we have y ′ i ⩾ λ/K = T -v 1+ v K -1 -1 1+ v and

Recall that here we utilize the assumption that K ⩽ T .

Step 2: bounding the second term. Plugging in the expression of Bregman divergence associated with log-barrier, we get

Now we look at each action i . Recall that, if the learning rate does not increase at round t , then ( 1 η t +1 ,i -1 η t,i ) h ( y ′ i w t +1 ,i ) is simply 0. Otherwise, we have η t +1 ,i > η t,i , and as a result, ( 1 η t +1 ,i -1 η t,i ) h ( y ′ i w t +1 ,i ) < 0 .

Therefore, if we let n i denote the total number of learning rate changes in action i , and let t n i denote the round when the last change happens (such that η T,i = η t n i +1 ,i = κη t n i ,i = κ n i η ), we have

By Lemma 9, when λ = T -v 1+ v K -1 1+ v , we have n i ⩽ log 2 ( T v 1+ v K 1 1+ v ) ⩽ log 2 T and η t,i ⩽ e log 2 T log T η ⩽ 5 η. Together with 1 -κ = 1 -e 1 / log T ⩽ -1 log T , we get

It is left to upper-bound term -h ( y ′ i w t n i +1 ,i ) . Noticing that

we get

Taking the summation over all actions, we finish bounding the second term as

Step 3: bounding the third term. Given the fact that ∇ ψ t ( ˜ w t +1 ) = ∇ ψ t ( w t ) -̂ ℓ t , we have

which implies that

Now we have

To proceed, we first show that if η ⩽ 1 10 M , it holds that

First, it is trivially true whenever ℓ ′ t,i ⩾ 0 as the left-hand-sight is non-negative. Since ℓ ′ t,i is at most as negative as -M and η t,i ⩽ 5 η , it is left to show that

which is clearly satisfied when η ⩽ 1 10 M .

By applying the fact that c -log(1 + c ) ⩽ c 2 , ∀ c ⩾ -0 . 5 to Eq. (31), we get

We now finish bounding the third term by deriving a high-probability bound on ∑ T t =1 ∣ ∣ ℓ ′ t,a t ∣ ∣ 1+ v (and multiplying it by 5 ηM 1 -v ).

Consider the martingale difference sequence ( ∣ ∣ ℓ ′ t,a t ∣ ∣ 1+ v -E [ ∣ ∣ ℓ ′ t,a t ∣ ∣ 1+ v ]) t ∈ [ T ] , we have

almost surely, and

By Lemma 11, we have with probability at least 1 -ζ that

At this point, we have bounded all three terms, and putting them together completes the proof

B.2.2 Bounding TRIMREG II

Lemma 3 (Upper bound on TRIMREG II) . For any fixed M > 0 and λ ∈ (0 , 1) , with probability at least 1 -ζ , it holds that

Proof of Lemma 3. By Hölder's inequality, we have

Define µ t,i = E ℓ ∼ P t,i [ | ℓ | · I {| ℓ | ⩽ M } ] , ∀ t ∈ [ T ] , i ∈ [ K ] , we have

Applying Lemma 10 to (∣ ∣ ℓ ′ t,i ∣ ∣ -µ t,i ) t ∈ [ T ] ,i ∈ [ K ] , since ∣ ∣ ∣ ∣ ℓ ′ t,i ∣ ∣ -µ t,i ∣ ∣ ⩽ 2 M and

we get that with probability at least 1 -ζ ,

which implies that

B.2.3 Bounding TRIMREG III

Lemma 4 (Upper bound on TRIMREG III) . For any fixed M > 0 and λ ∈ (0 , 1) , with probability at least 1 -ζ , it holds that

Proof of Lemma 4. We note that ( ⟨ y ′ , ̂ ℓ t -ℓ ′ t ⟩ /M ) t ∈ [ T ] 6 form a martingale sequence, and E [ ⟨ y ′ , ̂ ℓ t -ℓ ′ t ⟩ /M ∣ ∣ ∣ F t ] is clearly finite since w t,i ⩾ λ/K . Moreover, we have

where in the last step we utilize the facts that y ′ i ⩽ 1 and 1 /w t,i ⩽ ρ T,i .

Noting

by Lemma 12, we have with probability at least 1 -ζ that,

where ι := log(2 max { √ Tu 1+ v M -1 -v ⟨ y ′ , ρ T ⟩ , 1 , 2 ⟨ y ′ , ρ T ⟩} /ζ ) and step (a) is due to the elementary inequality √ x 1 x 2 ⩽ x 1 + x 2 2 , ∀ x 1 , x 2 ⩾ 0 .

We complete the proof by noticing that 1 ⩽ 2 K ⩽ ⟨ y ′ , ρ T ⟩ ⩽ 2 λ/K and apply it to ι .

B.2.4 Putting things together

By Lemmas 1, 2, 3, and 4, with probability at least 1 -ζ , we have

Choosing 1 -v -1

Finally, choosing η = min { 1 10 M , 1 40 M log( T ) log(8 KT/ζ ) } = 1 40 M log( T ) log(8 KT/ζ ) to cancel the terms containing ⟨ y ′ , ρ T ⟩ ensures that

C Omitted Details in Section 4

C.1 Concentration Results

In this subsection, we derive concentration results needed for the analysis, adapted from Bubeck & Slivkins (2012) to suit our heavy-tailed case.

Lemma 5 (Concentration on the trimmed importance-weighted estimator) . Suppose M t,i = u · ( t/K ) 1 1+ v · (log(log( T ) /ζ )) -1 3 v +1 . In both regimes, we have with probability at least 1 -ζ that, for any i ∈ [ K ] and t ∈ [ T ] , if ( K +1) ⩽ t ⩽ t sw, then

Proof. We fix some action i ∈ [ K ] . We first rewrite ̂ L t,i as ̂ L t,i = ∑ t s =1 ℓ ′ s,i I s,i w s,i . Note that for any s > τ i , we have w s,i = q i τ i s , and q i ⩾ w s,i ⩾ 1 /K for any s ⩽ τ i .

We define X s,i := ℓ ′ s,i I s,i w s,i -µ ′ s,i for any 1 ⩽ s ⩽ t . If t ⩽ t sw, then X 1 ,i , . . . , X t,i forms a martingale difference sequence. To have an upper bound on | X s,i | : 1) when s ⩽ τ i , we have

and 2) when s > τ i , we have

Combining two cases, we get

Moreover, we have

where the last inequality is because M s,i is non-decreasing in s.

By Lemma 11 we have that with probability at least 1 -ζ , for any t ⩽ t sw

By taking an union bound over all all actions i ∈ [ K ] , we have with probability at least 1 -ζ that

By Lemma 8, we have almost surely that

Combining two parts above, we get with probability at least 1 -ζ that, for any action i and round t ,

Lemma 6 (Concentration on the number of pulls) . It holds with probability at least 1 -ζ that, for any i ∈ [ K ] and t ∈ [ T ] , if t ⩽ t sw,

Proof of Lemma 6. We first fix some action i and round t . Recall that N t,i = ∑ t s =1 I s,i . Define X s,i := I s,i -w s,i . If t ⩽ t sw then X 1 ,i , . . . , X t,i forms a martingale difference sequence such that | X s,i | ⩽ 1 . Moreover, since w s,i is non-decreasing in s when s ⩽ τ i (and w τ i ,i = q i ), we have

Therefore, by Lemma 11, it holds with probability at least 1 -ζ that,

Taking union bounds over all actions i ∈ [ K ]

completes the proof.

Lemma 7 (Concentration on the trimmed empirical-mean estimator) . It holds with probability at least 1 -ζ that, for any i ∈ [ K ] and t ∈ [ K,t sw ] ,

Remark 9 . Note that in the special case of stochastic regime, the term ∑ t s =1 µ s,i I s,i N t,i is simply µ ( i ) (which is simply a scalar denoting the loss mean of action i in the stochastic regime as defined in Section 2), and this lemma has been proven in Bubeck et al. (2013). However, we are not aware of how to extend the analysis of stochastic case to adversarial case.

Proof. Recall that B s,i is defined in Algorithm 2 as B s,i := u ( N s,i log(2 T/ζ ) ) 1 1+ v , ∀ s ∈ [ T ] . Then we rewrite ̂ µ t,i as and define

Moreover, in this lemma we need a slightly different filtration. We define

Clearly, given any fixed round t and action i , X 1 ,i , . . . , X t,i form a martingale difference sequence adapted to F ′ 1 , . . . , F ′ t (since both B s,i and I s,i are deterministic conditioned on F ′ s ), with

and

where in the last inequality we utilize the fact that B s,i is non-decreasing in s .

Noting that

we apply Lemma 12 and get with probability at least 1 -ζ that, for any fixed t ,

Taking a (two-sided) union bound over all rounds and actions, we get with probability at least 1 -ζ that

Finally, we have

where step (a) follows from Lemma 8 and the last step relies on the fact that v ∈ (0 , 1] .

Applying triangle inequality to Eqs. (54) and (55) and then dividing both sides by N t,i completes the proof.

C.2 Logarithmic Regret in the Stochastic Regime

In this subsection, we provide the complete proof for logarithmic regret of Algorithm 2 in the stochastic regime.

Taking a union bound over Lemmas 5, 6, and 7, with probability at least 1 -ζ , for any i ∈ [ K ] and t ⩽ t sw, all of the following holds in either stochastic or adversarial regime (recall that β = 12 T 2 K log( T ) ):

Before we start to derive the regret bound, we simplify Eq. (56) a bit for convenience.

For any t such that τ i > t (i.e., action i is still active by round t ), we have w s,i ⩾ 1 /K, ∀ s ⩽ t , and Eq. (56) implies that

otherwise (when τ i ⩽ t ), we have

where in the first inequality we utilize the facts that q i ⩾ 1 /K and

and the second inequality is because

Combining two cases (whether τ i > t or not), Eq. (56) implies that

To show the regret bound, we first show that, in the stochastic regime, when all the three above events hold, we will never start to run Algorithm 1, i.e., tests (6)-(8) always fail (for all t < T ).

Test (6) always fails. This is simply implied by Eqs. (63) and (58) together with a triangle inequality.

Test (7) always fails. We first show that test (5) is never satisfied for action i ∗ , so we have i ∗ ∈ A t , ∀ t ⩽ t sw. To see this, for actions i, i ∗ ∈ A t -1 , by Eq. (63) we must have

which means that i ∗ is never eliminated from A t and hence stays active (due to test (5)).

Moreover, for any action i / ∈ A t , it must be deactivated at some round no later than t , so we have τ i ⩽ t and that test (5) is satisfied at round τ i (and is not satisfied at round τ i -1 ). Let j ∗ t ∈ argmin j ∈ A t -1 ̂ L t,j .

̸

Therefore, looking at round τ i (and any action i = i ∗ ), we have

where step (a) is from test (5) (which should hold now) and step (b) is due to Eq. (63) and the optimality of i ∗ (i.e., µ ( i ) -µ ( j ∗ τ i ) ⩽ µ ( i ) -µ ( i ∗ ) = ∆ i ).

̸

Moreover, looking at round τ i -1 (and any action i = i ∗ ), we have

where step (a) follows from the good event in Eq. (63), step (b) is because i ∗ ∈ A τ i -2 , and step (c) is due to test (5) (which now should not hold for action i since it is still active).

Now we can show that test (7) is never satisfied since in any round t for any i / ∈ A t :

where the last step is from Eq. (66).

Test (8) always fails. Since i ∗ ∈ A t , for any action i / ∈ A t , we have

where the first step is due to i ∗ ∈ A t and the last step is from Eq. (65).

Putting things together. Now we show two intermediate results, followed by bounding the regret. First, Eq. (66) implies that

which after rearranging implies that

Due to the definition of q i , we have

To see this, let i ′ denote the i -th earliest action that is deactivated, we have q i ′ ⩽ 1 K -i ′ +1 due to the algorithm design.

Now we have everything needed to arrive at the final result. By the definition of pseudo-regret in stochastic bandits, we have

where step (a) follows from Eq. (57), step (b) is due to Eq. (70), and step (c) is from Eq. (71). Choosing ζ = 1 /T < 1 /e , we have

C.3 Optimal Worst-case Regret in the Adversarial Regime

In this subsection, we provide the complete proof for (near-)optimal worst-case regret of Algorithm 2 in the adversarial regime.

For any action i / ∈ A t sw -1 , we have τ i ⩽ t sw -1 and test (8) is not satisfied for i at round t sw -1 (since the algorithm switch has not happened). We get

where step (a) is due to Eq. (63) (applied twice) and test (8). Therefore, we must have i ∗ t sw -1 ∈ A t sw -1 (which further implies i ∗ t sw -1 ∈ A s , ∀ s ⩽ t sw -1 ) since ∑ t sw -1 s =1 µ s,i ∗ t sw -1 -∑ t sw -1 s =1 µ s,I ∗ t sw -1 ⩽ 0 by the definition, otherwise a contradiction is incurred.

Now we can bound the cumulative regret up to the algorithm switch. Specifically, we rewrite the regret as

In the rewriting above, we use nothing but the simple fact that ∑ K i =1 N t sw -1 ,i = t sw -1 . We now bound each of the three terms (Part A, Part B, and Part C) separately.

Bounding Part A. We first rewrite Part A as

By Eq. (58), we have

By test (6) (which does not hold now), we have

Therefore, we can conclude that

Bounding Part B. Since i ∗ t sw -1 ∈ A t sw -1 as we have shown, for any action i / ∈ A t sw -1 , due to test (7) (which is not satisfied at round t sw -1 ), we have

For any other action i ∈ A t sw -1 , due to test (5), we have

Combining these two cases, we can claim that for any action i ∈ [ K ] ,

Bounding Part C. Simply due to Eq. (63), since i ∗ t sw -1 ∈ A t sw -1 , we have

Putting three parts together . Putting the three parts together, we get

where in the last step we apply Lemma 6 to bound N t sw -1 ,i in the term ∑ K i =1 N t sw -1 ,i Width ( t sw -1) τ i -1 . Now we bound each of the four terms one by one.

Bounding the first term. Applying Jensen's inequality, we get

and the first term is bounded as

Bounding the second term. Simply plugging in the definition of Width ( · ) , we have

Bounding the third term. By the fact that ∑ K i =1 q i = O (log K ) , we have

Bounding the forth term. Notice that in the algorithm design, each action will be pulled once in the initialization, and we clearly have τ i ⩾ K +1 . Therefore, we have

Combining the bounds on these four terms, we have

The regret incurred starting from round t sw + 1 is taken care of by Algorithm 1 (and the regret guarantee is given by Theorem 1). By taking a union bound, we have with probability at least 1 -ζ that

Choosing ζ = 1 /T , we have

D Heavy-tailed Adversarial Bandits with Huber Contamination

This section is dedicated to the adversarial regime, with the additional setup that the bandit feedback could be contaminated in the Huber model (Huber, 1996). In the stochastic MAB, Huber contamination has been studied in Guan et al. (2020); Agrawal et al. (2024); Wu et al. (2024). We first formally define the problem setup, and then present the algorithm design and the regret analysis. From the regret analysis one could readily see why removing Assumption 2 is necessary for a near-optimal regret upper bound. Lastly, we provide a matching lower bound which suggests that we obtain the near-optimal worst-case regret guarantee.

D.1 Problem Setup

The learning algorithm and environment perform the following interactions repeatedly in round t = 1 , . . . , T :

The algorithm samples action a t from [ K ] via a t ∼ w t := ( w t, 1 , . . . , w t,K ) ∈ Ω , i.e., the probability of sampling action i ∈ [ K ] is w t,i . The environment draws loss ℓ t,i from 'clean' distribution P t,i satisfying Assumption 1 for every action i ∈ [ K ] .

Let ℓ t,a t denote the feedback revealed to the algorithm associated with a t . With probability α ∈ (0 , 1] , the algorithm observes the contaminated feedback. That is, it observes ℓ t,a t = ˜ ℓ t,i , which is generated from an arbitrary 'bad' distribution Q t,i . With probability (1 -α ) , it observes the 'clean' loss ℓ t,a t = ℓ t,i .

The algorithm determines w t +1 based on all the revealed history so far.

The goal of the learning algorithm is to minimize the regret in the presence of contaminated feedback. For the ease of presentation, we consider the weaker expected pseudo-regret (rather than the highprobability version), which is defined as:

where the optimal action i ∗ and loss mean µ t are both still with respect to the 'clean' distributions, i.e., µ t,i := E ℓ t,i ∼ P t,i [ ℓ t,i ] and i ∗ ∈ argmin i ∈ [ K ] ∑ T t =1 µ t,i , while the expected pseudo-regret additionally includes the randomness from the contamination. Similar to all previous works, we assume that the 'contamination level' α is known to the algorithm.

Assumption 2 adapted to the contaminated case (to bound CONTRIMERR) becomes the following.

Assumption 3 (Truncated non-negative losses in the contaminated case) . Given any fixed M > 0 , the loss distributions of the optimal action i ∗ satisfy that E ℓ t,i ∗ ∼ P α t,i ∗ [ ℓ t,i ∗ · I {| ℓ t,i ∗ | > M } ] ⩾ 0 , ∀ t ∈ [ T ] .

Importantly, even if there is some prior knowledge regarding the 'clean' distribution P t,i , due to the arbitrary 'bad' distribution component contained in P α t,i ∗ , the adapted 'truncated non-negativity' may not hold any more and one may not obtain any meaningful regret guarantee. However, by taking advantage of the log-barrier regularizer, we naturally handle all possible 'bad' distribution Q t,i . In the following two subsections, we the details of regret upper bound and lower bound, respectively.

D.2 Regret Guarantee and Analysis

To obtain near-optimal high-probability regret guarantee again Huber contamination, we simply need to run our Algorithm 1 with different initial learning rate and trimming threshold, both of which now further depend on the contamination level. We formally state the theoretical guarantee below.

Theorem 3. For any failure probability ζ and contamination level α > 0 , by choosing initial learning rate and trimming threshold

Algorithm 1 ensures that with probability at least 1 -ζ ,

By further choosing ζ = 1 /T , Algorithm 1 ensures that

We first rewrite regret as

where N Q is the set containing all round indices in which the observation is contaminated (i.e., ℓ t,a t = ˜ ℓ t,a t ∼ Q t,a t ), and N P = [ T ] \ N Q (i.e., those rounds in which ℓ t,a t = ℓ t,a t ∼ P t,a t ).

Similar to the uncontaminated case, it is sufficient to choose some fixed trimming threshold M t,i = M .

D.2.1 Bounding CONTRIMERR I

Recall that | N Q | denotes the total number of rounds when the feedback is contaminated, which is exactly the sum of T independent random variables from Ber ( α ) . By standard concentration results (e.g., Lemma 10), we have with probability at least 1 -ζ that

where the last step is again due to the fact that √ x 1 x 2 ⩽ x 1 + x 2 2 , ∀ x 1 , x 2 ⩾ 0 .

For any fixed trimming threshold M , it holds almost surely that

which implies that, with probability at least 1 -ζ , we have

D.2.2 Bounding CONTRIMERR II

In round t ∈ N P , the losses are not contaminated. By Lemma 1, we have probability with at least 1 -ζ that,

D.2.3 Bounding CONTRIMREG

We first rewrite this part as

D.2.4 Bounding CONTRIMREGI

As long as it holds that η ⩽ 1 10 M , we have

and it is left to bound ∑ T t =1 ∣ ∣ ∣ ℓ ′ t,a t ∣ ∣ ∣ 1+ v .

We first rewrite it as

Similarly, with probability at least 1 -ζ , it holds that

As in the uncontaminated case, by Lemma 11, we have with probability at least 1 -ζ that

Taking a union bound, we get with probability at least 1 -ζ that

D.2.5 Bounding CONTRIMREGII

By Hölder's inequality, we have

We clearly have with probability at least 1 -ζ that

and with probability with at least 1 -ζ that

By taking a union bound, we get with probability at least 1 -ζ that

D.2.6 Bounding CONTRIMREGIII

We first rewrite CONTRIMREGIII as

To bound the first part, recall that for any t ∈ N Q , it holds that

and

Therefore, with probability at least 1 -ζ , it holds that

where ι ′ := log(4 max { √ ( 2 αT + 4 log(2 /ζ ) 3 ) ⟨ y ′ , ρ T ⟩ , 1 , 2 ⟨ y ′ , ρ T ⟩} /ζ ) .

For the uncontaminated part, with probability, by Lemma 12, we have with probability at least 1 -ζ that,

where ι ′′ := log(2 max { √ Tu -1+ v M 1 -v ⟨ y ′ , ρ T ⟩ , 1 , 2 M ⟨ y ′ , ρ T ⟩ /u } /ζ ) . Taking a union bound, we get with probability at least 1 -ζ that

D.2.7 Putting All Pieces Together

Combining all the parts, we have with probability at least 1 -ζ that

We first plug in λ and simplify the upper bound as

Assuming αT ⩾ 4 3 log(15 log( T ) /ζ ) and T ⩾ √ T log(15 log( T ) /ζ ) , we further simplify the upper bound as

where in the last inequality we apply K ⩽ T to upper bound terms of form Θ( αMT 1 1+ v K v 1+ v ) .

Now we are going to choose η = 1 u · min { ( K log( T ) T ) 1 1+ v , ( K log( T ) T ) 2+ v 2+2 v ( 1 α ) v 2+2 v } . We further assume that α v (1+ v )(2+ v ) T ⩾ K log( T ) , which implies that

Finally, we need to cancel the terms containing ⟨ y ′ , ρ T ⟩ to obtain the high-probability guarantee.

Case 1: u ⩾ M ⟨ y ′ , ρ T ⟩ . If max { u, M ⟨ y ′ , ρ T ⟩} = u , it would just introduce a ˜ O ( u ) term which can be ignored. In this case, η must satisfy that

Case 2: u < M ⟨ y ′ , ρ T ⟩ . With the current choice of M and η , we have

which implies that

Now it is sufficient to have η satisfying that

In summary, in either case, it is sufficient to have ηM ⩽ 1 110 log( T ) log(40 T 2 /ζ ) . Note that this is stronger than (and implies) the stability condition ηM ⩽ 0 . 1 . Starting again from Eq. (117), with any η > 0 , choosing

we have

Lastly, still choosing

which implies that meaning that

D.3 Regret Lower Bound with Contamination

Theorem 4 (Lower Bound for Heavy-tailed Adversarial Bandits with Huber Contamination) . For any bandit algorithm, there must exist one problem instance in which the algorithm suffers regret

Proof of Theorem 4. In the presence of heavy tails, every algorithm suffers Ω ( uT 1 1+ v K v 1+ v ) regret in the worst case, regardless of contamination level α (Bubeck et al., 2013).

Therefore, to show the lower bound with contamination, it suffices to show a lower bound of

given any α ∈ (0 , 1] .

This proof is a direct modification based on Wu et al. (2024, Appendix B). We use π to denote a bandit algorithm. We construct two environments, denoted by ν 1 and ν 2 , respectively. And then we show that any algorithm suffers the claimed regret in one of these two environments.

Environment ν 1 . In ν 1 , the loss of action 1 in every round t ∈ [ T ] is given by

where γ ⩽ 1 is some free parameter to choose at the last step of the proof. One can verify that E [ | ℓ 1 | 1+ v ] ⩽ u 1+ v via direct calculations.

̸

For any suboptimal action i = 1 , the loss in every round t ∈ [ T ] is given by

̸

One can verify that E [ | ℓ t,i | 1+ v ] ⩽ u 1+ v , ∀ i = 1 . Moreover, action 1 is the optimal one and we have the 'sub-optimality gap' ∆ := E [ ℓ t, 1 -ℓ t,i ] = u 5 γ v , ∀ i = 1 .

̸

Given algorithm π and environment ν 1 , we define i ′ = argmin i ∈{ 2 ,...,K } E π,ν 1 [ N T,i ] , and hence we have

E π,ν 1 [ N T,i ′ ] ⩽ T K -1 . Now we are able to construct the second environment.

Environment ν 2 . In this environment, everything is the same as in ν 1 , except that for action i ′ , now the loss follows

One can verify that E [ ℓ i ′ ] = 7 10 γ 1+ v , E [ | ℓ i ′ | 1+ v ] ⩽ u 1+ v and now the optimal action is i ′ . We use ˜ ν 2 to denote the contaminated version of ν ′ 2 (where the bad distributions are determined later).

And then, we use ˜ ν 1 ( ˜ ν 2 ) to denote the contaminated version of ν 1 ( ν 2 ). The bad distributions will be determined later.

Environments ˜ ν 1 and ˜ ν 2 . We choose γ = α 1 1+ v ∈ (0 , 1] . Then for any i ∈ [ K ] , we have TV( P t,i ∥ P ′ t,i ) ⩽ 2 5 γ 1+ v = 2 5 α ⩽ α 1 -α . According to Lemma 13, for any action i ∈ [ K ] , there exist bad distributions G i and G ′ i such that

where P t,i ( P ′ t,i ) denotes the loss distribution of action i in round t under environment 1 (2). We construct ˜ ν 1 and ˜ ν 2 by

where x i and x ′ i denote the loss distributions for action i in these two environments, respectively. Following from the regret definition, we first have

By adding them together, we have

where step (a) follows from the Bretagnolle-Huber inequality (Lattimore & Szepesvári, 2020, Theorem 14.2), and step (b) is due to the fact that ˜ ν 1 and ˜ ν 2 are identical under our construction.

Recall that ∆ = u 5 γ v and γ = α 1 1+ v , we arrive at

which completes the proof.

E BOBWGuarantees with Bounded Losses and Local Differential Privacy

We first give the definition of Differential Privacy followed by the learning setup.

Definition 1 (Differential Privacy (DP)) . For any given privacy budget ε > 0 , δ ⩾ 0 , a mechanism M : D → R m is said to be ( ε, δ ) -differentially private (DP) if for all datasets X,X ′ in D that differ on only one element and measurable subset E ⊂ R m , it holds that P ( M ( X ) ∈ E ) ⩽ exp( ε ) · P ( M ( X ′ ) ∈ E )+ δ. When δ = 0 , we refer to ( ε, δ ) -DP as ε -DP (pure DP), which is stronger than ( ε, δ ) -DP for some δ > 0 (approximate DP).

Definition 2 (Bandits with Bounded Losses and Local DP (LDP)) . All loss distributions ( P t,i ) t ∈ [ T ] ,i ∈ [ K ] have support bounded in [0 , 1] . Given any privacy budget ε ∈ (0 , 1] , δ > 0 , the bandit model is said to be ( ε, δ ) -LDP if a t +1 lies in the sigma-algebra generated by { a t ′ , M ( ℓ t,a t ′ ) } t ′ ∈ [ t ] in any round t ∈ [ T ] where M is an ( ε, δ ) -DP mechanism.

Roughly speaking, the algorithm should not touch true losses, and it observes privatized losses only. Here, we adopt the widely-used Laplace mechanism (Dwork et al., 2014). Specifically, when data are bounded in [0 , 1] , adding noise drawn from Lap ( ε ) 7 to them ensures ε -DP. By adopting it, the observed loss is the true loss plus an i.i.d. sample from Lap ( ε ) . That is, this setup could be viewed as a specific way of generating heavy-tailed losses (i.e., bounded true loss + Laplace noise for privacy).

In the literature, Agarwal & Singh (2017) and Tossou & Dimitrakakis (2017) investigated the adversarial regime and proposed algorithms that achieve ˜ O ( √ KT ε ) worst-case regret (in expectation) with ε -LDP protection, which matches the lower bound (Garcelon et al., 2021) up to some log T factor and implies that their algorithms are (nearly) minimax-optimal. In the stochastic regime, Ren et al.

(2020) proposed privatized Upper Confidence Bound-based algorithms with O ( ∑ i :∆ i > 0 log T ε 2 ∆ i ) gapdependent regret and O ( √ KT log T ε ) worst-case regret. They also provided a matching gap-dependent lower bound, which scales as Ω( ∑ i :∆ i > 0 log T ε 2 ∆ i ) when ε ⩽ 1 .

In the BOBW setting, Zheng et al. (2020) showed that by privatizing the Tsallis-INF (an optimal algorithm in the (non-private) BOBW setting when losses are bounded) with Gaussian noise, they achieved O ( √ KT ε ) regret in the adversarial regime and O ( ∑ i :∆ i > 0 log( T ) log(1 /δ ) ε 2 ∆ i ) regret in the stochastic regime with approximate ( ε, δ ) -LDP protection, which is weaker than pure ε -LDP. And these regret bounds hold in expectation only.

In what follows, we present the BOBW guarantee provided by our Algorithm 2 when the losses are privatized by noise from Lap ( ε ) .

Recall that the privatized loss ℓ t,i = ℓ t,i = ℓ t,i + z , where ℓ t,i is bounded in [0 , 1] and z ∼ Lap ( ε ) , we have

where in the last inequality we utilize the fact that distribution Lap ( ε ) has second moment bounded by 1 /ε 2 .

Therefore, plugging u = O (1 /ε ) and v = 2 into Theorem 2 yields the following high-probability BOBWregret guarantee with (pure) ε -LDP protection: O ( √ KT log( K )(log T ) 3 ε ) worst-case regret in the adversarial regime and O ( K log( K )(log T ) 4 ε 2 ∆ ) regret in the stochastic regime with high probability. To the best of our knowledge, this is the first BOBW regret guarantee in MAB with pure LDP protection. We believe that our BOBW results can also be generalized to the case when true losses (to be protected) are heavy-tailed rather than bounded (by properly trimming the true losses before privatization). Related work by Tao et al. (2022) investigated solely the stochastic regime.

F Auxiliary Lemmas

Lemma 8. For any fixed M t,i > 0 , it holds almost surely that ∣ ∣ µ t,i -µ ′ t,i ∣ ∣ ⩽ u 1+ v ( M t,i ) -v .

Proof of Lemma 8. Expanding the definitions of µ t,i and µ ′ t,i , we have

Lemma 9 (Adapted from Lemma 19 in Wei & Luo (2018)) . In Algorithm 1, for any fixed λ ∈ (0 , 1) , let n i be the number of times the learning rate of arm i changes (and in fact increases) in total, i.e., η T +1 ,i = κ n i η 1 ,i = κ n i η . We have n i ⩽ log 2 (1 /λ ) and η t,i ⩽ e log 2 (1 /λ ) log T η, ∀ t ∈ [ T ] , i ∈ [ K ] .

Proof of Lemma 9. Let t 1 , . . . , t n i be the (ordered) rounds when the learning rate of action i changes, we have

Clearly, we have n i ⩽ log 2 (1 /λ ) and therefore η t,i ⩽ κ n i η 1 ,i ⩽ e log 2 (1 /λ ) log T η.

Lemma 10 (Bernstein's inequality for independent random variables (Vershynin, 2018)) . Let X 1 , . . . , X T be zero-mean independent random variables such that | X t | ⩽ b, ∀ t ∈ [ T ] for some fixed constant b and V T = ∑ T i =1 E [ ( X t ) 2 ] . Then, for any ζ > 0 , we have with probability at least 1 -ζ that,

Lemma 11 (Freedman's inequality for martingales, Lemma 3 in Rakhlin et al. (2011)) . Let X 1 , . . . , X T be a martingale difference sequence adapted to filtration F 1 ⊂ · · · ⊂ F T such that | X t | ⩽ b almost surely for some fixed constant b and V t = ∑ t s =1 E [ ( X s ) 2 ∣ ∣ F s ] . Then, for any ζ < 1 / ( e log T ) and T ⩾ 4 , we have with probability at least 1 -ζ that for any t ∈ [ T ] ,

Lemma 12 (Adaptive Freedman's inequality, Theorem 9 in Zimmert & Lattimore (2022)) . Let X 1 , . . . , X T be a martingale difference sequence adapted to filtration F 1 ⊂ · · · ⊂ F T such that E [ X t |F t ] is finite almost surely. Then, for any ζ > 0 , it holds with probability at least 1 -ζ that

where V T = ∑ T i =1 E [ ( X t ) 2 ∣ ∣ F t ] and U T = max { 1 , max t ∈ [ T ] X t } .

Lemma 13 (Theroem 5.1 of Chen et al. (2018)) . Let R 1 and R 2 be two distributions on X . If for some α ∈ [0 , 1] it holds that TV( R 1 ∥ R 2 ) ⩽ α 1 -α , then there exist two distributions on the same probability space G 1 and G 2 such that

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: [NA]

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work focuses on theoretical Machine Learning.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies multi-armed bandits (MAB) with heavy-tailed losses.\nIn the heavy-tailed bandits literature, two common assumptions are made about the losses: either a known upper bound $u$ on (1+$v$)-th raw moments (where both/either $v$ and/or $u$ are known) or truncated non-negativity (or non-positivity) assumption.\nThis paper considers the former and proposes a Best-of-Both-Worlds (BOBW) policy based on the Online Mirror Descent (OMD) and detect-switch framework.\nThe authors present high-probability near-optimal regret bounds that can provide meaningful expected regret analysis. \nThe main techniques used in their proofs rely on variants of Freedman's inequality (Lemma 11 and 12).\n\n- - -\nAfter rebuttal:\n\nThe proof for the modified algorithm seems correct, though it introduces an additional logarithmic factor. \nWhile I believe a better result could potentially be achieved using the general FTRL framework in BOBW research instead of the switch-detect framework, further improving the current analysis could be explored as a separate research direction.\n\nAs a result, I am changing my score from 3 to 6 (and the soundness rating from 1 to 3).\n\nSTRENGTHS:\n### Quality and Clarity\nThe main manuscript is clearly written and easy to follow.\n\n### Originality and Significance\nThis paper provides high-probability regret bounds for both stochastic and adversarial settings. \nThe proposed policy is novel as it utilizes both OMD and a switch-detect policy specifically for (stochastic) heavy-tailed bandits.\nAdditionally, the results imply the achievability of BOBW performance in a pure local differential privacy setting.\n\nWEAKNESSES:\n### Assumption on losses\nAlthough the authors claim that their assumption, which requires knowledge of both $v$ and $u$, relaxes the undesirable truncated non-negative losses considered in Huang et al. (2022) or non-positivity assumption in Genalti et al. (2024), I disagree with this claim.\nGenalti et al. (2024) showed that there is **no** $u,v$-adaptive algorithm that can achieve the lower bound for known $u,v$ setting.\nI believe that these assumptions cannot be directly compared to determine which is generally weaker **in practice**.\nHowever, the truncated assumption appears to be more challenging problem in this context.\n\n### Proof of Lemma 2 - step 2 (p. 17)\nIn the equation between (25) and (26), the authors use $\\bar{w} = \\rho /2$, which is **incorrect** since $\\rho = 2/w$ not $2/\\bar{w}$ as shown in Algorithm 1.\nTherefore, the results in (26) and (27) are not necessarily true.\nThis implies that the authors should modify the choice of $\\eta$, which was designed to cancel terms related to the results in (27). \nThis issue seems very critical and needs to be addressed, even if it might not change the order of regret.\n\n### Proof for adversarial setting (Alg. 2)\nBy definition, for $s \\leq t\\_{sw}-1$, none of the tests (6), (7), and (8) should be satisfied. \nThis implies that for all $i\\in [K]$, (6) should not be satisfied, i.e., for all $i \\in [K]$,\n$$\n\\begin{equation*}\n       | {\\frac{\\hat{L}\\_{s,i}}{s} - \\hat{\\mu}\\_{s,i}} | > 9u(\\cdot)^{v/(1+v)} + 1[i \\in A\\_s] \\text{Width}(s)/s + 1[i \\not\\in A\\_s] \\text{Width}(\\tau\\_i)/\\tau\\_i.\n\\end{equation*}\n$$\nHowever, equation (69) used exactly the opposite condition. \nTherefore, the arguments related to bounding Part A seem incorrect.\n\nAlso, what is $\\hat{L}\\_{t\\_{sw}-1}'$?\nIs it different from the cumulative importance-weighted estimate?\n\n### Stochastic results\nFrom (64), the result is of order $K\\log K \\log T (\\log(T/\\xi))^3$, not $K\\log K \\log T (\\log(T/\\xi))^2$ stated in Theorem 2.\n\nQUESTIONS:\n1. What is $\\hat{y}\\_t$ in (2)?\n\n1. What is $t\\_{quit}$ in (13)? (just $t\\_{sw}$?)\n\n2. I cannot understand why (55) holds with $\\tau\\$ terms inside when $t > \\tau$. To be specific $(\\tau\\_i / K)^{(1-v)/(1+v)}$ term.\n\n3. Why (59) holds for all $i \\in [K]$ instead of $i \\in A\\_t$? The last step used the test (5), where only $i\\in A\\_t$ satisfies $\\leq$.\n\n4. Can you explain more on (63)?\n\n5. Although Algorithm 2 just used fixed constant, $c\\_1 = 6$, why it should be defined as a variable? It seems not necessary.\n\nLIMITATIONS:\nI follow the whole proofs up to Appendix C.\n\n### Minor comments\n1. Step 1 in p. 16: in Algorithm 1, $\\bar{w}_1$ is not defined since some necessary terms like $\\eta_0$ and $\\hat{\\ell}_0$ are not defined. Therefore, you should define $\\bar{w}_1 = w_1$ first.\nAlso, the last inequality of step 1 holds only when $K \\leq T$. This is a very minor one, but it should be specified in somewhere in Lemma.\n\n2. Line 629: we first some -> we first fix some?\n\n3. Line 631: why $w\\_{s,i}$ is non-decreasing? It would be true only if $i$ is activated.\n\n4. unfinished log term appears in (48).\n\n5. It would be better to specify that Lemma 8 is applied to derive the second inequality in (50).\n\n6. It would be better to specify that the result of (62) is applied to derive the third inequality in (64).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

comments: SUMMARY:
This paper considers the bandit problem for heavy-tailed losses and proposes an algorithm that achieves a nearly tight high-probability regret bounds.
The proposed algorithm has a best-of-both-worlds guarantee, i.e., it achieves nearly tight bounds in both adversarial and stochastic settings.
The proposed approach is also shown to be useful in terms of local differential privacy.

STRENGTHS:
- This study bypasses the assumption of truncated non-negative losses, which was required in the previous study by Huang et al. (2022).
- This paper shows high-probability regret bounds, which is rare in the contexts of heavy-tailed bandits and best-of-both-worlds algorithms.
- Obtained regret upper bounds are almost tight.

WEAKNESSES:
- The proposed algorithm requires prior knowledge of the parameters $u$ and $v$. This is a weakness when compared to algorithms that are adaptive to these parameters, e.g., by Huang et al.(2022) and  Genalti et al. (2024).
- There is a lack of mention or analysis of intermediate settings between stochastic and adversarial settings, e.g., corrupted environments.
A number of best-of-both-worlds algorithms are also effective in these settings (e.g., (Lee et al., 2020), (Zimmert and Seldin, 2021), (Dann et al., 2023)).

QUESTIONS:
How tight is the bound of the proposed algorithm in terms of its dependence on $\log T$?
Are there any known comparable lower bound?

LIMITATIONS:
The limitations and potential negative societal impact are adequately addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

comments: SUMMARY:
This work studied the adversarial bandit problem with heavy-tailed distribution. It also studied the best-of-both-world setting. It relaxed the non-negative assumption and analyzed near-optimal algorithms.



===================

I would like to keep the score after reading the rebuttal.

STRENGTHS:
1. This work discussed the earlier literature in detail.
1. It highlighted the technical challenges.
1. Overall, I think this work provided a good set of results and explained them well.

WEAKNESSES:
1. Is it possible to bound the expected regret of the proposed algorithm? What is the limitation?
1. As mentioned in Section 2, there are two possible definitions of regret in the adversarial setting. The author(s) should clarify which definition is used in the related works.
1. A table comparing all results is appreciated.

QUESTIONS:
Please refer to the 'Weaknesses' part.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors consider a best-of-both-worlds multi-armed bandits problem where the losses are not bounded and instead are heavy-tailed. \nTo be precise, in the stochastic setting, losses are generated from fixed distributions.\nIn the oblivious adversarial setting, the losses are drawn from distributions that are generated arbitrarily and can change from one round to the next.\nIn both cases, these distributions can sample heavy-tailed losses, which are defined such that the $1 + v$^{th} moment of the losses are bounded by $u^{1 + v}$ for some constants $u > 0$ and $v \\in (0, 1]$. \nThis setting is more challenging than the standard BOBW approach because of the generation process of the losses, which can lead to negative unbounded losses but also because the performance is evaluated in terms of expected regret rather than in terms of pseudo-regret, which is a more challenging measure. \n\nThe authors propose and analyze an algorithm based on the FTRL framework. They discuss that simply using regularization with Tsallis entropy (which is the state of the art for the standard BOBW MAB problem) cannot handle large negative losses and instead rely on the log barrier regularizer, which is a standard approach to handle problems where the stability is difficult to bound.\nWhile this approach itself is sufficient to derive bounds in the adversarial regime, achieving BOBW bounds requires supplementary tricks:\n\nThey then use a detect-switch strategy to monitor whether the environment is stochastic, in which case the arms are sampled at a rate that ensures stochastic guarantees, and otherwise make a definitive switch to the adversarial regime and use the previously discussed FTRL with log-barrier regularization.\n\nSTRENGTHS:\nThe authors tackle a challenging problem of BOBW bandits with heavy tail losses and propose a solution that achieves near-optimal results in both the adversarial and the stochastic regime. \n\nThe proposed methods combine well-studied methods in the BOBW literature and this paper highlights another topic of interest for the FTRL with log barrier framework.\nThe authors provide a detailed analysis of their methods and provide a very detailed explanation of their choices of methods.\n\nWEAKNESSES:\nWhile the presented results are novel and interesting, the proposed method is suboptimal by several logarithmic factors both in the stochastic and in the adversarial framework. Both the detect-switch method and FTRL with log barrier are known to be suboptimal in the standard BOBW MAB problem with bounded losses, which means that improving upon the existing results might require a completely different approach.\n\nQUESTIONS:\nDo you think that the detect-switch framework is necessary to achieve these BOBW methods or whether some more straightforward method (like FTRL with 1/2 Tsallis-Inf for the standard BOBW MAB with bounded losses)?\n\nLIMITATIONS:\nThis work is purely theoretical and the limitations of the applicability of the results are properly detailed in the conditions of each theorem.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}