Taming Heavy-Tailed Losses in Adversarial Bandits and the Best-of-Both-Worlds Setting
Authors: Duo Cheng
Xingyu Zhou
Bo Ji
Venue: NeurIPS 2024 poster (2024)
Keywords: bandits
regret minimization
the best of both worlds
heavy tails
online learning
Primary area: bandits

Abstract: In this paper, we study the multi-armed bandit problem in the best-of-both-worlds (BOBW) setting with heavy-tailed losses, where the losses can be negative and unbounded but have $(1+v)$-th raw moments bounded by $u^{1+v}$ for some known $u>0$ and $v\in(0,1]$. Specifically, we consider the BOBW setting where the underlying environment could be either (oblivious) adversarial (i.e., the loss distribution can change arbitrarily over time) or stochastic (i.e., the loss distribution is fixed over time) and is unknown to the decision-maker a prior, and propose an algorithm that achieves a $T^{\frac{1}{1+v}}$-type worst-case (pseudo-)regret in the adversarial regime and a $\log T$-type gap-dependent regret in the stochastic regime, where $T$ is the time horizon. Compared to the state-of-the-art results, our algorithm offers stronger \emph{high-probability} regret guarantees rather than expected regret guarantees, and more importantly, relaxes a strong technical assumption on the loss distribution. This assumption is needed even for the weaker expected regret obtained in the literature and is generally hard to verify in practice. As a byproduct, relaxing this assumption leads to the first near-optimal regret result for heavy-tailed bandits with Huber contamination in the adversarial regime, in contrast to all previous works focused on the (easier) stochastic regime. Our result also implies a high-probability BOBW regret guarantee when the bounded true losses are protected with pure Local Differential Privacy (LDP), while the existing work ensures the (weaker) \emph{approximate} LDP with the regret bounds in expectation only.

Taming Heavy-Tailed Losses in Adversarial Bandits and the Best-of-Both-Worlee Setting
Duo Cheng
Xingyu Zhou Wayne State University xingyu. whouwwaye... edu
Bo Ji Virginia Tech joj jojivvt.r edu
Virginia Tech duocheng@vt.co edu
Abstract
In this paper, we study the multi aulti-armed bandits problem in the best-of-bothe both-foothwoddds (BOBW) setting with heavy-tailed losses, where the losses can be negative and unbounded but have (1 + u u)-th raw moments bounded by u 1.+ for some known 0 and (0, 1). Specifically, underlying environment can be either (oblivious) adversarial (i.c., the loss distri- bution can change arbitrarily over time) or stochastic (i.e., the loss distribution is fixed over time), which is unknown to the recision-maker a prior. We propose
consider the BOBW setting where the
algorithm and prove that it achieves a T T' -type worst-case (pseudo-)reege in the adversarial regime and a log T-type gap-dependent regret in the stochastic regime, where T is the time horizon. Compared to the otate-of-the-art results, algorithm offers stronger high-probabilityy regret guarantees (vs. expected regret guarantees), and more importantly relaxes strong technical assumption on the loss distribution, which is generally hard to verify in practice. As a byproduct, relaxing this assumption leads to the first near-optimal regret result for heavy-tailed bandits with Huber contamination in the adversarial regime (vs. the easier stochas- tie regime studied in all previous works). Our result also implies a high-probability BOBW regret guarantee when the bounded true losses are protected with pure Local Differential Privacy (LDP), while the existing work ensures the (weaker) approximate LDP with the regret bounds in expectation only.
Introduction
Consider the multi-armed bandits (MAB) problem (Auer et al., 2002a,b), which a useful framework for sequential decision-make under uncertainty and be formulated repeated interactions each of the total T rounds indexed by t, the
between the environment and a (learning) algorithm. algorithm plays an action a, from fixed set of K actions (assuming K & T). Simultaneously. the environment determines the losses of all actions 4 E R The algorithm observes and suffers the loss associated with a; (denoted by (,,a)) The goal of the algorithm is to minimize the cumulative loss over T rounds, or equivalently, to minimize the regrer, defined as the difference between its cumulative loss and that incurred by playing the best-fixed action (in hindsight) all the time. Without observing the losses of the other actions, the algorithm must "infer" the optimal action through interactions the fly, facing the well-known trade-off between exploration and exploitation.
Depending on how the losses are determined, MAB problem is typically studied in two regimes: 1) the stochastic regime, where the loss of each action is drawn from fixed (but unknown) distribution over time; 2) the adversarial regime, where the losses can be arbitrary (within some known class). Typically, the losses are assumed to have support fundamental limits have been well understood: 1) in the stochastic regime, a number of optimism-
bounded interval (c.g.. [0, 1J), and the
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

based algorithms achieve the @ ((vKT) worst-case (pseudo-)rese and @ ai م20 /o (1/A.) log T)
gap-dependent regret, where A, (formally defined in Section 2) is the sub-optimality gap between action and the optimal action (Lai & Robbins, 1985; Auer et al., 2002a; Agrawal & Goyal, 2017); 2) in the adversarial regime, the (VVTT) worst-case regret can be achieved by classic Online Learning algorithms, such as Follow-the wewww-theeeguuee tetee-egulaized-Leeeeeeee Neflo-tthe-eeturbed--eeee
(FTRL), Online-Mirron Mirror-Descent (OMD), and (FTPL) (Audibert & Bubeck, 2009; Lee et al., 2024).
Despite these progresses, the optimal algorithms in the two regimes fall into different frameworks (i.e.. optimism baased algorithms for the stochastic regime vs. Online Learning algorithms for the adversarial regime). Moreover, while the former ones enjoy a logarithmic regret (i.e., (log T)) the stochastic regime, even their Õ(vKT) worst-case guarantees no longer hold when the environment deviates from the stochastic regime with empirical evidence provided in Zimmert & Seldin (2021).
On the other hand, while Online Learning algorithms always preserve worst-case optimality, they could be too "conservative" to enjoy logarithmic regrets in stochastic environments.
These performance discrepancies motivated the study of the Best-of- -ort-oh-thth--- Worlds (BOBW) setting. That is, one single algorithm preserves the optimal worst-case regret in the adversarial regime and adapts to the stochastic regime with a logarithmic regret, without knowing the type of the regime in advance. Bubeck & Slivkins (2012) initiated the study by proposing a detect-switch framework, which preserves the optimal ((VTT) regret in the adversarial regime and enjoys O ((lo T)²K/A) regret in the stochastic regime, where A == min;, A; is the smallest sub-optimality gap. Under this framework, Auer & Chiang (2016) improved the gap-dependent term from K/A to Ees.so (1/A.).
Another line of work showed that without explicit detect-switch, OMD, originally designed for the adversarial regime, can automatically adapt to stochastic environments with logarithmic regret (Wei & Luo, 2018; Zimmert& & Seldin, 2021). In particular, Zimmert& & Seldin (2021) achieved optimal regrets in the BOBW setting. Following these works, the power of Online Learning algorithms towards BOBW has been extended to various setups (Ito, 2021; Ito et al., 2022: Kong et al., 2023; Ito & Takemura, 2023; Dann et al., 2023; Jin et al., 2024; Lee et al., 2024; Tsuchiya et al., 2024).
While the aforementioned works require bounded losses, real-world data from application domains such as finance (Cont, 2001) and imaging (Hamza & Krim, 2001) often exhibits heavy-tailed distribution. Intuitively, heavy-tailed losses make learning problems harder (compared to bounded losses) as they are "noisier" and "less informative" (Zhang et al., 2020). When losses are unbounded but have (1 + w)-th raw moment bounded by u for some known u > 0 and v E (0, 1]. Bubeck et al. (2013) showed @(uk +he T+ worst-case regret and θ (E.s.>o(u' (1/+1/v/ / /A,) log T) gap-dependent regret² in the stochastic regime by integrating robust mean estimators (e.g.. trimmed mean and median-of-means) into optimism-based algorithms.
To address heavy-tailed losses in the adversarial regime and BOBW setting, Huang al. (2022) showed that with calibrated adaptive loss trimming thresholds, FTRL with Tsallis entropy regularizer (Audibert & Bubeck, 2009) enjoys the optimal BOBW expected regrets under the "truncated non- negative losses" assumption (see Assumption 2). Without this strong assumption, it is unclear whether the near-optimal worst-case regret can still be achieved in the adversarial regime, let alone the BOBW setting. The key technical challenge here is that heavy-tailed losses can be both negative and unbounded, which is known to break the regret guarantees of the Online Learning algorithms. Extensive discussions and insights for our solution are provided in Section 3.
Given this challenge, the gap raises an interesting question: In heavy-tailed MAB, are there any fun- damental barriers to the worst-case optimality in the adversarial regime and the BOBW guarantees?
We offer positive answer to the above question, which implies that there is no such barrier. Our main contributions are as follows:
In the adversarial regime, we propose an OMD-based algorithm achieving the (near- Joptimll ÖluK TH= T pseudo-regret with high probability. Our approach relaxes the undesired "truncated non-negative losses" assumption, which is needed in the state-of-the-att results
'We use the standard notations O(-), (2)). and ((-). Those with tilde hide poly-log terms in 1' and K. We use log(-) to denote the natural logarithm (log, otherwise the base æ is explicitly specified as log, (-). 'In the stochastic heavy-tailed MAB, the gap-dependent logarithmic bounds omit some ((exp(Ivv) (1))) factors; this is also the case in all previous works (Bubeck et al., 2013; Tao al., 2022; Huang et al., 2022).

---

Table 1: Comparison with related results in heavy-tailed MABs. In this table, column "(u, e)-free" indicates whether the algorithm ensures the stated guarantee for unknown (u, w); -Assumption-2- free" indicates whether the stated guarantee holds without Assumption 2: "Sto." and "Adv." abbreviations for "Stochastic" and "Adversarial", respectively; "High-prob." indicates whether the stated expected bound is implied by some (stronger) high- high-probability bound.
Algorithm Lower bounds (Bubeck et al., 2013)
[a. Asumpliin-2rre
Regime
Expected Regrei
High-proh.
Sto/Ady
N/A
N/A
N/A
Sto.
Q
Σ
log
15,50
RobustUCB (Bubeck et al., 2013)
Oluk F5 T T
log T)
X
Sto.
0
Σ
(
log
15,30
HTIXP (Huang al., 2022)
Adv.
O 7 -
x
Sto.
( )
log
AdaTINF (Huang al., 2022)
Adv.
ti (KT)
AdaR-UCB (Genalii et 2024)
V
Sto.
O Wh Foury (1a0)) log
Σ
o
ToRT
un.INF INF (Chen et al., 2024)
v
X
Ads.
O (uk (uuk ri ((log T log
x
Sto.
0 ( log log "
UMD-LB-HH HT (Algorithm 1)
X
Ads:
Oluk (10
SAC-ET (Algorithm 2)
Adv.
Oluk log (logg)))
Sto.
o K 1x
even for the weaker expected regret guarantec (Huang et al., 2022). Relaxing it also allows us to obtain the near-optimal worst-case guarantee against the Huber contamination, which, to our best knowledge, was only studied for the stochastic regime in the literature. This suggests broader implications our approach.
On top of the above advance in the adversarial regime, by leveraging the detect-switch framework, we further extend the (near- Joptimal regret guarantees to the BOBW setting, Specifically, our algorithm preserves the optimal Ō(uK 1.. T regret in the adversarial regime and enjoys O(K ((2+1//AA)// log( (K) (log (logT))*) gap-dependent logarithmic regret in the stochastic regime, both with high probability, which implies that there is no fundamen- tal barrier achieving the BOBW guarantees when the loss distributions are heavy-tailed. This result also immediately imply the first nigh-probability BOBW regret guarantees with pure Local Differential Privacy (LDP) protection on the true losses, while the existing result ensures the weaker approximate LDP protection with expected regret guarantees only.
bichnique-wise, we leverage the inherent stronger stability of log-barrier to relax a strong technical assumption made in previous works and utilize the wereeasinn-leannnng trick (Lee et al., 2020) to obtain the stronger gigh-probability guarantee in the adversarial regime. Moreover, we adapt the detect-switch framework by Bubeck & Slivkins (2012), originally designed for BOBW in the bounded-loss case, to the heavy-tailed setup. The adaptation introduces non-trivial challenges in the analysis due the history-dependent trimmed estimator. In particular, to obtain the desired concentration rate in the adversarial regime, the proof does not follow its existing counterpart in the stochastic regime. Beyond addressing these challenges, we identify novel use (i.c.. handling history-dependent trimming martingale concentrations) of an adaptive variant of Freedman's 's inequality (originally proposed by Lee et al. (2020) and improved by Zimmert & Lattimore (2022) for sigabprobbbylity regret in adversarial bandits), which may be of independent interest.
We refer the readers to Table I for a summary of the most relevant results, in which we also include adaptive results on the case when u, v are unknown. We also present comprehensive discussion about related work, which is deferred to Appendix A due to the page limit.
3

---

2 Problem Setup
In this section, we formally introduce the problem setup and define needed notations. To formulate heavy-tailed losses, we let E,; denote the loss of action in round t, which is drawn from distribution Pt, satisfying the following assumption:
Assumption 1. The (1 + --)thh (raw) moments of losses (which have potentially unbounded support
in R) are bounded by w for some constants u > 0 and E (0, 1]. i.e., Pe..:~Pt.s [|6,,]] [ <
u u'+", Vt [ E [K]. where [n]] denotes set { ( n for any integer > 1.
In the heavy-tailed MAB problem, the (learning) algorithm and environment perform the following interactions repeatedly in round & 1,...,??
1. The algorithm samples action ae from [K] via de wy == (Wt.1, wr, in the proba- bility simplex Ω = {ar [0, [0 Ei=1 x(i)= = 11. i.e., action E [K] is sampled with probability Wt, The environment draws loss l,, Pt, for every action i e [K].
2. The algorithm observes f,,,, only; the losses of all other actions are unrevealed.
3. The algorithm determines Wr++1 based on the history (w),a) 21, fi, Wt, (t, a.
Remark All of our algorithms and their regret bounds allow moment order (1 + e) E (1, 2] only. That is, one may not obtain any regret guarantee by running our algorithms with >> 1. If the losses have higher-order moments (v > 1), one can simply run our algorithms with == 1 and obtaining the corresponding regret bounds (since bounded higher-order moments imply lower-order ones). Note that Bubeck et al. (2013) showed that for all v e 1, the lower bounds in terms of both worst-case regret and gap-dependent regret are the same the case of u = 1.
We assume that heavy tail parameters and v and time horizon T are known to the algorithm priori.' 3 The objective of the algorithm is to minimize the pseudo-regret Ry, defined as
T
Ry = Σ (1r.ac - M,,5")
where pr; == Pr, [d,,]] denotes the mean loss of action i E [K] in round E [T]. and i" E argmin; Σt Ht, denotes any best-fixed action in hindsight.
Depending on how loss distributions are determined, we further define the following two regimes:
Stochastic regime: For every action i E [K]. the loss distributions are identical in all rounds. That is, we have Pi, and i" E argmin;e[k] ((i). We also define A := mdi)= - (i(i) and A == min A.
Pr, = P(i), implying Mi.
μT, uui)
(Oblivious) Adversarial regime: All loss distributions are chosen arbitrarily (by some adversary, with the full knowledge of the algorithm) before the interaction begins. Our regret definition and adversarial model are also considered in Huang et al. (2022).
Remark 2. It is not hard to see that the stochastic regime is a special (and "easy") case of the adversarial regime. There are two differences between heavy-tailed setup and the bounded-loss setup in the adversarial bandits literature: 1) Typically in the (bounded-loss) adversarial regime, the losses are considered be deterministic rather than randomized (and this difference also leads to some new challenges in the analysis when we adopt the detect-switch framework from the bounded case (Bubeck& & Slivkins, 2012) as we will show later in Section 4.2), and 2) stronger notion of regret, defined as Rr == ΣT Ei= lr,,, min Σ l,,5, considered; this is stronger than pseudo-regret since [[]]] > E. However, when adapted to the heavy-tailed case, it is natural to still consider randomized losses and pseudo-regret as in the stochastic regime: The "easiness" of the regime boils down to how heavy-tailed distributions are chosen. Moreover, a low stronger regret depends not only on playing good actions (with low loss means), but also on the realization potentially unbounded losses, which loses the standard meaning in evaluating the algorithm. Therefore, when switching from stochastic to adversarial regime in heavy-tailed bandits, we keep pseudo-regret as the metric."
We summarize (negative) results from the literature when u, u are unknown in Appendix A. Starting from this point, we always refer "pseudo-regret" (Ar in Eq. (1)) as "regret" in both regimes.

---

Our goal is to design one single algorithm that can achieve the near-optimal Ouk 140 T worst- case regret in the adversarial regime and enjoys log T -type regret when the regime is stochastic, without being informed of the regime type in advance.
closely related work by Huang et al. (2022) studied the same BOBW setup (i.e., achieving BOBW guarantee when u, are known). They proposed an FTRL-based algorithm with Tsallis entropy regularizer and carefully-chosen history-dependent trimming threshold for loss magnitude control and showed O =o (1/= A regret in the adversarial regime, both of which guarantees rely heavily on a strong technical assumption:
/ log T) regret in the stochastic regime and (ukKTi T expectation and optimal. However, their regret
Assumption 2 (Truncated non-negative losses (Huang et al., 2022)). Given any fixed M > 0, the loss distributions of the optimal action i" satisfy Pe,,, [4,,5- [[ > MJ > 0, VI E (T).
In the following sections, we first show that by resorting to the log-barrier regularizer, we obtain the near-optimal regret bound in the adversarial regime without Assumption and naturally extend it to the stronger high-probability guarantee (Section 3). On top of that, we further adapt the detect-switch framework in Bubeck Slivkins (2012) and obtain high-pro probbbbbbbybitty bounds in BOBW (Section 4),
Additional Notations. For any round and action i, we let 4, == Liae it denote whether action i is pulled in round and Ne. E Is.s denote the number of times when action is pulled before the end of round We use l.. == (ee, < /We, to denote the (trimmed) IW estimate with respect to some threshold M, and T4, == Σ:=1
15=11f. fa.: I T( < Ba,)/Ne,
to denote the (trimmed) empirical average with respect to some threshold Br We also use 14, Be, p-Pr ([Ke. & Mr,3]] to denote the mean of the trimmed loss and Tu Σ Eo. to denote the cumulative IW estimate.
3 High-probability Near-optimal Regret in the Adversarial Regime
This section is dedicated to high-probability regret in the adversarial regime. We first present detailed discussions on why Assumption 2 is needed in Huang et al. (2022) and how we get rid (Section 3.1), followed by the description of our algorithm design and regret guarantee (Section 3.2).
3.1 Technical Challenges and Insights
The main technical challenge we encounter comes from the potentially-unboundee negative losses (even for regret bounds in expectation only). We illustrate this challenge using the previous work of Huang et al. (2022) as an example. Their algorithm is running standard FTRL with (1 + e)- Tsalis entropy regularizer over the trimmed loss estimate sequence f,,..., ......,, ey with respect to some trimming threshold (Ma,))e (T) to be introduced later in this subsection.
To bound the expected regret, Huang et al. (2022) first rewrite it as
T
E [
E
(we v²' 14 -1/3
+ E
[e
v" sK1
K
T
E
Wt, (He m4r
+
Σ (Wi.s
Me,s-)
+ E
(w
Bx)
(2)
where y¹ is the K-dim vector such that the i-th entry is one and all the others are zero.
The analyses begin with Part III. The desired upper bound Part III holds only under the well-known "stability condition" associated with Tsallis entropy (Jin et al., 2024, Lemma C.5.3):
(w, r Ê = (w,(( -ox (wr,:) -,, (le, & M.,) > -C(u,v),
where constant C(u, v) > 0 depends on u and u only and the learning rate n is chosen as u - +t While this condition is trivially satisfied when losses are non-negative, due potentially
'The values of thresholds may change from time to time in this work but will be made clear given the context.
5

---

Algorithm 1 OMD with log-barrier and increasing learning rates for heavy-tailed MABs (OND-LB-HT)
Input:
failure probability Ç, initial learning rate 77. trimming threshold (MM,seetteee
2: Define: learning rate increase factor &
e2/logT: log-barrier regularizer
0r(x) = - i-1 log(z(i))/mn.i Bregman divergence Do, (2, x') = (z(i)/z'( -
K 1+ truncated probability log(z(1)/<((i)) - 1)/ sr simplex truncation parameter À = 7' =2
simplex Q' = {ar E Ω xxi) > A/K, Vi E [K]}
3: Initialization: For every action é E [K]. define W1,; = P1, = 2K, Mn =
4: for t = 1: do
5: Take action a, sampled from wy and observe leaa,
6: Construct loss estimate E,, af, 1111 Ce,il & M, Muslee,s/een Vi [K]
7:
Calculate Wz+11 arg (zz Ca) + Dir (Ir, wr)
8:
for i E [K] do
9: if 1/W1+1,3 > Pt. then P141,1 = 2/ 1 11-1, T1,sK
10: else Pt+1,i = Pt.i, Me41,1 1,,2
end for
12: end for
negative losses, threshold M1, here is chosen to be @(( wr,)) 14t) (in particular, to fully "cancel" the (wr,i)= from le,)). Otherwise, negative losses break this condition whenever Wr,a; is very small.
While such a threshold suffices to bound Part I and Part III with worst-case optimality (and even in the BOBW setting), applying the analysis for Part to Part II leads to an upper bound of form El ) on Part II, which is potentially unbounded since We.s* can be very close to zero. However, with the help of Assumption 2, Part II itself is non-positive and hence can be ignored.
Summary. The key issue above is that, due to unbounded and negative losses, Eq. (3) results in threshold My, which scales with (wv (in particular, for = i*), rendering Part II hard to bound. Remark3 This issue may not be fixed by simply shifting all loss estimates to become positive and satisfy Eq. (3). Roughly speaking, the reason is that obtaining desired regret bounds relies heavily on the well-bounded (1 + v)-th moment of the losses (before shifting). However, to ensure positive losses, the needed shift is too "significant" and breaks the "nice" moment conditions.
To handle this issue and relax Assumption 2, we resort to the log- barrierri regularizer, which offers the standard regret guarantee with the following stability condition (Agarwal al., 2017):
Th, Se, -0.5.
Importantly, this condition itself already provides a wi, (rather than the previous (w,,,) 1- the in Eq. (3)) to "cancel" the (w,,;) from l,,,, meaning that we do not need any additional Wt,s contributed from threshold Me,5. As a result, we can choose a different My. that scales with KT rather than the previous (wr,;) प.न such that all three parts are bounded by Ö(u ( ) without Assumption 2.
We further adopt the gerreasing-leaming--re trick (Lee et al., 2020) together with an adaptive variant of Freedman's 's inequality (stated in Lemma 12) to obtain the stronger high-probability regret.
3.2 Algorithm Overview and Regret Guarantee
We now discuss the algorithm design and present the full pseudo-code in Algorithm 1. At high level, our algorithm is running standard OMD over loss sequence ?,.....& & with respect to some fixed threshold M = u (T/K)++> The key ingredients (to obtain high- high-probability regret) are the special learning rate schedule and probability simplex truncation (in OMD update) (Wei & Luo, 2018; Lee et al., 2020), which we briefly introduce below.
Increasing Learning Rates. We vectors {pe (Petre(T) to keep track of smallest sampling probabilities throughout the interaction. To be more specific, for every action i, if we+ is small that 1 1/ > Pt.; holds, we increase the learning rate by a factor of k. > and set Pt+1,1 2/we+1.5 (Line 9). Otherwise, we keep the learning rate unchanged and set Pt+1, Pt, (Line 10). In the analysis, such

---

schedule introduces some negative term in the upper bound, which cancels out positive term that could potentially be very large due to the variance of loss estimates (Lee al., 2020).
Probability Simplex Truncation, We perform the OMD update over the truncated probability simplex 52' == (ar Ω xx(i) 3 X/K, Vi [K]} (where À controls the degree of truncation), of which the purpose is to ensure that Wr, is always at least X/K and hence to control the variance of the loss estimates. The value of À here 7 T K 1म. adapted to the heavy-tailed case and differs from the original value 1/T for the bounded-loss case in Lee et al. (2020).
The regret guarantee of Algorithm 1 is formally stated below with full proof' presented in Appendix B. Theorem 1. In the adversarial regime, for any failure probability Ç. by choosing initial learning rate n (40M (T'( log (8KT/C)) and trimming threshold Me,; u (T/K) ािं, Algorithm I ensures that with probability at least 1- - Ç. RT = O(uK 1- T (log Log(T/(), further choosing Ç = 1/T, Algorithm I ensures that IE: [RT] = O T The ))
Remark 4. Removing Assumption is crucial to obtain the first and near-optimal worst-case regret in heavy-tailed MAB when the feedback could be contaminated by the Huber model in the adversarial regime, in contrast to all previous works that study the (easier) stochastic regime (Guan et al., 2020; Agrawal et al., 2024; Wu et al., 2024). We provide all details in Appendix D.
4 High-probability Regrets in the Best-of-Both. Worldd Setting
With Algorithm 1 achieving high-probee probabbbbbbt optimal regret the adversarial regime, we further leverage the detect-switch framework proposed by Bubeck & Slivkins (2012) named SAO to achieve high-probabilityy bounds in the BOBW setting. We first present the BOBW guarantee, followed by an algorithm overview and analysis sketch. The complete proofs are provided in Appendix C.
Theorem 2. In the udversarial regime, for any failure probability ç, by choosing constant ei 6. Algorithm 2 ensures that with probability at least 1 -.
RT = "K JuKiiTTIte (log (log(B/()))) ((gg()(())))))))(3))))
which (by further choosing Ç 1/T) implies that E [Rr] = O (uK (uK (log In the stochastic regime, Algorithmm 2 ensures that with probability at least 1 -,
RT = O K log (og(()(( /A) log(T) (log(T/<)))),
which implies that E [Ry] = O (K (u
Remark 5. bigh-probbbility bounds also are powerful tools to handle adaptive adversaries (who determine the current distribution Pt,; based on past actions 01,...,, at-1). Following the literature (Audibert & Bubeck, 2010; Lee et al., 2020; Zimmert & Lattimore, 2022), based on our high- probability bounds against oblivious adversaries, one may further derive both high-probability and expected regret bounds against adaptive adversaries, which we leave as future investigation.
Remark 6. This theorem immediately implies a (iigh-probbbilit)) BOBW regret guarantee when the losses are bounded (in [O, 1)) and protected with s-Local Differential Privacy (LDP) via showing that the privatized losses (with Laplacian noise) have second moments bounded by O (Agarwal & Singh, 2017; Tossou & Dimitrakakis, 2017: Zheng et al., 2020: Ren et al., 2020). To the best of our knowledge, this is the first result showing (high probability) BOBW regret guarantee with pure LDP protection, while the of result (Zheng al., 2020) ensures (the weaker) approximate LDP protection. and only expected regret bounds are provided. Full details are given in Appendix E.
4.1 Algorithm Overview
Our algorithm design follows from the detect-switch framework by Bubeck & Slivkins (2012). The high-level idea is to keep performing statistical tests (to "identify" the environment) and carefully maintain the sampling distribution over all arms. Once some certain test fails (which implies that the environment unlikely stochastic), we switch to Algorithm 1 and run it over the remaining rounds.
In each round after playing action a sampled from distribution we, if any active action A, satisfies the test in Eq. (5), it is deactivated, and we T; and qi to store the round and the sampling
7

---

Algorithm 2 SAO for heavy-tailed MABs (SAO-HT)
1: Input: failure probability Ç; constant c₁
(1/K)
No. 170
2: Define: 3 127"Klog(" M,,
BA u [og(21773)
(log(lo((T)/())
(1)th(() = 12u +i प (3og( 3
3: Initialization: Play each action once; for every E le] wK+11 K. To T; set Ax = [K]]
4: for K + .,...., I do
5: Sample and play action ~ wy; observe f,
6: for E 1,-1 do
7: if
Ta,1- - min Tai > e Width( jEAc-1
(5)
8:
then A, = Ar- and qi Wt,i
9:
end if
10:
end for
11:
if any the three conditions holds then run Algorithm for the remaining rounds, let t&w = t, wtis T2 for every i Ar:
and let q:
12:
[og((/C)
I + E A,)
Width(()
3i E [K] such that [Ta//t - Tee
> 9m
N₁,1
Width(!)
+ I(i Φ Ac}
(6)
3i & A, such that (Ie,; min Lts)/t > (cr + ()dth - 1),
(7)
jEAr
Bi & A, such that (Le, min
Lr < (c1 4)Width(t)/r
(8)
jEA.
end if
11+1,1 Ii $ ())gn/(++))+++++++) A. (1- (1= /AAl.iie [K]
15: end for
16: q:
WT,i T3 = T Ar
probability when it is deactivated, respectively (Line 8). After that, the algorithm performs tests in Eqs. (6)-(8) for environment identification. If any of them is satisfied, the procedure is terminated, and we instcad run Algorithm over the remaining rounds. We use f.w to denote the round when the algorithm switch happens. Otherwise, update the distribution W4+1 for the next round in Line 14. To make notations and analyses well-defined, we deactivate all remaining arms in At,, when the algorithm switch happens.
4.2 Analysis Sketch
In this subsection, we present the key steps of the regret analysis the proof Theorem 2.
Before diving into the specific regime, we first derive a set of concentration results (good events). Informally, all of the following hold simultaneously with high probability in all rounds E [T]:
Width(!) + I(i & A.)
Width(t)
Le,s
Σ
Mas
(9)
T;
=
u
(10)
No
(No, I
N₂
δ (g:T. (1 + log t))
(11)
8

---

where Width(() KinKKieTTe (log(8/C) set1. All analyses below conditioned on these good events, and we omit "with high probability" in the arguments.
Remark 7. Our main non-trivial adaptation deviating from the case with bounded losses in Bubeck & Slivkins (2012) the good event Eq. (10) due to jointly randomized and heavy-tailed losses. In particular, we need the concentration result of trimmed mean [br,; specified in Eq. (10) in both regimes. In the stochastic regime, it been shown Bubeck et (2013). We need Eq. (10) adversarial regime (while Bubeck & Slivkins (2012) does not) as losses are deterministic therein. However, with heavy tails, the proof of Eq. (10) in the adversarial regime does not follow straightforwardly from the stochastic regime. More technical details are presented later in Remark 8. Nonetheless, this matter is specific to the nature of the trimmed estimator we use, which essentially could be replaced with any other estimator, long as the concentration rate is preserved. However, is unclear whether other estimators can handle non-identical distributions.
4.2.1 Analysis Overview in the Stochastic Regime
Step Showing that tests Eqs. (6)-(8 are never satisfied. We show by good events that tests in Eqs. (6)-(8) are never satisfied, implying that we never switch to Algorithm 1.
Step 2: Building the connection between A; and тj. From tests in Eqs. (7) and (8), can show that for every suboptimal action with A; > its sub-optimality gap A; = Ō((K/T:) (..).
Intuitively, an action with a smaller sub-optimality gap stays active for longer time.
Step 3: Bounding the total regret. By the definition of pseudo-regret, now have
RT
Σ N,, = Ō( Σ
A.TT) = Ō Σ Am.
(12)
:A:>0
:A::>0
We complete the proof by showing EtA, & (1/1) Ollog K).
4.2.2 Analysis Overview in the Adversarial Regime
In the adversarial regime, whenever we switch to Algorithm 1, it provides Ō(u(T - -.m) (high-probability) regret guarantee for the remaining (T (w) rounds. Therefore, it suffices to show that the cumulative regret before the switch is )x((t.))== K
In our analysis, we trivially bound the regret in the single round lxw by 2u, and it remains to show that the regret in the first (tsw 1) rounds is Ö(u(tsw 1) to K 2tf). which is explained in the following.
Step 1: Regret decomposition. We first get an regret upper bound in terms of it =
argmin;e(k] E.=1 Ma,i for / = lxw -
by
fw
-==l -1 Maiihe
Lim-1d
Dew f5m-1,6 1,8
Lee,, 1,72
Σ
Ma
Nim-1,
Nom-1,
Low
1gw
Part A
Part
bw
Pre- Ti
T
Harti
(13)
and then we bound Parts A, B, and C, separately.
Step 2: Bounding Part A. We rewrite Part A Part A
Σ Mostos Rem-1,1
New 1.
(De.- Lem-1.)) where the first term the right-hand side is O
(og(315))
due
ta
New Now--i.)
to Eq. (10) and the second term is O (B/C) Noo
Widthh (f.. due to test in Eq. (6).
T
Remark Due to heavy tails, the losses are trimmed by some history-dependen threshold B (which depends on the number of pulls N₁,) for rate-optimal concentration in good event (10). To

---

show this in the stochastic regime, one can treat the observed losses from one action as i.idd samples via the "reward tape/table" argument (Slivkins, 2019) and apply Bernstein's inequality for every fixed Ne.; (and set the uniform upper bound as Br.; associated with it) as in Bubeck et al. (2013). However, one cannot simply follow the same path in the adversarial regime, since the distributions are no longer identical, and has to follow the martingale-
analysis (Agarwal et al., 2021, Lemma 6.2).
Now one may readily see the issue: The desired uniform upper bound B,, c determined on-the-fly. while the standard Freedman's 's inequality for martingales (e.g., Lemma 11) requires a fixed uniform upper bound. To close this gap, we again exploit an adaptive variant of Freedman's inequality by Zimmert & Lattimore (2022) (Lemma 12; which was originally proposed for a totally different use, namely, obtaining high-probabilin bounds in adversarial bandits), in which we can replace the fixed uniform upper bound with the largest realization, satisfying our need perfectly.
Step 3: Bounding Part B. By considering two disjoint cases (i.e., action i E At or not), we can show Part B = Wiath(t,--)) using the tests in Eqs. (5) and (7). 7-1
Step 4: Bounding Part C. The good event in Eq. (9) simply implies Part C O (Width(4sw 1))-
Step 5: Putting all pieces together. Combing Steps 1-4 and the good event in Eq. (11) yields
Width - 11
Mssor
Ms
0
N 1,4
u (N
Width((!w 1)
Ti
K
0
u
(N₂ 1,1)
140 + 1
+ (lsw 1) 1) gi
i=1
K
1
(tyw
1)
+0
Width (1s -1)
Ti
1=1
= Ōuk - (lsw - 1) 11)
(14)
5
Conclusion
In this paper, we show that there is indeed no fundamental barrier to achieving the BOBW guarantee in heavy-tailed MAB by relaxing a strong, hard-to-verify technical assumption on the loss distributions of the optimal action needed for the stateoof-the-ar results. We further leverage the increasing mncreasing-eearning rates trick and the detect-switch framework to achieve the stronger high-probability guarantees. Our results also imply the first and near-optimal regret in the adversarial regime where the feedback could be contaminated in the Huber model, and the tigh-bpbbbbblityy BOBW regret guarantee when losses are bounded and protected with pure LDP, while the state- state-of-the- of-the-art result only ensures the weaker approximate LDP protection with regret guarantces in expectation.
One follow-up question is whether the gap-dependent term K (A)-1// can be improved to the refined En - which is achieved only in the stochastic regime (Bubeck et al., 2013). Under the detect-switch framework, we tend to believe this is possible by adapting more sophisticated tests designed in Auer & Chiang (2016) for the bounded-loss case, where the gap dependency is improved from K/A A to E...>o A), although a higher computational complexity is expected.
It will also be interesting to understand whether canonical Online Learning algorithms (i.c., without explicit detection and switch) provably enjoy BOBW guarantees in heavy-tailed MAB (and if so, whether a refined gap dependency can be achieved). In other words, do heavy tails break the implicit adaption of Online Learning algorithms the stochastic regime (in the worst case)? One promising direction is to still utilize log-barries regularizer together with potentially more advance learning rate and/or trimming threshold design. Notably, one unique advantage of it over the detect-switch framework is that it typically directly extends the BOBW regret guarantee to the corrupted regime, which an intermediate regime that smoothly extrapolates between the purely adversarial and stochastic regime (Zimmert & Seldin, 2021).
10

---

Acknowledgements
We thank anonymous paper reviewers for their insightful feedback, especially reviewer q85G, who carefully read the proof and pointed out some gaps from the analysis in an earlier version of the manuscript. This work is supported in part by the NSF grants under CNS-2312833, CNS-2312835, and CNS-2153220, the Commonwealth Cyber Initiative (CCI), and Nokia Corporation.
References
Agarwal, A., Luo, H., Neyshabur, B., and Schapire, R. E. Corralling a band of bandit algorithms. In Conference on Learning Theory, pp. 12-38. PMLR, 2017.
Agarwal, A., Jiang, N., Kakade, S. M., and Sun, W. Reinforcement learning: Theory and algorithms. 2021. URL https: https////hthhohtottreooooeo github. io/.
Agarwal, N. and Singh, K.
The price of differential privacy for online learning. In International Conference on Machine Learning, pp. 32-40. PMLR, 2017.
Agrawal, S. and Goyal, N. Near-optimal regret bounds for thompson sampling. Journal of the ACM (JACM), 64(5);1-22) 2017.
Agrawal, S., Juneja, S. K., and Koolen, W. M. Regret minimization in heavy-tailed bandits. In Conference on Learning Theory, pp. 26-62. PMLR, 2021.
Agrawal, S., Mathieu, T., Basu, D., and Maillard, O.-A. Crimed: Lower and upper bounds on regret for bandits with unbounded stochastic corruption. In International Conference on Algorithmic Learning Theory, pp. 1441 PMLR, 2024.
Ashutosh, K., Nair, J., Kagrecha, A., and Jagannathan, K. Bandit algorithms: Letting of regret for statistical robustness. In International Conference Artificial Intelligence and Statistics, pp. 622-630. PMLR, 2021.
of logarithmic
Audibert, J.-Y. and Bubeck, S. Minimax policies for adversarial and stochastic bandits. In COLT, pp. 217-226, 2009.
Audibert, J.-Y. and Bubeck, S. Regret bounds and minimax policies under partial monitoring. The Journal of Machine Learning Research, 11:2785-288 2010.
Auer, P. and Chiang, C.-K. An algorithm with nearly optimal pseudo-regret for both stochastic and adversarial bandits. In Conference on Learning Theory, pp. 116-120. PMLR, 2016.
Auer, P., Cesa-Bianchi, N., and Fischer, Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235-256, 2002a.
Auer, P., Cesa-Bianchi, N., Freund, Y., and Schapire, R. E. The nonstochastic multiarmed bandit problem. SIAM journal on computing, 32(1):48-77, 2002b.
Bubeck, S. and Slivkins, A. The best of both worlds: Stochastic and adversarial bandits. In Conference Learning Theory, pp. 42-1. JMLR Workshop and Conference Proceedings, 2012.
Bubeck, S., Cesa-Bianchi, N., and Lugosi, G. Bandits with heavy tail. IEEE Transactions on Information Theory, 59(1 (1 1):7711 17717, 2013.
Chen, M., Gao, C., and Ren, Z. Robust covariance and scatter matrix estimation under huber's contamination model. The Annals of Statistics, 46(5):1932-1960, 2018.
Chen, Y., Huang, J.. Dai, Y., and Huang, L. uniinf: www.of-oooooooohhoe algorithm for parameter-free taavy- mabs. arXiv preprint arXiv:2410.03284 2024.
Cont, R. Empirical properties of asset returns: stylized facts and statistical issues. Quantitative finance 1(2):223, 2001.
Dann, C., Wei, C.-Y., and Zimmert, J. blackbox approach to best of both worlds in bandits and beyond. In The Thirty Sixth Annual Conference on Learning Theory, pp. 5503-5570, PMLR, 2023,
11

---

Dwork, C., Roth, A., et al. The algorithmic foundations of differential privacy. Foundations and Trends@ Theoretical Computer Science, 9(3 (33-4):2114400 2014.
Foster, D. J., Gentile, C., Mohri, M., and Zimmert, J. Adapting to misspecification in contextual bandits. arXiv preprint 1rXiv:2107.05745. 2021.
Garcelon, E., Perchet, V., Fike-Burke, C., and Pirotta, M. Local differential privacy for regret minimization in reinforcement learning. Advances in Neural Information Processing Systems, 34: 10561-10573, 2021.
Genalti, G., Marsigli, L.,, Gatti, N., and Metelli, A. M. (c, -adaptive regret minimization in heavy-tailed bandits, 2024.
Guan, Z., Ji, K., Bucci Jr, D.J., Hu, T. Y., Palombo, J., Liston, M., and Liang, Y. Robust stochastic bandit algorithms under probabilistic unbounded adversarial attack. In Proceedings of the aaai conference on artificial intelligence, volume 34, pp. 4036-4043, 2020.
Hamza, A. B. and Krim, H. Image denoising: A nonlinear robust statistical approach. IEEE transactions on signal processing, 40)1333045333 2001.
Hsu, and Sabato, Heavy-tailed regression with a generalized median-of-means. International Conference on Machine Learning, pp. 37-45. PMLR, 2014.
Huang. J., Dai, Y., and Huang. L. Adaptive best-of-both, boot----ott-worrds algorithm for heavy-tailed multi- armed bandits. In international conference machine learning, pp. 9173-9200, PMLR, 2022.
Huang, J., Zhong, H., Wang. L., and Yang, L. Tackling heavy-tailed rewards in reinforcement learning with function approximation: Minimax optimal and instance-dependent regret bounds. Advances in Neural Information Processing Systems, 36, 2024.
Huber, P. J. Robust statistical procedures. SIAM, 1996.
Ito, S. Parameter-free multi-armed bandit algorithms with hybrid data-dependent regret bounds. In Conference on Learning Theory, pp. 2552-2583, PMLR, 2021.
Ito, S. and Takemura, K. thoe-ol-hhre linear bandit algorithm with variance-adaptive regret bounds. In The Thirty Sixth Annual Conference on Learning Theory, pp. 2653-26777 PMLR, 2023.
Ito, S., Tsuchiya, T., and Honda, J. Adversarially robust multi-armed bandit algorithm with variance- dependent regret bounds. Conference on Learning Theory, pp. 1421-1422. PMLR, 2022.
Jin, C., Jin, T., Luo, H., Sra, S., and Yu, T. Learning adversarial markov decision processes with bandit feedback and unknown transition. In International Conference on Machine Learning, pp. 4860-4869. PMLR, 2020.
Jin, T. and Luo, H. Simultaneously learning stochastic and adversarial episodic mdps with known transition. Advances in neural information processing systems, 33:16557-16566, 2020.
Jin, T., Huang, L., and Luo, H. The best both worlds: stochastic and adversarial episodic mdps with unknown transition. Advances in Neural Information Processing Systems, 34:20491-20502, 2021.
Jin, T., Liu, and Luo, H. Improved with general regularizers and multiple optimal arms. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
guarantees for multi-armed bandits: Ftrl
Jin, T., Liu, J., and Luo, H. Improved bott-of-bohh--oe wooth-wrrld guarantees for multi-armed bandits: Ftrl with general regularizers and multiple optimal arms. Advances Neural Information Processing Systems, 36, 2024.
Kato, M. and Ito, S. Best-of-both-wont linear contextual bandits. arXiv preprint arXiv:2312.1666 164.9 2023.
12

---

Kong, F., Zhao, C., and Li, S. Best-of Dest-of-three-wor three-herrddss analysis for linear bandits with follow-the- regularized- leader algorithm. In The Thirty Sixth Annual Conference on Learning Theory, pp. 657-673. PMLR, 2023.
Kuroki, Y., Rumi, A.. Tsuchiya, T., Vitale, F., and Cesa-Bianchi, N. cest-of-bott-wood algorithms for linear contextual bandits. arXiv preprint arXiv:2312.15433, 2023.
Lai, L. and Robbins, H. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics, 6(1):4-22, 1985.
Lattimore, T. and Szepesvári, C. Bandit algorithms. Cambridge University Press, 2020.
Lec, C.-W., Luo, H., Wei, C.-Y., and Zhang. M. Bias no more: bigh-probability data-dependens regret bounds for adversarial bandits and mdps. Advances in neural information processing systems, 33: 15522-15533, 2020.
Lee, C.-W., Luo, H., Wei, C.-Y., Zhang, M., and Zhang, X. Achieving instance-optimallty and minimax- optimalyy in stochastic and adversarial linear bandits simultancously. In International Conference Machine Learning, pp. 6142-6157 PMLR, 2021.
Lee, J., Honda, J., Ito, S., and Oh, M.-h. Follow-the- follow-the-perturbeed- peree-prer--dedddeeeee with fr (ce chet-type tail distributions: Optimality in adversarial bandits and bott--o-botthworldds arXiv preprint arXiv:2403.00 /334, 134, 2024.
Lee, K. and Lim, Minimax optimal bandits for heavy tail rewards. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Li, L.-F., Zhao, P., and Zhou, Z.-H. Improved algorithm for adversarial linear mixture mdps with bandit feedback and unknown transition. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS), 2024.
Lu, S., Wang. G., Hu, Y., and Zhang, L. Optimal algorithms for lipschitz bandits with heavy- rewards. In international conference machine learning, pp. 4154 PMLR, 2019.
Neu, G. Explore no more: Improved high-probability regret bounds for non-stochastic bandits. Advances in Neural Information Processing Systems, 28, 2015.
Rakhlin, A., Shamir, O., and Sridharan, K. Making gradient descent optimal for strongly convex stochastic optimization. arXiv preprint arXiv: 109.5647, 2011.
Ray Chowdhury, S. and Gopalan, A. Bayesian optimization under heavy-tailed payoffs. Advances in Neural Information Processing Systems, 32, 2019.
Ren, W., Zhou, X., Liu, J., and Shroff, N. B. Multi-armed bandits with local differential privacy. arXiv preprint arXiv:2007.0311 2020.
Shao, H., Yu, X., King, I., and Lyu, M. Almost optimal algorithms for linear stochastic bandits with heavy-tailed payoffs. Advances in Neural Information Processing Systems, 31, 2018.
Slivkins, A. Introduction to multi-armed bandits. arXiv preprint arXiv:1904.07272. 2019.
Tao, Y., Wu, Y., Zhao, P., and Wang, D. Optimal rates of (locally) differentially private heavy-tailed multi-armed bandits. In International Conference on Artificial Intelligence and Statistics, pp. 1546-1574. PMLR, 2022.
Tossou, and Dimitrakakis, C. Achieving privacy the adversarial multi-armed bandit. In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
Tsuchiya, T., Ito, S., and Honda, J. Sabbilit--eenltt--daapae follow-ther hewas-tterrgguliieee-leeeee Sparsity, game-dependency. and best-of-both-worlde Advances in Neural Information Processing Systems, 36, 2024.
Vershynin, R. dime-diin probability: An introduction with applications in data science, volume 47, Cambridge university press, 2018.
13

---

Wei, C.-Y. and Luo, H. More adaptive algorithms for adversarial bandits, In Conference On Learning Theory, pp. 1263-1291. PMLR, 2018.
Wu, Y. Zhou, X., Tao, Y., and Wang, D. On private and robust bandits. Advances in Neural Information Processing Systems, 36, 2024.
Xue, B., Wang, G., Wang, Y., and Zhang. L. Nearly optimal regret for stochastic linear bandits with heavy-tailed payoffs. In Proceedings of the Twenty-Ning International Conference on International Joint Conferences on Artificial Intelligence, pp. 2936 2942, 2021.
Zhang. J., Karimireddy, S. P., Veit, A., Kim, S., Reddi, S., Kumar, S., and Sra, S. Why are adaptive methods good for attention models? Advances in Neural Information Processing Systems, 33: 15383-15393, 2020.
Zheng, K., Cai, T., Huang, W., Li, Z.,, and Wang. L. Locally differentially private (contextual) bandits learning. Advances in Neural Information Processing Systems, 33: 13:0000-11310, 2020.
Zhong, H., Huang, J., Yang, L., and Wang, L. Breaking the moments condition barrier: No-regret algorithm for bandits with super heavy-tailed payoffs. Advances in Neural Information Processing Systems, 34:15710-155 15720, 2021.
Zhuang, V. and Sui, Y. No-regret reinforcement learning with heavy-tailed rewards. In International Conference on Artificial Intelligence and Statistics, pp. 3385-3393. PMLR, 2021.
Zimmert, J. and Lattimore, T. Return of the bias: Almost minimax optimal high probability bounds for adversarial linear bandits. In Conference on Learning Theory, pp. 3285-3312. PMLR, 2022.
Zimmert, and Seldin, Y. Tsallis-inf: An optimal algorithm for stochastic and adversarial bandits. Journal of Machine Learning Research, 22(28):1 1-49, 2021.
14

---

A Related Work
In this section, we give a comprehensive discussion on the related work.
Regret Minimization in the BOBW Setting (with Bounded Losses). There are mainly two different approaches towards enjoying BOBW regret guarantees. The first approach is the detect- switch framework initially proposed by Bubeck & Slivkins (2012). The framework runs in two-stage manner: In the first stage, the algorithm performs some tests in every round and carefully maintain the sampling distribution over all actions. Once any of the tests fails, it runs any off-the-shelf algorithm purely for the adversarial regime in the remaining rounds and inherits the regret guarantee of that algorithm. Bubeck & Slivkins (2012) proposed an algorithm achieving near-optimal regret in the adversarial regime and O (fog 2)2K/2) A) regret in the stochastic regime, both with high-probability. Auer & Chiang (2016) further refined the tests and improved the gap-dependent term to the optimal Ci:A,>0(1 A.). This framework has been generalized to setups including linear bandits (Lee et al., 2021) and bandits with feedback graph (Kong al., 2023).
Somewhat surprisingly, since the work by Wei & Luo (2018), there have been a large group of papers showing that Online Learning algorithms (e.g., OMD), originally for the adversarial regime, implicit adapts to stochastic environments and enjoys logarithmic regrets. BOBW guarantees of Online Learning algorithms have been established in quite broad setups including MAB (Wei & Luo, 2018; Zimmert& & Seldin, 2021; Jin et al., 2023). (contextual) linear bandits (Ito & Takemura, 2023; Kong al., 2023; Kuroki et al., 2023; Kato & Ito, 2023), and (tabular) Markov Decision Processes (MDPs) (Jin & Luo, 2020; Jin et al., 2021). One desirable advantage of this approach is that, thanks to the flexibility of Online Learning framework, it is possible to additionally enjoy some other adaptive regret bounds on top of the standard BOBW guarantee (Ito, 2021; Tsuchiya et al., 2024).
Regret Minimization in Heavy-tailed Stochastic Environments. There large body of works focusing on regret minimization in the stochastic environment where the losses are potentially heavy- tailed (Bubeck et al., 2013; Shao et al., 2018; Lu et al., 2019; Ray Chowdhury & Gopalan, 2019; Zhong et al., 2021; Agrawal et al., 2021; Xue et al., 2021; Lee & Lim, 2022; Zhuang & Sui, 2021;
Huang et al., 2024). The general recipe of the algorithm design is to derive concentration results for the loss mean estimation with robust estimators (e.g.. median-of-means (Hsu & Sabato, 2014), truncated/rimmed estimator (Bubeck et al., 2013), and Huber estimator (Huber, 1996)), and then integrate them with celebrated optimism- based algorithms.
bigh-probbbiility Regrets in Adversarial MAB and the BOBW Setting, The central piece towards gigb-probability bounds in adversarial MAB lies in balancing between the variance of the loss estimator and the total regret. In general, there are three types of approaches in the literature: 1) adding negative bonus to the Importance- Weighttd (IW) estimates (Auer et al., 2002b; Zimmert & Lattimore, 2022). 2) utilizing log-barrier regularizer together with increasing learning rates (Lee et al., 2020), and 3) replacing the IW estimator with the implicit-exploraaiii While the IX estimator enables simplified analysis and is thus widely-used (e.g., Jin et al. (2020); Li et al. (2024)), it heavily exploits the non-negativity of the losses, and it is unclear to us how to adapt it to the case of potentially-negatii
(IX) estimator (Neu, 2015).
losses. For comparison between the first two approaches, readers are referred to Foster et al. (2021) and Zimmert & Lattimore (2022). The detect-switch framework by Bubeck & Slivkins (2012) naturally provides high-proo siosaprobbbility bounds in BOBW, whenever subroutine algorithm achieving high- probabiibbby regret in the adversarial regime (e.g.. EXP3.P (Auer al., 2002b)) integrated in a plug-am and-play manner. While the Online Learning framework naturally provides BOBW guarantee via elegant analysis in expectation (Zimmert & Seldin, 2021), it is unclear how to adapt for high-probability guarantees.
Claptive/Parameter-Fee Heavy-tailed MAB. Throughout this work, we assume that the heavy tail parameters u and are known to the learning algorithm. In this case, Bubeck et al. (2013) showed min- imax lower bound 1ur K Te and gap-dependent lower bound S (Et., (u/ /// "(og T)) using stochastic environments, so these lower bounds also apply to the adversarial regime. However, in practice one may not know the values of u, v exactly in advance, therefore it is natural to ask whether it is possible to enjoy the same regret rate as if u, were known, which we refer to as "adaptation (to heavy tails) for free". On the negative side, Genalti et al. (2024) showed that there is no such an algorithm that can enjoy TF K )मल) worst-case regret for all unknown u, u in the stochastic regime. In terms of gap-dependent regret, similar "no adaptation for free" effect has been shown in earlier work by Ashutosh al. (2021). On the positive side, in the same work by
15

---

Genalti et al. (2024), it is shown that, somewhat surprisingly, with the same "truncated non-negative losses" assumption (Assumption 2), a UCB-based algorithm proposed by them enjoys "adaptation for free" in terms of both minimax and gap-dependent regrets in the stochastic regime. With the same assumption in the adversarial regime, Huang al. (2022) proposed variant of FTRL., which achieves the Ouu T + vKT) worst-case regret guarantee. In a very recent work by Chen al. (2024), FTRL-based algorithm named uni uNi shown enjoy the BOBW guarantee even when u. and both unknown, although the gap dependency of their logarithmic regret is K/A1/², and still Assumption 2 is needed. Exploring broader scenarios in which we can enjoy "adaptation for free" (and what extent) is interesting future direction.
B Omitted Details in Section 3
In this section, we provide complete proof for Theorem 1.
To begin with, first define le2 as the trimmed value of b,t with respect to M, i.e. E,s Me & MJ. Then, we decompose the pseudo-regret by
T
RT
U1,0r a.a + Ele. Me,3
t=1
TRIMERR
TRIMREG
B.1 Bounding TRIMERR
Lemma 1. For any fixed trimming threshold M M > 0 and ( < 1/(elogT) with probability least C, holds that
TRIMERR & 2 + ( MI (ogglo) + 8. IMoog(2log(T))/
Proof of Lemma 1. First note that (Kh,o, adapted to the filtration Fi....,Fy Fr where Fi == (a,, 4--, 1-1,4-111111 ( Moreover,
form martingale difference sequence
have 1h, -Ce < 2M and
(A/.0. 2
Σ B [1l, - (. (...) ii]
K
Wa,s (E [(t,,)) Fe, ar = =] + (2 244, [ee az = =i]
K
Wr,BB (2 Fr,
( at =
W [(Cb,) 1+ M F
2
Fr, az
M
(15)
where in the last inequality we utilize the fact that ΣK Wt, = 1 and Assumption 1. By Lemma 11, have with probability least 1 C.
T
(vha, -Ga,) & A " 4M log(log(T)/c)
(16)
Together with Lemma which bounds Pr,a, Phay we arrive at
Σ (Hr.r (.a,)) & Tu + T M (og(log(T)/() + 4M log(log(T)//) (17)
16

---

With exactly the same reasoning, we have with probability at least 1 S that
Σ (l. (0.--1,11) Tu 4
log(log boglo + 4M (Tg() (18)
Taking union bound over the two terms completes the proof.
B.2 Bounding TRIMREG
For any i E [K], let y be the K-dimensionnll one-hot vector such that the i-th element is 1, and all the others are zero. Define vector y' := (1 - A)y" + Awo, where (1/K, 1/K) is the uniform exploration. We can rewrite TRIMREG as
T
T
Σ
GG. -1,0) = Σ
(wr, Ee) -wv",
(ur V.
Σ W 5f
Σ w R E
(19)
TRIMREG
TRIMREG
TRIMREG
In the following subsections, bound each of the three terms in the right-hand-sight. To give preview:
TRIMREG is the standard online learning regret loss sequence E1, Er. and is handled using regret analysis OMD.
TRIMREG IT is bias term due to that we cannot directly compete with y" and have to choose close neighbor y' It is under control since y' and y" are close to each other.
TRIMREG III is the main challenge in getting eigh-probabilig bounds in adversarial bandits. While contains some terms that could be potentially large due to the high variance of le they would be cancelled by some negative terms TRIMREG introduced by the increasing learning rates that eventually the sum of the three terms can be bounded.
B.2.1 Bounding TRIMREG T
Lemma 2. Suppose & 10M and X = Tito to K itt, for any M = M, Algorithm / ensures that with probability at least --
lo
(y'
TRIMREGIS
5m - -T 1+
"
10n log T
+ 20m Tw Tw 1 log(log (To(gogg)(()))) + 10 (T)))))()))(().
Proof of Lemma 2. The proof is a combination of Agarwal et al. (2017, Lemma 12) and Lee et al. (2020, Lemma 2.1). For the (active) OMD update rule u++1 argmin (zz, D (I, w.) in Algorithm 1, it is equivalent to first obtaining some intermediate vector W141 such that Vih (W++1) Viha (w:) Eg. and then obtaining W141 argmingese Dv, (z, (.+1).
For any competitor E S', each round E [T], we have
(wr y, E.)
(wr y, Vihe (w:) Ve2a (We+1))
(a) Dv. (y, we) D (y) w + De, (We, 11+1)
D (y. wi) Dv. (u, w++1) + Do, (we, Wc+1);
(20)
where in step (a) we utilize the definition of Bregman divergence (or the so-called "three-point property") and step (b) is due to the generalized Pythagorean theorem.
17

---

Throughout this proof, choose À T K Iiu in Algorithm 1. Recall that we define y' := X1y + Xwn. Therefore, for any E [K]] we have v A/ K and hence y/ E Q2'
For any scalar O, we define function h(c) the summation over all T rounds and letting y = y' we get
logc, which is always non-negative. Taking
T
Σ w v Ex)
(Dec (y', wx) Dv (1/ w++1) Dve (Wi, w(+1))
T-1
T
Din Dvr w1) +
Σ (Dvi (V W6+1) Dee (V/ wx+1)) +
Σ
Dar (wi, Wx+1),
)
(21)
where in the inequality we get rid of - (3/,, WT+1) since Bregman divergence is always non- negative. In following, we bound each of these three terms the right-hand-side.
Step 1: bounding the first term. Plugging in the exact expression of Bregman divergence concerning
the log-barrier regularizer the we define in Algorithm we have
K
1/4
K
Vi
V/
))
D (3/ w1)
Σ
h
- 1 - log
W1,
W1,1
W1,$
i=1
log
(22)
Kyy
By choosing
T K T4", we have y/ 7 A/K = T+e K
and
K log ( K ाकक
1 log
K log T
Ky
7
7
Recall that here we utilize the assumption that K < T. Step 2: bounding the second term. Plugging in the expression of Bregman divergence associated with log-barrier, we get
E(D (V/, W1+1) Dev (v/, wx+1)) T-1
1
y/4
(23)
Me++1 17.,
We+1i
Now look each action i. Recall that, if the learning rate does not increase round f, then
h
of
is simply 0. Otherwise, we have M > ne, and as a result,
MEILL
De.
0.
11+11.1
Ve.
Therefore, we let n, denote the total number of learning rate changes action i, and let tn, denote the round when the last change happens (such that f)r.: Mo, +1,3 KMen k²"in), we have
1
1 Tee
Ví
1
y
<
h
M1+1i
WH+1,
1/t...
1,
Wt.., 1,
1
y/4
h
k
Wtn: #1,
K h
V/4
(24)
Wee.. 1,2
By Lemma 9, when À
T K I4"
have ni & logz (T to K K
< logg T and nh &
logg r"
1) & 51.. Together with
e/ logT
log
we get Ni
1- K h 17
(25)
Wh +1,
5m) log T
18

---

is left upper-bound term -h
Whny th.e )
Noticing that
V/
ViPT.
PT.
$ T,
(26)
Wt..:
1,2
2
2
Wing +1,
we get
V/4
ViPTs
ViPTA
V.PTa
-h
= -h
= log
+
< log T +
(27)
Win: 1,2
2
2
2
2
Taking the summation over all actions, we finish bounding the second term as
1/4
log I' + Mers
K log
PT)
(28)
1,+
T/e,
1,+
5r log T
"
10y log T
Step 3: bounding the third term. Given the fact that Vihe (w++1)
Viha (wa) - E. we have
-1 Pe.i.
(29)
We+1,
W2,s
which implies that
Wt,i
1 We
(30)
We+1,b
Now we have
Des Dos
(wi, (wt,
wt+1)
h
Wi,i
11+1.
T
-h
1 + nt We,
Th.
i=1
T
K
m,,we, - log 1 (
(31)
Th.
To proceed, we first show that if & 1oM it holds that
We, Re, > -0.5.
(32)
First, it trivially true whenever l, > as the as negative as -M and nh, & 5n, is left to show that
is non-negative. Since l, is at most
M
M Wtss
< 0.1,
(33)
which is clearly satisfied when 17 10M
By applying the fact that log(1
$ c 3.Vc -0.5 to Eq. (31), get
1
D (wi, W <
n
T
nha (we,a,
)2 )
T (Praa)2 55 Σ
< 5n
Σ
(34)
tm1
tm1
19

---

We now finish bounding the third term by deriving high tigh-probbbility bound on Σ 1=1 [r,a multiplying it by 5n) Consider the martingale difference sequence (1et
(and
1+2
E
we have
NE(TT]
|1/,0 1
E 1 har
[10)]
& M
almost surely, and
(1 1++ E
(to]) 21
(1125) - -R [1a 100)
2
E
(
Fe =
Fr, at
1=1
B |1a.3 22+2 Fr, =
mm
Wr Wri 2
| | Frz at =1]
< + M
(35)
By Lemma 11, have with probability least 1 that
T
1+v
1+t
+ 4v Tu + +
+ 2M
t=1
Tuuto +4, +4 Twlte Mi+e (og(log(T)/C) 2M log(log(7)/C)
(36)
At this point, we have bounded all three terms, and putting them together completes the proof
B.2.2 Bounding TRIMREG II
Lemma (Upper bound on TRIMREG II), For any fixed M > 0 and A E (0, with probability at least 1 - C, it holds that
TRIMREG II & AuKT +
+ Algg/()/
Proof of Lemma 3. By Hölder's inequality, we have
Σ wv 3¹, . |o' g- 11 Σ max(A/K,M1-1/E3) [1GI & A
ΣΣΣ M..l
(37)
Define Pe,e EewPe, [|e| [((e) M)] Mt. (T) E [K], have
Fe, EePr. [|e]] & Pt,P, [|e| 1111-00110 sto $
(38)
Applying Lemma 10 to (1%,,l - 14,)) [ since |1et.s - M4.,l & 2M and
E [[1e2,1 - ) < E [1 |] u we get that with probability at least C.
(39)
T
K
K
F4, + V2KTu log(1/() + M log(1/C),
3
1=1
which implies that
TRIMREG II & AuKT + Av A/ 22 log((/()) + AM log(1/9)
3
20

---

B.2.3
Bounding TRIMREG III
Lemma 4 (Upper bound on TRIMREG III. For any fixed M > 0 and À E (0,1), with probability at least 1 5. it holds that
TRIMREG III & 3TTuttM + 3M (V/ ,r) log maxx
AK/A3/S).
6 form a martingale sequence, and Proof of Lemma 4. We note that (W. M2) LE[T]
(// Pe / Fe is clearly finite since wr, 2 A/ Moreover, we have
E
22)/20)/
E
(1/,2.) ( Fr
(sla (C /22
E
Ft
(wt,ar)?
(u/)²
W6
E [(),)/M)²
Fe, a
(w.,,)
(w/)2 u 1+ -1--
Wi,
< + M (2)
(40)
where in the last step we utilize the facts that y < and 1/Wr,i & PL. Noting
K
K
2
-() /M
&
Σ
3/(
+ 1) <
y/
<
2(2
(41)
Wei
Wz,s
i=1
by Lemma 12, we have with probability at least - Ç that,
Σ w/ Ea - 4) & 3M VTw.ttttt-------- (w/, pT) + 2M max 2(y' 2(y',r))))
( 4.5 M 2M (w' pT) + 2M max 2(y/ pT)){
<3 (Tu 1ee - + M max (1, 2ky ,
(42)
where , == log(2mmll max Tu M pr). 1,2 PP))) and step (a) is due to the elemen- tary inequality v
F1 t2
Ve, 32 > 0.
We complete the proof by noticing that < 2K & (y', pT) & 2 x/x and apply it to e.
B.2.4 Putting things together
By Lemmas 2, 3, and 4, with probability at least - C. we have
Ry & 2T + " + T Mi log(8 (T)(( + SM log(s log(T))()
+ 2K log T (T) + 5mpM1~TTul
10nlog T
+ 20n Tulto M3- logg4)/))))/)) + 10nM log(4log(TT/C))
4
+ AuKT + Av 2KT 1+0 (4/C)
AM AM log(4/C)
3
+ 3Tu 4 - " 3M pr) log (4mxx( / + MM K/AAAA- 4K/A//s (43)
'(Re)SScaling by M is of course not necessary, which however simplifies the algebra calculations after applying Lemma 12.
21

---

Choosing = = u. (T/K) ife and X = TTo K 14u, we have
RT < 2uk BuKTiTTIS + 8V uTT K Ihe log(8log(T)) 8u7 K
2K logT
(v/ pr)
+
+ 5n Tuu!
10glog
+ 20v Tultoul - (T/K) 122 2 (og())g +
4 u u K TT log(4/C)
uT : uT K गेग (2log(4/S)
+ +3uK 3M pT) log
maxx
2K tov T 4K _ T }/(
< TuKTie (4/C+ 2K logT
W +
5n 1--p1-100 "Tu
10g log T
+ 20 uTTTi K f of log( (o (o((())g())))) 10n log( log
+ 3M log (8KT/9) + o [uK AiKiooTE log(T/<))
(44)
Finally, choosing 12 minf 10M 40M lom (og((((())) containing (v/ .P) ensures that
40 (oN to cancel the terms
R = O(K (log log(T/())
C Omitted Details in Section 4
C.1 Concentration Results
In this subsection, we derive concentration results needed for the analysis, adapted from Bubeck & Slivkins (2012) to suit our heavy-tailed case.
Lemma 5 (Concentration on the trimmed importance- weighted estimator). Suppose M1, (t/K) 44. (log(log(T) (ooog(TT//)))) any E [K] and E [T], if + 1) & & tww, then
/()) S6tI, In both regimes, we have with probability at least that.
min(s,,4} Σ
L, Σ Mad
< 6u
(t/K)
1
max(t - Tı, 0}
Krivziio max(t/Tr, 17
Wg.5
9:7;
(log(2Klog(T)) (Tg()))())
30-1
Proof. We fix some action E [K]. We first rewrite Le, as Le, Eiz
Note that for any
> Ts, we have Wg 4: and > Wass > 1/K for any & T₁.
We define X,, = K Me.s for any 1 & s < t. Ift & fows then X Xi....., Xe,; forms martingale difference sequence. To have an upper bound on XX,sl: 1) when s & T. we have
[x,il My,KK + M3,5 & 2KM,,
and 2) when s > Ti, we have
Ms
M Mas & 2K,
X
Ma,
K
Combining two cases, we get
2ut to K 1i max L 1}
1Xaal < 2KM max
m 1, < 2KM, max -
t
(45)
T
(log(log(T)// 3011
22

---

Moreover, we have
(c,,) (5,5)2as
Castad Wa
= E [xx
E
F
+ (2',,)2 -
Fi
(Wa,.) )22
s=1
((,,)²
I.
E
Fb
Σ (Mar))
(Wa,,a) )2
16%.1 1+1 (Mai)) 1-0 Fz
E
Waçt
$=
1
t mmax (t T2, 0f
< u (M,,))
Ws,i
9:78
min(r..t)
= (t/ +
mmax - Ts,
(log(log(T)/())
Wa
9:7%
(46)
where the last inequality is because M,,i is non-decreasing in 8. By Lemma 11 we have that with probability at least 1 S, for any t < fsw
(r
m (t Ti, 0}
<
(t(K/)) +
Σ
(log(log(T//{))
Woii
Wx,i
9:Ts
+ 4ul T+ K के
max(1/Tr, 13 (log(log(T)/()))) S
(47)
By taking an union bound over all all actions E [K]. we have with probability least that
min((r.()
Σ Mla ==1
& 4
t-c u (2(t/K)i
itv
Σ
mmaxft Tz, 0}
logg)))))(T)/C))
Wafi
9:7;
+ 4utr's K vte max(t/r,, 1) (log(2K log(T)/()) s011 Vt E [T], [K].
(48)
By Lemma 8, we have almost surely that
Ma.i 16.1 < +Maa)))) & uK (log(log (T)lggTT)/ol ({) J04T
< 2uK The T (log( (log(olgo/// (T)
(49)
Combining two parts above, we get with probability least that, for any action and round f,
min(r..)) Σ
Le, Σ Ma.i
& 6u
(t/
1
max (1 - Tr, 0}
+ K -h t पot
1}
Ws.i
9:Ti
1=1
==1
(TT())()))))) N....
(50)
Lemma 6 (Concentration on the number of pulls). It holds with probability at least 1 C for any E [K] and [T]] ift & taw.
No, < g. + logt) 44V + log t) log (K (og(Klog([T)) log(T)/C) + 2 log K log(T)/C).
23

---

Proof of Lemma 6. We first fix some action and round t. Recall that No, = E I. Define Xa. 11, W..s. If & ssw then ii....... X,, forms martingale difference sequence such that (Xa,i & 1. Moreover, since aa is non-decreasing in when 8 & Tj (and Wry, Q:). we have
R (X
Wa
9:
Σ
9: fT logt).
8=
Therefore, by Lemma 11, it holds with probability at least ( that,
N Σ Was & 4/44 9:7((1 log log(log( (T) + (logg()//))))
Taking union bounds over all actions E [K] completes the proof.
Lemma 7 (Concentration the trimmed empirical-mean estimator). It holds with probability at least 1 - C (hat, for any E [K] and & E [K, bsw)
Es Paailad Pa,s
log (og((4KT//))
140
< 9u
No,
Ne.
Remark 9. Note that in the special case of stochastic regime, the term Eimpuo is simply (u(t) Ne.
(which is simply scalar denoting the loss mean of action in the stochastic regime as defined in Section 2), and this lemma has been proven in Bubeck et al. (2013). However, we are not aware of how to extend the analysis of stochastic case to adversarial case.
14v
Proof. Recall that B, is defined in Algorithm 2 as Bas == u [og(21/3)
Vs E [7']. Then we
rewrite Ft, as
De. E.
la Aalaallllaa & Bayi)
N1.3
and define
Xe.5 (Co.IIlle.nl & B..i Ee, ResIfllord & B,,+)) L,s/u.
Moreover, in this lemma we need a slightly different filtration. We define
F. == o(a1, (1011.1.. f..........4 4,-1, (0-1166-1a
Clearly, given any fixed round & and action Xi......, Xt,, form martingale difference sequence adapted to FI....,FF (since both B. and T, are deterministic conditioned on Fi)), with
No.i
max & Bu,i/u= #tt
(1og(2T/2)
and
[X
12
Ft
Σ
(X 2
st1,,=1
Σ
E
1tv 1- /n² F(
ad1,;=1
1x
No.1 (lo((2T<{)
No (B
(-.))
No.
(51)
where in the last inequality we utilize the fact that B,, is non-decreasing in Noting that
1 No. < 2 (Nr,) < 2T.
No.i log(2TT())
Ne,i (og((2T(C)
2
$21
(52)
24

---

we apply Lemma 12 and get with probability at least 1 Ç that, for any fixed t,
Ni. (2g(())())
Σ x, < 3V (Nr.) -ooo (log(22/ _ 1og(27/()) +
12g(27/()
-(No,) (log(2T/<))
(53)
Taking a (two-sided) union bound over rounds and actions, we get with probability at least 1 - that
S
X < B(N (Iog(4KT*/C) 140 vt < tsw, 2 [K].
(54)
Finally, have
Pa. 1o, & Barl Mosss) Lofs
Eeg.erpnr (eatallllaal & Ba,ill Ma.k
No.
1 1+v
u
log(2T/()
N=1
Nos
u(log(2T/()) 1+
N
N=1
((log(4K7)/()) (1 (NV + 1)
< <u<(og(4KTT<<)) /() (N..)
(55)
where step (a) follows from Lemma 8 and the last step relies the fact that v (0, 1). Applying triangle inequality to Eqs. (54) and (55) and then dividing both sides by Ne, completes the proof.
C.2 Logarithmic Regret in the Stochastic Regime
In this subsection, we provide the complete proof for logarithmic regret of Algorithm 2 in the stochastic regime.
Taking a union bound over Lemmas 5, 6, and 7, with probability at least 1 - Ç. for any E [K] and & (sw. all of the following holds in either stochastic adversarial regime (recall that 8 127 K log(T)):
min(T:.t} Σ
Σ ==1
Mas
< 6u
(t/K)t= 1-
+
mmxx (1 - Ti, T2,00
+ Kiiv maxft/7;,11
Wg,s
9:7s
(log(3/()) J01I
(56)
No. & (1 + log
+ 4 9:7(1 log t) log(B/C) + (lg((9)))
(57)
Ei Maites
log(3/C) 1
Me.i
<
(58)
Ne.i
No.
Before we start to derive the regret bound, simplify Eq. (56) bit for convenience.
25

---

For any t such that 7, > t (i.e., action i is still active by round t), we have Was > 1/K, Vs < t, and Eq. (56) implies that
Σ s=1
Ma,
& 6a
(1/K) E Wa.i
+ KTtz 1T
(log(3/())
< 6u (t/KK) t Krivttiv (log(8/()) sन1ं
12nK Liv tito (log(3/() sit1 Width((),
(59)
otherwise (when Ti & D) we have
(V1KK) to K 12
KItet tl t /T/
L
Σ s=1
Mai
< 6u
+ K
(log(B/()) -
Ti
6u
tk tK 14v
11+0
K 1ov e tेo /Ti
(log(3/{)) 5
12uk -ों + (log (3/))) Ju /ri = Width(() 1/T:.
<
(60)
where in the first inequality we utilize the facts that q 2 1/ K and
T 2
Ts +
(61)
and the second inequality is because
tTtv
1-
t
triv
(62)
T
Th
T
Combining two cases (whether T; > or not), Eq. (56) implies that
L
Me.
llfi Ai)hiith(t)) + +Ifi & 3,j
(63)
T
To show the regret bound, we first show that, in the stochastic regime, when all the three above events hold, we will never start to run Algorithm 1, i.e., tests (6)-(8) always fail (for all < T).
Test (6) always fails. This is simply implied by Eqs. (63) and (58) together with a triangle inequality.
Test (7) always fails. We first show that test (5) is never satisfied for action i so we have i" Ag, VI low To see this, for actions i, E Ac-1. by Eq. (63) we must have
L - Li = (Lu.s- - L- (()(m))) (Iu,, - t- u(i)) + t. (u(t") m(i)
< 2Width(t) tA; < ea Width(t),
(64)
which means that i is never eliminated from A, and hence stays active (due to test (5)).
Moreover, for any action i & Ac, must deactivated at some round no later than 4, so we have T; & and that test (5) is satisfied at round (and is not satisfied at round 1). Let jt argmin Lej.
Therefore, looking at round t; (and any action i + i*), we have
e Width(T.) (a) LT,it Drish. (in,s - To ((())) (Ln.st - Ti -(jt,)) + T. (m(1) - m()7.))
2Width((T)) + (n(1) - (((i)) = 2Width(7.) +
(65)
26

---

where step (a) is from test (5) (which should hold now) and step (b) is due to Eq. (63) and the
optimality of 1n (i.e., u(ii ((i+,) & p(i) p(i²) A,).
Moreover, looking at round T; - (and any action * i*), we have
(T= 1]4 2Width(T; - 1) < (T,,,1,i (T, 1) uui)) (F (Fz- - 1) u(it))
(2)
+ (Ti 1) (n(i) (li"))
T -T:T.-1,=
(6)
T.,-1.1 11,,,
C1 Width(T, 1),
(66)
where step (a) follows from the good event in Eq. (63), step (b) is because it E Ar, and step (c) is due to test (5) (which now should not hold for action since it is still actíve).
Now can show that test (7) is never satisfied since in any round for any & Ac:
(Le, min L/s)tt < (Ii,a min Ltg)/t jEA,
jEAL-1
(Bus/t - m())) -
+ (u(1) - m(it))
Width(t)/ Width(t)/t+ D.
2Width()/(m-111) - 1) A
(2 + c1 + 22) 1),
(67)
where the last step is from Eq. (66).
Test (8) always fails. Since 1" Ar. for any action i & A., we have
(Te min P..)/t2 (T,, Tr,,-)/ )/t JEA,
Cnrst - p(i)) -Lo,--tt- + (u(i) - pli"))
> 2Width(t)/Ts + A;
(cr 2))itth
(68)
where the first step due to i E A, and the last step from Eq. (65).
Putting things together. Now we show two intermediate results, followed by bounding the regret. First, Eq. (66) implies that
A < (2 + ci - 11)(T---)))
(2 + ( 12n - (1; - 1) -ि (log(B/C)) उलन
< (2 + 12) - (T/22 पमक (log(3/() J641
< (2 0 24m -मन (T:) T (log( (/())
(69)
which after rearranging implies that
Ti = 0 ( k (1og(8/())1+*5) 3पे (
(70)
Due to the definition of gt. we have
K
Σ q
K K-i+1 < 14 log K.
(71)
To see this, let E' denote the é-th earliest action that is deactivated, we have qu K-P+1 algorithm design.
1
due to the
27

---

Now we have everything needed to arrive at the final result. By the definition of pseudo-regret in stochastic bandits, we have
RT = Σ A,NT,
:A,20
Σ A (1: log + 47, 4 (og (0000//3()(00033)))))
210g(3/())
Σ
A (2
log 6log(3/1))
:A,S0
m3
Σ 1,
0
Σ
4 log(T) + K (log(B/()) + /(A,) log(B/S)
7A,50
:A,30
0
u k log(T) (log(8/<)) /AF Σ
qi a log(39)
Σ A
:A,30
:A,20
< <0 K (0(K()
log() (log(8/()) (u - //A)+
(72)
where step (a) follows from Eq. (57), step (b) due to Eq. (70), and step (c) is from Eq. (71).
Choosing 1/T < 1/e, we have
1 =
E [RT] & 0 K (
=0 K (oog((K)(og T) / /))
(73)
C.3 Optimal Worst-case Regret in the Adversarial Regime
In this subsection, we provide the complete proof for (near-)optimal worst-case regret of Algorithm in the adversarial regime.
Let i argminge|k] E:=1 Maos and 1f argminge.. E.= Maji We first show that it A
For any action i A we have 7. Law and test (8) is not satisfied for i round lsw (since the algorithm switch has not happened). We get
la
Pa.i
aml
mm
1
Tnw 1io-1i/ 1,6) + Enw 1.1 Σ No. +
(T Tee 1.1
s=1
(a)
(lww 1)
Width(tow 1) Width(Asw - + (cz - 4) lsw - Width((Ew - 1)
Ts
T
(c) Lyw
Width(tsw 1) > 0,
(74)
Ti
where step (a) is due to Eq. (63) (applied twice) and test (8). Therefore, we must have i E A (which further implies -1 E A,, Vs Lew 1) since Σbu11 Maij the definition, otherwise contradiction is incurred.
Dte Ma & b
28

---

Now we can bound the cumulative regret to the algorithm switch. Specifically, we rewrite the regret as
by
K
1
Ma,0t
Ma
Me
==1
==1
s=1
i=1
==1
K
1m--l Mygitad
Eber Meitem -1
Now 11,8
N 11
tsw
K
Liw-1ii 1,8
De A1m-1,6
Dbw 1, -1
N bu 1,,8
===
Now 11
Esw
Part
Part
Lt 1.
Ma
(75)
Part
In the rewriting above, we use nothing but the simple fact that Σ New 11 bound each of the three terms (Part A, Part B, and Part C) separately.
tsw
We now
Bounding Part A. We first rewrite Part A
Part EEer
Dew 1,2 tsw
Bin
(76)
N1w
By Eq. (58), we have
(lg((3/()
Nim-1 P & Iu
(77)
Ni 11,
By test (6) (which does not hold now), we have
De 1,1
(og((3())
Width(t 1)
P
<
9u
(78)
fsw 1
N11
7-1
Therefore, we can conclude that
Part < 18u (og((//))
14
Width (tw
(79)
N1,5
Bounding Part B. Since i E Atn-11 1 as we have shown, for any action & A due to test (7)
(which is not satisfied round faw 1), we have
Lem- Lew 1,
T. 1, minjea Prem 1i < (c1 + 4
Width(Aww - 1)
<
(80)
tyw -1
tsw 1
Tr -
For any other action E A due test (5), we have
Pit - 1, Peem 1,8t ateu 1
I T minje Aun -2
Tiw--i -i <
Width( -
Width (f.w
Iaw
1sw
1yw
(81)
Combining these two cases, we can claim that for any action i E [K].
L 1, L
1,ii < (ci +
Width tsw
Part
(82)
Lew
Bounding Part C. Simply due to Eq. (63), since if.--1 - A4m-10 we have
Part C = Er 1,22
Most < Ci Width(t,w - 1).
(83)
29

---

Putting three parts together. Putting the three parts together, we get
Ma
Ma
Ha
==1
K
(1og(3/5)
K
Width( 1)
0
Nia-1,1
+0
Na-11
N₂ N11
+
O (Width(tsw -1))
N 1, (log(B/S)
I4
0
O (Width(tsw -1))
N1a-115
K
(9 + T)) uk 17v (tsw 1) 1o (log(1/()) S047 Ti --1
+0
K
uk r&o (tsw 1) т (log(B/<)) -m
+0
(og(3/C)
(84)
where in the last step we apply Lemma to bound 11 the term Σ 1
Width
Now bound each of the four terms by one.
Bounding the first term. Applying Jensen's inequality, we get
K
Nt. 1,4
Σ (N &
Σ
- (fuw - 1)
(85)
K
and the first term is bounded
K
(log(B/() it4 Nem-1.1 = =0 0 (uk r - 1) 1) (log(3/()) -
Now-1,i 1,
(86)
Bounding the second term. Simply plugging in the definition of Width(-), we have
Width(!ww 1) O (uK' Tho (Now 1) (log (ogg(3/()))) 1-30
Bounding the third term. By the fact that ΣK qi = Ollg K), we have
uK rī (tow 1) (log(B/()) B
(g.T((1 + log T))
Ts
(ogg(K) K)) log(T) (ogg 1ie (tsw pho (log(3/C)) set1
(87)
Bounding the forth term. Notice that in the algorithm design, each action will be pulled once the initialization, and we clearly have T; > K Therefore, we have
lBg(3/c) uk - (Isw 1) tt (log(B/()) 30
Zv
Ti
log(B/C) uk 1 (tsw i) (log(3/C)) 3041
K
uK io (tsw - 1) itv (log(B/{)) 3
(88)
Combining the bounds on these four terms, we have
30

---

Ms
Σ Nss = 0 (K) log (T)(T) rïv (tsw rov (log(B/<)) 1+ SoTT)
(89)
The regret incurred starting from round faw + 1 is taken care of by Algorithm 1 (and the regret guarantee is given by Theorem I). By taking union bound, we have with probability at least 1 that
Ry =0 (uK T The log K) log (TT(oo((()))( sप+ + +0 +0k tNv T (log T) log (3(()
=0 uk [KKii ito T log(K) (K (log (gg0)
(90)
Choosing ( = 1/T, we have
1 E [r 0 OuK log( (logg) + 2 uT O - log( () (log T)
(91)
D Heavy-tailed Adversarial Bandits with Huber Contamination
This section is dedicated to the adversarial regime, with the additional setup that the bandit feedback could be contaminated in the Huber model (Huber, 1996). In the stochastic MAB, Huber contamina- tion has been studied in Guan et al. (2020); Agrawal al. (2024); Wu et al. (2024). We first formally define the problem setup, and then present the algorithm design and the regret analysis. From the regret analysis one could readily see why removing Assumption is necessary for a near-optimal regret upper bound. Lastly, we provide a matching lower bound which suggests that we obtain the near-optimal worst-case regret guarantee.
D.1 Problem Setup
The learning algorithm and environment perform the following interactions repeatedly in round I:
t
1. The algorithm samples action a, from [K] via ~ wy := (Wt,1, (x,K) E S. i.e., the probability of sampling action E [K] Wr, The environment draws loss l,, from "clean" distribution Pt,, satisfying Assumption for every action i E [K].
2. Let 4,,, denote the feedback revealed the algorithm associated with ag. With probability a E (0, 1]. the algorithm observes the contaminated feedback. That is, observes 2r,, 2,,, which is generated from an arbitnary "bad" distribution Qt. With probability (1 - a).
it observes the "clean" loss le,a. = L,,i-
3. The algorithm determines W4+1 based on all the revealed history so far.
The goal of the learning algorithm is minimize the regret in the presence of contaminated feedback. For the ease of presentation, we consider the weaker expected pseudo-regret (rather than the high- probability version), which is defined as:
B := E
Σ w of Me)
(92)
where the optimal action i and loss mean br are both still with respect to the "clean" distributions, i.e. Me, Er,,-Pr.s [er,]] and i' E argmiin ΣxT Mr,i while the expected pseudo-regret additionally includes the randomness from the contamination. Similar to all previous works, we assume that the "contamination level" a is known the algorithm.
Assumption adapted to the contaminated case (to bound CONTRIMERR) becomes the following.
Assumption 3 (Truncatedd non-negative losses in the contaminated case). Given any fixed M > the loss distributions of the optimal action i" satisfy that E, B 1.3- [ -T -(( | > M] > 0, VI E (7].
31

---

Importantl, even if there is some prior knowledge regarding the "clean" distribution Pt,s. due to the arbitrary "bad" distribution component contained in Pi..., the adapted "truncated non-negativity" may not hold any more and one may not obtain any meaningful regret guarantee. However, by taking advantage of the log-barrier regularizer, we naturally handle all possible "bad" distribution Qt, In the following two subsections, we the details of regret upper bound and lower bound, respectively.
D.2 Regret Guarantee and Analysis
To obtain near-optimal high-probability regret guarantee again Huber contamination, simply need to run our Algorithm with different initial learning rate and trimming threshold, both of which now further depend the contamination level. We formally state the theoretical guarantee below.
Theorem 3. For any failure probability ( rate
and contamination level a > O, by choosing initial learning
1 u
K (log(T) T
1:30
=
and trimming threshold
24
M min
110g log log 40T33/C)
njc
Algorithm 1 ensures that with probability at least G
RT O (uati + uk -े= T (log(T))) (oo)))/oloog(//// (og((/)))
By further choosing
T, Algorithm ensures that
E E[Rr] = (ua 1fo T+ + uK 1. T (log(T))2.s)
We first rewrite regret as
Ri
Σ (14,
F..a)) Σ (Mi (Hi - -Z,,)+
Σ
(Ur Fraa.) - " [/10.)
NN
BENQ
BENp
BENt
CONTRIMERR!
CONTRIMERR
C CLa
(93)
CONTRIMREG
where No is the set containing all round indices in which the observation is contaminated (i.e., Zt,, l,a, Qt.a.), and Np [T)NNQ (i.e., those rounds in which Lt,,, E,,a, Pt.,,)- Similar to the uncontaminated case, it is sufficient to choose some fixed trimming threshold M,,1 M.
D.2.1 Bounding CONTRIMERRI
Recall that |NV2 denotes the total number of rounds when the feedback is contaminated, which is exactly the sum of T independent random variables from Ber(a). By standard concentration results (e.g., Lemma 10), we have with probability at least ( that
N2 < of 22Ta(1 a)) log(1/() 2log(1/S) < 2aT 4log(1/S)
(94)
where the last step is again due to the fact that V2122 & 22 Va 22 0. For any fixed trimming threshold M, it holds almost surely that Pt_ - Z, Mr,i 14,s ph., be,i u u 2M,
(95)
which implies that, with probability least Ç. we have
4log(1/())
CONTRIMERR & (2aT 3
(ul+uM -2MM
(96)
32

---

D.2.2 Bounding CONTRIMERR II
In round & Np, the losses not contaminated. By Lemma 1, we have probability with least 1- - that,
CONTRIMERR ıI & Np uu w/tpp//tMM" log
+ / + 8M (og(2)g((T)//)
< 6T
12M log
(97)
D.2.3 Bounding CONTRIMREG
We first rewrite this part as
CONTRIMREG
Σ
(wi, -1) ,E)
T
T
T
we y.
w
Σ
Z,)-
(98)
CONTRIMREGI
CONTRIMREGU
CONTRIMREGHH
D.2.4 Bounding CONTRIMREGI
As long as holds that < 10M we have
Σ
T
We v. 2
2K log
T 5 (y's + 10n log T
Σ
1+
(99)
1,1
1tv
and it left to bound
Σ Lea0
We first rewrite it as
Σ
T
Gas
1+v
1+
+
l+v
(100)
EEN
LENP
Similarly, with probability at least 1 C, it holds that
(20T+ 4log(1/5)
Σ
t,a
<
M+
(101)
NE
As in the uncontaminated case, by Lemma 11, we have with probability at least 1 that
Σ Gaaal
1+v
1+v
+ 4///N// u 4 +v log(log(T)/5) + 2M 1+v log(log(T)/c)
KEN,
NENP
Tu 1+
4. TT M log(log(T)/S) + 2
1+ log(log(T)/C).
(102)
Taking union bound, we get with probability at least - that
2K log T
(y' .T)
4 4log(2/C)
CONTRIMREGI
+ 5n 2aT +
M² + 5nMM-mTrrrrr
"
10n log
3
+ 20m/Tuu
log(2 log
(T)/) + 10 (og(2)o)(T)/())
2K log
(T) + 5n
4log(2/()
2aT +
M² 15m
17
10n log
3
20n log(2 log (T))().
(103)
33

---

D.2.5 Bounding COGTRIMREGHI
By Hölder's inequality, we have
T
T
Σ
3"
4
<
1/
1
t=1
t=1
max (2/K, A(1 1/K]} |//
K
Σ
+ Σ Σ
(104)
IENQ
BENP t=1
We clearly have with probability at least S that
C & KM 220T+ 4log(1/())
Σ
Σ
(105)
IeNo
and with probability with at least < that
K
Σ KENo 1=1
ΣΣR
1.d & uKT + v 2KTul+oMM log(1/()+ +
M log(1/)).
(106)
3
By taking union bound, we get with probability at least 1 ( that
CONTRIMREGI & ((2aM u) + /2KT + M1 (2/C) + 2KM log(2())
(107)
D.2.6 Bounding CONTRIMREGHT
We first rewrite CONTRIMRGGII as
CONTRIMREGHI Σ W Ea -
2 + Σ
( -
(108)
LENo
LENp
To bound the first part, recall that for any E No. it holds that
We/ Pe Eii /M < p())
(109)
and
E (10' 2
<3/M))
E
(10/ 22) Fi
M
(Va, )a (C (L, )22
E
Ft
M
(W) )
K
(v/)²
((2,)) Fe, =v]
M
(we):)) 2
K
(11)²
M²
M
We.5
(v/ .T))
(110)
34

---

Therefore, with probability at least 1 S, it holds that
Σ w. 2) & 3MM v (2aT+ + 4log(2/5)
(T)' + 2M max m Pr)
3
tENo
4.5M (2TT + 4log(2/()
3
2M (v/ ,T)d + 2M max ( ,rr)
2
3M [2aT= 4log(2/())
3M max .
(111)
where d' = log(4 max / (2xT+ + (2/g(2/5) (v', PT), 1,2 pt))/rrr)///)..
For the uncontaminated part, with probability, by Lemma 12, we have with probability at least 1 that,
Σ NEN
W 4) < 3u Tu 1-0 My pr)""" + 2u max/11, .t/////////
4.5uTu²M~ 2ud 1 2w sPT) + 2u (1, , pr)//uj("
2
< 3Tu 1u M max Myy M
(112)
where '" == max max
1, (//, pr) (ttp:'/rrr/////// pr)). Taking union bound, we
get with probability at least - that
4 4log(3/S)
CONTRIMREGME 3M (20T+
33
3
[2TT++ + 4log(3/C)
3M max(1, 2(y/, pT) log(6 m
2K/A//C))
3
3 (maxx(1,2M (r))) max //N/T/I////AAAAA/AAA
4MK/AH/9)
(113)
D.2.7 Putting All Pieces Together
Combining all the parts, we have with probability at least 1 < that
Ry < 4 aT 2log(5/()
(12tMM- + +2M) + 67u
12M log(llooT()T)///
2K log
(/.T))
(T + 2log(10/C)
+
10y + 10n
15 -
20 (og(0lo)TT))) (2a. uAAKT x 2KTTlttMM--
+ 2AK 2og 3M 2aI'+ 4log(15/()
+ 3M max(1,2(1 pr)} log (1x (20T+ + 4log(15/C))
K/A,2K/AH/C)
3
+
+ 5 maxfu, (()) log(10 max{ -TA
(114)
35

---

We first plug in À and simplify the upper bound as
2l0g(5/())
Rr & 4 aT +
1u + 2M) + +99Tu
12M loggo(00o
3
+ 2K logT
+ 101 aT + 2 2103(10/{) 10n
(0', PT)
M2 + 15m
3
+ 20n (1g(100og(()/() + 2aMT K +
uF K K + uT uT + K (200)(10/()
+ 2T K log((10/S) + 3M 2aT 4log(15/C)
3
[2TT+ 410g(15/5)
+ (M PT) log(15 1 max|
KT, KT, 2K/2
3
+5 (u, My/ pT)) l00 max 2V 1+v M AMKK/u,
(115)
Assuming a7 > 3 log(l)))))/// and T v
we further simplify the upper
bound
R1 < SaT (4 2M) + 97 + - 21MaT
2K logT T
( PT)
20naTM 35mM "Tu
10nlog T
+ 20 + 2a K - uf ih K : + uf K - / 22gg
+ 2t k + 6M PT) log max V V3, 2KT)
+ 5 max M (s' pr) log 10T max v 2u 1t0 M K, 4MK/AM/// ((/)))
< 32aMT + 17Tu to M + 40naTM2 + 35mM "Tu 1 2K log T
(r) + if K of 4. K... 22log(10/5)
log
+ (M ,P) log (gg(30KT/9)
+ max(u, Mwy pT) log(10T max 2u KM
(116)
where in the last inequality we apply K < T to upper bound terms of form ((a KT=
For any 7j > O, choosing M min{
10p J. we have
na
K log T Rr = 0 (ma))=v zto uu 2 (n) +T
"
(w',Pr) + 6M (// PT) log(30KT/s)
10n log T
max (u, (1' (3/ pr) log( max
AMMK//KK
O (ua uf ro. K ां. vlog(1/())
(117)
Now we are going to choose min ( assume that (140)(200) T > K log(T), which implies that
K log(T)) T
(Kloog(7)
(og(
(2) 2))2 }. We further
K log(T)
Itf
K logg(T
2-
1 2+
(118)
T
T
240 2420
and thus 7) = (T)
(2) 2720
T
36

---

Finally, we need to cancel the terms containing (v/, pr) to obtain the bygh-probability guarantee.
Case 1: u 3 (y)) If max(u, M (y/ pr)) = it would just introduce Ō(u) term which can be ignored. In this case, y) must satisfy that
1 10n (T)
+ 6M log (3g(3)/) < O,
(119)
which implies that
n < 60M lTg (og(30KT/C)
(120)
Case 2: u < M (W', pT).
With the current choice M and 7, we have
2+
1
T K log(T)
M <
& 0.lua
KT/K
(121)
10n
which implies that
lo0 max 2u 1+ M K, (MA4MK/////) & l00 max m ( /T)
< log(4407)
(122)
Now is sufficient to have y) satisfying that
10n (T) + 11M log( (0logg440////00 /5) < 0,
(123)
meaning that
7 110M (Tg(T) 1og(4002//))
(124)
In summary, in either case, it is sufficient to have nM & 110log(T) 40g))77))) stronger than (and implies) the stability condition n < 0.1. Starting again from Eq. (117), with any 1) >> O, choosing
Note that this is
u 1
I+C
M = minf u
110m (Tg log( (4072/5) }
a++"
na
we have
(na))too siv ul + + un) +eT (Iog(T) log(40T)/()) log T
Rr
(ma)
n
(W/PT)
10n log 6M (v/, pT) log(30KT/()
+ 5 maxfu, My pT) log (107 m
K,4MK////0/)
+ O (un T + uT 1. K =. (lg((1{5)
(125)
Lastly, still choosing
K (log(T)
+
K log(T) (
2420
I-2n
(Tg
2420
24
minf
}
T
T
T
yields that
RT 0
((uut uK 2420 24220 (log I" 2420 a 2420
uK uKK.. T 1. (log(T") + (log(407//()))
+0 ua - T uf - : Vlog(1/()
0 ua -:= T + uk - Tio (log(T))*--f) -मr (Iog(40T²) vlog(1/()
00 ua F T + uk - T rे. (log T) 1. log(40T2/6) (c)/ 10g(4)0//)))))))
(126)
37

---

D.3 Regret Lower Bound with Contamination
Theorem (Lower Bound for Heavy-tailed Adversarial Bandits with Huber Contamination). For any bandit algorithm, there must exist one problem inslunce in which the algorithm suffers regret
E R 52 uil 14 Ki4v 14v vifaatt))
Proof of Theorem 4. In the presence of heavy tails, every algorithm suffers Ω (uTTiiv K r in the worst regardless of contamination level (Bubeck et al., 2013). Therefore, to show the lower bound with contamination, suffices to show lower bound of
regret
Ω (uTa
given any (0,1). 1].
This proof is a direct modification based on Wu et (2024, Appendix B). We use π to denote bandit algorithm. We construct two environments, denoted by 1/2 and V2, respectively. And then we show that any algorithm suffers the claimed regret in one of these two environments.
Environment V1, In v1, the loss of action in every round E [7] is given by
u/7,
with probability ""
1+v
4,1
O,
with probability 7
where 7. < 1 is some free parameter to choose at the last step of the proof. One can verify that [[]]]=++]+ & ul+x via direct calculations.
For any suboptimal action * 1, the loss in every round t [T] given by
4/7
with probability
1+c
10
le,
3
0,
with probability
1+v
10"
One can verify that El|,, [+1] & w Vi 1. Moreover, action 1 is the optimal one and we have the "sub-optimalitty gap" B(G,,1 l,, "", Vi 1.
Given algorithm and environment VI, we define i' = argmin E [Nr,)], and hence we have iE (2...... KTT.
Now we are able to construct the second environment. E Nr.e] < K-1
T'
Environment 1/2 In this environment, everything is the same in v1, except that for action i', now the loss follows
w/7s
with probability 7
1+
10
4
7
with probability
10" u 1+v
One can verify that B[(+] -t. Bll [+ & u Io and now the optimal action is i'. We D5 to denote the contaminated version of 1/2 (where the bad distributions determined later).
And then, we use V1 (V2) to denote the contaminated version of P1 (22). The bad distributions will be determined later.
Environments Di and i2. We choose 7
+ (0, 1]. Then for any i [K]. we have Za & r According to Lemma 13, for any action E [K]. there exist bad distributions G; and G², such that
< 2 (P
(1 - aL + aG; (1 a)Pk, + aG',
38

---

where Pt.: (Pi,)) denotes the loss distribution of action i in round t under environment (2). We construct 1/1 and Bn by
M = [z; = (1 (1- (1- aa; [K]]
2/2 [a] = (1- SG; : [K]],
where x; and of denote the loss distributions for action in these two environments, respectively. Following from the regret definition, we first have
AT
T
(T(T, = A
T
E [Tr.a)
>
P
NT,1
&
2
2
AT
T
(T( (22 = A [NT.1++
Σ
2A
E
[NT
P n,in
NT,1 3
2
2
if(11
By adding them together, have
AT
T
)
(T( Rr(a (x,
P
Nr.1
+ Px,
NN,1 >
2
2
AT
exp (-KL(P,, 7,ūr [[PT,,2))
4
AT
exp (0),
where step (a) follows from the Bretagnolle-Hube inequality (Lattimore & Szepesvári, 2020,
Theorem 14.2), and step (b) is due to the fact that 7/1 and 82 are identical under our construction.
Recall that 7 and 7 = -t.. we arrive at
> M(Tart=))
which completes the proof.
E BOBW Guarantees with Bounded Losses and Local Differential Privacy
We first give the definition of Differential Privacy followed by the learning setup.
Definition 1 (Differential Privacy (DP)). For any given privacy budget e >> 0, 3> 0, a mechanism M : 'D - Rn is said to be (c, d)-fifferentially private (DP) for all datasets X, X' in 'D that differ only one element and measurable subset & C I it holds that P(M(X) E E) & exp(e) - PP P(M(X') E E) + 8. When 5 = 0, we refer to 8)-B) as s-DP (pure DP), which is stronger than (s, DDP for some 5 (approximate DP).
Definition 2 (Bandits with Bounded Losses and Local DP (LDP)). All loss distributions
(Pt.))eez),.eE have support bounded in [0, 1]. Given any privacy budget s E (0, 1],5 > 0, the ban- dit model is said to be (s, 5)LDP if a lies in the sigma-algebra generated by (ae, ((Maaa))))eee in any round t [2'] where M is an (s, 6DD mechanism.
Roughly speaking, the algorithm should not touch true losses, and it observes privatized losses only. Here, we adopt the widely-used Laplace mechanism (Dwork et al., 2014). Specifically, when data are bounded in [0, 1]. adding noise drawn from Lap(e)? to them ensures D-D.. By adopting it, the observed loss is the true loss plus an i.i.d. sample from Lap(e). That is, this setup could be viewed as specific way of generating heavy-tailed losses (i.e., bounded true loss + Laplace noise for privacy).
In the literature, Agarwal & Singh (2017) and Tossou & Dimitrakakis (2017) investigated the adversarial regime and proposed algorithms that achieve Ō /KT) worst-case regret (in expectation) with c-LDP protection, which matches the lower bound (Garcelón et al., 2021) up some log T factor and implies that their algorithms are (nearly) minimax-optimal. In the stochastic regime, Ren et al.
'aaplace distribution with parameter is denoted by Lap(n) and has Probability Density Function fx(+) exp(-x)=p))/////pppo Vz H.
39

---

(2020) proposed privatized Upper Confidence Bound-based algorithms with O(Eis,>
Ing T EA! gap-
dependent regret and O O T worst-case regret. They also provided matching gap-dependent when 1.
lower bound, which scales as 2 RAE Tog =A,
In the BOBW setting. Zheng et al. (2020) showed that by privatizing the Geallis-NNE (an optimal algorithm in the (non-private) BOBW setting when losses are bounded) with Gaussian noise, they achieved O (VKT)
regret in the adversarial regime and O (Ens,>0 log(T) log(1/8) regret in the
stochastic regime with approximate (e, 6--LDP protection, which weaker than pure e-LDP. And these regret bounds hold in expectation only.
In what follows, we present the BOBW guarantee provided by our Algorithm 2 when the losses privatized by noise from Lap(s).
Recall that the privatized loss E,,5 = l El,1 + 2, where &,s is bounded in [0, 1] and z Lap(e), we have
B EI Bl|e,, 21² < (E,,)) + 2 < 2 2(1/))² O(1/E²),
(127)
where in the last inequality we utilize the fact that distribution Lap(=) has second moment bounded by 1/e2.
Therefore, plugging ((1/e) and v 2 into Theorem 2 yields the following high-probability BOBW regret guarantee with (pure) s-LDP protection: O / (o) worst-case regret in
the adversarial regime and O K log( (1ogg regret in the stochastic regime with high probability. To the best of our knowledge, this is the first BOBW regret guarantee in MAB with pure LDP protection. We believe that our BOBW results can also be generalized to the case when true losses (to be protected) are heavy-tailes rather than bounded (by properly trimming the true losses before privatization). Related work by Tao et al. (2022) investigated solely the stochastic regime.
2²A
F Auxiliary Lemmas
Lemma 8. For any fixed M > 0, it holds almost surely that |Pt. - -vl, & u ul (MMe
Proof of Lemma 8, Expanding the definitions of frts and ph. we have
|4, -
E Pr,m Pe.
[R,
E Pe,
[les Iflles & Mull
E [lr, 1lle,al Mc.l] Pr.
IE P1,
[|ee,al T(1l,11 > Mr.HHl
E |1e,,11 1: (M,,)) .Ileedd M,Jl
<
E.
[1eez1 1to (M, -
< u 1+0 (Ale.)))
Lemma (Adapted from Lemma 19 in Wei Luo (2018)). Algorithm for any fixed X E (0, 1), let n; be the number of times the learning rute of arm i chunges (and in fact increases) in total, i.e. Mh+11 = k" = =.. We have n4 < (1/)( and This <e
lorz(1/3)
log
n, Vl E [T), [K].
Proof of Lemma 9. Let Let tn, be the (ordered) rounds when the learning rate of action changes,
we have
K
Playae > 2pt., 1,4
2n 27
Pe = 2 K. lorg(1/A) log
X
Wingg 2d
Clearly, we have n, < (1/g2(1/A) and therefore nh,s K² n &
17.
40

---

Lemma 10 (Bernstein's ' inequality for independent random variables (Vershynin, 2018)). Let XI....,Xy be zero-mean independent random variables such that [Xel & b, Vt E [or some fixed constant b and Vi = Eim1 [. Then, for any Ç > 0, we have with probability at least 1 - that,
T
2b 2blg((1/C)
Σ X1 vV log(//)) + 1=1
3
Lemma 11 (Freedman's inequality for martingales, Lemma 3 in Rakhlin et al. (2011)). Let X1...., Xr be martingale difference sequence adapted to filtration F1 C... Fr such that [Xel & b almost surely for some fixed constant b and V, E E [(x,)²(F.). Then, for any
Ç 1/(elog T) and T 2 4, we have with probability least 1 that for any E (T),
Σ X; max Vo, (og(oo(())(]]] (logo///)))////o/ogg((g(()())))) (lg((og(()//)) &
Lemma 12 (Adaptive Freedman's inequality, Theorem 9 in Zimmert & Lattimore (2022)). Let Xi...., Xr be a martingale difference sequence adapted to filtration Fi C ... C Fr such that E
[x. is finite almost surely. Then, for any ( 0, it holds with probability at least S that
T
maxx(Ur, vVrl) + 2UT log
mmax(Ur: vvr]
Σ
X1
& 31
VT log
1=1
where Vr = E [xx,) Fi] and Un = (() X1).
Lemma 13 (Theroem 5.1 of Chen et al. (2018)), Let RG and R2 two distributions on X. ffo some [0, it holds that TV(R RR2) < I-a then there exist two distributions on the same probability space G1 and G2 such that
(1 (1 a1 + = (1-a)Ra + G2-
41

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
42

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
43

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
wwwpabmmsssooooooiiiooooooooooee moylicyyy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //iips.coo. wttpccccpppccceeeeccecccccccccct wetiic/ggge///eeeeeeeeeeiieieeee /Codeeubeeeeeeeeee / ssi wweeinsssoooooolo oniioniyllcy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
44

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: : /ttp///eerrp/.e//ee//eee/ cc cc/ppublitt/thhhccclddeeeliie
didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work focuses theoretical Machine Learning.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
45

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi sneerswwihcodeee thcode. com/ datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
46

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
47


Reviews:
[{"review_id": "s4Uc0xJaaL", "reviewer": "Reviewer_FpXy", "summary": "In this work, the authors consider a best-of-both-worlds multi-armed bandits problem where the losses are not bounded and instead are heavy-tailed. \nTo be precise, in the stochastic setting, losses are generated from fixed distributions.\nIn the oblivious adversarial setting, the losses are drawn from distributions that are generated arbitrarily and can change from one round to the next.\nIn both cases, these distributions can sample heavy-tailed losses, which are defined such that the $1 + v$^{th} moment of the losses are bounded by $u^{1 + v}$ for some constants $u > 0$ and $v \\in (0, 1]$. \nThis setting is more challenging than the standard BOBW approach because of the generation process of the losses, which can lead to negative unbounded losses but also because the performance is evaluated in terms of expected regret rather than in terms of pseudo-regret, which is a more challenging measure. \n\nThe authors propose and analyze an algorithm based on the FTRL framework. They discuss that simply using regularization with Tsallis entropy (which is the state of the art for the standard BOBW MAB problem) cannot handle large negative losses and instead rely on the log barrier regularizer, which is a standard approach to handle problems where the stability is difficult to bound.\nWhile this approach itself is sufficient to derive bounds in the adversarial regime, achieving BOBW bounds requires supplementary tricks:\n\nThey then use a detect-switch strategy to monitor whether the environment is stochastic, in which case the arms are sampled at a rate that ensures stochastic guarantees, and otherwise make a definitive switch to the adversarial regime and use the previously discussed FTRL with log-barrier regularization.", "questions": "Do you think that the detect-switch framework is necessary to achieve these BOBW methods or whether some more straightforward method (like FTRL with 1/2 Tsallis-Inf for the standard BOBW MAB with bounded losses)?", "limitations": "This work is purely theoretical and the limitations of the applicability of the results are properly detailed in the conditions of each theorem.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors tackle a challenging problem of BOBW bandits with heavy tail losses and propose a solution that achieves near-optimal results in both the adversarial and the stochastic regime. \n\nThe proposed methods combine well-studied methods in the BOBW literature and this paper highlights another topic of interest for the FTRL with log barrier framework.\nThe authors provide a detailed analysis of their methods and provide a very detailed explanation of their choices of methods.", "weaknesses": "While the presented results are novel and interesting, the proposed method is suboptimal by several logarithmic factors both in the stochastic and in the adversarial framework. Both the detect-switch method and FTRL with log barrier are known to be suboptimal in the standard BOBW MAB problem with bounded losses, which means that improving upon the existing results might require a completely different approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Comment 1: Both the detect-switch method and FTRL with log barrier are known to be suboptimal in the standard BOBW MAB problem with bounded losses, which means that improving upon the existing results might require a completely different approach.\n\nRe: We agree that both log-barrier (for adversarial bandits) and detect-switch method (for BOBW) do not achieve the optimal regret (i.e., suffering extra $\\\\log T$ factors) even with bounded losses, although to our knowledge, currently they are the only known approaches that are promising to handle heavy-tailed bandits in BOBW (and previous approaches need additional assumptions), and how to narrow the gap towards the optimal regret is still largely open, for which a totally different algorithm design seems to be necessary.\n\n> Comment 2: Do you think that the detect-switch framework is necessary to achieve these BOBW methods or whether some more straightforward method (like FTRL with 1/2 Tsallis-Inf for the standard BOBW MAB with bounded losses)?\n\nRe: As we discussed in the last paragraph in the main body, it is largely unknown whether purely online algorithms (e.g., FTRL) alone can achieve BOBW regrets in heavy-tailed bandits. Even considering the adversarial regime only, log-barrier seems to be necessary, so one promising direction is to show that OMD/FTRL alone (without detect-switch framework) with log-barrier can achieve BOBW guarantee, although involved theoretical analysis is expected.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3gGu7OEWBj", "reviewer": "Reviewer_t2SX", "summary": "This work studied the adversarial bandit problem with heavy-tailed distribution. It also studied the best-of-both-world setting. It relaxed the non-negative assumption and analyzed near-optimal algorithms.\n\n\n\n===================\n\nI would like to keep the score after reading the rebuttal.", "questions": "Please refer to the 'Weaknesses' part.", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work discussed the earlier literature in detail.\n1. It highlighted the technical challenges.\n1. Overall, I think this work provided a good set of results and explained them well.", "weaknesses": "1. Is it possible to bound the expected regret of the proposed algorithm? What is the limitation?\n1. As mentioned in Section 2, there are two possible definitions of regret in the adversarial setting. The author(s) should clarify which definition is used in the related works.\n1. A table comparing all results is appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Comment 1: Is it possible to bound the expected regret of the proposed algorithm? What is the limitation?\n\nRe: We believe that the reviewer is asking whether it is possible to bound the stronger regret $\\mathbb{E}[\\bar{R}_T]$ defined in line 147 in the adversarial regime. We now state the challenge of that in heavy-tailed bandits.\n\nBy definition, we would like to bound\n$$\\\\mathbb{E}[\\\\bar{R}\\_T] = \\\\mathbb{E}[ \\\\mathbb{E}[\\\\sum\\_{t=1}^Y \\\\langle w\\_t - y\\_{i^*}, \\\\ell\\_t \\\\rangle]|\\\\ell\\_1,\\\\dots,\\\\ell\\_T].$$\n\nNote that here $i^*:=\\\\text{argmin}\\_i \\\\sum\\_{i=1}^T \\\\ell\\_{t,i}$, which depends on the realization of loss sequence. Looking at the inner expectation, the quantity $\\sum_{t=1}^Y \\langle w_t - y_{i^*}, \\ell_t \\rangle$ depends on not only the policy ($w_t$), **but also** the realization (the scale of losses). In other words, as long as the losses have large scale, the absolute quantity of regret is also large, even if the learning algorithm is good.\n\nTherefore, one potential solution is to bound $\\mathbb{E}[\\bar{R}_T]$ hopefully thanks to the outer-level expectation (i.e., the realizations cannot be always very large due to the heavy tail definition). It is unclear to us whether this can be well bounded. Notably, this was also our intuition for why it is reasonable to consider pseudo-regret in heavy-tailed bandits: strong regret could be no longer meaningful as a performance metric.\n\n> Comment 2: As mentioned in Section 2, there are two possible definitions of regret in the adversarial setting. The author(s) should clarify which definition is used in the related works.\n\nRe: Thanks for the suggestion. We will clarify this in future versions. Roughly speaking, in the heavy-tailed case, both [1] and us consider pseudo-regret (as we explained in our Remark 2), and in the bounded-loss case, the stronger regret is typically considered and can be handled.\n\n> Comment 3: A table comparing all results is appreciated.\n\nRe: We thank the reviewer for the suggestion. We will consider adding a table to summarize the existing results for a clear comparison between our work and previous ones.\n\nReferences\n\n[1] Huang, Jiatai, Yan Dai, and Longbo Huang. \"Adaptive best-of-both-worlds algorithm for heavy-tailed multi-armed bandits.\" international conference on machine learning. PMLR, 2022. https://proceedings.mlr.press/v162/huang22c.html", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W9N09B4sEp", "reviewer": "Reviewer_LBNn", "summary": "This paper considers the bandit problem for heavy-tailed losses and proposes an algorithm that achieves a nearly tight high-probability regret bounds.\nThe proposed algorithm has a best-of-both-worlds guarantee, i.e., it achieves nearly tight bounds in both adversarial and stochastic settings.\nThe proposed approach is also shown to be useful in terms of local differential privacy.", "questions": "How tight is the bound of the proposed algorithm in terms of its dependence on $\\log T$?\nAre there any known comparable lower bound?", "limitations": "The limitations and potential negative societal impact are adequately addressed.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This study bypasses the assumption of truncated non-negative losses, which was required in the previous study by Huang et al. (2022).\n- This paper shows high-probability regret bounds, which is rare in the contexts of heavy-tailed bandits and best-of-both-worlds algorithms.\n- Obtained regret upper bounds are almost tight.", "weaknesses": "- The proposed algorithm requires prior knowledge of the parameters $u$ and $v$. This is a weakness when compared to algorithms that are adaptive to these parameters, e.g., by Huang et al.(2022) and  Genalti et al. (2024).\n- There is a lack of mention or analysis of intermediate settings between stochastic and adversarial settings, e.g., corrupted environments.\nA number of best-of-both-worlds algorithms are also effective in these settings (e.g., (Lee et al., 2020), (Zimmert and Seldin, 2021), (Dann et al., 2023)).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Comment 1: The proposed algorithm requires prior knowledge of the parameters $u$ and $v$. This is a weakness when compared to algorithms that are adaptive to these parameters, e.g., by Huang et al.(2022) and Genalti et al. (2024).\n\nRe: While there are $(u,v)$-adaptive algorithms proposed in [1, 2], they handle **one single regime only** (adversarial regime in [1] and stochastic regime [2]) and hence are not directly comparable to our BOBW result. Moreover, these adaptive algorithms require the truncated non-negative loss assumption.\n\nThe setup we consider in this work is the BOBW setting with the knowledge of $u$ and $v$. Under this setup, the **only** existing result is from [1], which is optimal in both regimes and requires the truncated non-negative loss assumption. Compared to that, we still achieve BOBW guarantee but try to get rid of the additional assumption.\n\n> Comment 2: There is a lack of mention or analysis of intermediate settings between stochastic and adversarial settings, e.g., corrupted environments. A number of best-of-both-worlds algorithms are also effective in these settings (e.g., (Lee et al., 2020), (Zimmert and Seldin, 2021), (Dann et al., 2023)).\n\nRe: That is a good point. Online Learning-based algorithms naturally ensure regret guarantee in the corrupted regime through advanced and elegant analysis, and there is no existing result on achieving regret guarantee in the corrupted regime using the detect-switch framework. It is unclear whether/how the detect-switch framework can do that (by, e.g., explicitly detecting the degree of corruption $C\\in[0, \\Theta(T)]$). This is indeed an informative remark we may consider adding to future versions.\n\n> Comment 3: How tight is the bound of the proposed algorithm in terms of its dependence on $\\log T$? Are there any known comparable lower bounds?\n\nRe: There are two main sources of $\\log T$ factors in our work, including 1) the log-barrier regularizer and 2) high-probability guarantees (from concentrations). For the latter one, we mean that the $\\log T$ factor is inevitable for a high-probability guarantee, even considering the adversarial regime only, as shown in [3].\n\nIn terms of in-expectation regret, the worst-case lower bound in the adversarial regime is $\\Omega(u K^{\\frac{v}{1+v}} T^{\\frac{1}{1+v}})$, and the gap-dependent lower bound in the stochastic regime is $\\Omega(\\sum_{i\\neq i^*}\\frac{\\log T}{(\\Delta_i)^{1/v}})$ as we mentioned in the paper.\n\nIn terms of high-probability regret, when heavy tails are involved, (to our knowledge) there is no regret lower bound showing the refined dependency on $\\log T$.\n\nReferences\n\n[1] Huang, Jiatai, Yan Dai, and Longbo Huang. \"Adaptive best-of-both-worlds algorithm for heavy-tailed multi-armed bandits.\" international conference on machine learning. PMLR, 2022. https://proceedings.mlr.press/v162/huang22c.html\n\n[2] Genalti, Gianmarco, Lupo Marsigli, Nicola Gatti, and Alberto Maria Metelli. \"$(ε, u) $-Adaptive Regret Minimization in Heavy-Tailed Bandits.\" In The Thirty Seventh Annual Conference on Learning Theory, pp. 1882-1915. PMLR, 2024. https://proceedings.mlr.press/v247/genalti24a.html\n\n[3] Gerchinovitz, Sébastien, and Tor Lattimore. \"Refined lower bounds for adversarial bandits.\" Advances in Neural Information Processing Systems 29 (2016). https://proceedings.neurips.cc/paper_files/paper/2016/hash/2f37d10131f2a483a8dd005b3d14b0d9-Abstract.html", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3rRXnEftPN", "reviewer": "Reviewer_q85G", "summary": "This paper studies multi-armed bandits (MAB) with heavy-tailed losses.\nIn the heavy-tailed bandits literature, two common assumptions are made about the losses: either a known upper bound $u$ on (1+$v$)-th raw moments (where both/either $v$ and/or $u$ are known) or truncated non-negativity (or non-positivity) assumption.\nThis paper considers the former and proposes a Best-of-Both-Worlds (BOBW) policy based on the Online Mirror Descent (OMD) and detect-switch framework.\nThe authors present high-probability near-optimal regret bounds that can provide meaningful expected regret analysis. \nThe main techniques used in their proofs rely on variants of Freedman's inequality (Lemma 11 and 12).\n\n- - -\nAfter rebuttal:\n\nThe proof for the modified algorithm seems correct, though it introduces an additional logarithmic factor. \nWhile I believe a better result could potentially be achieved using the general FTRL framework in BOBW research instead of the switch-detect framework, further improving the current analysis could be explored as a separate research direction.\n\nAs a result, I am changing my score from 3 to 6 (and the soundness rating from 1 to 3).", "questions": "1. What is $\\hat{y}\\_t$ in (2)?\n\n1. What is $t\\_{quit}$ in (13)? (just $t\\_{sw}$?)\n\n2. I cannot understand why (55) holds with $\\tau\\$ terms inside when $t > \\tau$. To be specific $(\\tau\\_i / K)^{(1-v)/(1+v)}$ term.\n\n3. Why (59) holds for all $i \\in [K]$ instead of $i \\in A\\_t$? The last step used the test (5), where only $i\\in A\\_t$ satisfies $\\leq$.\n\n4. Can you explain more on (63)?\n\n5. Although Algorithm 2 just used fixed constant, $c\\_1 = 6$, why it should be defined as a variable? It seems not necessary.", "limitations": "I follow the whole proofs up to Appendix C.\n\n### Minor comments\n1. Step 1 in p. 16: in Algorithm 1, $\\bar{w}_1$ is not defined since some necessary terms like $\\eta_0$ and $\\hat{\\ell}_0$ are not defined. Therefore, you should define $\\bar{w}_1 = w_1$ first.\nAlso, the last inequality of step 1 holds only when $K \\leq T$. This is a very minor one, but it should be specified in somewhere in Lemma.\n\n2. Line 629: we first some -> we first fix some?\n\n3. Line 631: why $w\\_{s,i}$ is non-decreasing? It would be true only if $i$ is activated.\n\n4. unfinished log term appears in (48).\n\n5. It would be better to specify that Lemma 8 is applied to derive the second inequality in (50).\n\n6. It would be better to specify that the result of (62) is applied to derive the third inequality in (64).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "### Quality and Clarity\nThe main manuscript is clearly written and easy to follow.\n\n### Originality and Significance\nThis paper provides high-probability regret bounds for both stochastic and adversarial settings. \nThe proposed policy is novel as it utilizes both OMD and a switch-detect policy specifically for (stochastic) heavy-tailed bandits.\nAdditionally, the results imply the achievability of BOBW performance in a pure local differential privacy setting.", "weaknesses": "### Assumption on losses\nAlthough the authors claim that their assumption, which requires knowledge of both $v$ and $u$, relaxes the undesirable truncated non-negative losses considered in Huang et al. (2022) or non-positivity assumption in Genalti et al. (2024), I disagree with this claim.\nGenalti et al. (2024) showed that there is **no** $u,v$-adaptive algorithm that can achieve the lower bound for known $u,v$ setting.\nI believe that these assumptions cannot be directly compared to determine which is generally weaker **in practice**.\nHowever, the truncated assumption appears to be more challenging problem in this context.\n\n### Proof of Lemma 2 - step 2 (p. 17)\nIn the equation between (25) and (26), the authors use $\\bar{w} = \\rho /2$, which is **incorrect** since $\\rho = 2/w$ not $2/\\bar{w}$ as shown in Algorithm 1.\nTherefore, the results in (26) and (27) are not necessarily true.\nThis implies that the authors should modify the choice of $\\eta$, which was designed to cancel terms related to the results in (27). \nThis issue seems very critical and needs to be addressed, even if it might not change the order of regret.\n\n### Proof for adversarial setting (Alg. 2)\nBy definition, for $s \\leq t\\_{sw}-1$, none of the tests (6), (7), and (8) should be satisfied. \nThis implies that for all $i\\in [K]$, (6) should not be satisfied, i.e., for all $i \\in [K]$,\n$$\n\\begin{equation*}\n       | {\\frac{\\hat{L}\\_{s,i}}{s} - \\hat{\\mu}\\_{s,i}} | > 9u(\\cdot)^{v/(1+v)} + 1[i \\in A\\_s] \\text{Width}(s)/s + 1[i \\not\\in A\\_s] \\text{Width}(\\tau\\_i)/\\tau\\_i.\n\\end{equation*}\n$$\nHowever, equation (69) used exactly the opposite condition. \nTherefore, the arguments related to bounding Part A seem incorrect.\n\nAlso, what is $\\hat{L}\\_{t\\_{sw}-1}'$?\nIs it different from the cumulative importance-weighted estimate?\n\n### Stochastic results\nFrom (64), the result is of order $K\\log K \\log T (\\log(T/\\xi))^3$, not $K\\log K \\log T (\\log(T/\\xi))^2$ stated in Theorem 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We first would like to thank Reviewer q85G for carefully reading our work and providing constructive comments, which significantly help us improve our work. We are also glad to further engage in interactive discussions with the reviewer.\n\n> Cmt 1: Why Eq. (55) holds\n\nRe: This is indeed a mis-calculation. After correction, Eq. (55) should be\n\n$$\\\\left| \\\\widehat{L}\\_{t,i} - \\\\sum_{s=1}^t \\\\mu\\_{s,i} \\\\right| \\\\leq \\\\text{Width}(t)\\\\cdot t/ \\\\tau\\_i ,$$ where basically the original $\\text{Width}(\\tau_i)$ is now replaced with $ \\text{Width}(t)$.\nEq. (51) now implies that\n$$ \\\\left|\\\\hat{L}\\_{t,i} - \\\\sum\\_{s=1}^t\\mu\\_{s,i}\\\\right| \\\\leq  \\\\mathbb{I}\\\\{i\\\\in A\\_t\\\\} \\text{Width}(t) +\\mathbb{I}\\\\{i\\notin A_t\\\\} \\text{Width}(t)\\frac{t}{\\tau_i}. $$\nIn other words, before the deactivation ($t\\leq \\tau_i$), the tightness of our control on $\\\\left|\\\\widehat{L}\\_{t,i} - \\sum_{s=1}^t \\mu_{s,i}\\\\right|$ is unchanged, and after that, it becomes looser (with $\\\\text{Width}(\\\\tau\\_i)$ replaced by $\\text{Width}(t)$).\n\nWhile this correction leads to changes in tests (6)-(8), we justify that we will still get the claimed regret guarantee (albeit an extra log factor in the adversarial regime). The **key intuition** is that, to fix this gap, we are replacing $\\text{Width}(\\tau_i)$ with $\\text{Width}(t)$ in some steps, which is the same thing we did in the last step of Eq. (75) in the current proof. In other words, these $\\text{Width}(\\tau_i)$ in the analysis eventually becomes $\\text{Width}(t_{\\text{sw}})$, sooner or later.\n\nWe provide the complete roadmap below.\n\n**A. New tests (6)-(8)**\n\nThe corrected tests (6)-(8) should be like:\n\n$$\\\\text{test (6):} \\\\exists i\\\\in[K] \\\\text{\\~such that\\~}\\\\left| \\\\frac{\\\\widehat{L}\\_{t,i}}{t} - \\\\widehat{\\\\mu}\\_{t,i} \\\\right|> 9u\\\\left(\\\\log(\\\\beta/\\\\zeta)/ N\\_{t,i} \\\\right)^{\\\\frac{v}{1+v}} +  \\\\mathbb{I}\\\\{i\\\\in A\\_t\\\\}\\\\frac{\\\\text{Width}(t)}{t} +  \\\\mathbb{I}\\\\{i\\\\notin A\\_t\\\\}\\\\frac{\\\\text{Width}(t)}{\\\\tau\\_i},$$\n$$\\\\text{test (7):} \\\\exists i\\\\notin A\\_t \\text{\\~such that\\~}  (\\\\hat{L}\\_{t,i} - \\\\min\\_{j\\\\in A_t} \\\\hat{L}\\_{t,j})/t > (c_1 + 4)\\\\text{Width}(t)/(\\\\tau_i -1),$$\n$$\\\\text{test (8):} \\\\exists i\\\\notin A\\_t \\\\text{\\~such that\\~}  (\\\\hat{L}\\_{t,i} - \\\\min\\_{j\\\\in A_t} \\hat{L}_{t,j})/t \\leq (c_1 - 4)\\text{Width}(t)/\\tau_i,$$\n\n**B. Stochastic regime**\n\nAfter correcting Eq. (55) and tests (6)-(8), the bound in stochastic regime is unchanged, following the three steps in Sec. 4.2.1 (lines 274-280). That is, we are still able to show that (modified) tests (6)-(8) never fail. Then, steps 2 and 3 are not affected (step 2 is only about the round up to $\\tau_i$, not after, and step 3 is only about the sampling strategy), so we get the same bound.\n\n**C. Adversarial regime**\n\nThe key is whether the modified tests will impact the analysis and the guarantee in the adversarial regime, which we will show below is not the case.\n\n**C.1. Regret decomposition**\n\nTo get the regret decomposition in Eq. (13), it is crucial to show that Eq. (66) is still true. With the modified tests, we can still show that (again with $\\text{Width}(\\tau_i-1)$ replaced with $\\text{Width}(t_{\\text{sw}}-1)$):\n$$\\sum_{s=1}^{t_{\\text{sw}}-1} \\mu_{s,i} - \\sum_{s=1}^{t_{\\text{sw}}-1} \\mu_{s,I^*_{t_{\\text{sw}}-1}} > (c_1-6) \\frac{t_{\\text{sw}}-1}{\\tau_i-1} \\text{Width}(t_{\\text{sw}}-1) \\geq 0.$$\nTherefore, the regret decomposition in Eq. (13) still holds. The next step is to bound parts A, B, and C therein.\n\n**C.2. Parts A and B**\n\nWith the modified tests, now we have\n$\\text{part A} = O\\left(\\left(\\frac{\\log(\\beta/\\zeta)}{N_{t_{\\text{sw}}-1,i}}\\right)^{\\frac{v}{1+v}} + \\frac{\\text{Width}(t_{\\text{sw}}-1)}{\\tau_i-1} \\right),$\nand\n$\\text{part B} = O\\left(\\text{Width}(t_{\\text{sw}}-1)/(\\tau_i -1) \\right).$\nBoth of them have the original $\\text{Width}(\\tau_i-1)$ replaced with $\\text{Width}(t_{\\text{sw}}-1)$.\n\n**C.3. Part C**\n\nSince part C is related to action $i^*_{t_{\\text{sw}}-1}$ only, which is active in round $t_{\\text{sw}}-1$ as shown in C.1 above, it is not affected, and we still have\n$$\\text{part C} = O\\left(\\text{Width}(t_{\\text{sw}}-1)\\right)$$\n**C.4. Final calculation**\n\nCombing three terms, now Eq. (75) becomes\n$$\\\\sum\\_{s=1}^{t\\_{\\\\text{sw}}-1}\\\\mu\\_{s,a\\_t} - \\\\sum\\_{s=1}^{t\\_{\\\\text{sw}}-1}\\\\mu\\_{s,i^*} = O\\\\left(\\\\sum\\_{i=1}^K N\\_{t\\_{\\\\text{sw}}-1,i} \\\\cdot u\\\\left( \\\\frac{\\\\log(\\\\beta/\\\\zeta)}{N\\_{t\\_{\\\\text{sw}}-1,i}} \\\\right)^{\\\\frac{v}{1+v}}   \\\\right)+ O\\\\left( \\\\underbrace{\\\\sum\\_{i=1}^K N\\_{t\\_{\\\\text{sw}}-1,i}\\\\frac{\\\\text{Width}(t\\_{\\\\text{sw}}-1)}{\\\\tau_i -1}}\\_{\\\\text{* term}} \\\\right) + O\\\\left( \\\\text{Width}(t\\_{\\\\text{sw}}-1) \\\\right).$$\nThe first and third term are the same as original ones (and can be well bounded), and in the second term (* term), the previous $\\text{Width}(\\tau_i-1)$ becomes $\\text{Width}(t_{\\text{sw}}-1)$, which will be our focus below.\n\nApplying Lemma 6 to $N\\_{t\\_{\\\\text{sw}}-1,i}$ and expanding $\\text{width}(\\cdot)$, the * term is bounded by\n\n$$O\\\\left( \\\\sum_{i=1}^K\\\\left( q_i \\\\tau_i(1+\\\\log T)\\\\right)\\\\frac{u K^{\\\\frac{v}{1+v}} (t_{\\\\text{sw}}-1)^{\\frac{1}{1+v}} \\\\left(\\\\log(\\\\beta/\\\\zeta) \\\\right)^{\\\\frac{3v}{3v+1}}}{\\\\tau_i -1}  + \\\\sum_{i=1}^K\\\\log(\\\\beta/\\\\zeta)\\\\frac{u K^{\\\\frac{v}{1+v}} (t_{\\\\text{sw}}-1)^{\\\\frac{1}{1+v}} \\\\left(\\\\log(\\\\beta/\\\\zeta)\\\\right)^{\\\\frac{3v}{3v+1}}}{\\\\tau_i -1}\\\\right).$$\n\nThe first part is not new and is bounded using $\\\\sum_{i=1}^K q_i = O(\\\\log K)$, and the second part (which used to be a lower-order term) is also under control since $\\\\tau_i \\\\geq K+1$ due to the initialization in Algorithm 2.\n\nFinally, now the bound becomes\n\n$$\\\\sum_{s=1}^{t_{\\\\text{sw}}-1} \\\\mu_{s,a_t} - \\\\sum_{s=1}^{t_{\\\\text{sw}}-1} \\\\mu_{s,i^*} = O\\\\left( (\\\\log K) (\\\\log T) u K^{\\frac{v}{1+v}} (t_{\\\\text{sw}}-1)^{\\\\frac{1}{1+v}} \\\\left(\\\\log(\\\\beta/\\\\zeta)\\\\right)^{1 + \\\\frac{3v}{3v+1}} \\\\right).$$\n\nAfter correction, it has an extra $ \\\\left( \\\\log(\\\\beta/\\\\zeta) \\\\right)$ factor, which is coming from the second part of the * term.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper tackles the best-of-both-world problem for multi-armed bandits with heavy-tailed losses. This work removes the restrictive assumption made in Huang et al. (2022).  Although there are concerns about the sub-optimality of the regret bounds, the consensus is that the ideas developed in this paper are valuable and interesting.
Decision: Accept (poster)