Title: Stochastic Multi-armed Bandits: Optimal Trade-off among Optimality, Consistency, and Tail Risk
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi-armed bandit

worst-case optimality

instance-dependent consistency

light-tailed risk
Abstract: We consider the stochastic multi-armed bandit problem and fully characterize the interplays among three desired properties for policy design: worst-case optimality, instance-dependent consistency, and light-tailed risk. We show how the order of expected regret exactly affects the decaying rate of the regret tail probability for both the worst-case and instance-dependent scenario. A novel policy is proposed to achieve the optimal regret tail risk for any regret threshold. Concretely, for any given $\alpha\in[1/2, 1)$ and $\beta\in[0, 1)$, our policy achieves a worst-case expected regret of $\tilde O(T^\alpha)$ and instance-dependent expected regret of $\tilde O(T^\beta)$, while enjoys a probability of incurring an $\Omega(T^\delta)$ regret that decays exponentially with a polynomial $T$ term. Such decaying rate is proved to be best achievable. We also generalize our analysis to the stochastic multi-armed bandit problem with non-stationary baseline rewards, where in each time period $t$, the decision maker pulls one of $K$ arms and collects a reward which is the sum of three terms: the mean of the pulled arm, an independent noise, and a non-stationary baseline reward as a function of $t$. Our results reveal insights on the trade-off between expected regret and tail risk for both worst-case and instance-dependent scenario, indicating that more sub-optimality and inconsistency leaves space for more light-tailed risk of incurring a large regret.

Stochastic Multi-armed Bandits: Optimal Trade-off among Optimality, Consistency, and Tail Risk

Abstract

We consider the stochastic multi-armed bandit problem and fully characterize the interplays among three desired properties for policy design: worst-case optimality, instance-dependent consistency, and light-tailed risk. We show how the order of expected regret exactly affects the decaying rate of the regret tail probability for both the worst-case and instance-dependent scenario. A novel policy is proposed to achieve the optimal regret tail risk for any regret threshold. Concretely, for any given α ∈ [1 / 2 , 1) and β ∈ [0 , 1) , our policy achieves a worst-case expected regret of ˜ O ( T α ) and instance-dependent expected regret of ˜ O ( T β ) , while enjoys a probability of incurring an Ω( T δ ) regret that decays exponentially with a polynomial T term. Such decaying rate is proved to be best achievable. We also generalize our analysis to the stochastic multi-armed bandit problem with non-stationary baseline rewards, where in each time period t , the decision maker pulls one of K arms and collects a reward which is the sum of three terms: the mean of the pulled arm, an independent noise, and a non-stationary baseline reward as a function of t . Our results reveal insights on the trade-off between expected regret and tail risk for both worst-case and instance-dependent scenario, indicating that more sub-optimality and inconsistency leaves space for more light-tailed risk of incurring a large regret.

1 Introduction

The stochastic multi-armed bandit (MAB) problem is a well-established area of research in sequential decision-making under uncertainty, applications of which include online advertising, recommendation systems, digital clinical trials, etc. Let T be a pre-fixed time horizon. In a standard MAB problem, the decision maker selects one of several arms (rewards of which are unknown a priori) in each time period t ∈ [ T ] with the goal of maximizing the expected sum of rewards over all time periods. The MAB problem highlights the exploration-exploitation trade-off: the decision maker must balance between exploring arms with relatively unknown reward distributions and exploiting arms with relatively known high expected rewards. There is a significant amount of literature on MAB, with a comprehensive review provided in [25].

In order to evaluate policy performance and guide policy design, much of the MAB literature uses the metric of maximizing the expected cumulative reward , or equivalently minimizing the expected

regret (where regret is generally defined as the difference between the cumulative reward of pulling the best arm and the cumulative reward of a policy). However, it has recently been documented that if an MAB policy design only focuses on minimizing the expected regret , the policy may be exposed to heavy-tailed risks of incurring a large regret; see [11] and [24] for detailed documentation. Roughly speaking, designing a policy that focuses only on the expected regret could be analogous to designing an investment portfolio that focuses only on the expected return without looking at other risks. As recently documented in [24], many renowned algorithms (e.g., Successive Elimination (SE) [10], Thompson Sampling (TS) [20]), despite of enjoying optimality on expected regret, can incur a 'heavy-tailed risk' - the probability of incurring a linear regret slowly decays at a polynomial rate Ω(poly(1 /T )) as T tends to infinity. In contrast, a 'light-tailed' risk in this MAB setting means that the probability of a policy incurring a linear regret decays at an exponential rate exp( -Ω( T γ )) for some γ > 0 .

Two important and popular notions in the MAB literature to describe properties of a policy - worstcase optimality and instance-dependent consistency -are both defined in terms of expected regret. There lacks an understanding in the literature about how much regret tail risk will arise by pursuing optimality and/or consistency for policy design. In particular:

How do different levels of optimality and consistency respectively affect the tail risk? What is the optimal trade-off among optimality, consistency, and light-tailed risk?

Motivated by these questions, we summarize our contributions in Section 1.1. To facilitate describing the results on regret orders and function orders, we use O ( · ) ( ˜ O ( · ) ) and Ω( · ) ( ˜ Ω( · ) ) to present upper and lower bounds on the growth rate up to constant (logarithmic) factors, respectively, and Θ( · ) ( ˜ Θ( · ) ) to characterize the rate when the upper and lower bounds match up to constant (logarithmic) factors. We use o ( · ) and ω ( · ) to present strictly dominating upper bounds and strictly dominated lower bounds, respectively.

1.1 Our Contributions

1. We characterize the connections and interplays among worst-case optimality, instance-dependent consistency, and light-tailed risk. We show that relaxing worst-case regret order or instance-dependent regret order can help make the regret tail lighter in an information-theoretic way. Given the family of policies with a worst-case regret of ˜ O ( T α ) and instance-dependent regret of ˜ O ( T β ) , we demonstrate how fast we can best hope the probability of incurring an Ω( T δ ) decays with T , for any feasible tuple ( α, β, δ ) . (This can be viewed as a best-achievable 'lower bound" result. The next item describes a matching 'upper bound" result.)

2. We design a simple policy that, for any given α and β , obtains ˜ O ( T α ) worst-case regret and ˜ O ( T β ) instance-dependent regret, and simultaneously obtains the best achievable regret tail probability for both worst-case and instance-dependent scenarios. Our regret tail bounds are explicit formulas showing that the probability of incurring a worst-case regret larger than any x is bounded by exp( -˜ Ω(( x/T 1 -α ) ∧ T β )) , and that the probability of incurring an instance-dependent regret larger than any x is bounded by exp( -˜ Ω( T β )) . Our policy design builds upon the idea of successive elimination, and constructs novel bonus terms to balance among worst-case optimality, instancedependent consistency, and light-tailed risk.

3. Our theory is generalized to an advanced MAB model that allows a common non-stationary baseline among all arms for each time period. Different time periods can have different baseline rewards. The model may accommodate applications with structured non-stationarity beyond those from standard multi-armed bandit problems where arm rewards are assumed to be stationary. We apply a simple and novel technique of random permutation to deal with this additional feature. Last, a brief account of experiments are conducted to illustrate our theoretical findings. We discuss how tuning parameters affect the performance of our policy, and reiterate the insight that relaxing worstcase optimality and instance-dependent consistency (or allowing sub-optimality and inconsistency) may leave space for regret distribution being more light-tailed.

1.2 Related Work

Our work builds upon the stochastic multi-armed bandit (MAB) literature, and reviews can be found in [4, 20, 25, 17]. However, the tail risk of stochastic bandit algorithms is not well understood. [2, 21] showed that the policy regret generally concentrates only at a polynomial rate. That is, the probability of incurring c (ln T ) p regret (with c > 0 and p > 1 fixed) is polynomially decaying with T . [1] showed that a policy that obtains logarithmic regret can be sensitive to mis-specified risk parameter. That is, if one underestimated the risk parameter σ for subGaussian noises, then one may obtain a large regret. Note that the goal here is still to minimize the expected regret.

Our work is inspired by [11, 24]. [11] first analyzed heavy-tailed risk for bandit algorithms and showed that the family of information-theoretically optimized bandit policies suffers from serious heavy-tailed risk: the probability of incurring a linear regret is at least Ω(1 /T ) . They also modified UCB algorithms to reach a desired tail risk polynomially dependent on T , making the algorithms more robust to mis-specifications. [24] showed the incompatibility between instance-dependent consistency and light-tailed risk. They also proposed a simple policy design showing that worst-case optimality (expected regret bounded by ˜ O ( √ T ) ) permits light-tailed risk (tail bound exponentially decaying with √ T ). However, the interplay among optimalilty, consistency, and tail risk remains unclear. This is the question we want to fully address in this work.

Our work is also related with multi-tasking bandits, where minimizing expected regret is not the only goal (see, e.g., [8, 9, 28, 29, 23]). In particular, [23] studied the optimal trade-off between efficiency (obtaining low regret) and statistical power (estimating arm gaps correctly). They also extended their analysis to the MAB problem with non-stationary baseline rewards. We note that their task is still centered on expectation, and their results are entirely based on an instance-dependent perspective. Another line of related work is risk-averse formulations of the stochastic MAB problem (see, e.g., [22, 12, 18, 31, 27, 5, 26, 19, 30, 3, 14, 7, 6]). These formulations consider different objectives other than expected regret, such as mean-variance and value-at-risk. Our work's objective and metric (levels of tail risks) are different, which leads to different policy design and analysis.

1.3 Organization and Notation

The rest of the paper is organized as follows. In Section 2, we discuss the setup and introduce the key concepts. In Section 3, we show our main results. In Section 4, we extend our main results. In Section 5, we conclude. Experiments are provided in the supplementary material. Before proceeding, we introduce some notation. For any a, b ∈ R , a ∧ b = min { a, b } , a ∨ b = max { a, b } , and a + = max { a, 0 } . We denote [ N ] = { 1 , · · · , N } for any positive integer N . Next, we state our problem setting and describe all the notation used in the problem setting.

2 The Setup

2.1 The Model

Fix T as the time horizon and K as the number of different arms. Without loss of generality, we presume that K ≥ 2 , T ≥ 3 , and T ≥ K . In each time period t ∈ [ T ] , the decision maker (DM) has access to all prior information and then chooses an arm a t ∈ [ K ] to pull. Upon pulling the arm, the DMobserves and receives a reward r t,a t . Specifically, let H t = { a 1 , r 1 ,a 1 , · · · , a t -1 , r t -1 ,a t -1 } be the history information prior to time t . H 1 is set as ∅ for definition completeness.

In this work we consider policies that are allowed to utilize the knowledge of T . When we say π is policy, we mean that π is composed of a series of 'sub-policies": π (1) , · · · , π ( T ) , · · · . That is, with the prior knowledge of T , π executes π ( T ) throughout the whole time horizon. At time t , the action taken is determined by π t ( T ) : H t ∪ { T } ↦- → a t , which can potentially be random. Different T 's may lead to completely different sub-policies.

The timet reward is independently sampled as r t,a t = θ a t + ϵ t,a t , where θ a t is the mean reward of arm a t , and ϵ t,a t is a zero-mean random noise, independent across time periods. The random noise ϵ t,a t is assumed to be σ -subGaussian. That is, there exists a σ > 0 such that for any arm k and time t ,

Let θ = ( θ 1 , · · · , θ K ) be the vector of means. Let θ ∗ = max { θ 1 , · · · , θ K } be the optimal mean reward among the K arms. Note that the DM does not know b t or θ a priori, except that b t ∈ [0 , B ] (for some B ≥ 0 ) and θ ∈ [0 , 1] K . The empirical regret of the policy π = ( π 1 , · · · , π T ) under the mean vector θ and the noise parameter σ over a time horizon of T is defined as

Let ∆ k = θ ∗ -θ k denote the gap between the optimal arm and the k th arm. Define ∆ 0 > 0 such that

Denote n t,k = ∑ t s =1 1 { a s = k } as the number of times arm k has been pulled up to time t . We sometimes use n k = n T,k to denote the total number of times arm k pulled through the entire time horizon. We denote t k ( n ) as the time period that arm k is pulled for the n -th time. We define the pseudo regret as

and the aggregated random noise as

Therefore, the empirical regret can also be written as ˆ R π θ,σ ( T ) = R π θ,σ ( T ) -N π ( T ) . In this work, for notation simplicity, we will write ˆ R π θ ( T ) instead of ˆ R π ( T ) θ,σ ( T ) and R π θ ( T ) instead of R π ( T ) θ,σ ( T ) . The following simple lemma gives the mean and the tail probability of the aggregated noise N π ( T ) .

In the worst-case scenario, x is taken as Ω( √ T ) , and so the tail in Lemma 2.1 is negligible. In the instance-dependent scenario, x may take o ( √ T ) , making the tail in Lemma 2.1 no longer ignorable. We note that the empirical regret unavoidably incurs a tail in Lemma 2.1 due to the appearance of genuine noise, even if we always pull the optimal arm. Therefore, we will mainly focus on pseudo regret in our subsequent discussions.

2.2 Worst-case Optimality, Instance-dependent Consistency, Light-tailed Risk

Now we describe concepts that are needed to formalize the policy design and analysis.

Worst-case α -optimality. Fix α ∈ [1 / 2 , 1) . A policy is said to be worst-case α -optimal or simply, α -optimal , if for any ε > 0 , the policy has that

In brief, a policy is α -optimal if the worst-case expected regret can never be growing in T at a polynomial rate faster than T α . Intuitively, the smaller the α is, the better performance a policy has in terms of worst-case expected regret order.

Instance-dependent β -consistency. Fix β ∈ [0 , 1) . A policy is called instance-dependent β -consistent or simply β -consistent, if for any underlying true mean vector θ and any ε > 0 , the policy has that

In brief, a policy is β -consistent if the expected regret can never grow faster than T β for any fixed instance. We note that the 'worst-case" notion and the 'instance-dependent" notion, in these two items, are most commonly used in the bandits literature, and both notions care about the expectation of the regret distribution. The next notion concerns the tail of regret distribution.

3. ( δ, γ ) -tail. Fix δ ∈ (0 , 1] and γ ∈ [0 , 1] . We differentiate between two scenarios: worst-case and instance-dependent.

(a). A policy is called worst-case ( δ, γ ) -tailed, if there exists constants c ∈ (0 , 1 / 2) and C > 0 such that

Both c and C are allowed to rely on ( δ, γ ) . More intuitively, the above definition means that a policy is worst-case ( δ, γ ) -tailed if the worst-case probability of incurring a regret of T δ can be bounded by an exponential term of polynomial T γ :

All else equal, one would prefer a policy with larger γ so that the tail probability of incurring a big regret is lighter.

(b). A policy is called instance-dependent ( δ, γ ) -tailed, if for any underlying true mean vector θ , there exists constants c ∈ (0 , 1 / 2) and C > 0 such that

Note that the constant C may be dependent on both c and θ . In brief, a policy is instance-dependent ( δ, γ ) -tailed if the instance-dependent probability of incurring a regret of T δ can be bounded by an exponential term of polynomial T γ :

Remarks.

When defining the tail, we impose c ∈ (0 , 1 / 2) to avoid the corner case when δ = 1 . In such case, if c ≥ 1 , the tail probability is zero because θ ∈ [0 , 1] K . We note that when δ < 1 , the condition c ∈ (0 , 1 / 2) is not essential, and here we retain it for simplicity of exposition.

It is well known that for the stochastic MAB problem, one can design algorithms to achieve both 0 -consistency and 1 / 2 -optimality. Among them, two types of policies are of prominent interest: Successive Elimination (SE) and Upper Confidence Bound (UCB). The bonus term (or, the confidence radius) rad ( n ) is typically set as

with η > 0 being some tuning parameter. That said, both the SE policy and UCB policy may not perform well in terms of tail probability of incurring a large regret, as documented in [11] and [24].

3 Main Results

3.1 Tail Lower Bound: The Best to Hope

In this section, we show how fast the regret tail can decay as a function of T , given that a policy is α -optimal or/and β -consistent. More concretely, if a policy is α -optimal or/and β -consistent, Theorem 3.1 shows what is the fastest decaying rate we can hope for the probability that the pseudo regret is at least Ω( T δ ) (in either the worst-case or instance-dependent scenario).

Theorem 3.1. We have the following arguments.

Fix α ∈ [1 / 2 , 1) . If a policy π is α -optimal, then for any δ > α and γ > δ + α -1 , π can never be worst-case ( δ, γ ) -tailed.

Fix β ∈ [0 , 1) . If a policy π is β -consistent, then for any δ > β and γ > β , π can never be instance-dependent ( δ, γ ) -tailed.

In the proof of Theorem 3.1, briefly, for the worst-case scenario, we construct a series of instance pairs such that the gap between two arms shrink with a polynomial rate of 1 /T . For the instancedependent scenario, we fix a pair of instances and investigates how the tail probability scales in the two environments as T increases. An intermediate step towards doing the complete proof is a lemma showing that if the policy is 'effective", i.e., achieves a sub-linear regret under either case, then the estimation of the sub-optimal arm becomes more precise in probability as T increases. Details are provided in the supplementary material.

Theorem 3.1 implies Corollary 3.2, which is a direct application of the following argument - if a policy is not instance-dependent ( δ, γ ) -tailed, then it is also not worst-case ( δ, γ ) -tailed.

Corollary 3.2. Fix α ∈ [1 / 2 , 1) and β ∈ [0 , 1) . If a policy π is both α -optimal and β -consistent, then we have the following arguments.

For any δ > α and γ > ( δ + α -1) ∧ β , π can never be worst-case ( δ, γ ) -tailed.

For any δ > β and γ > β , π can never be instance-dependent ( δ, γ ) -tailed.

To help better understand Corollary 3.2 in an intuitive way, we let x = Θ( T δ ) , then the critical values of T γ for different scenarios and cases are listed in Table 1. That said, the best we can hope for the order of the regret tail bounds cannot decay faster than the critical values.

3.2 Tail Upper Bound: The Best to Achieve

In this section, we show that 'the best we can hope" is achievable by concrete policies. Given α and β , our goal is to design a policy such that the decaying rate of the log tail probabilities match the critical values indicated in Table 1. Namely, we design a single policy that is α -optimal and β -consistent, and enjoys the optimal tail rate in both worst-case and instance-dependent scenarios.

Algorithm 1 is designed upon the standard Successive Elimination (SE) algorithm. First, for each phase i , we add a random permutation step prior to taking the actions. This randomization step will be useful for extending our theories. Second, the bonus term rad ( n ) in SEwRP is different from standard bonus terms like (1) or those used in [24] - we need to reach balance among optimality, consistency, and tail risk. The concrete design is given in Theorem 3.3, which fully characterizes the regret tail bound of Algorithm 1 in both worst-case and instance-dependent scenarios.

Theorem 3.3. For the K -armed bandit problem, π = SEwRP with

satisfies the following properties: fix any η 1 , η 2 ≥ 0 , for any x > 0 , we have

1. (worst-case regret tail)

2. (instance-dependent regret tail)

The design of our bonus term is novel and hopefully provides additional insights, as follows. The first component can be interpreted as controlling the worst-case tail risk, while the second one can be regarded as controlling the instance-dependent tail risk. There exhibits a phase transition with respect to the size of confidence interval. At the beginning ˜ Θ( T 2 α -β ) time periods, the second term dominates, and so the confidence interval shrinks at a rate of 1 / √ n , suggesting that we focus more on exploration within the consistency constraint. While in the remaining time periods, the first term dominates, and so the confidence interval shrinks at a rate of 1 /n , suggesting that we focus more on exploitation within the optimality condition. Our policy design suggests that in real-world practice, to achieve more light-tailed risk, it might be beneficial to have two different phases in the policy design: more exploration at the beginning, and more exploitation afterwards.

The following proposition shows that our policy has ˜ O ( T α ) worst-case expected regret and ˜ O ( T β ) instance-dependent expected regret.

Proposition 3.4. For the K -armed bandit problem, π = SEwRP with (2) satisfies the following expected regret bounds (ignoring additive and multiplicative constant terms). If β > 0 or β = 0 and η 2 = Ω(1 ∨ σ ) , then

̸

Proof road-map. Without loss of generality, we assume arm 1 is optimal. Let the regret threshold x ≥ K . For any k = 1 , we define

S k = { Arm 1 is not eliminated before arm k } .

In the following, we differentiate between two scenarios.

For the worst-case scenario, we define

̸

Then with some basic algebra, one can build

̸

̸

We need to emphasize that introducing A ∗ allows us to establish ∆ k = Ω( x/T ) , otherwise we may only have ∆ k = Ω( x/KT ) . The goal of defining A ∗ is to make sure our analysis aligns with our new bonus design (we have a 1 /K α factor in the numerator) such that our tail probabilities have optimal dependence on the number of arms K .

We decompose the tail risk of an algorithm by two types of events: (i) spending too much time before correctly discarding a sub-optimal arm; (ii) wrongly discarding the optimal arm.

1. Consider the case when S k happens. This corresponds to the tail risk of spending too much time before correctly discarding a sub-optimal arm [24]. For either k ∈ A ∗ or k / ∈ A ∗ , we fix some non-random number n 0 < n k (the choice of n 0 is important: to be precise, we let n 0 = ⌈ ( x -K ) / (2 K ∆ k ) ⌉ when j ∈ A ∗ and n 0 = T/K when j / ∈ A ∗ ) where n 0 is a phase when a sub-optimal arm k is not eliminated by the optimal arm 1 . Then arm 1 and k are both not eliminated after each of them being pulled n 0 times, which means

The probability of this event can be bounded using concentration properties of sub-Gaussian random variables.

2. Now consider the case when ¯ S k happens. This corresponds to the tail risk of wrongly discarding the optimal arm [11, 24]. Then after some phase n , the optimal arm 1 is eliminated by some arm k ′ , while arm k is not eliminated. We emphasize that k = k ′ does not necessarily hold when K > 2 . As a result, the following two events hold simultaneously:

The two events leads to, respectively,

Then the tail probability can be further bounded by

which yields the second term in (3).

For the instance-dependent scenario, the proof follows similarly to that for the worst-case scenario, but with some amendment. For each k = 1 , we treat ∆ k as a constant, and instead of (5), we use

̸

Here, we do not need A ∗ to control the dependence on K . Also, when bounding the probability when S k happens, we accordingly define

The detailed proof is left to the supplementary material. We would like to emphasize the technical novelty compared to that in [24]. In general, since α and β become flexible constants, the proof requires more delicate controls on probability bounds. When proving the worst-case upper bound, we need a careful manipulation on rad ( n ) since we are dealing with the minimum of two different types of bonus terms. When proving the instance-dependent upper bound, we require a careful division of the tail event to make the bound as tight as possible, depending on specific instances ( θ ). In [24], the tail bound is only concerned with the worst-case scenario with α = 1 / 2 , and hence the aforementioned challenges do not exist.

Finally, we would like to note that although applying the naive explore-then-commit (ETC) strategy might be able to circumvent heavy-tailed risk, the approach has two main issues:

From the regret expectation perspective, the ETC policy can only achieve worst-case O ( T 2 / 3 ) expected regret bound. When α ∈ [1 / 2 , 2 / 3) , we still have to consider other types of policies. Further, it seems unlikely that without knowing the arm gaps { ∆ k } , the explorethen-commit policy can achieve ˜ O ( T α ) worst-case regret and ˜ O ( T β ) instance-dependent regret simultaneously with β < α .

From the regret tail risk perspective, the tail probability of incurring a large regret may not be optimal in the worst-case. Consider a simple 2 -armed bandit case with arm gap ∆ . An ETC policy with m = Θ( T α ) ( α ∈ [2 / 3 , 1) ) steps of exploration has a probability of Ω(exp( -m ∆ 2 )) committing to a wrong arm. In the worst-case scenario, let ∆ = T ( α -1) / 2 and the regret threshold be x = T ( α +1) / 2 / 4 ∈ ( m ∆ , ( T -m )∆) . Then if we incur a regret of x , it means we commit to a wrong arm after the exploration, which suggests P ( R π θ ( T ) > x ) = Ω(exp( -m ∆ 2 )) = Ω(exp( -T 2 α -1 )) = ω (exp( -x/T 1 -α )) .

4 Extension with Structured Non-stationarity

In this section, we extend our results to a more general stochastic MAB model. In certain applications, the assumption that the rewards for one arm are i.i.d. generated may fail to hold. On the other hand, the non-stationary MAB models are at the other extreme being a bit too conservative, where no structural properties exist. The stochastic MAB model with non-stationary baseline rewards tries to strike a balance between stationary stochastic MAB and fully adversarial MAB. Such a structure is receiving attentions in the literature (see, e.g., [13, 16, 15, 23]). We restate the setting as follows and highlights those different from the standard stochastic MAB model introduced in Section 2.

At each time t , upon pulling an arm a t , a reward is independently sampled as r t,a t = b t + θ a t + ϵ t,a t , where b t is the baseline reward chosen by the nature at time t , θ a t is the mean reward of arm a t , and ϵ t,a t is a zero-mean random noise, independent across time periods. The random noise ϵ t,a t is assumed to be σ -subGaussian. Let θ = ( θ 1 , · · · , θ K ) be the vector of means and θ ∗ = max { θ 1 , · · · , θ K } be the optimal mean reward among the K arms. Note that the DM does not know b t or θ a priori, except that b t ∈ [0 , B ] (for some B ≥ 0 ) and θ ∈ [0 , 1] K . The empirical regret of the policy π = ( π 1 , · · · , π T ) under the mean vector θ and the noise parameter σ over a time horizon of T is defined as

Note that here although b t appears in the model, the term get cancelled in the regret. Let ∆ k = θ ∗ -θ k denote the gap between the optimal arm and the k th arm. Same as Section 2.1, ∆ 0 , the pseudo regret and the aggregated random noise are defined accordingly.

Theorem 4.1. For the K -armed bandit problem with baseline rewards, π = SEwRP with (2) satisfies the following properties: fix any η 1 , η 2 ≥ 0 , for any x > 0 , we have

1. (worst-case regret tail)

(instance-dependent regret tail)

Proposition 4.2. For the K -armed bandit problem with baseline rewards, π = SEwRP with (2) satisfies the following expected regret bounds (ignoring additive and multiplicative constant terms). If β > 0 or β = 0 and η 2 = Ω(1 ∨ σ ) , then

The proof of Theorem 4.1 follows that of Theorem 3.3, except that when bounding the mean of error terms, we now have not only the true noise ϵ , but also a baseline b t . Lemma 4.3 is established to characterize the concentration of these combined error terms. We would like to note that the randomization step in SEwRP is crucial to hedge against the baseline rewards b t , for which deterministic algorithms may fail. With uniform permutation, although the estimation of one arm is still biased, the difference of estimations between two different arms becomes unbiased.

Lemma 4.3. Let τ i be the (random) time period that we pull the last arm in A i . For any n ≥ 1 and k, k ′ ∈ [ K ] , define

We have, for any x > 0 ,

5 Conclusion

In this work, we study the stochastic MAB problem with the goal of obtaining light-tailed regret distribution under given degree of worst-case optimality and instant-dependent consistency. We fully characterize the tradeoff among optimality, consistency, and tail risk via (i) proving lower bounds for decaying rate of incurring a large regret and (ii) proposing a novel algorithm with explicit regret tail upper bounds that match the lower bounds. We extend our analysis to the stochastic MAB model with non-stationary baseline rewards, and obtain similarly tight regret tail bounds. We study the empirical performance via synthetic simulations, and present insights of our results.

An open question for this work is whether our results can be generalized to the any-time case where the policy has no knowledge of T . For the lower bound, it has been pointed out in [21] that not knowing T may have a significant impact on the concentration of regret distribution for standard UCB policies. For the upper bound, as is pointed out in [24], an SE type policy will fail to obtain light-tailed regret distribution under the any-time design even for the basic MAB model. Meanwhile, it seems nontrivial to incorporate randomization into confidence bound based policies to handle the stochastic MAB model with non-stationary baseline rewards. We leave the question for future work.

References

K. Ashutosh, J. Nair, A. Kagrecha, and K. Jagannathan. Bandit algorithms: Letting go of logarithmic regret for statistical robustness. In International Conference on Artificial Intelligence and Statistics , pages 622-630. PMLR, 2021.

J.-Y. Audibert, R. Munos, and C. Szepesvári. Exploration-exploitation tradeoff using variance estimates in multi-armed bandits. Theoretical Computer Science , 410(19):1876-1902, 2009.

D. Baudry, R. Gautron, E. Kaufmann, and O. Maillard. Optimal thompson sampling strategies for support-aware cvar bandits. In International Conference on Machine Learning , pages 716-726. PMLR, 2021.

S. Bubeck and N. Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. arXiv preprint arXiv:1204.5721 , 2012.

A. Cassel, S. Mannor, and A. Zeevi. A general approach to multi-armed bandits under risk criteria. In Conference On Learning Theory , pages 1295-1306. PMLR, 2018.

J. Q. Chang and V. Y. Tan. A unifying theory of thompson sampling for continuous risk-averse bandits. In Proc. of the 36th AAAI Conference on Artificial Intelligence. AAAI Press , 2022.

N. Chen and S. Yang. Bridging adversarial and nonstationary multi-armed bandit. arXiv preprint arXiv:2201.01628 , 2022.

A. A. Deshmukh, U. Dogan, and C. Scott. Multi-task learning for contextual bandits. Advances in neural information processing systems , 30, 2017.

A. Erraqabi, A. Lazaric, M. Valko, E. Brunskill, and Y.-E. Liu. Trading off rewards and errors in multi-armed bandits. In Artificial Intelligence and Statistics , pages 709-717. PMLR, 2017.

E. Even-Dar, S. Mannor, Y. Mansour, and S. Mahadevan. Action elimination and stopping conditions for the multi-armed bandit and reinforcement learning problems. Journal of machine learning research , 7(6), 2006.

L. Fan and P. W. Glynn. The fragility of optimized bandit algorithms. arXiv preprint arXiv:2109.13595 , 2022.

N. Galichet, M. Sebag, and O. Teytaud. Exploration vs exploitation vs safety: Risk-aware multi-armed bandits. In Asian Conference on Machine Learning , pages 245-260. PMLR, 2013.

K. Greenewald, A. Tewari, S. Murphy, and P. Klasnja. Action centered contextual bandits. Advances in neural information processing systems , 30, 2017.

N. Khajonchotpanya, Y. Xue, and N. Rujeerapaiboon. A revised approach for risk-averse multi-armed bandits under cvar criterion. Operations Research Letters , 49(4):465-472, 2021.

G.-S. Kim and M. C. Paik. Contextual multi-armed bandit algorithm for semiparametric reward model. In International Conference on Machine Learning , pages 3389-3397. PMLR, 2019.

A. Krishnamurthy, Z. S. Wu, and V. Syrgkanis. Semiparametric contextual bandits. In International Conference on Machine Learning , pages 2776-2785. PMLR, 2018.

T. Lattimore and C. Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

O.-A. Maillard. Robust risk-averse stochastic multi-armed bandits. In International Conference on Algorithmic Learning Theory , pages 218-233. Springer, 2013.

L. Prashanth, K. Jagannathan, and R. K. Kolla. Concentration bounds for cvar estimation: The cases of light-tailed and heavy-tailed distributions. In Proceedings of the 37th International Conference on Machine Learning , pages 5577-5586, 2020.

D. J. Russo, B. Van Roy, A. Kazerouni, I. Osband, Z. Wen, et al. A tutorial on thompson sampling. Foundations and Trends® in Machine Learning , 11(1):1-96, 2018.

A. Salomon and J.-Y. Audibert. Deviations of stochastic bandit regret. In International Conference on Algorithmic Learning Theory , pages 159-173. Springer, 2011.

A. Sani, A. Lazaric, and R. Munos. Risk-aversion in multi-armed bandits. Advances in Neural Information Processing Systems , 25, 2012.

D. Simchi-Levi and C. Wang. Multi-armed bandit experimental design: Online decision-making and adaptive inference. Available at SSRN 4224969 , 2022.

D. Simchi-Levi, Z. Zheng, and F. Zhu. A simple and optimal policy design for online learning with safety against heavy-tailed risk. arXiv preprint arXiv:2206.02969 , 2022.

A. Slivkins et al. Introduction to multi-armed bandits. Foundations and Trends® in Machine Learning , 12(1-2):1-286, 2019.

A. Tamkin, R. Keramati, C. Dann, and E. Brunskill. Distributionally-aware exploration for cvar bandits. In NeurIPS 2019 Workshop on Safety and Robustness on Decision Making , 2019.

S. Vakili and Q. Zhao. Risk-averse multi-armed bandit problems under mean-variance measure. IEEE Journal of Selected Topics in Signal Processing , 10(6):1093-1111, 2016.

F. Yang, A. Ramdas, K. G. Jamieson, and M. J. Wainwright. A framework for multi-a (rmed)/b (andit) testing with online fdr control. Advances in Neural Information Processing Systems , 30, 2017.

J. Yao, E. Brunskill, W. Pan, S. Murphy, and F. Doshi-Velez. Power constrained bandits. In Machine Learning for Healthcare Conference , pages 209-259. PMLR, 2021.

Q. Zhu and V. Tan. Thompson sampling algorithms for mean-variance bandits. In International Conference on Machine Learning , pages 11599-11608. PMLR, 2020.

A. Zimin, R. Ibsen-Jensen, and K. Chatterjee. Generalized risk-aversion in stochastic multiarmed bandits. arXiv preprint arXiv:1405.0833 , 2014.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of trading off problem-dependent and worst-case regret and \"tail risk\" of the regret in bandits. Here, the tail risk means the probability that the regret is larger than $\\Omega(T^\\delta)$ for some $\\delta>0$. Recent studies have shown that the usual algorithms in bandits have unavoidably a linear regret with probability $O(T^{-1})$, and that without changing the scaling in $T$ of the confidence bounds of standard UCB algorithms one cannot hope to do better than a polynomially decreasing tail risk. Knowing if an exponentially decreasing tail risk is achievable is an interesting question in my opinion, and the authors answer favorably to it in the paper.\n\nProviding a worst-case upper bound of $\\Omega(T^\\alpha)$ for $\\alpha\\geq1/2$ and a problem-dependent bound of $\\Omega(T^\\beta)$ for $\\beta>0$, the authors determine the values $(\\delta, \\gamma)$ for which a bound of the kind $P(R(T) \\geq cT^\\delta)\\leq \\exp(-CT^\\gamma)$ is achievable. They then provide a successive elimination strategy based on the UCB principle, that matches the best guarantees established given values of $(\\alpha, \\beta)$. They then provide an extension of their algorithm for a structured non-stationary setting.\n\nSTRENGTHS:\nThe paper has several merits. In my opinion it tackles an interesting problem in an original way. Though it is a bit notation-heavy, overall every notion is clearly stated and relatively easy to understand. The theorems are also clear, both regarding the lower bounds and the upper bounds for the SEwRP algorithm. I think that the focus on this kind of successive elimination strategy is a good thing for the clarity of the paper.\n\nOverall, I have a positive opinion of the paper, even if I believe that it requires some changes and clarifications before publication (see comments below).\n\nWEAKNESSES:\nEven if the paper is relatively easy to follow, the message of the paper is in my opinion not so clear and some changes may largely improve its clarity.\n\n* In my opinion, starting from $(\\alpha, \\beta)$ to derive the bounds on the tail risk is not very natural. It seems more natural to first consider a given tail-risk constraint (e.g. imposed by a law-maker), and then try to propose an algorithm satisfying this constraint with the best regret guarantees. Furthermore, I tend to think that once we drop the possibility of logarithmic regret then the problem-dependent guarantees may not matter much. For this reason, I would tend to suggest a much simpler version of the paper where the authors would simply provide the best achievable worst case regret bound given a tail risk. In my opinion, this would make the paper just as interesting, and much clearer.\n\n* From reading the main paper the intuition of the proofs and the technically difficult points are not very clear. I think that extended discussions on the results may bring value to the paper.\n\n* Following previous point, to me it is not very clear that SEwRP is actually needed, and maybe a much simpler explore-then-commit strategy may satisfy all criteria.  \n\n* Section 4 does not bring much insights on the table and is more suited for the appendices in my opinion. I suggest to remove this section in order to provide more intuitions on the results and their proof in previous sections\n\n* Practicality of the algorithms: it is folklore that the confidence bounds of UCB are already generally rather conservative. With such inflated confidence bounds, it is natural to wonder what happens when running the algorithm for reasonably large horizons. It is a bit disappointing that the experiment section is only in appendix, and only present regret distributions only (which is interesting) and not standard regret curves too. In particular, I wonder if the regret is really sub-linear for e.g $T=10^4$ and $9$ Bernoulli distributions with $\\mu=0.5$ and one with $\\mu=0.6$. Furthermore, a comparison of the performance of SewRP and standard bandit algorithms (or just standard UCB) is necessary in my opinion to also assess the \"experimental\" trade-off, even if their theoretical guarantees are not comparable and it is certain that the safe algorithm will perform worse on average. \n\nThe previous points are more opinions than objective statements, so I would be happy to discuss them with the authors in the discussion phase. \n\n------------- Post-Rebuttal ----------------\n\nThe authors provided convincing answers to the previous points in their rebuttal, making me believe that none of these points is a major issue. Although I am still unsure on the practicality of the algorithm, I believe that the theoretical contribution of this work is strong enough so that it will be a nice Neurips paper.\n\nQUESTIONS:\nSee previous section.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the stochastic multi-armed bandit (MAB) problem. The authors investigate the relationship between worst-case optimality, instance-dependent consistency, and light-tailed risk in policy design. Three main properties are considered for policy design: worst-case optimality, instance-dependent consistency, and light-tailed risk. \n\n- The authors show the interplay between worst-case optimality, instance-dependent consistency, and light-tailed risk is characterized, indicating that relaxing the worst-case or instance-dependent regret order can lighten the regret tail in an information-theoretic way.\n\n- A novel policy is designed that achieves an optimal trade-off among worst-case optimality, instance-dependent consistency, and light-tailed risk.\n\n- The theory is generalized to a MAB model that allows for non-stationary baseline rewards, a common situation among all arms for each time period.\n\nSTRENGTHS:\nThis paper makes contributions to the understanding of the stochastic multi-armed bandit problem and presents a novel policy design, which provides further insight into worst-case optimality, instance-dependent consistency, and light-tailed risk.\n\nThe authors have provided a detailed characterization of the interplays among worst-case optimality, instance-dependent consistency, and light-tailed risk. They successfully illustrate how different levels of these properties affect tail risk, and have determined an optimal trade-off among them.\n\nThe authors propose a unique policy Successive Elimination with Random Permutation (SEwRP) that achieves the optimal regret tail risk for any regret threshold, exhibiting desirable qualities in both worst-case and instance-dependent scenarios. This policy builds upon the concept of successive elimination, introducing novel bonus terms to balance the three key properties of policy design. For any given $\\alpha$ and $\\beta$, the proposed policy obtains optimal worst-case regret and instance-dependent regret, while also achieving the best possible regret tail probability for both scenarios.\n\nThe authors have successfully generalized their analysis to include a stochastic multi-armed bandit problem with non-stationary baseline rewards. This extension could prove useful in various applications dealing with structured non-stationarity.\n\nWEAKNESSES:\nThough the paper presents a novel and intriguing perspective, its primary reliance on theoretical analysis may be viewed as a limitation. Incorporating empirical validation of the proposed algorithm within the main body of the text could offer a more persuasive argument by substantiating the theoretical outcomes.\n\nOne limitation of this study is the unresolved question of whether the presented results can be extended to 'any-time' scenarios, where the policy has no prior knowledge of the time horizon $T$. The inability to validate the presented approach in the 'any-time' setting, a more realistic and complex scenario, leaves a gap in the research. This complexity should be addressed in future work to enhance the general applicability and robustness of the proposed algorithm.\n\nQUESTIONS:\nThe paper is well-written, and the authors make the content accessible and easy to understand.\n\nGiven your paper's discussion in L228 on the phase transition regarding the size of the confidence interval in the design of the novel bonus term, could you elaborate on its interpretation and implications? Specifically, how does this phase transition impact the overall performance and robustness of the proposed policy? Additionally, could you provide further insight into the practical significance of this transition from the dominance of the second term to the first term in real-world applications?\n\nLIMITATIONS:\nYes, the authors seem to have adequately addressed the limitations of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
The submission studies the stochastic multi-armed bandits. The arm-selection policy is required to be worst-case optimal, instance-dependent consistency, and have low tail risks (worst-case and instance-dependent) simultaneously. Lower bounds for achieving these goals at the same time are provided in Theorem 3.1. The submission proposed a policy (SEwRP) that matches the lower bounds (Theorem 3.3 and Proposition 3.4). The main results are also generalized to non-stationary scenarios of baseline rewards (Theorem 4.1 and Proposition 4.2).

STRENGTHS:
- (a) Studying the interplay between worst-case bound, consistency, and risk provides a new perspective to understanding MAB.
- (b) A novel confidence interval (rad(n)) is designed to address different criteria at different phases of the learning process. 
- (c) Besides the rigorous analysis, the treatment for handling the confidence interval and the choice of tail event to tighten the bound constitute the technical contributions.
- (d) Fluent arguments and clarifying intuitions provide a comfortable reading experience.

WEAKNESSES:
- (e) The submission, with its supplementary, is a complete and self-content paper.

QUESTIONS:
- (f) In Algorithm 1, does i represent the number of elimination iterations, and t represent the number of arm pulls?
- (g) Is it correct that the reward's formulation (Line 124) contains the common Gaussian reward and normal reward?
- (h) What is the role of empirical regret? It is defined in Line 130 but does not appear in the key results and derivations.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper presents an insightful investigation into the trade-off between optimality, measured by expected regret, and risk, defined as the probability of large regret, in the context of Multi-Armed Bandit problem algorithm design. The authors have made several significant contributions:

* They have demonstrated an "impossibility" result (Theorem 3.1), which indicates that a lower expected regret and a reduced rate of large regret cannot be achieved simultaneously.
* The authors have proposed an algorithm capable of achieving a Pareto trade-off between expected regret and tail rate.
* They have also extended these findings to a structured non-stationary bandit setting.

STRENGTHS:
I find the authors' results compelling and their technical contributions to be of high value. This is indeed a noteworthy piece of work. The result provided could provide a general guideline (what can and cannot be acheived) for the MAB algorithm design.

WEAKNESSES:
NA

QUESTIONS:
* It would be beneficial to provide a clearer explanation of the intuition behind the "impossibility" result.
* Please elucidate the necessity for random permutation within the proposed algorithm.
* It would be interesting to explore the influence of the non-stationary constant $b_t$ on the performance of the algorithm. While Lemma 4.3 provides valuable insight, it would be helpful to understand why the result in Lemma 4.3 holds true despite $b_t$ potentially following any sequence (even one chosen in an adversarial manner).
* A numerical example illustrating the superior tail behavior of regret under the proposed algorithm, as compared to the conventional successive elimination algorithm, would be a beneficial addition.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06