Title: Improved Best-of-Both-Worlds Guarantees for Multi-Armed Bandits: FTRL with General Regularizers and Multiple Optimal Arms
Venue: NeurIPS 2023
Accepted: yes
Keywords: multi-armed bandit

best of both worlds

Follow-the-Regularized-Leader

Tsallis entropy

Shannon entropy

Log-barrier
Abstract: We study the problem of designing adaptive multi-armed bandit algorithms that perform optimally in both the stochastic setting and the adversarial setting simultaneously (often known as a best-of-both-world guarantee). 
A line of recent works shows that when configured and analyzed properly, the Follow-the-Regularized-Leader (FTRL) algorithm, originally designed for the adversarial setting, can in fact optimally adapt to the stochastic setting as well. 
Such results, however, critically rely on an assumption that there exists one unique optimal arm. 
Recently, Ito [2021] took the first step to remove such an undesirable uniqueness assumption for one particular FTRL algorithm with
the 1/2-Tsallis entropy regularizer. 
In this work, we significantly improve and generalize this result, showing that uniqueness is unnecessary for FTRL with a broad family of regularizers and a new learning rate schedule. 
For some regularizers, our regret bounds also improve upon prior results even when uniqueness holds. We further provide an application of our results to the decoupled exploration and exploitation problem, demonstrating that our techniques are broadly applicable.

Improved Best-of-Both-Worlds Guarantees for Multi-Armed Bandits: FTRL with General Regularizers and Multiple Optimal Arms

Abstract

We study the problem of designing adaptive multi-armed bandit algorithms that perform optimally in both the stochastic setting and the adversarial setting simultaneously (often known as a best-of-both-world guarantee). A line of recent works shows that when configured and analyzed properly, the Follow-the-RegularizedLeader (FTRL) algorithm, originally designed for the adversarial setting, can in fact optimally adapt to the stochastic setting as well. Such results, however, critically rely on an assumption that there exists one unique optimal arm. Recently, Ito [2021] took the first step to remove such an undesirable uniqueness assumption for one particular FTRL algorithm with the 1 / 2 -Tsallis entropy regularizer. In this work, we significantly improve and generalize this result, showing that uniqueness is unnecessary for FTRL with a broad family of regularizers and a new learning rate schedule. For some regularizers, our regret bounds also improve upon prior results even when uniqueness holds. We further provide an application of our results to the decoupled exploration and exploitation problem, demonstrating that our techniques are broadly applicable.

1 Introduction

We study the problem of multi-armed bandits (MAB) where a learner sequentially interacts with an environment for T rounds. In each round, the learner selects one of the K arms and observes its loss. The goal of the learner is to minimize her regret, which measures the difference between her total loss and that of the best fixed arm in hindsight. Depending on how the losses are generated, two settings have been heavily studied in the literature: the stochastic setting, where the loss of each arm at each round is an i.i.d. sample of a fixed and unknown distribution, and the adversarial setting, where the losses can be arbitrarily decided by an adversary. In the stochastic setting, the UCB algorithm [Lai and Robbins, 1985, Auer et al., 2002a] attains the instance-optimal regret O ( ∑ i :∆ i > 0 log T ∆ i ) , where the sub-optimality gap ∆ i is the difference between the expected loss of arm i and that of the optimal arm. On the other hand, in the adversarial setting, the minimax-optimal regret is known to be of order Θ( √ KT ) [Auer et al., 2002b, Audibert and Bubeck, 2009], achieved via the well-known Follow-the-Regularized-Leader (FTRL) framework.

Junyan Liu ∗

Given the rather different algorithmic ideas in UCB and FTRL, it is natural to ask whether there exists an adaptive algorithm that achieves the best of both worlds (BOBW), simultaneously enjoying the instance-optimal (log T ) -regret in the stochastic setting and minimax-optimal √ T -regret in the adversarial setting. This question was first answered affirmatively in [Bubeck and Slivkins, 2012], followed by a sequence of improvements and extensions in the past decade. Among these works, a somewhat surprising result by Wei and Luo [2018] shows that, when configured and analyzed properly, FTRL, an algorithm originally designed for the adversarial setting to achieve √ T -type regret, in fact can also achieve (log T ) -type regret in the stochastic setting automatically. This result was latter significantly improved to optimal by Zimmert and Seldin [2019, 2021] using the 1 / 2 -Tsallis entropy regularizer and extended to many other problems. In fact, these algorithms not only achieve BOBW, but also automatically adapt to intermediate settings with a regret bound that interpolates smoothly between the two extremes.

A key drawback of such FTRL-based approaches, however, is that their analysis for the stochastic setting critically relies on a uniqueness assumption, that is, there exists one unique optimal arm (with ∆ i = 0 ). A recent work by Ito [2021] took the first step to address this issue and proposed a novel analysis showing that the exact same Tsallis-INF algorithm of [Zimmert and Seldin, 2019] in fact works even without this assumption. Unfortunately, his analysis is specific to the 1 / 2 -Tsallis entropy regularizer with an arm-independent learning rate, and it is highly unclear how to extend it to other regularizers which often require an arm-dependent learning rate. For example, extending it to the log-barrier regularizer was explicitly mentioned as an open problem in [Ito, 2021].

In this work, we significantly improve and generalize the analysis of [Ito, 2021], greatly deepening our understanding on using FTRL to achieve BOBW. Our improved analysis allows us to obtain a suite of new results, all achieved without the uniqueness assumption. Specifically, we consider a new and unified arm-dependent learning rate schedule and the following regularizers (plus a small amount of extra log-barrier); see also Table 1 for a summary.

Log-barrier: with our new learning rate schedule, we show a regret bound of order O ( G log T ) for the stochastic setting and O ( √ KT log T ) for the adversarial setting. Here, G = | U | ∆ MIN + ∑ i ∈ V 1 ∆ i measures the difficulty of the instance, with U = { i : ∆ i = 0 } being the set of optimal-arms, V being the set of all remaining sub-optimal arms, and ∆ MIN = min i ∈ V ∆ i being the minimum non-zero sub-optimality gap. Our bound for the stochastic setting improves those in [Wei and Luo, 2018, Ito, 2021], both of which require the uniqueness assumption.

Shannon entropy: we show a regret bound of order O ( G (log T ) 2 ) for the stochastic setting and O ( √ KT log T ) for the adversarial setting. This improves [Ito et al., 2022] (when applying their more general results to MAB) in two ways: first, their result requires the uniqueness assumption while ours does not; second, their G is defined as K ∆ MIN , strictly larger than ours.

̸

β -Tsallis entropy: we also consider Tsallis entropy with a general parameter β ∈ (0 , 1) , and show a regret bound of order O ( G log T β (1 -β ) ) for the stochastic setting and O (√ KT (log T ) I { β =1 / 2 } β (1 -β ) ) for the adversarial setting. The only prior work that uses β -Tsallis entropy for BOBW in MAB is [Zimmert and Seldin, 2021], but their algorithm is infeasible (unless β = 1 / 2 ) since the learning rate is tuned in terms of the unknown sub-optimality gaps. We not only address this issue with our new learning rate schedule, but also remove the uniqueness assumption. While β = 1 / 2 leads to be best bounds in MAB, following [Rouyer and Seldin, 2020], we showcase the importance of other values of β (specifically, β = 2 / 3 ) in the so-called Decoupled Exploration and Exploitation (DEE-MAB) setting, and again significantly improve their results (see Table 2).

It is worth noting that, as it is common for FTRL-based approaches, our algorithms also automatically adapt to more general corrupted settings (or the so-called adversarial regime with a self-bounding constraint [Zimmert and Seldin, 2021]). The complete statements of our results can be found in Section 3.

Techniques Inspired by [Ito, 2021], we decompose the regret into three parts: the regret related to the sub-optimal arms, the regret related to the optimal arms, and the residual regret. Bounding each of them requires new ideas, as discussed below (see Section 4 for details).

̸

̸

a To be more precise, a small amount of extra log-barrier has been omitted in this table for simplicity.

b To be clear, the setting in [Ito et al., 2022] is more general than MAB. Here, the comparison is solely based on their results specified to MAB.

To bound the regret related to the sub-optimal arms by a so-called self-bounding quantity (the key to achieve BOBW using FTRL), we design a novel arm-dependent learning rate schedule (which is also our key algorithmic contribution). For example, when using β -Tsallis entropy, this schedule balances the corresponding stability term and penalty term of a sub-optimal arm i to O ( ∑ T t =1 ( p t i ) 1 -β / γ t +1 i ) , which is then bounded by a self-bounding quantity. Apart from removing the uniqueness assumption, as mentioned this learning rate schedule also enables us to achieve the first BOBW guarantees for β -Tsallis entropy with any value of β , and also to improve the bound of Ito et al. [2022] for Shannon entropy, which are notable results on their own.

Then, to bound the regret related to the optimal arms, we greatly extend the idea of [Ito, 2021] that is highly specific to the simple form of 1 / 2 -Tsallis entropy with an arm-independent learning rate. Specifically, we develop a new analysis based on a key observation of a certain monotonicity of Bregman divergences. Such monotonicity only requires two mild conditions on the regularizer that are usually satisfied, allowing us to apply it to a broad spectrum of regularizers.

Our arm-dependent learning rate does make the residual regret much more complicated compared to [Ito, 2021]. To handle it, we carefully consider two cases and show that in both cases it can be related to some self-bounding quantities.

Finally, we note that various places of our analysis require the learner's distribution over arms to be stable in a multiplicative sense between two consecutive rounds. We achieve this by adding an extra small amount of log-barrier, a technique first proposed in [Bubeck et al., 2018]. While we do not know how to prove the same results without this extra tweak, we conjecture that it is indeed unnecessary.

Related work For early results solely for the stochastic setting or solely for the adversarial setting, we refer the readers to the systematic survey in [Lattimore and Szepesvári, 2020]. The study of BOBW for MAB starts from the pioneering work of Bubeck and Slivkins [2012], followed by many improvements via different approaches [Seldin and Slivkins, 2014, Auer and Chiang, 2016, Seldin and Lugosi, 2017, Wei and Luo, 2018, Lykouris et al., 2018, Gupta et al., 2019, Zimmert and Seldin, 2019, 2021] and many extensions from MAB to other problems such as semi-bandits [Zimmert et al., 2019], linear bandits [Lee et al., 2021], MAB with feedback graphs [Ito et al., 2022, Erez and Koren, 2021, Rouyer et al., 2022], MAB with switching cost [Rouyer et al., 2021, Amir et al., 2022], model-selection [Pacchiano et al., 2022], partial monitoring [Tsuchiya et al., 2023], and Markov Decision Process (MDP) [Lykouris et al., 2019, Jin and Luo, 2020, Jin et al., 2021, Chen et al., 2021]. Among these works, the FTRL-based approach is particularly appealing since it is simple in both the algorithm design and the analysis, and also extends seamlessly to other more general settings (such as the corrupted setting). The uniqueness assumption used to be critical for the analysis of this approach, but plays no role in other methods such as [Auer and Chiang, 2016, Seldin and Lugosi, 2017]. Following the first step by [Ito, 2021], our work further demonstrates that this was merely due to the lack of a better analysis (and sometimes a better learning rate schedule). We believe that our techniques shed light on removing the uniqueness assumption for using FTRL in more complicated problems such as semi-bandits and MDPs.

2 Preliminaries

In multi-armed bandits (MAB), a learner is given a fixed set of arms [ K ] = { 1 , 2 , · · · , K } and has to interact with an environment for T ≥ K rounds. In each round t , the learner chooses an arm i t ∈ [ K ] while simultaneously the environment decides a loss vector ℓ t ∈ [0 , 1] K . The learner then suffers and observes the loss ℓ t i t of the selected arm for this round. The goal of the learner is to minimize her (pseudo) regret, which measures the difference between her expected cumulative loss and that of the best arm in hindsight. Formally, the regret is defined as Reg T = E [ ∑ T t =1 ℓ t i t -∑ T t =1 ℓ t i ⋆ ] , where i ⋆ ∈ argmin i ∈ [ K ] E [ ∑ T t =1 ℓ t i ] is one of the best arms in hindsight, and E [ · ] denotes the expectation with respect to the internal randomness of both the algorithm and the environment.

Adversarial setting versus stochastic setting We consider two different settings according to how the loss vectors are decided by the environment. In the adversarial setting, the environment decides the loss vectors in an arbitrary way with the knowledge of the learner's algorithm. In this case, the minimax optimal regret is known to be Θ( √ KT ) [Audibert and Bubeck, 2009].

In the stochastic setting, following prior work such as [Zimmert and Seldin, 2021], we consider a situation much more general than the vanilla i.i.d. case (sometimes called the adversarial regime with a self-bounding constraint). Formally, we assume that the loss vectors satisfy the following condition: there exists a gap vector ∆ ∈ [0 , 1] K and a constant C ≥ 0 such that

where Pr[ i t = i ] denotes the learner's probability of taking arm i in round t . This condition subsumes the well-studied i.i.d. setting (discussed in Section 1) where the loss vectors are independently sampled from a fixed but unknown distribution and thus Condition (1) holds with equality, C = 0 , and ∆ i = E [ ℓ t i -ℓ t i ⋆ ] being the sub-optimality gap of arm i (independent of t ). In this case, the instance-optimal regret is O ( ∑ i :∆ i > 0 log T ∆ i ) , achieved by the UCB algorithm [Auer et al., 2002a]. More generally, Condition (1) covers the corrupted i.i.d. setting where the loss vectors are first sampled from a fixed distribution and then corrupted by an adversary in an arbitrary way as long as the expected cumulative ℓ ∞ distance between the corrupted loss vector and the original one is bounded by C ∈ [0 , T ] . While these two examples both involve iidness, note that Condition (1) itself is much more general and does not necessarily require that.

Uniqueness assumption Prior work using FTRL to achieve BOBW crucially relies on a uniqueness assumption when analyzing the regret under Condition (1). Specifically, it is assumed that there exists one and only one arm ˚ i with ∆ ˚ i = 0 . In the special i.i.d. case, this simply means that there

Algorithm 1 FTRL for BOBW without Uniqueness

Input : coefficient θ , learning rate α , Tsallis entropy parameter β , log-barrier coefficient C LOG , number of arms K , number of rounds T (not necessary if a doubling trick is applied; see [Ito, 2021, Section 5.3])

̸

exists a unique optimal arm ( ˚ i = i ⋆ ). Under this uniqueness assumption, the Tsallis-INF algorithm of [Zimmert and Seldin, 2021] achieves Reg T = O ( ∑ i = ˚ i log T ∆ i + √ C ∑ i = ˚ i log T ∆ i ) . The recent work [Ito, 2021] takes the first step to remove such an assumption for the Tsallis-INF algorithm and develops a refined analysis with regret bound O ( ∑ i :∆ i > 0 log T ∆ i + √ C ∑ i :∆ i > 0 log T ∆ i + D + K ) ,

̸

where D is such that E [ ∑ T t =1 max i :∆ i =0 E t [ ℓ t i -ℓ t i ⋆ ] ] ≤ D, and E t [ · ] is the conditional expectation with respect to the history before round t . Note that, in the i.i.d. setting, D is simply 0 , while in the corrupted setting, D is at most C . We significantly generalize and improve this result to a borad family of algorithms, and all our results hold without the uniqueness assumption.

We denote by U = { i ∈ [ K ] : ∆ i = 0 } the set of arms with a zero gap, and V = [ K ] \ U the set of arms with a positive gap. In the i.i.d. setting, U is simply the set of optimal arms and V is the set of sub-optimal arms. We also define ∆ MIN = min i ∈ V ∆ i to be the minimum nonzero gap.

3 Algorithms and Results

The pseudocode of our algorithm is presented in Algorithm 1. It is based on the general FTRL framework, which finds p t , the distribution of selecting arms in around t , via solving the optimization problem p t = argmin p ∈ Ω K 〈 p, ∑ τ<t ̂ ℓ τ 〉 + ϕ t ( p ) . Here, Ω K is the set of all possible distributions over K arms, ̂ ℓ τ is an loss estimator for ℓ τ , and ϕ t is a regularizer. The learner then samples arm i t from the distribution p t and observes the suffered loss ℓ t i t . With this feedback, the algorithm constructs the standard unbiased importance-weighted loss estimator: ̂ ℓ t i = I { i t = i } ℓ t i p t i , ∀ i ∈ [ K ] , where I {·} denotes the indicator function.

We consider three different reuglarizers ϕ t : log-barrier, β -Tsallis entropy, and Shannon entropy; see Algorithm 1 for definitions. By now, they are standard reuglarizers used extensively in the MAB literature, each with different useful properties, but there are some small tweaks in our definitions: 1) a linear term (from p i log( p i /e ) = p i log p i -p i ) is added to the canonical form of Shannon entropy, which is critical to ensure a certain type of monotonicity of Bregman divergences (see Section 4); 2) for technical reasons, we also incorporate a small amount of extra log-barrier (with coefficient C LOG ), which ensures multiplicative stability of the algorithm.

More importantly, we propose the following unified arm-dependent learning rate:

where α ∈ [0 , 1] and θ ∈ R > 0 are parameters (set differently for different regularizers). The clipping of p τ i to 1 /T is because ( p t i ) 1 -2 α itself could be unbounded for α ∈ ( 1 / 2 , 1] when p t i is too small. This learning rate is not only conceptually simpler than those in [Ito, 2021, Ito et al., 2022] and important for removing the uniqueness assumption, but also leads to better bounds in some cases as we discuss below.

Main results We now present our main results. Our regret bounds in the stochastic setting are expressed in terms of an instance complexity measure G = | U | ∆ MIN + ∑ i ∈ V 1 ∆ i , which is of the same order as the standard complexity measure ∑ i ∈ V 1 ∆ i when | U | = O (1) (in particular, this is the case when the uniqueness assumption holds). Similar to [Ito, 2021], our bounds are also in terms of the constant D defined in Section 2. We start with the following result for the log-barrier regularizer.

Theorem 3.1. When using the log-barrier regularizer with C LOG = 162 , α = 0 , and θ = √ 1 / log T , Algorithm 1 ensures Reg T = O ( √ KT log T ) always, and simultaneously the following regret bound when Condition (1) holds: Reg T = O ( G log T + √ C G log T + K 2 √ log T + K log T + D ) .

Log-barrier was first used to achieve BOBW in [Wei and Luo, 2018] and later improved in [Ito, 2021], both of which require the uniqueness assumption. The O ( √ KT log T ) bound for the adversarial setting is almost minimax optimal except for the extra √ log T factor (a common caveat for log-barrier). On the other hand, the bound under Condition (1) matches that of [Ito, 2021] when uniqueness holds and generalizes it otherwise. 2 It is worth noting that this bound (and the same for our other results) suffers an O ( D ) term, which unfortunately can be as large as C , making the bound weaker than those always with only √ C dependence under the uniqueness assumption. It is unclear to us whether such O ( D ) dependence is necessary when we do not make the uniqueness assumption.

Next, we present our results for Shannon entropy.

Theorem 3.2. When using the Shannon entropy regularizer with C LOG = 162log K , α = 1 , and θ = √ 1 / log T , Algorithm 1 ensures Reg T = O ( √ KT log T ) always, and simultaneously the following regret bound under Condition (1): Reg T = O ( G (log T ) 2 + √ C G (log T ) 2 + K 2 log 3 / 2 T + D ) .

The recent work [Ito et al., 2022] is the first to discover that Shannon entropy, used in the very first adversarial MAB algorithm EXP3 [Auer et al., 2002b], in fact also achieves BOBW when configured and analyzed properly (assuming uniqueness). Their results are for the more general setting of MAB with a feedback graph, and when specified to standard MAB, their regret bound under Condition (1) is worse than ours with G defined as K/ ∆ MIN . The key of our improvement comes from our very different and arm-dependent learning rate schedule. Note that there are extra log T factors in the regret for both the adversarial setting and the stochastic setting, which is also the case in [Ito et al., 2022]. While this makes the bounds worse compared to other regularizers, in the more general setting with a feedback graph, Shannon entropy is known to be critical for achieving the right dependence on the independence number of the feedback graph, and we believe that our results shed light on how to remove the uniqueness requirement in this more general setting using Shannon entropy.

Finally, we present our results for Tsallis entropy.

Theorem 3.3. For any β ∈ (0 , 1) , when using the β -Tsallis entropy regularizer with C LOG = 162 β 1 -β , α = β , and θ = √ (1 -β ) /β , Algorithm 1 ensures Reg T = O (√ 1 β (1 -β ) KT (log T ) I { β = 1 2 } ) always, and simultaneously the following regret bound under Condition (1): Reg T = O ( G log T β (1 -β ) + √ C G log T β (1 -β ) + D + K 2 √ β (1 -β ) 3 / 2 + βK log T 1 -β ) .

̸

When β = 1 / 2 , our learning rate γ t i simply becomes √ t (which is arm-independent), and our algorithm exactly recovers Tsallis-INF [Zimmert and Seldin, 2021]. In this case, our result is essentially the same as what the improved anlaysis of [Ito, 2021] shows, which does not require

̸

̸

uniqueness. For β = 1 / 2 , while such regularizers were also analyzed in [Zimmert and Seldin, 2021] (under uniqueness), their algorithm is infeasible since the learning rates are tuned based on the unknown ∆ i 's. On the other hand, our algorithm not only uses a simple and feasible learning rate schedule, but also works without uniqueness. The bound for the adversarial setting has an extra √ log T factor when β = 1 / 2 though, which we conjecture can be removed (as it is the case when using a fixed learning rate [Audibert and Bubeck, 2009, Abernethy et al., 2015]); see Remark C.2.2. We find it surprising that our learning rate exhibits totally different behavior when β < 1 / 2 versus when β > 1 / 2 (recall that α is set to β ): in the former, γ t i increases in the previous p τ i ( τ < t ), while in the latter, it decreases in p τ i .

̸

It might not be clear at this point what the value is to consider β = 1 / 2 -after all, our bounds are minimized when β = 1 / 2 . It turns out that, however, other values of β play important roles in other problems, as for example demonstrated by Rouyer and Seldin [2020] in a decoupled exploration and exploitation setting. Below, we generalize our results to this setting, showcasing the broad applicability of our techniques.

Decoupled exploration and exploitation The Decoupled Exploration and Exploitation MAB (DEE-MAB) problem, first considered in [Avner et al., 2012], is a variant of MAB where in each round t , the learner picks an arm i t to exploit and an arm j t to explore, and then suffers the loss ℓ t i t while observing the feedback ℓ t j t . The performance of the learner is still measured by the same regret definition in terms of the exploitation arms i 1 , . . . , i T . The standard MAB can be seen as a special case where i t and j t must be the same. In DEE-MAB, it turns out that the adversarial setting is as difficult as standard MAB with a lower bound Ω( √ KT ) , but one can do much better in the stochastic setting with a T -independent regret bound. For example, [Rouyer and Seldin, 2020] uses FTRL with 2 / 3 -Tsallis entropy to achieve O ( √ KT ) in the adversarial setting and simultaneously O (√ ∑ i ∈ V K ∆ i · ∆ MIN ) in the i.i.d. setting assuming a unique optimal arm.

Using our techniques, we not only remove the uniqueness requirement, but also improve their bounds. Specifically, we consider the exact same algorithm as theirs, which can be described using the framework of Algorithm 1: take C LOG = 0 , θ = K 1 / 6 , α = 1 / 2 , and β = 2 / 3 for the Tsallis entropy regularizer; sample the exploitation arm i t according to p t as before and the exploration arm j t according to a different distribution g t with g t i ∝ ( p t i ) 2 / 3 ; finally, construct the importance-weighted estimator using the exploration information: ̂ ℓ t i = I { j t = i } ℓ t i g t i . Our results are as follows.

Theorem 3.4. For the DEE-MAB problem, the algorithm described above ensures Reg T = O ( √ KT ) always, and simultaneously the following regret bound under Condition (1): Reg T = O (√ ∑ i ∈ V K ∆ 2 i + √ C · ( ∑ i ∈ V K ∆ 2 i ) 1 / 4 + K + D ) .

Note that in the i.i.d. setting (where C = D = 0 ), we improve their bound from O (√ ∑ i ∈ V K ∆ i · ∆ MIN ) to O (√ ∑ i ∈ V K ∆ 2 i ) (in addition to removing the uniqueness assumption).

4 Analysis

In this section, we take β -Tsallis entropy as an example to illustrate the key ideas of our analysis in proving the MAB results under Condition (1). As in all prior work, the analysis relies on a so-called self-bounding technique. Specifically, our goal is to bound the regret as follows (ignoring all minor terms, including the dependence on β ):

where the two terms above enjoy a self-bounding property since they can be related back to the regret under Condition (1). To see this, we apply AM-GM inequality followed by Condition (1) to show the

following for any z ≥ 0 :

Rearranging and picking the optimal z yields the regret bound under Condition (1) in Theorem 3.3.

The key is thus to prove Eq. (3), which is not that difficult under uniqueness (when | V | = K -1 ) but turns out to be much more complicated without uniqueness. To proceed, we start with some key concepts and ideas from [Ito, 2021] which we follow. First, define the skewed Bregman divergence for two time steps s, t ∈ N as

and its variant restricted to any subset I ⊆ [ K ] as D s,t I ( x, y ) = ϕ s I ( x ) -ϕ t I ( y ) -⟨∇ ϕ t I ( y ) , x -y ⟩ , where ϕ t I ( x ) = -C LOG ∑ i ∈I log x i -1 1 -β ∑ i ∈I γ t i x β i (that is, ϕ t restricted to I ). The standard Bregman divergence associated with ϕ t , which we denote by D t ( x, y ) , is then a shorthand for D t,t ( x, y ) . One key idea of [Ito, 2021] is to carefully choose the right benchmark for the algorithm when there is a unique optimal arm, the benchmark basically has to be this unique optimal arm, but when multiple optimal arms exist, the benchmark can now be any distribution over these arms, and it can even be varying over time. Indeed, for round t , the following benchmark was used in [Ito, 2021]:

which follows the same definition of p t but is restricted to Ω U = { p ∈ Ω K : ∑ i ∈ U p i = 1 } , the set of distributions over the zero-gap arms. As the second equality shows, q t is also the projection of p t onto Ω U w.r.t. the Bregman divergence D t . With these time-varying benchmarks, Ito [Ito, 2021] proves

The rest of the analysis is where we start to deviate from that of [Ito, 2021] (thought still largely inspired by it), which is critical for our algorithms that use arm-dependent learning rates. First, we introduce an important intermediate point ¯ p t +1 = ¯ p t +1 U + ¯ p t +1 V where ¯ p t +1 U and ¯ p t +1 V are defined as

By definition, ¯ p t +1 is obtained from p t by redistributing the weights among arms in U and those in V , in a way that minimizes an FTRL objective similar to that of p t +1 . We note that Ito [2021] also uses the same ¯ p t +1 U is his analysis, but we introduce ¯ p t +1 V (and thus ¯ p t +1 ) as well since it importantly allows us to decompose each Bregman divergence difference term in Eq. (6) as follows.

In the rest of this section, we proceed to bound each of the three terms in Eq. (44) (see also Table 3 for a summary of bounds for each of these terms and each of the three regularizers).

Regret on Sub-Optimal Arms The regret related to the sub-optimal arms (or more formally arms in V ) is the most straightforward to deal with, since our objective is to arrive at the self-bounding terms in Eq. (3) which are exactly only in terms of arms in V . Indeed, we can write this term as (with p t V being the vector with the same value as p t for coordinates in V and 0 for coordinates in U )

and then apply standard arguments to show that the stability term is of order ∑ i ∈ V ( p t i ) 1 -β / βγ t +1 i while the penalty term is of order ∑ i ∈ V ( γ t +1 i -γ t i )( p t i ) β / 1 -β . In the Tsallis-INF algorithm, we have β = 1 / 2 and γ t i = √ t , and thus the stability term and the penalty term are of the same order. This inspires us to design a learning rate for general β with the same objective. Indeed, it can be verified that our particular learning rate schedule makes sure that the penalty term is of the same order as the stability term, meaning D t,t +1 V ( p t , ¯ p t +1 ) = O ( ∑ i ∈ V ( p t i ) 1 -β / βγ t +1 i ) . Further plugging in the learning rate and summing over t , we arrive at the following with z t i = max { p t i , 1 / T } :

Finally, applying the following technical lemma shows that the above is of the same order as the first term in our objective Eq. (3). More details can be found in Section A.2.

Lemma 4.2. Let { x t } T t =1 be a sequence with x t > 0 for all t . Then, for any α ∈ [0 , 1] , we have

Regret on Optimal Arms Next, we show that the regret on optimal arms (or more formally arms in U ), D t,t +1 U ( p t , ¯ p t +1 ) -D t,t +1 U ( q t , q t +1 ) , is nonpositive, which corresponds to the intuition that pulling optimal arms incur no regret. Ito [2021] proves something similar for 1 / 2 -Tsallis entropy via a certain monotonicity property of Bregman divergence, but his proof is highly specific to 1 / 2 -Tsallis entropy. Instead, we develop the following general monotonicity theorem which applies to a broad spectrum of regularizers as long as they satisfy two mild conditions.

Theorem 4.3 (Monotonicity of Bregman divergence) . For any t ∈ N , let f t : Ω → R be a continuously-differentiable and strictly-convex function defined on Ω ⊆ R . Suppose that the following two conditions hold for all z ∈ Ω : (i) ( f t ) ′ ( z ) is differentiable and concave; (ii) ( f t +1 ) ′ ( z ) ≤ ( f t ) ′ ( z ) . Then, for any x, m ∈ R with x ≤ m , and y, n ∈ R such that ( f t +1 ) ′ ( y ) -( f t ) ′ ( x ) = ( f t +1 ) ′ ( n ) -( f t ) ′ ( m ) = ξ for a fixed scalar ξ , we have D t,t +1 ( x, y ) ≤ D t,t +1 ( m,n ) , where D t,t +1 ( u, v ) = f t ( u ) -f t +1 ( v ) -( u -v ) · ( f t +1 ) ′ ( v ) is the skewed Bregman divergence.

While we state the theorem for the one-dimensional case, it trivially extends to multi-dimensional regularizers as long as they decompose over the coordinates (which is the case for all our regularizers).

Take Tsallis entropy as an example: we only need to apply the theorem with f t ( z ) = -γ t i z β 1 -β for each i and then sum up the conclusions. The two conditions stated in the theorem also hold for all regularizers we consider. In particular, Condition (ii) holds as long as the learning rate γ t i is non-decreasing in t and the regularizer itself is non-increasing (thus with nonpositive first derivative). This explains the additional linear term in our definition of Shannon entropy: this way it is strictly decreasing. Note that Condition (ii) also trivially holds if f t is independent of t , in which case the theorem states the monotonicity for the standard (non-skewed) Bregman divergence.

After verifying the conditions, we can now apply this theorem to show D t,t +1 U ( p t , ¯ p t +1 ) ≤ D t,t +1 U ( q t , q t +1 ) . For each i ∈ U , we take x = p t i and m = q t i . Since by definition q t is obtained by projecting p t onto Ω U , it can be shown via KKT conditions that p t i ≤ q t i indeed holds for all i ∈ U . Then, we define an intermediate point z such that ∇ ϕ t +1 U ( z ) -∇ ϕ t U ( p t ) = ∇ ϕ t +1 U ( q t +1 ) -∇ ϕ t U ( q t ) and show D t,t +1 U ( p t , ¯ p t +1 ) ≤ D t,t +1 U ( p t , z ) . Finally, taking y = z i and n = q t +1 i and applying Theorem 4.3 finishes the proof; see Section A.3 for details.

Residual Regret Finally, bounding the residual regret D t +1 (¯ p t +1 , p t +1 ) by the self-bounding terms in Eq. (3) that are only in terms of arms in V is another key challenge in our analysis, especially given the arm-dependent learning rates. We start by developing a new analysis that leads to tighter bounds compared to [Ito, 2021] on the Lagrangian multipliers associated with Eq. (7), which reveals that the key to analyze D t +1 (¯ p t +1 , p t +1 ) is to bound the following term (or terms of a similar form)

Again, for the case of 1 / 2 -Tsallis entropy with an arm-independent learning rate γ t i = √ t , removing all dependence on i ∈ U in Eq. (12) is relatively straightforward as shown by Ito [2021]. Indeed, in this case, Eq. (12) simplifies to 1 √ t ( ∑ i ∈ V ( p t i ) 3 / 2 ) ( ∑ i ∈ U ( p t i ) 2 ) ( ∑ i ∈ [ K ] ( p t i ) 3 / 2 )( ∑ i ∈ U ( p t i ) 3 / 2 ) . By splitting ∑ i ∈ [ K ] ( p t i ) 3 / 2 into

two summations, one over i ∈ V and another over i ∈ U , and further applying x + y ≥ x 2 / 3 y 1 / 3 for any x, y > 0 , we have ∑ i ∈ [ K ] ( p t i ) 3 / 2 ≥ ( ∑ i ∈ V ( p t i ) 3 / 2 ) 2 / 3 ( ∑ i ∈ U ( p t i ) 3 / 2 ) 1 / 3 and thus Eq. (12) is bounded by

where importantly, in the second step we use the fact ∥ x ∥ 2 ≤ ∥ x ∥ 3 / 2 to drop all dependence on i ∈ U , eventually arriving at the self-bounding term of Eq. (3).

Unfortunately, with an arm-dependent learning rate, it is unclear to us how to analyze Eq. (12) in a similar way. Instead, we propose a different analysis with the following rough idea: we propose a condition under which Eq. (12) can be bounded by ∑ i ∈ V ( p t i ) 2 -β / γ t i and then further related to a self-bounding term similarly to the analysis of the regret on sub-optimal arms. If, on the other hand, the condition does not hold, then we show that the probability p t i of selecting an optimal arm i ∈ U must be no more than the total probability of selecting sub-optimal arms ∑ j ∈ V p t j . Using this fact again allows us to convert dependence on i ∈ U to i ∈ V . Since we apply this technique to all i ∈ U , it leads to the extra | U | factor in the second self-bounding term of Eq. (3), which eventually translates to | U | log T ∆ min in our instance complexity G . All details can be found in Section A.4.

5 Conclusions

In this work, we improve and generalize the analysis of [Ito, 2021], showing that many FTRL algorithms can achieve BOBW without the uniqueness assumption. Specifically, we propose a unified arm-dependent learning rate schedule and novel analytical techniques to remove the uniqueness assumption for a broad family of regularizers, including log-barrier, β -Tsallis entropy, and Shannon entropy. With these new techniques, our regret bounds improve upon prior results even when the uniqueness assumption holds. We further apply our results to the decoupled exploration and exploitation setting, showing that our techniques are broadly applicable.

There are many natural future directions, including (1) removing the | U | ∆ MIN term in our regret bounds; (2) improving the dependence on D (which as mentioned could be as large as the corruption level C ); (3) understanding whether the extra the log-barrier regularizer is necessary or not; (4) and finally generalizing our results to other problems such as semi-bandits and Markov Decision Processes.

Acknowledgments and Disclosure of Funding

TJ and HL are supported by NSF Award IIS-1943607.

References

D. Abernethy, C. Lee, and A. Tewari. Fighting bandits with a new kind of smoothness. Advances in Neural Information Processing Systems , 28, 2015.

Amir, G. Azov, T. Koren, and R. Livni. Better best of both worlds bounds for bandits with switching costs. Advances in neural information processing systems , 2022.

J.-Y. Audibert and S. Bubeck. Minimax policies for adversarial and stochastic bandits. In Proceedings of the Annual Conference on Learning Theory , 2009.

Auer and C.-K. Chiang. An algorithm with nearly optimal pseudo-regret for both stochastic and adversarial bandits. In Proceedings of the Annual Conference on Learning Theory , 2016.

Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 2002a.

Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77, 2002b.

Avner, S. Mannor, and O. Shamir. Decoupling exploration and exploitation in multi-armed bandits. In Proceedings of the 29th International Coference on International Conference on Machine Learning , 2012.

Bubeck and A. Slivkins. The best of both worlds: Stochastic and adversarial bandits. In Proceedings of the 23rd Annual Conference on Learning Theory , 2012.

Bubeck, M. Cohen, and Y. Li. Sparsity, variance and curvature in multi-armed bandits. In Algorithmic Learning Theory , 2018.

Chen, S. S. Du, and K. Jamieson. Improved corruption robust algorithms for episodic reinforcement learning. Proceedings of the International Conference on Machine Learning , 2021.

Erez and T. Koren. Towards best-of-all-worlds online learning with feedback graphs. In Advances in Neural Information Processing Systems , 2021.

Gupta, T. Koren, and K. Talwar. Better algorithms for stochastic bandits with adversarial corruptions. In Proceedings of the Annual Conference on Learning Theory , 2019.

Ito. Parameter-free multi-armed bandit algorithms with hybrid data-dependent regret bounds. In Proceedings of Thirty Fourth Conference on Learning Theory , 2021.

Ito, T. Tsuchiya, and J. Honda. Nearly optimal best-of-both-worlds algorithms for online learning with feedback graphs. In Advances in Neural Information Processing Systems , 2022.

Jin and H. Luo. Simultaneously learning stochastic and adversarial episodic mdps with known transition. Advances in neural information processing systems , 2020.

Jin, L. Huang, and H. Luo. The best of both worlds: stochastic and adversarial episodic mdps with unknown transition. Advances in Neural Information Processing Systems , 2021.

L. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics , 6(1):4-22, 1985.

Lattimore and C. Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

C.-W. Lee, H. Luo, C.-Y. Wei, M. Zhang, and X. Zhang. Achieving near instance-optimality and minimax-optimality in stochastic and adversarial linear bandits simultaneously. Proceedings of the International Conference on Machine Learning , 2021.

Lykouris, V. Mirrokni, and R. Paes Leme. Stochastic bandits robust to adversarial corruptions. In Proceedings of the Annual ACM SIGACT Symposium on Theory of Computing , 2018.

Lykouris, M. Simchowitz, A. Slivkins, and W. Sun. Corruption robust exploration in episodic reinforcement learning. arXiv preprint arXiv:1911.08689 , 2019.

Pacchiano, C. Dann, and C. Gentile. Best of both worlds model selection. In Advances in Neural Information Processing Systems , 2022.

Rouyer and Y. Seldin. Tsallis-inf for decoupled exploration and exploitation in multi-armed bandits. In Proceedings of Thirty Third Conference on Learning Theory , 2020.

Rouyer, Y. Seldin, and N. Cesa-Bianchi. An algorithm for stochastic and adversarial bandits with switching costs. In International Conference on Machine Learning , pages 9127-9135. PMLR, 2021.

Rouyer, D. van der Hoeven, N. Cesa-Bianchi, and Y. Seldin. A near-optimal best-of-both-worlds algorithm for online learning with feedback graphs. In Advances in Neural Information Processing Systems , 2022.

Seldin and G. Lugosi. An improved parametrization and analysis of the exp3++ algorithm for stochastic and adversarial bandits. In Proceedings of the Annual Conference on Learning Theory , 2017.

Seldin and A. Slivkins. One practical algorithm for both stochastic and adversarial bandits. In Proceedings of the International Conference on Machine Learning , 2014.

Tsuchiya, S. Ito, and J. Honda. Best-of-both-worlds algorithms for partial monitoring. Algorithmic Learning Theory , 2023.

C.-Y. Wei and H. Luo. More adaptive algorithms for adversarial bandits. In Proceedings of the Annual Conference On Learning Theory , 2018.

Zimmert and Y. Seldin. An optimal algorithm for stochastic and adversarial bandits. In The International Conference on Artificial Intelligence and Statistics , 2019.

Zimmert and Y. Seldin. Tsallis-inf: An optimal algorithm for stochastic and adversarial bandits. Journal of Machine Learning Research , 22(28):1-49, 2021.

Zimmert, H. Luo, and C.-Y. Wei. Beating stochastic and adversarial semi-bandits optimally and simultaneously. In Proceedings of the International Conference on Machine Learning , 2019.

Appendix

A Proofs for MAB Results in Table 1

In this section, we provide details of our novel analysis framework and take a closer look at our learning rate schedule. Importantly, our analysis framework is able to be applied to all configurations of Algorithm 1 for different regularizers in a unified way. As mentioned, the key term in the regret is ∑ T t =1 D t,t +1 ( p t , p t +1 ) -D t,t +1 U ( q t , q t +1 ) with q t defined in Eq. (5). We thus define the preprocessing cost (PRECOST) as

which we later show is small. Then, we use Lemma 4.1 to decompose the key term into the regret related to the sub-optimal arms (REGSUB), the regret related to the optimal arms (REGOPT) and the residual regret (RESREG):

Our goal is to bound all these four terms in terms of two self-bounding quantities for some x > 0 (plus other minor terms):

These two quantities enjoy a certain self-bounding property which is known to be critical to achieve the gap-dependent bound in the stochastic setting. Specifically, as we illustrate at the beginning of Section 4, under Condition (1), these self-bounding quantities can be related back to the regret itself when the set V coincides with that of Condition (1).

Lemma A.1 (Self-Bounding Quantities) . Under Condition (1), we have for any z > 0 :

Proof. For any z > 0 , we have

where the second step follows from the AM-GM inequality: 2 √ xy ≤ x + y for any x, y > 0 , and the last step follows from Condition (1).

By similar arguments, we have S 2 ( x ) bounded for any z > 0 as:

In Table 3, we summarize the bounds of PRECOST, REGSUB, REGOPT and RESREG when using Algorithm 1 with Shannon entropy, β -Tsallis entropy, and log-barrier, respectively, together with the corresponding lemmas. Importantly, though the bounds are stated in separated lemmas for different regularizers, the proof ideas and techniques are almost the same with only slight modifications to adapt to the specific regularizer and the choice of parameters.

Using the bound on each of there four terms, we now show how to prove our main theorems, starting with the case for β -Tsallis entropy. First, we simply sum up all the bounds to arrive at the following adaptive regret bound.

Lemma A.2 (Regret Bound for β -Tsallis Entropy) . For any β ∈ (0 , 1) , when using β -Tsallis entropy with C LOG = 162 β 1 -β , α = β , and θ = √ 1 -β β , Algorithm 1 guarantees:

We emphasize that this bound holds for any subset U ⊆ [ K ] , not just the U defined with respect to Condition (1) (so a slight abuse of notations here). This is important for proving the regret bound in the adversarial case, as shown in the following proof for Theorem 3.3.

Proof of Theorem 3.3. First, for the adversarial setting, we set U = { i ⋆ } so that D = 0 by its definition. Therefore, we have the self-bounding quantity S 1 ( | U | log T β (1 -β ) ) bounded as:

where the second step follows from the fact | U | = 1 and ∑ T t =1 ∑ i ∈ [ K ] p t i = T . Similarly, we bound the other self-bounding quantity S 2 ( log T β (1 -β ) ) as:

where the second step uses the Cauchy-Schwarz inequality. Combining the bounds together concludes that Reg T = O (√ KT log T β (1 -β ) ) in the adversarial setting.

Next, suppose that Condition (1) holds. We now set U = { i ∈ [ K ] : ∆ i = 0 } as in Section 2, which makes D the same as that defined in Section 2 as well. We further denote W = D + K 2 √ β (1 -β ) 3 / 2 + βK log T 1 -β to simplify notations. Then, we have Reg T bounded for any z > 0 by

where κ > 0 is an absolute constant (based on Lemma A.2), and the second step applies Lemma A.1 (with the arbirary z there set to z 2 κ ) together with our complexity measure G = | U | ∆ MIN + ∑ i ∈ V 1 ∆ i .

For any z ∈ (0 , 1) , we can further rearrange and arrive at

where we define x = 1 -z z > 0 in the second step. Picking up the optimal x to minimize the right hand side gives

where the second step follows from the fact that √ x + y + z ≤ 3 ( √ x + √ y + √ z ) for any x, y, z ≥ 0 , and the third step uses the AM-GM inequality 2 √ xy ≤ x + y for any x, y ≥ 0 .

For the Shannon entropy regularizer and the log-barrier regularizer, we again summarize the adaptive regret bound in the following two lemmas. Theorem 3.2 and Theorem 3.1 then follow immediately. Since the arguements are exactly the same as above, we omit them for simplicity.

Lemma A.3 (Regret Bound for Shannon Entropy) . When using the Shannon entropy regularizer with C LOG = 162log K , α = 1 , and θ = √ 1 / log T , Algorithm 1 ensures

LemmaA.4 (Regret Bound for Log-barrier) . When using the log-barrier regularizer with C LOG = 162 , α = 0 , and θ = √ 1 / log T , Algorithm 1 ensures

In the following four subsections, we discuss how to bound PRECOST, REGSUB, REGOPT, and RESREG respectively. We recall the following definitions discussed in Section 4: the skewed Bregman divergence for two time steps s, t ∈ N is

and its variant restricted to any subset I ⊆ [ K ] is

where ϕ t I ( x ) = -C LOG ∑ i ∈I log x i -1 1 -β ∑ i ∈I γ t i x β i (that is, ϕ t restricted to I ). The standard Bregman divergence associated with ϕ t is denoted by D t ( x, y ) , a shorthand for D t,t ( x, y ) . For any subset I ⊆ [ K ] and any vector x ∈ R K , we define x I ∈ R K such that its entries in I are the same as x while those in [ K ] \I are 0 . Finally, we use 1 ∈ R K to denote the all-one vector.

A.1 Preprocessing Cost

Due to the extra log-barrier added to stabilize the algorithm, we consider distribution ˜ q t defined as

where 1 V ∈ R K , according to our earlier notations, is the vector with all entries in V being 1 and the remaining being 0 . This specific distribution moves a small amount of weights of q t from U to V , which guarantees that our framework can also work for the log-barrier and log(1 / ˜ q t i ) is bounded for ∀ i ∈ V with a proper ϵ .

Lemma A.1.1. For any ϵ ∈ (0 , 1) , any subset U ⊆ [ K ] and V = [ K ] \ U , q t defined in Eq. (5) , and ˜ q t defined in Eq. (15) , Algorithm 1 ensures

Proof. By the definition of ˜ q t in Eq. (15), one can show

Therefore, we have

where the second step uses Eq. (16), and the last step follows from the law of total expectation.

According to Lemma A.1.1, our main goal here is to bound 〈 ̂ ℓ t , p t -˜ q t 〉 . To this end, we present the following decomposition lemma which is helpful for our further analysis.

Lemma A.1.2 (Lemma 16, Ito [2021]) . For p t in Algorithm 1 and ˜ q t in Eq. (15) , we have

Armed with Lemma A.1.2, we are now ready to introduce our main result of the regret preprocessing. It is worth noting that the following lemma does not rely on the specific form of regularizers, and thus, we can use it for all the regularizers we consider.

Lemma A.1.3. Suppose that there exist B 1 , B 2 ∈ R such that

Let B = B 1 + B 2 , and we have

In other words, we have PRECOST bounded by E [ D 1 ( ˜ q 1 , p 1 ) + O ( ϵKT ) + B + D ] .

Proof. Adding and subtracting D t +1 ( ˜ q t +1 , p t +1 ) from the bound in Lemma A.1.2 , we have

First, we consider the term D t +1 ( ˜ q t +1 , p t +1 ) -D t,t +1 ( ˜ q t , p t +1 ) as follows:

where the second step follows from the fact that ˜ q t +1 -˜ q t = (1 -ϵ ) ( q t +1 -q t ) according to Eq. (15); the fifth step follows from facts that ∇ ϕ t +1 U ( q t +1 ) = ∇ ϕ t +1 U ( p t +1 ) + c · 1 U for a Lagrange multiplier c ∈ R and 〈 c · 1 K , q t +1 -q t 〉 = 0 ; the last step uses the fact that ∇ ϕ t +1 ( p t +1 ) = ̂ L t +1 + c ′ · 1 K where ̂ L t ≜ ∑ τ<t ̂ ℓ τ is the cumulative loss vector prior to round t and c ′ ∈ R is another Lagrange multiplier.

By Eq. (17), Eq. (18), and the definition B = B 1 + B 2 , we can further show

Then, we can show the following by reorganizing the summation:

where the second step follows ̂ L t U -̂ L t +1 U = -̂ ℓ t U and the last step uses q t i ≤ 1 for ∀ t, i . Finally, by telescoping, we have

where the second step holds since the Bregman divergence is non-negative. Combining all the inequalities above concludes the proof.

Lemma A.1.4 (PRECOST for Shannon Entropy) . When using the Shannon entropy regularizer with α = 0 and θ = √ 1 / log T , Algorithm 1 ensures

Eq. (17) with B 1 = 4 ϵT log K +1 ;

Eq. (18) with B 2 = 0 ;

Therefore, according to Lemma A.1.3, by picking ϵ = 1 T , our algorithm ensures

Proof. We first show Eq. (18). For the entries in V , we have

where the second step follows from the definition of ˜ q in Eq. (15) which states ˜ q t i = ϵ | V | for any i ∈ V .

For the entries in U , by direct calculation, we have

Note that, taking the summation of Eq. (19) over all rounds yields

where the second and third step follow from the fact -1 ≤ (1 -ϵ ) log (1 -ϵ ) ≤ 0 , and the last step uses the definition of the learning rates.

On the other hand, by summing Eq. (20) over all rounds, we have

where the second step uses the fact that x log x -x ≤ 0 for any x ∈ [0 , 1] ; the third step follows from the definition of γ t i which ensures γ T +1 i ≤ √ 1 + T · T ≤ 2 T ; the last step follows from Jensen's inequality.

Finally, we bound D 1 ( ˜ q 1 , p 1 ) as:

where the second step follows from the fact that x log x ≤ 0 for x ∈ [0 , 1] ; the third step follows from the facts that p 1 i = 1 K and γ 1 i = √ 1 / log T for any i ∈ [ K ] ; the forth step uses the fact that log ( 1 1 -ϵ ) = log ( 1 + ϵ 1 -ϵ ) ≤ ϵ 1 -ϵ for any ϵ ∈ (0 , 1) .

Lemma A.1.5 (PRECOST for Log-barrier) . When using the log-barrier regularizer with α = 1 and θ = √ 1 / log T , Algorithm 1 ensures

Therefore, according to Lemma A.1.3, by picking ϵ = 1 T , our algorithm ensures

Proof. We first show that Eq. (18) holds. For the entries in V , we have

where the second step follows from the definition of ˜ q t in Eq. (15) that ˜ q t i = ϵ | V | for all i ∈ V . Taking the summation over all rounds, we have

which could be bounded with S 2 ( · ) with our specific learning rate schedule as:

For the entries in U , we have

By summing over all rounds and direct calculation, we have

Finally, we bound D 1 ( ˜ q 1 , p 1 ) = ϕ 1 ( ˜ q 1 ) -ϕ 1 ( p 1 ) -〈 ∇ ϕ 1 ( p 1 ) , ˜ q 1 -p 1 〉 = ϕ 1 ( ˜ q 1 ) -ϕ 1 ( p 1 ) , because ∑ i ∈ [ K ] ˜ q 1 i = ∑ i ∈ [ K ] p 1 i and p 1 = 1 K · 1 K which implies that all entries of ∇ ϕ 1 ( p 1 ) are equal. Note that p 1 = 1 K · 1 K , ˜ q 1 U = 1 -ϵ | U | · 1 U , and ˜ q 1 V = ϵ | V | · 1 V . For ϕ 1 V ( ˜ q 1 ) -ϕ 1 V ( p 1 ) , we have

where C LOG ∑ i ∈ V ( log ( ˜ q 1 i ) -log ( p 1 i )) is bounded via a similar way used to bound the Shannon entropy. Similarly, for ϕ 1 U ( ˜ q 1 ) -ϕ 1 U ( p 1 ) , we have

Lemma A.1.6 (PRECOST for β -Tsallis Entropy) . For any β ∈ (0 , 1) , when using the β -Tsallis entropy regularizer with α = β and θ = √ 1 -β β , Algorithm 1 ensures

Eq. (17) with B 1 = 2 ϵKT √ β (1 -β ) ;

Eq. (18) with B 2 = 0 ;

Therefore, according to Lemma A.1.3, by picking ϵ = 1 T , our algorithm further ensures

Proof. Let us first show that Eq. (18) holds. For the entries in V , we have

where the second step follows from the definition of ˜ q t in Eq. (15) that ˜ q t i = ϵ | V | for any i ∈ V . Since ϕ t +1 V ( ˜ q t +1 ) -ϕ t V ( ˜ q t ) ≤ 0 , we have ∑ T t =1 ϕ t +1 V ( ˜ q t +1 ) -ϕ t V ( ˜ q t ) ≤ 0 , which gives B 2 = 0 for Eq. (18).

For the entries in U , we have

By summing over all rounds and direct calculation, we have

and the last step uses the fact that (1 -ϵ ) β ≥ 1 -ϵ for any β ∈ (0 , 1) . Finally, we have by direct calculation:

where the first step uses the fact that 〈 ϕ 1 ( p 1 ) , ˜ q 1 -p 1 〉 = 0 , since ∇ ϕ 1 ( p 1 ) = c · 1 K for some constant c ∈ R and ˜ q 1 , p 1 ∈ Ω K ; the third and forth steps follow the definitions that p 1 = 1 K · 1 K , ˜ q 1 i = ϵ | V | for any i ∈ V , and γ 1 i = √ 1 -β β · 1 K .

A.2 Regret on Sub-Optimal Arms

One important property of our learning rate schedule is that:

holds for any arm i ∈ [ K ] and t ∈ [ T ] , which controls the increase of learning rates from round t to t +1 .

On the other hand, our algorithm is somehow stabilized by the extra log-barrier, i.e., it ensures the following multiplicative relation between p t , ¯ p t +1 , and p t +1 :

according to Lemma C.3.2, Lemma C.3.3, and Lemma C.3.4.

Then, we are ready to analyze the regret related to the sub-optimal arms (REGSUB). For the skewed Bregman divergence D t,t +1 V ( p t , ¯ p t +1 ) , we can decompose it into the stability and penalty terms as

where the first step follows from Lemma C.5.1, and the second step follows from the definition of skewed Bregman divergence.

Lemma A.2.1 (Bound on Stability) . With C LOG ≥ 162 , Algorithm 1 ensures

where ∥ x ∥ M ≜ √ x ⊤ Mx is the quadratic norm of x ∈ R K with respect to some positive semi-definite matrix M ∈ R K × K , and ∇ -2 ϕ t V ( p t ) denotes the Moore-Penrose inverse of the Hessian ∇ 2 ϕ t V ( · ) .

Specifically, the Hessian ∇ 2 ϕ t V ( x ) is a diagonal matrix:

where ∂ 2 ϕ t V ( x ) ∂ ( x i ) 2 is the second order derivative of ϕ t V ( x ) with respect to the variable x i . According to the definition of ϕ t V ( · ) , ∂ 2 ϕ t V ( x ) ∂ ( x i ) 2 = 0 for any arm i ∈ U . Therefore, the Moore-Penrose inverse of ∇ 2 ϕ t V ( x ) is also a diagonal matrix, and it holds that

where ( ∇ -2 ϕ t V ( x ) ) i,i denotes the i -th diagonal element of the matrix ∇ -2 ϕ t V ( x ) .

Proof. To prove this result, we introduce the minimizer z ∈ R K ≥ 0 defined as:

By Lemma C.3.5, the extra log-barrier also guarantees the multiplicative relation between p t and z , that is, z i / 2 ≤ p t i ≤ 2 z i holds for any arm i ∈ V .

According to Lemma C.5.2, we then have

where ξ is some intermediate point between z and p t , that is, ξ = ρz +(1 -ρ ) p t for some ρ ∈ [0 , 1] . With the multiplicative relation between z and p t , we have 1 2 p t i ≤ ξ i ≤ 2 p t i for any arm i ∈ V .

Therefore, one can verify

for all the regularizers of Algorithm 1. Thus, we can further bound ∥ ∥ ∥ ̂ ℓ t V ∥ ∥ ∥ 2 ∇ -2 ϕ t V ( ξ ) as:

where the first step follows from the definition of quadratic norm, and the second step follows from Eq. (25).

Lemma A.2.2 (REGSUB for Log-barrier) . When using the log-barrier regularizer with C LOG = 162 , α = 0 , and θ = √ 1 / log T , Algorithm 1 ensures REGSUB = O ( S 2 (log T )) .

Proof. For the log-barrier regularizer, the multiplicative relation between p t , ¯ p t +1 , and p t +1 in Eq. (22) is guaranteed by Lemma C.3.2.

We first consider the stability term. By Lemma A.2.1, we have

where the first step follows from the facts that the Hessian ∇ 2 ϕ t V ( p t ) is a diagonal matrix and its diagonal element for arm i ∈ V is C LOG + γ t i ( p t i ) 2 .

On the other hand, we bound the penalty term ϕ t V (¯ p t +1 ) -ϕ t +1 V (¯ p t +1 V ) as

where the third step follows from the fact that γ t i ≤ γ t +1 i for any arm i . Therefore, we have REGSUB bounded as

where the last step follows from the multiplicative relation p t +1 i ≤ 2 p t i for any arm i . By our learning rate schedule, we further have

where the first step follows from the fact p t i ≤ max { p t i , 1 / T } ; the third step uses the Newton-Leibniz formula ∫ b a du √ 1+ u = 2 √ 1 + u ∣ ∣ b a .

Lemma A.2.3 (REGSUB for β -Tsallis Entropy) . For any β ∈ (0 , 1) , when using the β -Tsallis entropy regularizer with C LOG = 162 β 1 -β , α = β , and θ = √ 1 -β β , Algorithm 1 ensures

Proof. First, the multiplicative relation between p t , ¯ p t +1 , and p t +1 in Eq. (22) is ensured by Lemma C.3.2. According to Lemma A.2.1, we have

since the diagonal of ∇ 2 ϕ t V ( p t ) is βγ t i ( p t i ) 2 -β + C LOG ( p t i ) 2 for any arm i ∈ V .

Therefore, we have REGSUB bounded as

t t +1

where the last step follows from the multiplicative relation between p and ¯ p in Eq. (22). Let p 0 i = 1 K for any arm i for notational convenience. For the stability term, we have

where the first step follows from the multiplicative relation between p t +1 i and p t i . On the other hand, we can bound the penalty term as:

where the first step uses Eq. (21). This shows that the stability term and the penalty term are of the same order, and the rest of the proof boils down to the following final calculation:

where the first step follows from the definition of γ t i ; the second step applies Lemma 4.2 ; the last step follows from the fact that max { p t i , 1 / T } ≤ p t i + 1 T .

Lemma A.2.4 (REGSUB for Shannon Entropy) . When using the Shannon entropy regularizer with C LOG = 162log K , α = 1 , and θ = √ 1 / log T , Algorithm 1 ensures REGSUB = O ( S 2 ( log 2 T )) .

Proof. Similarly, we apply Lemma A.2.1 and have

since the diagonal of ∇ 2 ϕ t V ( p t ) is γ t i p t i + C LOG ( p t i ) 2 for arm i .

For the penalty term, we have ϕ t V (¯ p t +1 ) -ϕ t +1 V (¯ p t +1 V ) bounded by

where the second step uses the fact that x log ( e x ) is monotonically increasing for x ∈ [0 , 1 / T ] ; the third step follows from Eq. (21); the last step follows from the multiplicative relation between p t and ¯ p t +1 so that max { p t i , 1 / T } ≥ 1 2 max { ¯ p t +1 i , 1 / T } .

Therefore, we have REGSUB bounded as

by combining the bounds for the stability and penalty terms.

According to our learning rate schedule, we have

where the second step applies Lemma 4.2; the third step uses the fact that max { p t i , 1 / T } ≥ 1 / T .

A.3 Regret on Optimal Arms

We show that REGOPT ≤ 0 holds for all regularizers used in this paper with a specific technique. Lemma A.3.1 (REGOPT for All Regularizers) . For any of the three regularizers, Algorithm 1 ensures REGOPT ≤ 0 .

Proof. According to the KKT conditions of the optimization problem for q t , we have

where ξ ′ ∈ R is a Lagrange multiplier for q t +1 . Let z ∈ R K > 0 be a vector that satisfies

By definition, we have

Also, by [Ito, 2021, Lemma 18], we have

where the last step uses the fact that the Bregman divergence is non-negative. Hence, to show D t,t +1 U ( p t , ¯ p t +1 ) ≤ D t,t +1 U ( q t , q t +1 ) , it suffices to show D t,t +1 U ( p t , z ) ≤ D t,t +1 U ( q t , q t +1 ) , which is done below by applying x = p t i , y = z , m = q t i , n = q t +1 i , and ξ = -̂ ℓ t i + ξ ′ in Theorem 4.3.

First, we show that p t i ≤ q t i for any arm i ∈ U . By the KKT conditions for q t and p t , we know that ∇ ϕ t U ( q t ) = ∇ ϕ t U ( p t U ) + η · 1 U for a Lagrange multiplier η ∈ R . Due to the facts that ∑ i ∈ U q t i = 1 ≥ ∑ i ∈ U p t i and ∇ ϕ t ( · ) is monotone-increasing, η is non-negative and q t i ≥ p t i holds for any arm i ∈ U .

It is clear that all our regularizers can be written in the form of ϕ t ( p ) = ∑ i ∈ [ K ] f t ( p i ) , and we will apply Theorem 4.3 with such f t : [0 , 1] → R . It suffices to check the two required conditions on f t . First, all regularizers used in Algorithm 1 satisfy condition (i): ( f t ) ′ ( z ) is differentiable and concave. Second, as mentioned in Section 4, we only need to show that the learning rate γ t i is non-decreasing in t (which is true in our case) and the regularizer itself is non-increasing, the latter of which can be verifed for all our regularizers as long as we have p t , z, q t , q t +1 ∈ (0 , 1] K . As z ∈ R K > 0 , we still need to check z i ≤ 1 . Indeed, from Eq. (26), we have ∇ ϕ t +1 ( z ) ≤ ∇ ϕ t +1 ( q t +1 ) (entry-wise inequality), and since q t +1 i ≤ 1 for ∀ i ∈ U and ∇ ϕ t +1 ( · ) is monotone-increasing entry-wise, we have z i ≤ 1 . To sum up, Theorem 4.3 indeed implies

which further implies D t,t +1 U ( p t , ¯ p t +1 ) ≤ D t,t +1 U ( q t , q t +1 ) . Taking the summation over all the rounds concludes the proof.

Theorem A.3.2 (Restatement of Theorem 4.3) . For any t ∈ N , let f t : Ω → R be a continuouslydifferentiable and strictly-convex function defined on Ω ⊆ R . Suppose that the following two conditions hold for all z ∈ Ω : (i) ( f t ) ′ ( z ) is differentiable and concave; (ii) ( f t +1 ) ′ ( z ) ≤ ( f t ) ′ ( z ) . Then, for any x, m ∈ R with x ≤ m , and y, n ∈ R such that ( f t +1 ) ′ ( y ) -( f t ) ′ ( x ) = ( f t +1 ) ′ ( n ) -( f t ) ′ ( m ) = ξ for a fixed scalar ξ , we have D t,t +1 ( x, y ) ≤ D t,t +1 ( m,n ) , where D t,t +1 ( u, v ) = f t ( u ) -f t +1 ( v ) -( u -v ) · ( f t +1 ) ′ ( v ) is the skewed Bregman divergence.

Proof of Theorem 4.3. For shorthand, we denote F t ( z ) = ( f t ) ′ ( z ) and use ( F t ) ′ ( z ) to denote the first-order derivative of F t with respect to z . Taking the derivative with respect to x on both sides of ( f t +1 ) ′ ( y ) -( f t ) ′ ( x ) = ξ , we have

We now see y as a function of x and D t,t +1 ( x, y ) = f t ( x ) -f t +1 ( y ) -( x -y ) F t +1 ( y ) as a function of x as well. Taking the derivative with respect to x , we have

The implies D ( x, y ) ≤ D ( m,n ) given the condition x ≤ m .

A.4 Residual Regret

Throughout this section, we write all our regularizers in the form ϕ t ( x ) = -C LOG ∑ i ∈ [ K ] log x i + ∑ i ∈ [ K ] γ t i ψ ( x i ) for a proper choice of ψ ( · ) (e.g., ψ ( x i ) = x i log ( x i e ) for the Shannon entropy regularizer). Also, we will present the Hessian ∇ 2 ϕ t ( p t ) as a diagonal matrix whose diagonal element for arm i is denoted as w t i , that is, ∇ 2 ϕ t ( p t ) = diag { w t i : i ∈ [ K ] } . Clearly, it always holds that w t i = C LOG ( p t i ) 2 + γ t i ψ ′′ ( p t i ) .

With these notations, we are now ready to introduce our decomposition of E t [ D t +1 (¯ p t +1 , p t +1 ) ] into the term we discussed in Section 4 plus other terms.

Lemma A.4.1. Under Eq. (22) , Algorithm 1 ensures that E t [ D t +1 (¯ p t +1 , p t +1 ) ] is bounded by

Proof. By Lemma C.5.2, for any c ∈ R , we have D t +1 (¯ p t +1 , p t +1 ) bounded as

where the first step follows from the fact that ¯ p t +1 , p t +1 ∈ Ω K , thus 〈 ¯ p t +1 -p t +1 , c · 1 K 〉 = 0 holds for any c ∈ R ; the second step follows from Lemma C.5.2, where ξ is some intermediate point between ¯ p t +1 and p t +1 ; the thirds step follows from the multiplicative relation between ¯ p t +1 and p t +1 , and the fact that γ t +1 i ≥ γ t i for all t, i .

Then, we analyze the difference ∇ ϕ t +1 (¯ p t +1 ) -∇ ϕ t +1 ( p t +1 ) . According to the KKT conditions of the optimization problems in the FTRL framework, we have

where λ U , λ V , λ K are corresponding Lagrange multipliers for the ¯ p t +1 and p t +1 .

To give a tighter bound of the Lagrange multipliers λ U and λ V , we greatly extend the approach of Ito [2021] in Lemma C.4.3. Together with the multiplicative relation between p t , ¯ p t +1 and p t +1 , we obtain λ U 's upper and lower bounds:

Similarly, we have the following upper and lower bounds of λ V :

With the help of these notations, for any c ∈ R , we have

where in the second step we define c ′ = c + λ K . Picking c ′ to minimize this term yields

where the second step uses ( x -y ) 2 ≤ 2( x 2 + y 2 ) for any x, y ∈ R . In Eq. (28), the first term is only related to λ U , and the second term depends on λ V . By upper and lower bounds of λ U and λ V ,

we have Eq. (28) bounded as

Then, we further bound the first and the third term in Eq. (29). For the first term, we have

̸

where the first step uses the fact that ̂ ℓ t i · ̂ ℓ t j = 0 for i = j , and the second step follows from the definition of importance-weighted loss estimator. Similarly, we have the third term bounded as

For the second term in Eq. (29), we have

where the first step follows from the Cauchy-Schwarz inequality. Similarly, we have the following bound for the forth term

Plugging Eq. (30), Eq. (31), Eq. (32), and Eq. (33) into Eq. (29) finishes the proof.

As mentioned earlier in Section 4, the first term in Lemma A.4.1 is the leading term in the stochastic setting which will yield a term of the order O ( | U | log T ∆ MIN ) in the regret bound. In the following lemma, we further decompose this term into two parts where the first part is related to S 2 and the second part is related to S 1 .

Lemma A.4.2. The following holds:

Proof. We first rewrite the fraction on left hand side as:

To bound the right-hand side of Eq. (34), we introduce an indicator function for any t and any i ∈ U :

to separate the term into two parts as:

Case 1. By direct calculation, we have

Case 2. For arm i that A t i = 0 , we have

where the second step follows from the fact that ( ∑ j ∈ V p t j )( ∑ j ∈ V 1 p t j w t j ) ≥ ∑ j ∈ V 1 w t j . Therefore, we have

where 1 -A t i = I { A t i = 0 } ≤ I { p t i ≤ ∑ j ∈ V p t j } holds due to the fact that I { A } ≤ I { B } for any events A,B with A ⊆ B . Finally, combining bounds for these two cases finishes the proof.

Based on Lemma A.4.1 and the careful decomposition in Lemma A.4.2, we now bound RESREG by

Then, we are ready to bound these three terms with different configurations of Algorithm 1 in Lemma A.4.3, Lemma A.4.4, and Lemma A.4.5. The corresponding ψ ′ ( x ) 's are -βx β -1 1 -β for the β -Tsallis entropy regularizer, -1 x for the log-barrier regularizer, and log( x ) for the Shannon entropy regularizer.

Lemma A.4.3 (RESREG for β -Tsallis Entropy) . For any β ∈ (0 , 1) , when using the β -Tsallis entropy regularizer with C LOG = 162 β 1 -β , α = β , and θ = √ 1 -β β , Algorithm 1 ensures:

which leads to the following bound of RESREG :

Proof. Clearly, the first inequality relates to that of the stability term since

where the first step uses the fact that w t i ≥ γ t i β ( p t i ) 2 -β , and the second step follows from the similar argument of the analysis in Lemma A.2.

Next, by direct calculation, we have

where the first step uses the fact that w t i ≥ βγ t i ( p t i ) 2 -β ; the second step follows from the multiplicative relation between p t i and p t -1 i (we set p 0 i = 1 / K = p 1 i for ∀ i ∈ [ K ] for convenience); the third step

applies Lemma C.2.1, a weighted variant of Lemma 4.2, and again the multiplicative relation; the last step follows from max { p t i , 1 / T } 1 -2 β ≤ T for β ∈ (0 , 1) and max { p t i , 1 / T } ≤ p t i + 1 / T .

To further bound this term, we here have to use the Cauchy-Schwarz inequality which leads to the extra | U | ∆ MIN dependency in the final regret bound:

where the first step uses the property of indicator that p t i I { p t i ≤ ∑ j ∈ V p t j } ≤ ∑ j ∈ V p t j . Taking the expectation finishes the proof for the second inequality.

For the last inequality in the statement, for any round t , we have

where the first step applies w t i ≥ βγ t i ( p t i ) 2 -β and Eq. (21).

Taking the summation over all the rounds, we have

which finishes the proof.

Lemma A.4.4 (RESREG for Log-barrier) . When using the log-barrier regularizer with C LOG = 162 , α = 0 , and θ = √ 1 / log T , Algorithm 1 ensures:

which leads to the following bound of RESREG :

Proof. By direct calculation, we have

where the second step use the same arguments used in Lemma A.2.2.

Following the same idea used for the β -Tsallis entropy regularizer, we have

where the second step follows from Lemma 4.2 and w t i ≥ γ t i / ( p t i ) 2 ; the second step uses the CauchySchwarz inequality; the last follows from the definition of the self-bounding quantity S 1 in Eq. (14). Finally, by direct calculation, for any round t , we have

Taking the summation over all rounds yields that

which finishes the proof.

Lemma A.4.5 (RESREG for Shannon Entropy) . When using the Shannon entropy regularizer with C LOG = 162log K , α = 1 , and θ = √ 1 / log T , Algorithm 1 ensures:

which leads to the following bound of RESREG :

Proof. Similar to the analysis for β -Tsallis entropy, we also have the first inequality relates to the bound of the stability term as:

where the second step follows form similar analysis of the stability term in Lemma A.2.4.

By the same arguments in the proof of Lemma A.4.3, we have

according to our learning rate schedule, where the extra log T factor is casued by θ . By direct calculation, for any round t , we have

Note that, for any arm i , the term log 2 ( p t i ) p t i max { p t i , 1 / T } can be bounded as :

Finally, taking the summation over all rounds yields

which finishes the proof.

B Proof of Theorem 3.4: Decoupled-Tsallis-INF

For the Decoupled-Tsallis-INF algorithm [Rouyer and Seldin, 2020], the learning rate schedule is γ t i = K 1 / 6 √ t for all i ∈ [ K ] at round t (henceforth denoted by γ t for conciseness), and the loss estimator is

Although our analysis of Decoupled-Tsallis-INF follows similar ideas as the case for MAB, we need a finer decomposition of D t,t +1 ( p t , p t +1 ) since we do not add an extra log-barrier regularizer to ensure the multiplicative relation between p t +1 and p t (which is important to avoid any T -dependence). To present this new decomposition, we first introduce some definitions. For ¯ p t +1 , we maintain the same definition of ¯ p t +1 U in Eq. (7), but slightly adjust the definition of ¯ p t +1 V as:

Compared to the definition in Eq. (7) used for the analysis of MAB, the new definition here changes ϕ t +1 to ϕ t . Such a skewness of the regularizer eventually allows us to get ∑ i ∈ V ( p t ) 2 / 3 rather than ∑ i ∈ V (¯ p t +1 ) 2 / 3 in the regret bound, which is important since we cannot easily convert between these two anymore as we did for the MAB analysis due to the aforementioned lack of multiplicative relation. Furthermore, as ϕ t +1 is used for U and ϕ t is used for V , we introduce another intermediate point ˜ p t +1 to bridge this skewness, which is defined as:

where ˜ ϕ t ( x ) = ϕ t V ( x ) + ϕ t +1 U ( x ) can be regarded as a specific intermediate regularizer between ϕ t V and ϕ t +1 U . To distinguish the Bregman divergence D t ( x, y ) defined on regularizer ϕ t , we define D ˜ t ( x, y ) on regularizer ˜ ϕ t as:

Similarly, we define the following two notions of skewed Bregman divergence for s, t ∈ N :

With the help of definitions above, our new decomposition for D t,t +1 ( p t , p t +1 ) -D t,t +1 U ( q t , q t +1 ) is as follows (see Lemma B.1.1)

Now, we are ready to use this decomposition to analyze the regret bound for Decoupled-Tsallis-INF. As we do not add an extra log-barrier, we just set ϵ = 0 , i.e., q t = ˜ q t where ˜ q t is given in Eq. (15). As a result, the only preprocessing cost for Decoupled-Tsallis-INF is D . To show Theorem 3.4, we present the following theorem.

Theorem B.1. Decoupled-Tsallis-INF ensures

for any T 0 ≥ 16 K , any subset U ⊆ [ K ] , V = [ K ] \ U , and D = E [ ∑ T t =1 max i ∈ U E t [ ℓ t i -ℓ t i ⋆ ] ] .

Proof. With the decomposition in Lemma B.1.1, we have the following upper bound for Reg T :

where we bound the regret of the first T 0 rounds by O ( √ KT 0 ) via Lemma B.5.3, and D 1 ( q 1 , p 1 ) by

First, we bound the regret on sub-optimal arms using Lemma B.2.1:

The regret related to the optimal arms for any given t is again non-positive (see Appendix B.3):

Finally, we turn to bound the residual regret (see Appendix B.4). By Lemma B.4.1, the residual regret is bounded as

Combining all the above, we complete the proof.

Armed with the result in Theorem B.1, we are now ready to prove Theorem 3.4.

Proof of Theorem 3.4. Theorem B.1 shows that there exists a constant z ∈ R > 0 such that

holds for any T 0 ≥ 16 K . For the adversarial setting, we simply pick T 0 = T and V = [ K ] \{ i ⋆ } (so that D = 0 ), and then the claimed bound O ( √ KT ) follows.

̸

For the stochastic setting, let V = { i : ∆ i = 0 } as in Section 2 and define

as an instance complexity measure for DEE-MAB. We have the following bound of Reg T for any η > 0 and T 0 ≥ 16 K :

where the second step uses Condition (1); the forth step uses [Rouyer and Seldin, 2020, Lemma 8]; the fifth step follows from the fact ∑ T t = T 0 +1 t -3 / 2 ≤ ∫ T T 0 +1 t -3 / 2 dt ≤ 2 √ T 0 +1 ≤ 2 √ T 0 ; the sixth step uses (1 + η ) 3 ≤ 4(1 + η 3 ) for any η > 0 ; the last step uses √ K ≤ √ KT 0 as T 0 ≥ 16 K . Picking the optimal η yields

where the second step uses the fact ( x + y ) 2 / 3 ≤ 3 ( x 2 / 3 + y 2 / 3 ) for any x, y ≥ 0 , and the third step follows from the fact x 1 / 3 y 2 / 3 ≤ x + y for any x, y ≥ 0 .

For C ≤ ˜ G 1 / 2 K 1 / 2 , by picking T 0 = max { ˜ G , 16 K } , we have

where the first step uses C ≤ ˜ G 1 / 2 K 1 / 2 ; the second step follows from ˜ G ≤ T 0 ≤ ˜ G +16 K ; the last step follows from the fact √ x + y ≤ √ x + √ y for x, y ≥ 0 .

For C ≥ ˜ G 1 / 2 K 1 / 2 , we set T 0 = max { ˜ G 1 / 2 C K 1 / 2 , 16 K } and have Eq. (43) further bounded by

where the first step uses ˜ G 1 / 2 C K 1 / 2 ≤ T 0 ≤ ˜ G 1 / 2 C K 1 / 2 +16 K ; the second step follows from the facts that x 1 / 3 y 2 / 3 ≤ x + y for any x, y ≥ 0 , and that ˜ G 1 / 3 K 2 / 3 = ( ˜ G 1 / 2 K 1 / 2 ) 2 / 3 · ( K ) 1 / 3 ; the third step uses C ≥ ˜ G 1 / 2 K 1 / 2 .

Combining these two cases together yields the claimed bound:

B.1 Regret Decomposition for Decouple-Tsallis-INF

In this subsection, we present the proposed regret decomposition using ¯ p t +1 (with the new definition) and ˜ p t .

Proof. We proceed as follows:

where the first step follows from Lemma C.5.1 according to the definitions of p t ; the second step adds and subtracts ˜ ϕ t ( p t +1 ) ; the third step follows from the definition of D ˜ t,t ; the forth step uses Lemma C.5.2; the last step follows from the definition of ˜ ϕ t which implies ˜ ϕ t U ( p t +1 ) = ϕ t +1 U ( p t +1 ) .

Then, with the help of ¯ p t +1 , we can further decompose the term D t, ˜ t ( p t , ˜ p t +1 ) as:

where the last term is the residual regret.

For the regret on optimal arms, we show that the term ϕ t U ( p t ) -˜ ϕ t U (¯ p t +1 ) -〈 ∇ ˜ ϕ t U ( ˜ p t +1 ) , p t U -¯ p t +1 U 〉 in Eq. (45) is exactly D t,t +1 U ( p t , ¯ p t +1 ) :

where the first step uses the KKT conditions of ¯ p t +1 and ˜ p t +1 , which indicate ∇ ˜ ϕ t U ( ˜ p t +1 ) = ∇ ϕ t +1 U (¯ p t +1 ) + c · 1 U for a constant c ∈ R ; the second step follows from the fact ∑ i ∈ U p t i = ∑ i ∈ U ¯ p t +1 i , which guarantees 〈 c · 1 U , p t U -¯ p t +1 U 〉 = 0 for any c ∈ R ; the third step follows from the definition of ˜ ϕ t which implies ˜ ϕ t U (¯ p t +1 ) = ϕ t +1 U (¯ p t +1 ) .

Following the same idea of handling Eq. (45), we have:

where the first step follows from the definition of ˜ ϕ t that ˜ ϕ t V (¯ p t +1 ) = ϕ t V (¯ p t +1 ) and the KKT conditions of ¯ p t +1 and ˜ p t +1 , which indicate that ∇ ˜ ϕ t V ( ˜ p t +1 ) = ∇ ϕ t V (¯ p t +1 ) + c ′ · 1 V for a constant c ′ ∈ R . Finally, combining these bounds together, we have

which concludes the proof.

B.2 Regret on Sub-Optimal Arms

Lemma B.2.1. For the Decoupled-Tsallis-INF algorithm, the following holds:

for any T 0 ≥ 16 K and any subset V ⊆ [ K ] .

Proof. The stability term D t V ( p t , ¯ p t +1 ) , in conditional expectation, is bounded as:

where the first step applies Lemma C.5.3; the third step uses the definition of g t i ; the forth step follows from the fact ∑ i ∈ [ K ] ( p t i ) 2 / 3 ≤ K 1 / 3 ; the last step uses the definition of γ t = K 1 / 6 √ t .

On the other hand, the penalty term can be bounded as

where the second step uses γ t +1 -γ t ≤ K 1 / 6 √ t +1 . By summing up Eq. (50) and Eq. (51) over all t from T 0 +1 to T , we arrive at the desired bound.

B.3 Regret on Optimal Arms

Since the standard 2 / 3 -Tsallis entropy regularizer is twice differentiable and its partial derivatives are concave (that is, ( -x 2 / 3 ) ′ = -2 / 3 · x -1 / 3 is concave on R > 0 ), the first condition of Theorem 4.3 holds. As the Decoupled-Tsallis-INF algorithm adopts γ t = K 1 / 6 √ t , one can easily verify that the second condition of the theorem also holds. Hence, one can apply Theorem 4.3 to bound the regret on optimal arms by zero, i.e., D t,t +1 U ( p t , ¯ p t +1 ) -D t,t +1 U ( q t , q t +1 ) ≤ 0 for ∀ t .

B.4 Residual Regret

In this section, our goal is to prove the following lemma, which upper-bounds the residual regret by a self-bounding term.

Lemma B.4.1. For the Decoupled-Tsallis-INF algorithm, the following holds for any T 0 ≥ 16 K and any subset V ⊆ [ K ] :

To show Lemma B.4.1, we start from a decomposition of the residual regret. Similar to the analysis of the residual regret for MAB in Section A.4, here, bounding the residual regret requires us to carefully analyze the following KKT conditions:

where λ U , λ V , λ K are corresponding Lagrange multipliers for ¯ p t +1 and ˜ p t +1 . Note that, according to the definition of ¯ p t +1 V in Eq. (36) and the fact that ˜ ϕ t ( x ) = ϕ t V ( x ) + ϕ t +1 U ( x ) for any x , the second and the forth conditions are slightly different from those in Section A.4.

We start from the following decomposition of D ˜ t ( ¯ p t +1 , ˜ p t +1 ) . For any c ∈ R , we have

where the second step uses the fact 〈 c · 1 K , ¯ p t +1 -˜ p t +1 〉 = 0 for any c ∈ R ; the third step follows from the facts that ∇ ˜ ϕ t +1 U ( ˜ p t +1 ) = ∇ ϕ t +1 U ( ˜ p t +1 ) = ∇ ϕ t +1 U (¯ p t +1 ) -λ U · 1 U + λ K · 1 K and similarly that ∇ ˜ ϕ t +1 V ( ˜ p t +1 ) = ∇ ϕ t +1 V (¯ p t +1 ) -λ V · 1 V + λ K · 1 K , which are derived from the KKT conditions above, and 〈 λ K · 1 K , ¯ p t +1 -˜ p t +1 〉 = 0 ; the forth step follows from the fact that 〈 λ U · 1 U , ¯ p t +1 V -¯ p t +1 V 〉 = 〈 λ V · 1 V , ¯ p t +1 U -¯ p t +1 U 〉 = 0 ; the fifth step uses the definition of ˜ ϕ t .

Now, we choose c as

which minimizes a subsequent bound (Eq. (53)) below. With this choice of c , we apply Lemma C.5.3 (the condition required by the lemma is verified at the end of Appendix B.5) to get for any t ≥ T 0 +1 ,

where the third step applies the choice of c and the last step uses ( x -y ) 2 ≤ 2( x 2 + y 2 ) for any x, y ∈ R .

To further bound the regret in Eq. (54), we need to know the range of λ U and λ V . According to Lemma C.4.2, we use Eq. (80) and Eq. (81) to obtain the upper and lower bounds of λ U , and apply Eq. (82) and Eq. (83) to get upper and lower bounds of λ V , which are summarized as:

where the last step of the lower bound λ V ≥ 0 follows from ¯ p t +1 i ≥ 0 and ̂ ℓ t i ≥ 0 for any t, i .

In what follows, we continue to bound the λ U -related part of Eq. (54) and the λ V -related part respectively. As our goal is to bound E [ ∑ T t = T 0 +1 D ˜ t (¯ p t +1 , ˜ p t +1 )] , it suffices to bound these terms in conditional expectation for every round t ≥ T 0 +1 .

Bounding the λ U -related term First, we consider the term in Eq. (54) related to λ U and decompose it as:

For the case of I { λ U ≥ 0 } , we use the upper bound of λ U to show

For the case of I { λ U < 0 } , we use the lower bound of λ U to show

Now, our goal is to bound Eq. (55) and Eq. (56), which are shown below respectively.

Bounding Eq. (55) For this part, we bound it as

̸

where the first step follows from the fact ̂ ℓ t i · ̂ ℓ t j = 0 for any i = j and the definition of ̂ ℓ t i ; the second step uses

the third step applies Lemma B.5.1 and Corollary B.5.2 to obtain the multiplicative relation on U and V , respectively; and the last step holds since p t is deterministic given the history. Furthermore, by direct calculation, we have

where Eq. (58) follows from

Eq. (59) and Eq. (60) use the following, respectively:

and Eq. (61) holds by the facts ∑ i ∈ [ K ] ( p t i ) 2 / 3 ≤ K 1 / 3 and γ t = K 1 / 6 t . Thus, the cumulative regret of the term in Eq. (55) from round T 0 +1 to round T can be bounded as:

Bounding Eq. (56) For this term, we have

where the first step follows from γ t +1 -γ t ≤ K 1 / 6 √ t , γ t = K 1 / 6 t , and Eq. (57); the second step applies Corollary B.5.2 to obtain the multiplicative relation; the third step applies √ ∑ i ∈ V ( p t i ) 4 / 3 ≤ ∑ i ∈ V ( p t i ) 2 / 3 ; the last step divides (∑ i ∈ U ¯ p t +1 i ) 2 = ( (∑ i ∈ U ¯ p t +1 i ) 4 / 3 ) 3 / 2 for both the numerator and the denominator. By using the bound above, we have

where the second step uses ∑ i ∈ U ( p t i ∑ j ∈ U p t j ) 4 / 3 ≥ 1 | U | 1 / 3 ≥ 1 K 1 / 3 and the last step follows from T 0 ≥ K ≥ √ K . Thus, the cumulative regret of this part, starting from t = T 0 +1 to t = T , can be bounded by

Bounding the λ V -related term We now consider the regret of Eq. (54) associated with λ V . Note that as λ V ≥ 0 always holds, we do not need to consider two cases. Repeating the argument used to bound Eq. (55), we have

̸

where the first step applies the upper bound of λ V and uses the fact that ̂ ℓ t i · ̂ ℓ t j = 0 for any i = j together with the definition of ̂ ℓ t i ; the second step bounds the fraction ∑ i ∈ U ( ¯ p t +1 i ) 4 / 3 / ∑ i ∈ [ K ] ( ¯ p t +1 i ) 4 / 3 by one; the last step applies Corollary B.5.2 to obtain the multiplicative relation. By a similar argument used for λ U , we have

where the last step uses ∑ i ∈ V ( p t i ) 2 ≤ ( ∑ i ∈ V ( p t i ) 2 / 3 )( ∑ i ∈ V ( p t i ) 4 / 3 ) , ∑ i ∈ [ K ] ( p t i ) 2 / 3 ≤ K 1 / 3 , and γ t = K 1 / 6 √ t . To conclude, the cumulative regret of the term related to λ V is bounded as

Finally, combining Eq. (62), Eq. (63), and Eq. (65) yields the result of Lemma B.4.1.

B.5 Auxiliary Lemmas for Analysis of Decoupled-Tsallis-INF

Since the regularizer of Decoupled-Tsallis-INF does not have an extra log-barrier, we cannot expect an entry-wise multiplicative relation as used in our MAB analysis. Hence, we follow Lemma 22 and Lemma 28 in [Ito, 2021] to show group multiplicative relation for 2 / 3 -Tsallis entropy.

Lemma B.5.1. For any given t and index set I ⊆ [ K ] , if x, y ∈ [0 , 1] K satisfies ∑ i ∈I x i = ∑ i ∈I y i and for 2 / 3 -Tsallis entropy ϕ and c ∈ R , we have

then, the following holds.

Proof. From Eq. (66), we have

which implies that

If c ≤ 0 , then, we have

Since the above holds for every i ∈ I , the desired claim is immediate. Now, we consider c ≥ 0 . For all i ∈ I\{ i t } , we have √ t +1( y i ) -1 / 3 ≤ √ t ( x i ) -1 / 3 , which gives ( x i ) 1 / 3 √ t ≤ ( y i ) 1 / 3 √ t +1 . Rearranging it, we arrive at

Let us define

By z t +1 i ≥ z t i for all i ∈ I\{ i t } and ∑ i ∈I z t +1 i ≤ ∑ i ∈I z t i = 1 , we can show

where the last step uses Lemma 28 in [Ito, 2021] by replacing the power 3 / 2 by 4 / 3 .

Recall that ∇ ϕ t +1 U (¯ p t +1 ) = ∇ ϕ t U ( p t ) -̂ ℓ t U + λ U · 1 U , which implies that Eq. (66) and ∑ i ∈I x i = ∑ i ∈I y i in Lemma B.5.1 hold by applying I = U , y = ¯ p t +1 , and x = p t . Hence, we have ∑ i ∈ U ( ¯ p t +1 i ) 4 / 3 ≤ 8 ∑ i ∈ U ( p t i ) 4 / 3 . By repeating the same reasoning in Lemma B.5.1 and changing γ t +1 to γ t , the following corollary is immediate.

Corollary B.5.2. For any given t and index set I ⊆ [ K ] , if x, y ∈ [0 , 1] K satisfies ∑ i ∈I x i = ∑ i ∈I y i and for 2 / 3 -Tsallis entropy ϕ and c ∈ R , we have ϕ t I ( y ) = ϕ t I ( x ) -̂ ℓ t I + c · 1 I , then, ∑ i ∈I ( y i ) 4 / 3 ≤ 8 ∑ i ∈I ( x i ) 4 / 3 .

Recall that as ¯ p t +1 V is computed via ϕ t rather than ϕ t +1 and ∇ ϕ t V (¯ p t +1 ) = ∇ ϕ t V ( p t ) -̂ ℓ t V + λ V · 1 V , Corollary B.5.2 implies the multiplicative relation: ∑ i ∈ V ( ¯ p t +1 i ) 4 / 3 ≤ 8 ∑ i ∈ V ( p t i ) 4 / 3 . √

Next, we present a lemma that is used to bound the regret for the first T 0 rounds by simply O ( KT 0 ) . Lemma B.5.3. For q t defined in Eq. (5) , p t defined in Algorithm 1, and any t ∈ [ T ] , we have

Proof. By similar arguments of [Ito, 2021, Lemma 24], we arrive at

where z t +1 ∈ R K is the unconstrained projection such that ∇ ϕ t +1 ( z t +1 ) = ∇ ϕ t ( p t ) -̂ ℓ t (see more details in [Ito, 2021]). Then, we have for any i ∈ [ K ] and any t ,

which implies that for any i ∈ [ K ] and any t ,

One can bound the stability term in conditional expectation by

where the first step applies Lemma C.5.3 and uses a similar approach of Eq. (49) (changing V to [ K ] ); the second step uses the fact that when p t i = 1 /K for all i ∈ [ K ] , the square of sum is maximized. The penalty term is bounded by

where the first step uses a similar argument of Eq. (51); the second step applies Eq. (69); the last step follows from the fact ∑ i ∈ [ K ] ( p t i ) 2 / 3 ≤ K 1 / 3 since p t i = 1 / K maximizes the value of ∑ i ∈ [ K ] ( p t i ) 2 / 3 . Combining Eq. (70) and Eq. (71), we complete the proof.

Sanity check for the condition in Lemma C.5.3 Recall that we defer the sanity check for the condition of Lemma C.5.3 in Appendix B.4. This condition requires us to check that ( ¯ p t +1 j ) 1 / 3 ( λ V -c ) γ t for any j ∈ V and ( ¯ p t +1 j ) 1 / 3 ( λ U -c ) γ t for any j ∈ U can be lower-bounded by a fixed negative constant (that is, β 1 -β ( e β -1 β -1 ) = 2 ( e -1 2 -1 ) ≈ -0 . 78693 when β = 2 / 3 ).

Because the lower bound of λ V is zero, which is larger than that of λ U , we will verify that ( ¯ p t +1 j ) 1 / 3 ( λ U -c ) γ t can be lower-bounded for any j ∈ U and t ≥ T 0 + 1 , and the other one can be similarly bounded. For any j ∈ U , we first show that ( ¯ p t +1 j ) 1 / 3 λ U t is lower bounded as:

γ

where the first step uses the lower bound of λ U ; the second step follows from the definition of learning rate γ t ; the third step follows from the fact that x 1 / 3 y ≤ x 4 / 3 + y 4 / 3 for any x, y ≥ 0 ; the last step uses t ≥ T 0 +1 ≥ 16 K +1 .

Then, we show that ( ¯ p t +1 j ) 1 / 3 c γ t is upper-bounded as:

where first step applies the choice of c (see Eq. (52)); the second step uses upper bounds of λ U and λ V ; the third step bounds ∑ i ∈ [ K ] ( ¯ p t +1 i ) 4 / 3 ≥ K -1 / 3 ; the fourth step uses γ t = K 1 / 6 √ t and bounds ( p t i t ) 1 / 3 ≤ 1 ; the last step uses t ≥ T 0 +1 ≥ 16 K +1 .

Finally, combining the bounds above, we have

which satisfies the condition of Lemma C.5.3.

C Supplementary Lemmas

C.1 Proof of Lemma 4.1

Proof. As D t,t +1 ( p t , p t +1 ) = D t,t +1 V ( p t , p t +1 ) + D t,t +1 U ( p t , p t +1 ) , we first decompose D t,t +1 V ( p t , p t +1 ) . By the definition of skewed Bregman divergence, we have

where the third step follows from the fact ∇ ϕ t +1 V (¯ p t +1 ) -∇ ϕ t +1 V ( p t +1 ) = c · 1 V for a Lagrange multiplier c ∈ R and the fact ∑ i ∈ V p t i = ∑ i ∈ V ¯ p t +1 i .

By similar arguments, we have

Combining these two parts together, we have

Subtracting D t,t +1 U ( q t , q t +1 ) from both sides completes the proof.

C.2 Proof of Lemma 4.2

To prove Lemma 4.2, we first consider a more general version that takes a weight sequence { a t } T t =1 into consideration. Note that, by simply setting a t = 1 for all t , one recovers Lemma 4.2.

Lemma C.2.1. Let { x t } T t =1 and { a t } T t =1 be some sequences with that x t , a t > 0 for all t . Then, for any α ∈ [0 , 1] , we have

Proof. For any η > 0 , we have

where the second step uses the AM-GM inequality.

Note that, we have

where the first step bounds the fraction with the integral of 1 u from 1 + ∑ t -1 s =1 x 1 -2 α s to 1 + ∑ t s =1 x 1 -2 α s , and the last step follows from the Newton-Leibniz formula.

Therefore, the following bound holds for any η > 0

Finally, picking the optimal η finishes the proof.

Remark C.2.2. For any β ∈ (0 , 1) , when using the β -Tsallis entropy regularizer with C LOG ≥ 162 β 1 -β , α = β , and θ = √ 1 -β β , Algorithm 1 ensures (ignoring some lower-order terms)

in the adversarial setting. Note that, when β = 1 2 , our learning rate schedule becomes γ t i = √ t which is the same as that of the Tsallis-INF algorithm, and the regret bound above can be simplified as

where the second step follows from the Cauchy-Schwarz inequality: ∑ i ∈ [ K ] √ p t i ≤ √ K ∑ i ∈ [ K ] p t i = √ K , and the last step uses the fact that ∑ T t =1 1 √ t +1 = O ( √ T ) . On the other hand, for the extreme case where β = 0 , we have

where the first step bounds the fraction by the integral of 1 √ 1+ u from ∑ t -1 s =1 p s i to ∑ t s =1 p s i ; the second step follows from the Newton-Leibniz formula; the last step follows from the Cauchy-Schwarz inequality.

These two bounds for β = 0 and β = 1 / 2 inspire us to conjecture the following: for any β ∈ (0 , 1) , any sequence of distributions { p t } T t =1 , we have

̸

Clearly, if this conjecture holds, then, the extra log T factor in the regret bounds for the adversarial setting can be removed when β = 1 / 2 .

C.3 Multiplicative Relation

The added fixed amount log-barrier in the regularizer ensures multiplicative relation among ¯ p t +1 , p t , and p t +1 . To show this, we start from the following general lemma.

Lemma C.3.1. Consider any constant C LOG > 0 , any learning rate vectors γ, γ ′ ∈ R d > 0 satisfying γ ′ i ≥ γ i for all i ∈ [ d ] , any loss vectors L, L ′ ∈ R d > 0 , and any convex function ψ ( x ) which ensures ψ ′′ ( x ) 4 ≤ ψ ′′ ( z ) ≤ 4 ψ ′′ ( x ) for any x ∈ R > 0 and any z ∈ [ x 2 , 2 x ] . Define ϕ ( p ) = ∑ i ∈ [ d ] γ i ψ ( p i ) as the regularizer with learning rate γ , ϕ ′ ( p ) = ∑ i ∈ [ d ] γ ′ i ψ ( p i ) as the regularizer with learning rate γ ′ , ϕ L ( p ) = -C LOG ∑ i ∈ [ d ] log p i as the log-barrier regularizer, F ( p ) = ⟨ p, L ⟩ + ϕ ( p ) + ϕ L ( p ) , and F ′ ( p ) = ⟨ p, L ′ ⟩ + ϕ ′ ( p ) + ϕ L ( p ) . Further define x, y ∈ Ω ⊆ R d > 0 as: x = argmin p ∈ Ω F ( p ) and y = argmin p ∈ Ω F ′ ( p ) . If C LOG satisfies:

then, for any i ∈ [ d ] , we have

Proof. For simplicity, we denote H as the Hessian ∇ 2 F ( x ) , and H L as the Hessian ∇ 2 ϕ L ( x ) which is a diagonal matrix with C LOG x 2 i on its diagonal for every entry i ∈ [ d ] . Our goal is to show that ∥ y -x ∥ H ≤ 1 , which is enough to guarantee Eq. (74) because 1 ≥ ∥ y -x ∥ H ≥ ∥ y -x ∥ H L , and

where the last step follows from the condition C LOG ≥ 9 .

To prove ∥ y -x ∥ H ≤ 1 , it suffices to show that for any z ∈ Ω that ensures ∥ z -x ∥ H = 1 , it holds that F ′ ( z ) ≥ F ′ ( x ) . To see this, note that the level set A = { p ∈ Ω : F ′ ( p ) ≤ F ′ ( x ) } is a convex set that contains x and y . Clearly, any intermediate point between x and y belongs to A , and does not belong to the boundary set ∂B = { p : ∥ p -x ∥ H = 1 } where B = { p : ∥ p -x ∥ H < 1 } also contains x . Therefore, y belongs to the set B and guarantees that ∥ y -x ∥ H ≤ 1 .

To this end, we first bound F ′ ( z ) for any z ∈ Ω with ∥ z -x ∥ H = 1 as

where the first step follows the Taylor expansion of F ′ ( z ) at x with ξ being an intermediate point between x and z ; the third step holds due to the optimality condition of x , i.e., ∇ F ( x ) ⊤ ( z -x ) ≥ 0 ; the forth step uses the fact that ∇ 2 F ′ ( ξ ) ⪰ ∇ 2 F ( ξ ) ; the fifth step applies the multiplicative relation between ξ and x : 1 2 x i ≤ ξ i ≤ 2 x i for any i ∈ [ d ] , which indicates ψ ′′ ( ξ i ) ≥ ψ ′′ ( x i ) 4 according to the property of ψ ; the last step uses the condition that ∥ z -x ∥ H = 1 .

To finish the proof, we only need to show that ( ∇ F ′ ( x ) -∇ F ( x )) ⊤ ( z -x ) ≥ -1 8 . We bound ( ∇ F ′ ( x ) -∇ F ( x )) ⊤ ( z -x ) as

where the second step uses Hölder's inequality, and the forth step follows from the fact that H -1 ⪯ H -1 L , which ensures ∥ p ∥ H -1 ≤ ∥ p ∥ H -1 L for any p ∈ R d .

Note that, we have

where the second step follows from the fact that ( x + y ) 2 ≤ 2 ( x 2 + y 2 ) for any x, y ; the second step uses Eq. (73). Plugging this inequality back to Eq. (75) finishes the proof.

In what follows, we show three applications of Lemma C.3.1 for the β -Tsallis entropy (Lemma C.3.2), the Shannon entropy (Lemma C.3.3), and the log-barrier (Lemma C.3.4), respectively.

Lemma C.3.2 (Multiplicative Relation for β -Tsallis Entropy) . For any β ∈ (0 , 1) , when using the β -Tsallis entropy regularizer with C LOG ≥ 162 β 1 -β , α = β , and θ = √ 1 -β β , Algorithm 1 guarantees

for all t ∈ [ T ] and arm i ∈ [ K ] .

Proof. We first apply Lemma C.3.1 to show 1 2 p t i ≤ p t +1 i ≤ 2 p t i . In particular, we set Ω as a probability simplex, L = ∑ τ<t ℓ τ i , L ′ = ∑ τ ≤ t ℓ τ i , γ = γ t , and γ ′ = γ t +1 where the loss estimators and learning rates are defined in Algorithm 1. These choices naturally give x = p t and y = p t +1 . Then, we only need to check that choosing C LOG = 162 β 1 -β can guarantee that Eq. (73) holds. Clearly, we have C LOG ≥ 9 . From the definition of ̂ ℓ t i , one can show

Finally, by Eq. (21), we have

where the third step uses the inequality that max { p t i , 1 / T } ≤ p t i + 1 T .

To prove 1 2 p t i ≤ ¯ p t +1 i ≤ 2 p t i for all arms i ∈ [ K ] , we show 1 2 p t i ≤ ¯ p t +1 i ≤ 2 p t i for all arms i ∈ U and all arms i ∈ V , respectively. Since the proof ideas of both are the same, we only show the one related to set U . We set Ω = { x ∈ R K ≥ 0 : ∑ i ∈ U x i = ∑ i ∈ U p t i , ∑ i ∈ V x i = 0 } and maintain all settings the same as the above. Note that, Eq. (73) only depends on loss estimators and learning rate schedules, and thus, it holds for new decision space Ω . Repeating a similar argument, we can obtain the multiplicative relation related to arms in V .

Lemma C.3.3 (Multiplicative Relation for Shannon Entropy) . When using the Shannon entropy regularizer with C LOG ≥ 162 log K , α = 1 , and θ = √ 1 log T , Algorithm 1 guarantees

for all t ∈ [ T ] and arm i ∈ [ K ] .

Proof. We consider applying Lemma C.3.1 to prove the multiplicative relation between p t and p t +1 with similar setups in the proof Lemma C.3.2. For the Shannon entropy regularizer in Algorithm 1, we have ψ ( x ) = x log ( x e ) and ψ ′ ( x ) = log( x ) .

By the same argument, we know that 162 log K ≥ 32 ≥ 32 ∑ i ∈ [ K ] ( ̂ ℓ t i ) 2 ( p t i ) 2 and 162 log K ≥ 9 . Then, we only need to verify that 162 log K ≥ 32 ∑ i ∈ [ K ] ( γ t +1 i -γ t i ) 2 ψ ′ ( p t i ) 2 ( p t i ) 2 as:

where the first step follows from Eq. (21) that γ t +1 i -γ t i ≤ 1 log T max { p t i , 1 / T } -1 γ t +1 i ; the second step follwos from the fact that 0 ≥ p t i log ( p t i ) ≥ max { p t i , 1 / T } log (max { p t i , 1 / T } ) as x log x is monotonically decreasing in [0 , 1 / T ] ; the forth step follows from the fact that 0 ≤ -log (max { p t i , 1 / T } ) ≤ log T ; the fifth step utilizes the concavity of -x log x .

Finally, following the same steps in Lemma C.3.2 finishes the proof of the multiplicative relation between ¯ p t +1 and p t .

Lemma C.3.4 (Multiplicative Relation for Log-barrier) . When using the log-barrier regularizer with C LOG ≥ 162 , α = 0 and θ = √ 1 log T , Algorithm 1 guarantees

for all t ∈ [ T ] and arm i ∈ [ K ] .

Proof. Similarly, we only need to verify that 162 log K ≥ 32 ∑ i ∈ [ K ] ( γ t +1 i -γ t i ) 2 ψ ′ ( p t i ) 2 ( p t i ) 2 as:

where the first step follows from the definition of the log-barrier regularizer that ψ ′ ( p t i ) = -1 p t i ; the second step applies Eq. (21).

Lemma C.3.5. With C LOG ≥ 162 , Algorithm 1 guarantees 1 2 p t i ≤ z i ≤ 2 p t i for any arm i ∈ [ K ] and round t ∈ [ T ] where z ∈ R K ≥ 0 is defined as

Proof. Clearly, p t V is the solution of the following optimization problem:

since p t V satisfies all the KKT conditions.

Therefore, we are able to apply Lemma C.3.1 by setting L = ∑ τ<t ̂ ℓ τ V , L ′ = ∑ τ ≤ t ̂ ℓ τ V , Ω being the corresponding simplex, γ = γ ′ = γ t where the loss estimators and learning rates are defined in Algorithm 1. As γ ′ = γ and C LOG = 162 > 9 , we only need to verify that C LOG ≥ 32 ∑ i ∈ V ( ̂ ℓ t i ) 2 ( p t i ) 2 .

By the definition of the importance weighted loss estimator ̂ ℓ t , we have

which finishes the proof.

C.4 Generalization of [Ito, 2021, Lemma 21]

In this section, we greatly generalize [Ito, 2021, Lemma 21] which is critical to analyze the residual regret. The approach of Ito [2021] requires a closed form solution of the optimization problem from the FTRL framework, which is not always guaranteed. Our approach removes this constraint, and thus, it can be applied to β -Tsallis entropy regularizers when β = 1 / 2 , and even more complicated regularizers such as those hybrid ones.

̸

Lemma C.4.1. Given Ω ⊆ R and twice-differentiable functions f, g : Ω → R , if their corresponding first-order derivatives f ′ , g ′ are invertible and the inverse functions ( f ′ ) -1 , ( g ′ ) -1 are differentiable and convex, then, for any p, q ∈ Ω and z ∈ R satisfying g ′ ( q ) = f ′ ( p ) -z , we have

Proof. We first show Eq. (76):

where the third step follows from the convexity of ( g ′ ) -1 , which ensures ( g ′ ) -1 ( u ) ≥ ( g ′ ) -1 ( v ) + ( ( g ′ ) -1 ) ′ ( v ) · ( u -v ) for any u, v in the domain of ( g ′ ) -1 ; the last step uses the inverse function theorem.

On the other hand, we have

where the forth step uses g ′ ( q ) = f ′ ( p ) -z ; the fifth step follows from the convexity of ( g ′ ) -1 ; the last step uses the inverse function theorem. Rearranging this inequality yields Eq. (77).

By swapping p with q , f with g , and flipping z to -z , repeating the steps above yields that

Rearranging these inequalities finishes the proof of Eq. (79) and Eq. (78), respectively.

In the following, we show an application of Lemma C.4.1 in the FTRL framework.

Lemma C.4.2. Let I ⊆ [ K ] be an index set. Suppose ∇ ϕ t +1 I ( q ) = ∇ ϕ t I ( p ) -̂ ℓ t I + c · 1 I where ϕ t is the regularizer with ϕ t ( x ) = ∑ i ∈ S γ t i ψ ( x i ) , ψ : Ω → R for Ω ⊆ R is a strictly convex function such that γ t i ψ satisfies all conditions in Lemma C.4.1 for all t, i , ̂ ℓ t I is the loss estimator, and c ∈ R . If p, q satisfy ∑ i ∈I p i = ∑ i ∈I q i , then we have:

Consequently, for ∇ ϕ t I ( q ) = ∇ ϕ t I ( p ) -̂ ℓ t I + c · 1 I where all definitions remain the same as above, we have

Proof. By directly applying Eq. (76) in Lemma C.4.1 with f ( · ) = γ t i ψ ( · ) , g ( · ) = γ t +1 i ψ ( · ) , p = p i , q = q i , and z = ̂ ℓ t i -c , we have

Since ∑ i ∈I p i = ∑ i ∈I q i , we sum Eq. (84) over all i ∈ I and then rearrange to get Eq. (80). On the other hand, by applying Eq. (79) in Lemma C.4.1 with the same choices, we have

We again sum Eq. (85) over all i ∈ I and then rearrange to get Eq. (81).

Lemma C.4.2 is applicable to the analysis of the Decoupled-Tsallis-INF algorithm. For the analysis of the MAB algorithms, since we add some extra log-barrier to the regularizers, we will need the following similar results.

Lemma C.4.3. Under the same setup of Lemma C.4.2 and let θ ∈ R > 0 be given, if the regularizer is ϕ t ( x ) = ∑ i ∈ S γ t i ψ ( x i ) + θr ( x i ) where ψ ( · ) , r ( · ) are strictly convex and twice differentiable, γ t i ψ ( · ) + θr ( · ) satisfies all conditions in Lemma C.4.1 for all t, i , we have

Proof. For any i ∈ I , applying Eq. (76) in Lemma C.4.1 with f = γ t +1 i ψ ( · ) + θr ( · ) , g = γ t i ψ ( · ) + θr ( · ) , p = p i , q = q i , and z = ̂ ℓ t i -c yields that

By the condition that ∑ i ∈I p i = ∑ i ∈I q i , taking the summation on both sides gives us:

which proves the first inequality. Similarly, the other inequalities in the statement can be obtained by applying Eq. (77), Eq. (78), and Eq. (79) in Lemma C.4.1.

C.5 Analysis of FTRL Framework and Bregman Divergence

Lemma C.5.1 (Lemma 17 in Ito [2021]) . For any regularizers ϕ, ϕ ′ , and L, L ′ ∈ R K , denote z, z ′ as

where Ω = { x ∈ D | Ax = b } for some convex set D and A ∈ R K × K , b ∈ R K . Define the divergences F 1 and F 2 as:

Since the following lemma only uses standard Bregman divergence and the analysis is for a given round t , we drop t for D t ( x, y ) and use D ( x, y ) .

Lemma C.5.2. For any L, ℓ ∈ R K and a Legendre regularizer ϕ , let p, q be the vectors such that

We then have

Moreover, there exists an intermediate point ξ = η · p +(1 -η ) · q for some η ∈ [0 , 1] , such that

Proof. Since ϕ is Legendre, we have ∇ ϕ ( p ) = -L + c · 1 K for some c ∈ R . Thus, it holds for any z ∈ Ω K that

Since q maximizes the last two terms, we have for any z ∈ Ω K ,

Let F ( x ) = ⟨ x, L ⟩ + ϕ ( x ) and G ( x ) = ⟨ x, L + ℓ ⟩ + ϕ ( x ) so that p is the minimizer of F ( x ) and q is the minimizer of G ( x ) . By ∇ ϕ ( p ) = -L + c · 1 K and direct calculation, we have

where we apply Taylor's expansion with ξ = η · p + (1 -η ) · q for some η ∈ [0 , 1] being an intermediate point between p and q , followed by the first-order optimality condition. On the other hand, we have

where the second step follows from the optimality of p , and the last step applies Hölder's inequality. Combining these two inequalities gives us that

for some η ∈ [0 , 1] such that ξ = η · p +(1 -η ) · q .

Lemma C.5.3. Consider the following regularizer for any β ∈ (0 , 1) and learning rate γ i > 0 :

For any q, x ∈ R K ≥ 0 and ℓ ∈ R K satisfying ( q i ) 1 -β ℓ i γ i ≥ β 1 -β ( e β -1 β -1 ) for all arm i , we have

Proof. Let w = argmin x ∈ R K > 0 ⟨ q -x, ℓ ⟩ -D ( x, q ) be the maximizer. By setting the gradient to zero, we have -ℓ -∇ ϕ ( w ) + ∇ ϕ ( q ) = 0 , which equivalently implies that for any arm i ∈ [ K ] ,

By direct calculation, we have

where the first step uses -ℓ -∇ ϕ ( w ) + ∇ ϕ ( q ) = 0 , and the fifth step uses Eq. (86). Moreover, we have for any arm i

where the second step uses Eq. (86), and the third step follows from the fact that (1 + z ) α < 1 + αz + α ( α -1) z 2 for α < 0 and z ≥ exp ( 1 α ) -1 with α = β β -1 = 1 -1 1 -β < 0 for β ∈ (0 , 1) and z = 1 -β γ i β ( q i ) 1 -β ℓ i ≥ e β -1 β -1 = exp ( 1 α ) -1 .

Plugging Eq. (88) into Eq. (87) yields

which concludes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new algorithm for multi-armed bandit problems, leveraging the FTRL framework and the flexible $\\beta$-Tsallis entropy family of regularizers, where $\\beta \\in [0,1]$. This algorithm firstly uses a new learning rate schedule to offer best-of-both-worlds guarantees for a wide range of regularization parameters. Secondly, it eliminates the requirement of the uniqueness assumption of the optimal arm. Thirdly, it improves the stochastic bounds for Shannon entropy and Log-barrier regularization.\n\nSTRENGTHS:\n- The paper introduces new elegant learning rates for $\\beta$-Tsallis entropy, providing a best-of-both-worlds guarantee.\n- It generalizes the regret analysis approach by Ito (2021) by removing the uniqueness assumption of the optimal arm.\n- As $\\beta$-Tsallis entropy is an important regularization in bandit algorithm, the results could be useful in other settings too.\n\nWEAKNESSES:\n- The results presented in the paper for the plain multi-armed bandit, in my view, lack interest and significance compared to the algorithmic and analysis novelties. Specifically, there is no improvement in terms of regret bounds in the plain multi-armed bandit problem, as FTRL with $1/2$-Tsallis regularizer (1/2-Tsallis-INF algorithm) already achieves the optimal bound in both adversarial and stochastic regimes, and the uniqueness assumption has already been addressed by Ito (2021). The only potential value lies in applying the same ideas to other settings, such as the decoupled exploration and exploitation problem explored, as discussed in the paper.\n\n- The bounds in intermediate regimes between stochastic and adversarial, where $C \\neq 0$ seem to be suboptimal as they do not interpolate well between the optimal bounds of the two regime. Read question 2 and 3 for further clarification of this issue.\n\n- There are few undefined notations used in the analysis. For instance, Equation (6) introduces the notation $D_U$ without providing a clear definition, and the same issue applies to $\\phi_U(x)$ and $\\phi_V(x)$ in Equation (7). If the used notation for this part were consistent with Ito (2021), $\\phi_U(x)$ must accept $x$ from $\\mathbb{R}^{|U|}$ but in Equation (7) it takes $x \\in \\mathbb{R}^{K}$. This inconsistency requires revision for clarity and accuracy.\n\nQUESTIONS:\n1- As stated in the paper, the regularizer always includes a small amount of extra log-barrier. Typically, the use of log-barrier introduces an additional multiplicative factor of $\\sqrt{\\log T}$ in the adversarial bound. However, your adversarial bound in Theorem 3.3 does not reflect this when $\\beta = 1/2$. Can you elaborate on this?\n\n2- Regarding the results on regime so called adversarial regime with a self-bounding constraint, In lines 183 and 184, you claim that your bound smoothly interpolates between $\\log T$ and $\\sqrt{T \\log T}$ as $C$ ranges from $0$ to $T$. However, the presence of $D$ in your bound raises doubts about this smooth interpolation since $D$ can be as large as $C$, and $C$ can be on the order of $\\mathcal{O}(T)$. This essentially shows that your bound has no robustness to corruption. Could you address this inconsistency?\n\n3- Following up on the previous question, there is an improvement in the analysis of the $1/2$-Tsallis-INF algorithm by Masoudian and Seldin (2021), which demonstrates smooth interpolation between the two optimal bounds, $\\mathcal{O}(\\sum_{i \\neq i^*} \\frac1{\\Delta_i}\\log T)$ and $\\mathcal{O}(\\sqrt{KT})$ as $C$ increases. Is there any way to achieve a similar improvement for your algorithm? If not, discuss the challenges involved in obtaining the same improvement for intermediate regimes?\n\n4- In the contribution section, for log-barrier regularization, the authors claim to have removed the uniqueness assumption of the optimal arm utilized in Ito (2021) and improved the stochastic bound. However, it should be noted that the algorithm proposed by Ito does not align completely with the algorithm presented in this paper, as Ito employs optimistic follow the regularized leader with different learning rates. Please provide further clarification.\n\n\nReferences:\n\n- Shinji Ito, Parameter-Free Multi-Armed Bandit Algorithms with Hybrid Data-Dependent Regret Bounds, COLT 2021\n\n- Saeed Masoudian and Yevgeny Seldin, Improved analysis of the tsallis-inf algorithm in stochastically constrained adversarial bandits and stochastic bandits with adversarial corruptions, COLT 2021\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors focus on best-of-both-worlds (BOBW) algorithms based on follow-the-regularized-leader (FTRL) in multi-armed bandits.\nThe theoretical guarantees for most existing FTRL-based BOBW algorithms were based on the assumption that the best arm is unique in order to take advantage of self-bounding techniques.\nIt is known that this assumption can be removed by the paper in [13], but its analysis was only applicable to the case of Tsallis-INF (FTRL with 1/2-Tsallis entropy), one of the most representative BOBW algorithms.\nExtending the analysis of [13], the authors show that a BOBW guarantee can be obtained with FTRL with more general regularizers, i.e., negative Shannon entropy, log-barrier, and FTRL with $\\beta$-Tsallis entropy, without the assumption of an unique optimal arm.\nFurthermore, by using the new theory, the authors improve the regret upper bound in the stochastic regime in the decoupled setting.\n\nSTRENGTHS:\n- The paper is very well organized and well written.\n- The paper greatly advances the theory of [13], excluding the unique optimal arm assumption for a wide range of typical regularizers. The assumption have been employed for constructing BOBW algorithms with FTRL, and this is an interesting and important technical contribution to the community.\n- In addition, the authors affirmatively answer the question of whether it is possible to achieve BOBW without knowing $\\Delta_{\\min}$ when $\\beta$-Tsallis entropy with $\\beta \\neq 1/2$ (which was unresolved in Zimmert and Seldin [31]), and the question of whether it is possible to achieve BOBW with Shannon entropy using ${\\Delta_{\\min}}$ to $\\Delta_i$-wise in the stochastic setting (unresolved in Ito et al. [14]). The both of contributions are interesting and important. The related points are listed in weakness.\n\nWEAKNESSES:\n- There do not appear to be any major weaknesses in this paper.\n- One weakness would be a discussion of whether removing the assumption of unique optimal arm actually improves or worsens the performance of algorithms (The reviewer expects the algorithm becomes more conservative, and the performance becomes worse.)\n- Since the discussion excluding the assumption of unique optimal arm is cumbersome on its own, it would be desirable to have a discussion of which techniques in the paper contributed to resolving the problems. More specifically, which components of the algorithm play an important role in resolving the problems of [31] and [14], which are mentioned in the above Strengths part? In addition, if we accept the assumption of unique optimal arm, can we achieve improvements in [31] and [14] with a much more similar argument?\n\nMinor issues and typos:\n- line 87: Sepcifically -> Specifically\n\nQUESTIONS:\n- Shannon entropy relies on $(\\log T)^2$ rather than $\\log T$, which is the case for all published algorithms using FTRL with Shannon entropy, but do the authors think it is possible to make it to $\\log T$?\n- In addition to the above, the reviewer expect the authors to answer the questions listed in Weaknesses.\n- Along with them, the reviewer hopes that the author will address the questions that have been pointed out in the \"Weaknesses\" section above.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper studies the problem of designing adaptive multi-armed bandit algorithms that perform optimally in both the stochastic setting and the adversarial setting simultaneously (often known as a best-of-both-world guarantee). The authors show that the uniqueness assumption is unnecessary for FTRL with a broad family of regularizers and a new learning rate schedule. For some regularizers, their regret bounds also improve upon prior results even when uniqueness holds.

STRENGTHS:
1.	The considered problem, i.e., best-of-both-world for multi-armed bandit, is important in the bandit literature.
2.	The theoretical analysis looks sound, and the improvement is significant.
3.	This paper is well-written and clearly organized.

WEAKNESSES:
1.	This paper does not provide any experimental result. It would improve the paper if the authors could conduct empirical evaluation for their algorithms and compare to existing BOBW algorithms, to validate their theoretical results.

QUESTIONS:
Please see the weaknesses above.

LIMITATIONS:
Please see the weaknesses above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of proving best of both worlds guarantees for algorithms based on the FTRL framework for the multi-armed bandits problem. While it has been demonstrated in (Zimmert and Seldin (2019,2021)) that Tsallis-INF (FTRL with the $1/2$-Tsallis entropy regularizer) achieves optimal regret in both the adversarial and stochastic settings simultaneously, their analysis for the stochastic case relied on the assumption that the optimal arm is unique. The more recent work of Ito (2021) showed that Tsallis-INF still enjoys $\\\\log T$ regret in the stochastic case even if the optimal arm is not unique. In this paper, the authors generalize the analysis of Ito (2021) to other regularizers. Namely, they prove, without the uniqueness assumption, best-of-both-worlds guarantees for FTRL with any $\\\\beta$-Tsallis regularizer (including the log barrier and the Shannon entropy regularizers) using a new arm-dependent learning rate, albeit all regularizers are mixed with the log barrier for technical reasons.\n\nSTRENGTHS:\n- This work provides best of both worlds guarantees without the unique optimal arm assumption for FTRL with a broad family of regularizers. While the use of the $1/2$-Tsallis regularizer is the optimal choice (and already analyzed by Ito (2021)) for the standard bandits problem, other choices are still useful in closely related problems as illustrated in the decoupled exploration and exploitation problem.\n- Moreover, this work seems to be the first to provide BOBW guarantees (without requiring prior knowledge of the suboptimality gaps) for the $\\\\beta$-Tsallis regularizer when $\\\\beta$ is not $1/2$.\n- Overall, the paper is well written and the presentation is clear. A concise sketch of the analysis technique is provided in the last section, and the proofs seem mostly well written and easy to follow.\n\nWEAKNESSES:\n- Unlike Ito(2021), the provided bounds include an added term of order $|U| \\\\log(T) / \\\\Delta_\\\\min$ where $U$ is the set of optimal arms and $\\\\Delta_\\\\min$ is the smallest sub-optimality gap. Thus, the bounds are negatively affected when there are many optimal arms. While this is still an improvement in cases where prior works only achieved a $K \\\\log(T) / \\\\Delta_\\\\min$ dependence (as in the Shannon entropy case), in other cases (most notably for the $1/2$-Tsallis regularizer analyzed by Ito (2021) without the uniqueness assumption) the provided results are inferior to prior works.\n- The fact that all regularizers are summed with a log barrier term is a little unsatisfactory. For instance, in the Shannon entropy case, we potentially lose the appealing property of having closed form expressions for the predictions of FTRL.\n- Though probably curable with a doubling trick, the fact that the proposed approach sometimes requires prior knowledge of the time horizon is a minor weakness.\n\nQUESTIONS:\n- It seems that the log barrier was not added to the Tsallis regularizer in the decoupled exploration and exploitation problem, was it to avoid the $K \\\\log T$ term? Why did the analysis go through in this case but not in the standard bandits problem?\n\nLIMITATIONS:\nThe authors did address some of the limitations of their work. Notably the fact that their analysis requires adding a log barrier term to all the considered regularizers. The authors also acknowledged, though a little less explicitly, the extraneous dependence of their bounds on the number of optimal arms.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}