Title: An Exploration-by-Optimization Approach to Best of Both Worlds in Linear Bandits
Venue: NeurIPS 2023
Accepted: yes
Keywords: bandit

linear bandit

best of both worlds

exploration by optimization
Abstract: In this paper, we consider how to construct best-of-both-worlds linear bandit algorithms that achieve nearly optimal performance for both stochastic and adversarial environments.  For this purpose, we show that a natural approach referred to as exploration by optimization [Lattimore and Szepesvári, 2020] works well. Specifically, an algorithm constructed using this approach achieves $O(d \sqrt{ T \log{T}})$-regret in adversarial environments and $O(\frac{d^2 \log T}{\Delta_{\min}} )$-regret in stochastic environments.  Symbols $d$, $T$ and $\Delta_{\min}$ here represent the dimensionality of the action set, the time horizon, and the minimum sub-optimality gap, respectively.  We also show that this algorithm has even better theoretical guarantees for important special cases including the multi-armed bandit problem and multitask bandits.

An Exploration-by-Optimization Approach to Best of Both Worlds in Linear Bandits

Abstract

In this paper, we consider how to construct best-of-both-worlds linear bandit algorithms that achieve nearly optimal performance for both stochastic and adversarial environments. For this purpose, we show that a natural approach referred to as exploration by optimization [Lattimore and Szepesvári, 2020b] works well. Specifically, an algorithm constructed using this approach achieves O ( d √ T log T ) -regret in adversarial environments and O ( d 2 log T ∆ min ) -regret in stochastic environments. Symbols d , T and ∆ min here represent the dimensionality of the action set, the time horizon, and the minimum sub-optimality gap, respectively. We also show that this algorithm has even better theoretical guarantees for important special cases including the multi-armed bandit problem and multitask bandits.

1 Introduction

The linear bandit problem [Abernethy et al., 2008] is a fundamental sequential decision-making problem in which a player chooses a d -dimensional vector a t from a given action set A ⊆ R d and incurs loss f t . The expectation of f t is assumed to be expressed as an inner product of glyph[lscript] t and a t with an unknown loss vector glyph[lscript] t ∈ R d in each round t = 1 , 2 , . . . , T . This paper focuses on the (expected) regret defined as R T = max a ∗ ∈A E [ ∑ T t =1 〈 glyph[lscript] t , a -a ∗ 〉 ] . It is known that the order of achievable regret can vary greatly depending on the structure of the environment that generates losses. For example, in stochastic environments, i.e., if the loss vector glyph[lscript] t is fixed at glyph[lscript] ∗ for all t , there exists an algorithm that achieves regret of O (log T ) . By way of contrast, for adversarial environments in which the loss vector changes arbitrarily, the optimal regret bound is Θ( √ T ) [Abernethy et al., 2008, Bubeck et al., 2012]. While this clearly suggests the importance of choosing an algorithm that is well-suited to the environment, in real-world applications it is often quite difficult, if not virtually impossible, to know the type of environment in advance.

One promising approach to this difficulty is to apply best-of-both-worlds (BOBW) algorithms [Bubeck and Slivkins, 2012] that work nearly optimally both for stochastic and adversarial environments. The first BOBW algorithm for linear bandit problems was proposed by Lee et al. [2021]; it achieves O ( √ dT log( T ) log( T |A| )) -regret in adversarial environments and O ( c ∗ log( T |A| ) log T ) -regret in stochastic environments, where c ∗ = c ( A , glyph[lscript] ∗ ) represents an instance-dependent quantity characterizing the instance-optimal asymptotic bound for stochastic environments [Lattimore and Szepesvari, 2017]. This algorithm by Lee et al. [2021] also has the notable advantage that these regret guarantees hold with high probability. On the other hand, their regret bound for stochastic environments includes an extra log T factor. 1 This issue of extra log T factors has been resolved in a recent study by Dann et al. [2023b]. They have proposed algorithms achieving regret bounds with optimal dependency on T , which are summarized here in Table 1. Their regret bounds for stochastic environments depend

Kei Takemura

NEC Corporation

on the minimum sub-optimality gap ∆ min . Note that we have c ∗ ≤ O ( d/ ∆ min ) [Lee et al., 2021, Lemma 16] and that this inequality can be arbitrarily loose [Lattimore and Szepesvari, 2017]. The study by Dann et al. [2023b] involves the proposal of a general reduction technique, which has the remarkable advantage of being applicable to a variety of sequential decision-making problems in addition to linear bandit problems, though it does tend to complicate the structure and implementation of the algorithms.

The main contribution of this paper is to show that BOBW algorithms can be constructed via a (conceptually) simple approach referred to as exploration by optimization (EXO) [Lattimore and Szepesvári, 2020b, Lattimore and Gyorgy, 2021, Foster et al., 2022]. In this approach, we update the reference distribution q t using the exponential weight method and then compute sampling distribution p t for action a t (by modifying q t moderately) and loss estimator g t by solving an optimization problem so that an upper bound on regret is minimized. As shown by Lattimore and Gyorgy [2021] and Foster et al. [2022], this natural approach has a connection to both the information ratio [Russo and Van Roy, 2014] and the decision-estimation coefficient [Foster et al., 2021], and it achieves nearly optimal worst-case regret bounds for some online decision-making problems. These existing studies on EXO, however, focus on adversarial environments, and not much has yet been shown about the potential of its working for stochastic environments. In this paper, we show that the EXO approach also works for stochastic environments to achieve O (log T ) -regret.

The regret bounds with the EXO approach and in existing studies are summarized in Table 1. This paper shows the regret bounds of O ( √ αdT log T ) in adversarial environments and of O ( βd log T ) in stochastic environments, where α is a parameter that depends on A and β is a parameter depending on A and glyph[lscript] ∗ . Bounds on α and β shown in this paper are summarized in Table 2. For general action sets, the EXO approach reproduces the regret bound shown by Dann et al. [2023b, Corollary 7]. Further, for some special cases, such as multi-armed bandits and hypercube linear bandits, it achieves improved regret bounds that are tight in stochastic regimes parametrized by the sub-optimality gap ∆ min .

To show BOBW regret bounds, we consider here the EXO approach combined with the continuous exponential weights method [Hoeven et al., 2018], in which we update reference distributions q t over a convex set. In our regret analysis of the exponential weight, the ratio of the variance of the sampling distribution p t to the variance of q t plays an important role: the larger the former, the better the guarantee obtained. To show O (log T ) -regret bounds for stochastic environments, we exploit the fact that q t is a log-concave distribution [Lovász and Vempala, 2007]. Intuitively speaking, as any log-concave distribution has most of its weight concentrated around its mean (see, e.g., [Lovász and Vempala, 2007, Lemma 5.17]), we may choose sampling distribution p t so that it has a larger variance than q t which leads to improved regret bounds.

Our regret analysis uses the self-bounding technique [Wei and Luo, 2018, Zimmert and Seldin, 2021] to show regret bounds for stochastic environments. One advantage of this technique is that a regret bound also leads to stochastic environments with adversarial corruptions or corrupted stochastic environments [Lykouris et al., 2018, Li et al., 2019]. As with existing algorithms based on the self-bounding technique, our algorithm achieves a regret bound of O ( αdT log T + √ CαdT log T ) for corrupted stochastic environments with the corruption level C ≥ 0 . More generally, these regret bounds are achieved for environments in adversarial regimes with self-bounding constraints [Zimmert and Seldin, 2021], details of which are provided here in Section 3.

Another contribution of this paper is to establish a connection between the EXO approach and the SCRiBLe algorithm [Abernethy et al., 2008, 2012] (and its extensions [Dann et al., 2023b, Ito and Takemura, 2023]). The SCRiBLe algorithm uses a follow-the-regularized-leader (FTRL) method with self-concordant barrier regularization to select a point in the action set, rather than a probability distribution. Abernethy et al. [2008] have shown that this approach achieves O ( √ d 3 T ) -regret bounds for adversarial environments. Recently, Dann et al. [2023b] and Ito and Takemura [2023] have shown that it can also achieve O ( d 3 log T ∆ min ) -regret in stochastic environments, through modification of the sampling distributions and loss estimators. To better understand the connection between these studies and the EXO approach, we propose a variant that we refer to as mean-oriented EXO and which includes EXO with the continuous exponential weight method as a special case. Given this correspondence, existing SCRiBLe-type algorithms can be interpreted as EXO-based methods in which the optimization is roughly solved with approximation ratios ≥ Ω( d ) .

The results of this paper establish a connection between the framework of exploration by optimization and best-of-both-worlds regret guarantees. We would like to emphasize that there is potential to extend this result to a broader class of sequential decision-making beyond linear bandits, such as partial monitoring problems and episodic MDPs. In fact, the EXO framework can be applied to a general model of sequential decision-making with structured observation [Foster et al., 2022] and is known to have a connection to several key concepts in sequential decision-making, such as the Decision-Estimation Coefficients [Foster et al., 2021] and information-directed sampling [Lattimore and Gyorgy, 2021, Russo and Van Roy, 2014], which has been extensively applied to partial monitoring [Lattimore and Szepesvári, 2020b] and linear partial monitoring [Kirschner et al., 2020]. Building further relationships between these generic frameworks and BOBW regret analysis will be an important topic to work on in the future.

2 Related work

For adversarial linear bandit problems, various approaches have been considered, including algorithms based on self-concordant barriers (SCRiBLe) [Abernethy et al., 2008, 2012, Rakhlin and Sridharan, 2013], discrete exponential weights (Geometric Hedge) [Dani et al., 2008, Bubeck et al., 2012, Cesa-Bianchi and Lugosi, 2012], and continuous exponential weights (continuous Geometric Hedge) [Hazan and Karnin, 2016, Bubeck and Eldan, 2019], all of which can be interpreted as methods in the FTRL framework. We can see that continuous exponential weight methods can be interpreted as an FTRL approach with an entropic barrier, a d -self-concordant barrier [Chewi, 2021], which implies a close relationship between the SCRiBLe algorithm and the continuous Geometric Hedge, as Bubeck and Eldan [2019] have noted. One key difference lies in how the sampling distribution p t is chosen; the SCRiBLe algorithm employs a sampling scheme supported on the Dikin ellipsoid while the continuous Geometric Hedge designs p t by mixing the reference distribution q t with some exploration basis (e.g., a G-optimal design distribution). As a result of this difference, the latter can achieve an O ( d -1 / 2 ) -times better regret bound than the former. Recent studies by Dann et al. [2023b, Theorem 3] and Ito and Takemura [2023] have proposed new sampling schemes based on Dikin ellipsoid sampling to achieve BOBW regret bounds. This paper takes these approaches a step further, determining a sampling distribution itself on the basis of an optimization problem for minimizing regret, which leads to improved performance.

The self-bounding technique [Wei and Luo, 2018, Zimmert and Seldin, 2021] is known as a promising approach for designing and analyzing BOBW algorithms. The greatest impetus for the widespread use of this technique is considered to come from the Tsallis-INF algorithm [Zimmert et al., 2019] for the multi-armed bandit problem. Since the study on Tsallis-INF, FTRL methods with Tsallisentropy regularizer (with some modification) have been used successfully in a variety of sequential

decision-making problems with limited (structured) observation, including combinatorial semibandits [Zimmert et al., 2019], bandits with switching costs [Rouyer et al., 2021, Amir et al., 2022], bandits with delayed feedback [Masoudian et al., 2022], online learning with feedback graphs [Erez and Koren, 2021, Rouyer et al., 2022]. dueling bandits [Saha and Gaillard, 2022], and episodic MDPs [Jin and Luo, 2020, Jin et al., 2021, Dann et al., 2023a]. On the other hand, a few (rather limited number of) studies have constructed BOBW algorithms using regularizer functions other than the Tsallis entropy. For example, Ito [2021] proposed a BOBW multi-armed bandit algorithm by using log-barrier regularization equipped with adaptive learning rates. More recently, Shannon-entropy regularization with some specific adjustment methods for learning rates has been shown to provide BOBWregret guarantees for several problems including online learning with feedback graphs [Ito et al., 2022b], partial monitoring [Tsuchiya et al., 2023], and linear bandits [Kong et al., 2023]. It seems that these Shannon-entropy-based methods have the strength of being versatile while these have the weakness that their regret bounds for stochastic environments include extra O (log T ) -factors. The approach in this paper may appear to be related to the previous studies using the Shannon entropy [Ito et al., 2022b, Tsuchiya et al., 2023, Kong et al., 2023] as it employs exponential weight methods, but it is more related to the studies using log-barrier regularization [Ito, 2021]. In fact, both log barriers and continuous exponential weight can be captured in the category of self-concordant barriers as we will discuss in Sections 5 and 6. These also have in common the type of regret bounds: O (log T ) in stochastic environments and O ( √ T log T ) in adversarial environments.

3 Problem setting

In linear bandit problems, the player is given an action set A ⊆ R d , which is a closed bounded set of d -dimensional vectors. Without loss of generality, we assume that A spans all vectors in R d . In each round t , the environment chooses a loss vector glyph[lscript] t ⊆ R d while the player selects an action a t ∈ A without knowing glyph[lscript] t . The player then gets feedback of f t ∈ [ -1 , 1] that is drawn from a distribution M a t , where the series of distributions M = ( M a ) a ∈A is in M glyph[lscript] t := { ( M a ) a ∈A |∀ a ∈ A , E f ∼ M a [ f ] = 〈 glyph[lscript] t , a 〉} . The condition that the support of M a is included in [ -1 , 1] for all a ∈ A implies that 〈 glyph[lscript] t , a 〉 ∈ [ -1 , 1] holds for all a ∈ A and t ∈ [ T ] . In other words, it is assumed that glyph[lscript] t ∈ L for any t , where we define L ⊆ R d by L = { glyph[lscript] ∈ R d | ∀ a ∈ A , | 〈 glyph[lscript],a 〉 | ≤ 1 } . The performance of the player is evaluated by means of regret defined as

Regimes of environments In stochastic environments , glyph[lscript] t is assumed to be round-invariant, i.e., there exists glyph[lscript] ∗ ∈ L such that glyph[lscript] t = glyph[lscript] ∗ holds for all t . In adversarial environments , glyph[lscript] t can be chosen in an adversarial manner depending on the history so far (( glyph[lscript] s , a s )) t -1 s =1 . Stochastic environments with adversarial corruptions correspond to an intermediate setting between these two types of environments. These environments are parametrized by the corruption level C ≥ 0 , in which the loss vectors ( glyph[lscript] t ) are assumed to satisfy E [ ∑ T t =1 max a ∈A | 〈 glyph[lscript] t -glyph[lscript] ∗ , a 〉 | ] ≤ C . To capture these settings of environments, we define an adversarial regime with a ( glyph[lscript] ∗ , a ∗ , C, T ) -self-bounding constraint [Zimmert and Seldin, 2021] in which the environments choose losses so that

holds for any algorithms. This regime includes (corrupted) stochastic environments. In fact, any corrupted stochastic environments with the true loss glyph[lscript] ∗ ∈ L and the corruption level C ≥ 0 are included in an adversarial regime with a ( glyph[lscript] ∗ , a ∗ , C, T ) -self-bounding constraint. Our regret analysis for (corrupted) stochastic environments relies only on the condition of (2), as in some existing studies [Zimmert and Seldin, 2021, Dann et al., 2023b].

Reduction to convex action set Let X = conv( A ) represent the convex hull of A . We may consider the action set X instead of A , i.e., if we can construct an algorithm Alg X for the problem with action set X , we can convert it to an algorithm Alg A for the problem with A . In fact, for any

output a ′ t ∈ X of Alg X , we can pick a t ∈ A so that E [ a t | a ′ t ] = a ′ t , 2 When defining a t as the output of Alg A , we have E [ f t | a ′ t ] = E [ 〈 glyph[lscript] t , a t 〉 | a ′ t ] = 〈 glyph[lscript] t , a ′ t 〉 , which means that the expectation of the feedback f t given a ′ t is indeed a linear function in a ′ t and that the regret for Alg A coincides with that for Alg X .

4 Exploration by optimization

4.1 Algorithm framework and regret analysis

Let P ( X ) denote the set of all distributions over X . For any distribution p ∈ P ( R d ) , denote µ ( p ) = E x ∼ p [ x ] ∈ R d , H ( p ) = E x ∼ p [ xx glyph[latticetop] ] ∈ R d × d , and V ( p ) = H ( p ) -µ ( p ) µ ( p ) glyph[latticetop] ∈ R d × d . We consider algorithms that choose a t following a sampling distribution p t ∈ P ( X ) in each round t . To determine p t , we compute an (approximately) unbiased estimator g t ( x ; a t , f t ) of the loss function 〈 glyph[lscript] t , x 〉 . Let G be the set of all estimators g that are linear in the first variable, i.e., G = { g : X × X × [ -1 , 1] → R | ∃ h : X × [ -1 , 1] → R d , g ( x ; a, f ) = 〈 h ( a, f ) , x 〉} . For any distribution p ∈ P ( X ) , let G u ( p ) ⊆ G represent the set of all unbiased estimators:

A typical choice of g given p is the estimator defined as follows:

where H ( p ) is assumed to be non-singular. This is an unbiased estimator, i.e., g defined by (4) is an element of G u . In fact, if M ∈ M glyph[lscript] , we have E a ∼ p,f ∼ M a [ h ( a, f )] = E a ∼ p [( H ( p )) -1 aa glyph[latticetop] glyph[lscript] ] = ( H ( p )) -1 H ( p ) glyph[lscript] = glyph[lscript] .

Using { g s } t s =1 , we set a reference distribution q t over X by the continuous exponential weights method as follows: q t ( x ) ∝ q 0 ( x ) exp( -η t ∑ t -1 s =1 g s ( x ; a s , f s )) , where q 0 is an arbitrary initial distribution over X and η t is a learning rate such that η 1 ≥ η 2 ≥ ··· ≥ η T > 0 . We may choose η t depending on the historical actions and observations so far (( a s , f s )) t -1 s =1 . Let D KL ( q ′ , q ) represent the Kulllback-Leibler divergence of q ′ from q . Note that the exponential weights method can be interpreted as a follow-the-regularized-leader approach over P with the regularizer function q ↦→ D KL ( q, a 0 ) , i.e., the distribution q t is a solution to the following optimization problem:

We may determine the sampling distribution p t on the basis of q t in an arbitrary way. The regret is then bounded as follows:

Lemma 1. For any distribution p ∗ ∈ P ( X ) such that D KL ( p ∗ , q 0 ) < ∞ , the expected value of the regret R T ( a ∗ ) for a ∗ ∼ p ∗ is bounded by

This follows from the standard analysis for exponential update methods or FTRL (see, e.g., [Lattimore and Szepesvári, 2020a, Exercise 28.12]), and all omitted proofs can be found in the appendix. For the case of the exponential weights, the stability term stab( g, q ) can be bounded as

The bias term bias( g, p, q, glyph[lscript] ) corresponds to the bias of the estimater g ( · ; a, f ) for the fucntion 〈 glyph[lscript], ·〉 , under the condition that a ∼ p , and E [ f | a ] = 〈 a, glyph[lscript] 〉 . We can easily see that g ∈ G u ( p ) implies bias( g, p, q, glyph[lscript] ) = 0 for all q ∈ P ( X ) and glyph[lscript] ∈ L .

For any reference distribution q ∈ P ( X ) and learning rate η > 0 , we define Λ q,η ( p, g ) by

From Lemma 1, the regret is bounded as

In the exploration-by-optimization (EXO) approach [Lattimore and Szepesvári, 2020b, Lattimore and Gyorgy, 2021], we choose g t and p t so that the value of Λ q t ,η t ( g t , p t ) is as small as possible, i.e., we consider the following optimization problem:

We denote the optimal value of this problem by Λ ∗ q t ,η t .

Remark 1. In most linear bandit algorithms based on exponential weights [Dani et al., 2008, Bubeck et al., 2012, Cesa-Bianchi and Lugosi, 2012, Hazan and Karnin, 2016, Besbes et al., 2019], p t is defined as q t mixed with a small component of exploration basis π 0 and g t is given by (4) with p = p t . The exploration basis π 0 here is a distribution over X such that G ( π 0 ) := sup a ∈X a glyph[latticetop] H ( π 0 ) a is bounded. It is known that any action set X ⊆ R d admits an exploration basis π 0 such that G ( π 0 ) ≤ d , which is called the G-optimal design or the John's ellipsoid exploration basis. Given such a π 0 , we set p t = (1 -γ t ) q t + γ t π 0 with some γ t ∈ (0 , 1) . Under the condition of η t = Ω(1 /d ) and γ t = Θ( dη t ) , we have Λ q t ,η t ( p t , g t ) = O ( γ t + η t d ) = O ( η t d ) , which implies that Λ ∗ q,η t = O ( η t d ) holds for any q ∈ P ( X ) . Hence, from (8), if D KL ( p ∗ , q 0 ) ≤ B , by setting η t = η = Θ( √ B/ ( dT )) , we obtain E a ∗ ∼ p ∗ [ R T ( a ∗ )] = O ( √ BdT ) .

4.2 Sufficient condition for best-of-both-worlds regret guarantee

In this section, we show that a certain type of upper bounds on Λ ∗ q,η that depend on reference distribution q will lead to best-of-both-worlds regret bounds. In the following, we set the initial disturibution q 0 to be a uniform distribution over X . Denote ∆ glyph[lscript] ∗ ( x ) = 〈 glyph[lscript] ∗ , x -a ∗ 〉 for any glyph[lscript] ∗ ∈ L . The following lemma provides a sufficient condition for best-of-both-worlds regret guarantees:

Theorem 1. Suppose that there exist α > 0 , β > 0 and η 0 > 0 such that

for all t and η ≤ η 0 . Then, an algorithm based on the EXO approach achieves the following regret bounds simultaneously: (i) In adversarial environments, we have R T = O ( √ αdT log T + d log T η 0 ) . (ii) In environments satisfying (2) , we have R T = O (( β + 1 η 0 ) d log T + √ Cβd log T ) .

In the proof of this theorem, we consider the algorithm that chooses ( p t , g t ) to be an optimal solution to (9) and that sets learning rates η t by η t = min { η 0 , √ d log T α + ∑ t -1 s =1 η -1 s Λ ∗ qs ,η s } . A complete proof is given in the appendix.

Remark 2. In the implementation of the algorithm, we do not necessarily need to solve the optimization problem (9) exactly. In fact, to achieve the regret upper bounds in Theorem 1, it is sufficient to compute p t , g t , and z t > 0 that satisfy η -1 t Λ q t ,η t ( p t , g t ) ≤ z t ≤ min { α, β · ∆ glyph[lscript] ∗ ( µ ( q t )) } for some α > 0 and β > 0 , under the assumption of η t ≤ η 0 . In this case, setting η t = min { η 0 , √ d log T/ √ α + ∑ t -1 s =1 z s } will work.

4.3 Regret bounds for concrete examples

In this section, we will see how we can bound Λ ∗ q t ,η t . For q ∈ P ( X ) , define ω ( q ) , ω ′ ( q ) ≥ 0 by

Note that ω ( q ) ≤ ω ′ ( q ) holds. In fact, from the definition of ω ′ ( q ) , there exists p ∈ P ( X ) satisfying µ ( p ) = µ ( q ) and d · V ( q ) glyph[precedesequal] ω ′ ( q ) · H ( p ) . For such a distribution p , we have H ( p ) -1 · V ( q ) = tr( H ( p ) -1 / 2 V ( q ) H ( p ) -1 / 2 ) ≤ tr( d -1 · ω ′ ( q ) · I ) = ω ′ ( q ) , which implies that ω ( q ) ≤ ω ′ ( q ) .

Using these, we can construct a bound Λ ∗ q,η as the following lemma provides:

Lemma 2. Suppose that q is a log-concave distribution. If η ≤ 1 /d , we then have Λ ∗ q,η = O ( ηω ( q )) . If η ≤ 1 , we then have Λ ∗ q,η = O ( ηω ′ ( q )) .

Bounds on Λ ∗ q,η in this lemma can be achieved by p = γ ˜ p +(1 -γ ) π 0 and g defined by (4), where ˜ p is the minimizer on the right-hand side of (11) or (12), π 0 is a G-optimal design for X , and some mixing weight parameter γ ∈ (0 , 1) . The proof of this lemma is given in the appendix.

General action set We can provide an upper bound on ω ( q ) by exploiting the property of logconcave distributions . A distribution q ∈ P ( X ) is called a log-concave distribution if it has a distribution function that is proportional to exp( -h ( x )) for some convex function h . Note that the reference distribution q t is a log-concave function.

Lemma 3. Suppose that q is a log-concave distribution. Let glyph[lscript] ∗ ∈ L be an arbitrary loss vector such that a ∗ ∈ arg min a ∈A 〈 glyph[lscript] ∗ , a 〉 is unique. Denote ∆ glyph[lscript] ∗ , min = min a ∈A\{ a ∗ } ∆ glyph[lscript] ∗ ( a ) . We then have ω ′ ( q ) = O ( d · min { 1 , ∆ glyph[lscript] ∗ ( µ ( q )) / ∆ glyph[lscript] ∗ , min } ) .

We here provide a proof sketch for this lemma. We consider a distribution p ∈ P ( A ) of a ∈ A generated by the following procedure: (i) Pick x ∈ X following q . (ii) Let ξ be an arbitrary distribution over A such that µ ( ξ ) = x . In other words, compute ξ : A → R ≥ 0 such that ∑ a ′ ∈A ξ ( a ′ ) = 1 and ∑ a ′ ∈A ξ ( a ′ ) a ′ = x . (iii) Pick a ∈ A following ξ . It is then clear that µ ( p ) = µ ( q ) holds. Further, it follows from the assumption that q is a log-concave distribution that (1 -p ( a ∗ )) u glyph[latticetop] V ( p ) u = Ω( u glyph[latticetop] V ( q ) u ) holds for any a ∗ ∈ A and any u ∈ R d \ { 0 } , which implies d · V ( q ) glyph[precedesequal] O ((1 -p ( a ∗ )) d ) · V ( p ) glyph[precedesequal] O ((1 -p ( a ∗ )) d ) H ( p ) . Finally, since we have (1 -p ( a ∗ )) ≤ ∆ glyph[lscript] ∗ ( µ ( p )) / ∆ glyph[lscript] ∗ , min , the bound on ω ′ ( q ) in Lemma 3 follows. A full proof can be found in the appendix.

From Lemmas 2 and 3, we obtain the following bounds on α and β :

Corollary 1. For arbitrary action set A , the condition (10) in Theorem 1 holds with α = O ( d ) , β = O ( d/ ∆ glyph[lscript] ∗ , min ) , and η 0 = O (1) .

Hence, there exists an algorithm that achieves R T = O ( d √ T log T ) in adversarial environments and R T = O ( d 2 ∆ glyph[lscript] ∗ , min log T + √ Cd 2 ∆ glyph[lscript] ∗ , min log T ) in stochastically constrained adversarial environments.

Multi-armed bandits Suppose a problem setting in which the action set is the canonical bases: A = A d := { e 1 , e 2 , . . . , e d } = { a ∈ { 0 , 1 } d | ‖ a ‖ 1 = 1 } . For this action set, we have the following bound on ω ( q ) :

Lemma 4. If A = A d , for any log-concave distribution q , ω ( q ) defined in (11) is bounded as ω ( q ) ≤ O ( ∑ d i =1 µ i ( q )(1 -µ i ( q )) ) ≤ O (min 1 ≤ i ≤ d { 1 -µ i ( q ) } ) .

glyph[negationslash]

Let glyph[lscript] ∗ ∈ L be an a loss vector such that e i ∗ ∈ arg min 1 ≤ i ≤ d glyph[lscript] ∗ i is unique. We then have ∆ glyph[lscript] ∗ ( µ ) = ∑ i = i ∗ ∆ glyph[lscript] ∗ ( e i ) µ i ≥ ∆ glyph[lscript] ∗ , min (1 -µ i ∗ ) , which, combined with Lemma 4, implies ω ( q ) ≤ ∆ glyph[lscript] ∗ ( µ ( q )) / ∆ glyph[lscript] ∗ , min . From this and Lemma 2, we have the following bounds on α and β :

Corollary 2. For the multi-armed bandit problem A = A d , the condition (10) in Theorem 1 holds with α = O (1) , β = O (1 / ∆ glyph[lscript] ∗ , min ) and η 0 = O (1 /d ) .

Product-space action sets Suppose that A ⊆ R d is expressed as a product space of m sets, i.e., A = A (1) × A (2) × · · · × A ( m ) , where A ( j ) ⊆ R d j for each j and ∑ m j =1 d j = d . Denote X ( j ) = conv( A ( j ) ) . This setting is referred to as the multi-task bandit problem [Cesa-Bianchi and Lugosi, 2012], in which the player chooses a ( j ) t ∈ A ( j ) for each j in parallel, and then gets only a single feedback corresponding to the sum of the losses for all m actions. For such an action set, probability distribution q t given by continuous exponential weights can be expressed as a product measure of m log-concave measures q ( j ) t ∈ P ( X ( j ) ) . In fact, if a probability density function is expressed as q ( x ) = c exp( 〈 ηL, x 〉 ) for some c > 0 η > 0 , and L = ( L (1) , . . . , L ( m ) ) ∈ R d , it can be decomposed as q ( x ) = c exp( ∑ m j =1 〈 ηL ( j ) , x ( j ) 〉 ) = ∏ m j c ( j ) exp( 〈 ηL ( j ) , x ( j ) 〉 ) = ∏ m j q ( j ) ( x ( j ) ) , where x ( j ) ∈ X ( j ) and c ( j ) = ∫ x ∈X ( j ) exp( 〈 ηL ( j ) , x ( j ) 〉 )d x . This fact leads to the following bounds on ω ( q ) and Λ ∗ q,η :

Lemma 5. If A = A (1) ×··· × A ( m ) and q ∈ P ( X ) can be expressed as a product measure of q ( j ) ∈ P ( X ( j ) ) for j = 1 , . . . , m , we have ω ( q ) ≤ ∑ m j =1 ω ( q ( j ) ) .

Corollary 3. If A = A d 1 × · · · × A d m , the condition (10) in Theorem 1 holds with α = O ( m ) , β = O (1 / ∆ glyph[lscript] ∗ , min ) , and η 0 = O (1 /d ) . If A = { 0 , 1 } d , the condition (10) in Theorem 1 holds with α = O ( d ) , β = O (1 / ∆ glyph[lscript] ∗ , min ) , and η 0 = O (1 /d ) .

4.4 Computational complexity

At this time, it is not known if there is an efficient algorithm to solve the optimization problems (9). However, it is worth noting that the optimization problem corresponding to the right-hand side of (11) is a convex optimization problem. Further, as the minimum can be achieved by p ∈ P ( A ) , we can reduce the problem into a convex optimization problem over |A| -dimensional space, which can be solved efficiently if |A| is not very large.

Note that it is not always necessary to solve the optimization problem (9) exactly, as mentioned in Remark 2. For example, we can implement an algorithm achieving regret bounds of Corollary 1 given a separation oracle (or equivalently, a linear optimization oracle) for cA , without solving (9) exactly. Indeed, to achieve the regret bounds of Corollary 1, it suffices to find p and g such that Λ q t ,η t ( p, g ) is bounded by the RHS of (10). The construction of such p and g is provided in the proof of Lemmas 2 and 3, which can be performed using a separation oracle for A . In fact, we can obtain samples from p by using the techniques for log-concave sampling (e.g., [Lovász and Vempala, 2007]) and for computing convex combination expression (cf. Carathéodory's theorem for convex hull and [Schrijver, 1998, Corollary 14.1g]). However, the analysis of log-concave sampling and calculations of H ( p ) (which is required for constructing g ) including consideration of calculation errors can be highly complicated, and the computational cost can be very large, although on the order of polynomials (e.g., [Hazan and Karnin, 2016, Corollary 6.2]).

5 Mean-oriented FTRL and SCRiBLe

For linear bandit problems, we have an alternative approach that we referred to as mean-oriented FTRL , in which we compute a reference mean w t given as follows:

where h s ( a s , f s ) is an estimator of glyph[lscript] s and ψ is a convex regularization function over X such that min x ∈X ψ ( x ) = 0 . Let D ψ ( x ′ , x ) denote the Bregman divergence associated with ψ : D ψ ( x ′ , x ) = ψ ( x ′ ) -ψ ( x ) -〈∇ ψ ( x ) , x ′ -x 〉 . This approach ensures the following regret bound:

Lemma 6. For any point x ∗ ∈ X , we have

One example of linear bandit algorithms based on mean-oriented FTRL is the SCRiBLe algorithm [Abernethy et al., 2008, 2012], which employs self-concordant barriers as regularizer functions.

We can also consider an EXO approach for mean-oriented FTRL. For any reference point w ∈ X and learning rate η > 0 , define

and Λ ∗ w,η = inf p ∈P ( X ) , ˆ glyph[lscript] : X× R → R d Λ w,η ( p, ˆ glyph[lscript] ) .

This achieves the following BOBW regret bounds, under assumptions on Λ ∗ w,η :

Lemma 7. Suppose that ψ is a ν -self-concordant barrier. Suppose that there exist α > 0 , β > 0 and η 0 > 0 such that

for all t and η ≤ η 0 . Then an algorithm based on the EXO approach achieves the following regret bounds simultaneously: (i) In adversarial environments, we have R T = O ( √ ανT log T + ν log T η 0 ) . (ii) In environments satisfying (2) , we have R T ( a ∗ ) = O (( β + 1 η 0 ) ν log T + √ Cβν log T ) .

5.1 Connection to the SCRiBLe algorithm and variants of it

Abernethy et al. [2008, 2012] have proposed a sampling scheme w t ↦→ p t supported on the Dikin ellipsoid, which leads to Λ w t ,η t ( p t , h t ) = O ( η t d 2 ) with an appropriately defined unbiased estimator h t . This algorithm referred to as SCRiBLe hence achieves R T = O ( √ d 2 νT log T ) , the adversarial regret bound in Lemma 7 with α = O ( d 2 ) . Recent studies by Dann et al. [2023b, Theorem 3] and Ito and Takemura [2023] have shown that a modified sampling scheme based on the Dikin ellipsoid sampling achieves Λ w t ,η t ( p t , h t ) = O ( η t d 2 · min { 1 , ∆ glyph[lscript] ∗ ( w t ) / ∆ glyph[lscript] ∗ , min } ) . Consequently, these modified algorithms achieve BOBW regret bounds in Lemma 7 with α = O ( d 2 ) and β = O ( d 2 / ∆ glyph[lscript] ∗ , min ) . These results suggest that the following lemma holds:

Lemma 8 ([Dann et al., 2023b, Ito and Takemura, 2023]) . If ψ is a self-concordant barrier, (15) holds with α = O ( d 2 ) , β = O ( d 2 / ∆ glyph[lscript] ∗ , min ) and η 0 = O (1 /d ) .

5.2 Connection to EXO with continuous exponential weights

The entropic barrier is a ν -self-concordant barrier with ν ≤ d [Chewi, 2021], for which a definition and properties are given, e.g., in [Bubeck and Eldan, 2019]. We can see that mean-oriented EXO with entropic-barrier regularization coincides with the EXO approach with exponential weights. In fact, if we compute w t using (13) with ψ the entropic barrier, from the definition of the entropic barrier, w t is equal to the mean of the distribution q t given by the continuous exponential weights. Given this correspondence, we can see that bounds on Λ ∗ w t ,η follow immediately from Corollary 1:

Lemma 9. If ψ is the entropic barrier for X , (15) holds with α = O ( d ) , β = O ( d/ ∆ glyph[lscript] ∗ , min ) and η 0 = O (1) .

It is worth noting that these bounds are O (1 /d ) -times better than the results for Lemma 8 that follow from previous studies.

6 Limitations and future work

A limitation of this work is that the regret bounds for (corrupted) stochastic environments require the assumption that the optimal arm a ∗ ∈ arg min a ∈A 〈 glyph[lscript] ∗ , a 〉 is unique. While this assumption is common in analyses of best-of-both-worlds algorithms based on the self-bounding technique [Zimmert and Seldin, 2021, Dann et al., 2023b], a limited number of studies on the multi-armed bandit problem [Ito, 2021, Jin et al., 2023] have removed this uniqueness assumption by careful analyses of regret bounds for follow-the-regularized-leader methods. As these analyses rely on structures specific to the multi-armed bandit problems, extending them to other problem settings does not seem trivial. Extending this analysis technique to linear bandits and removing the uniqueness assumptions will be an important future challenge.

Another future direction is to work toward a tighter regret bound of O ( c ∗ log T ) in stochastic environments while satisfying BOBW regret guarantee, where c ∗ is an instance-dependent quantity that characterizes the optimal asymptotic bound (see, e.g., [Lattimore and Szepesvari, 2017, Corollary 2]). As c ∗ ≤ O ( d/ ∆ min ) [Lee et al., 2021, Lemma 16.] and the gap between these two quantities can be arbitrarily large [Lattimore and Szepesvari, 2017, Example 4], our bounds depending on 1 / ∆ min can be far from tight. In the multi-armed bandit problem, a special case of linear bandits, the quantity c ∗ can be expressed as c ∗ = ∑ a ∈A :∆ glyph[lscript] ∗ ( a ) > 0 2 ∆ glyph[lscript] ∗ ( a ) , and known FTRL-type BOBW algorithms with O ( c ∗ log T ) -regret bounds employs the Tsallis entropy with Θ(1 / √ t ) -learning rates: ψ t ( w ) = - √ t ∑ d i =1 √ w i [Zimmert and Seldin, 2021] or logarithmic barriers with entry-wise adaptive learning rates: ψ t ( w ) = -∑ d i =1 1 η ti log w i [Ito, 2021, Ito et al., 2022a]. The latter would be more closely related to our results in this paper as logarithmic barriers are examples of self-concordant barriers. A major difference between the approach in this paper and the one by Ito [2021] is that the former only considers regularizer functions expressed as a scalar multiple of a fixed function, while the latter even changes the shape of regularizer functions for each round. Such a more flexible adaptive regularization may be necessary for pursuing tighter regret bounds. Another promising approach is to extend the Tsallis-entropy approach to linear bandits while the connection between the Tsallis-INF [Zimmert and Seldin, 2021] and this paper may be somewhat tenuous as the Tsallis entropy is not a self-concordant barrier.

References

Abernethy, E. Hazan, and A. Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. In Conference on Learning Theory , 2008.

D. Abernethy, E. Hazan, and A. Rakhlin. Interior-point methods for full-information and bandit online learning. IEEE Transactions on Information Theory , 58(7):4164-4175, 2012.

Amir, G. Azov, T. Koren, and R. Livni. Better best of both worlds bounds for bandits with switching costs. Advances in Neural Information Processing Systems , 35, 2022.

Besbes, Y. Gur, and A. Zeevi. Optimal exploration-exploitation in a multi-armed bandit problem with non-stationary rewards. Stochastic Systems , 9(4):319-337, 2019.

Bubeck and R. Eldan. The entropic barrier: Exponential families, log-concave geometry, and self-concordance. Mathematics of Operations Research , 44(1):264-276, 2019.

Bubeck and A. Slivkins. The best of both worlds: Stochastic and adversarial bandits. In Conference on Learning Theory , pages 42-1. PMLR, 2012.

Bubeck, N. Cesa-Bianchi, and S. M. Kakade. Towards minimax policies for online linear optimization with bandit feedback. In Conference on Learning Theory , pages 41-1. JMLR Workshop and Conference Proceedings, 2012.

Cesa-Bianchi and G. Lugosi. Combinatorial bandits. Journal of Computer and System Sciences , 78(5):1404-1422, 2012.

Chewi. The entropic barrier is n -self-concordant. arXiv preprint arXiv:2112.10947 , 2021.

Dani, S. M. Kakade, and T. P. Hayes. The price of bandit information for online optimization. In Advances in Neural Information Processing Systems , pages 345-352, 2008.

Dann, C.-Y. Wei, and J. Zimmert. Best of both worlds policy optimization. In International Conference on Machine Learning , pages 6968-7008. PMLR, 2023a.

Dann, C.-Y. Wei, and J. Zimmert. A blackbox approach to best of both worlds in bandits and beyond. In Conference on Learning Theory , pages 5503-5570. PMLR, 2023b.

Erez and T. Koren. Towards best-of-all-worlds online learning with feedback graphs. Advances in Neural Information Processing Systems , 34, 2021.

J. Foster, S. M. Kakade, J. Qian, and A. Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 , 2021.

J. Foster, A. Rakhlin, A. Sekhari, and K. Sridharan. On the complexity of adversarial decision making. In Advances in Neural Information Processing Systems , 2022.

Hazan and Z. Karnin. Volumetric spanners: an efficient exploration basis for learning. The Journal of Machine Learning Research , 17(1):4062-4095, 2016.

Hoeven, T. Erven, and W. Kotłowski. The many faces of exponential weights in online learning. In Conference On Learning Theory , pages 2067-2092. PMLR, 2018.

Ito. Parameter-free multi-armed bandit algorithms with hybrid data-dependent regret bounds. In Conference on Learning Theory , pages 2552-2583. PMLR, 2021.

Ito and K. Takemura. Best-of-three-worlds linear bandit algorithm with variance-adaptive regret bounds. In Conference on Learning Theory , pages 2653-2677. PMLR, 2023.

Ito, S. Hirahara, T. Soma, and Y . Yoshida. Tight first-and second-order regret bounds for adversarial linear bandits. In Advances in Neural Information Processing Systems , volume 33, 2020.

Ito, T. Tsuchiya, and J. Honda. Adversarially robust multi-armed bandit algorithm with variancedependent regret bounds. In Conference on Learning Theory , pages 1421-1422. PMLR, 2022a.

Ito, T. Tsuchiya, and J. Honda. Nearly optimal best-of-both-worlds algorithms for online learning with feedback graphs. In Advances in Neural Information Processing Systems , volume 35, 2022b.

Jin and H. Luo. Simultaneously learning stochastic and adversarial episodic mdps with known transition. Advances in Neural Information Processing Systems , 33:16557-16566, 2020.

Jin, L. Huang, and H. Luo. The best of both worlds: stochastic and adversarial episodic mdps with unknown transition. Advances in Neural Information Processing Systems , 34, 2021.

Jin, J. Liu, and H. Luo. Improved best-of-both-worlds guarantees for multi-armed bandits: Ftrl with general regularizers and multiple optimal arms. arXiv preprint arXiv:2302.13534 , 2023.

Kirschner, T. Lattimore, and A. Krause. Information directed sampling for linear partial monitoring. In Conference on Learning Theory , pages 2328-2369. PMLR, 2020.

Kong, C. Zhao, and S. Li. Best-of-three-worlds analysis for linear bandits with follow-theregularized-leader algorithm. In Proceedings of Thirty Sixth Conference on Learning Theory , pages 657-673. PMLR, 2023.

Lattimore and A. Gyorgy. Mirror descent and the information ratio. In Conference on Learning Theory , pages 2965-2992. PMLR, 2021.

Lattimore and C. Szepesvari. The end of optimism? an asymptotic analysis of finite-armed linear bandits. In Artificial Intelligence and Statistics , pages 728-737. PMLR, 2017.

Lattimore and C. Szepesvári. Bandit Algorithms . Cambridge University Press, 2020a.

Lattimore and C. Szepesvári. Exploration by optimisation in partial monitoring. In Conference on Learning Theory , pages 2488-2515. PMLR, 2020b.

C.-W. Lee, H. Luo, C.-Y. Wei, M. Zhang, and X. Zhang. Achieving near instance-optimality and minimax-optimality in stochastic and adversarial linear bandits simultaneously. In International Conference on Machine Learning , pages 6142-6151. PMLR, 2021.

Li, E. Y. Lou, and L. Shan. Stochastic linear optimization with adversarial corruption. arXiv preprint arXiv:1909.02109 , 2019.

Lovász and S. Vempala. The geometry of logconcave functions and sampling algorithms. Random Structures & Algorithms , 30(3):307-358, 2007.

Lykouris, V. Mirrokni, and R. Paes Leme. Stochastic bandits robust to adversarial corruptions. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pages 114-122, 2018.

Masoudian, J. Zimmert, and Y. Seldin. A best-of-both-worlds algorithm for bandits with delayed feedback. Advances in Neural Information Processing Systems , 35, 2022.

Mirrokni, R. P. Leme, A. Vladu, and S. C.-w. Wong. Tight bounds for approximate carathéodory and beyond. In International Conference on Machine Learning , pages 2440-2448. PMLR, 2017.

Rakhlin and K. Sridharan. Online learning with predictable sequences. In Conference on Learning Theory , pages 993-1019. PMLR, 2013.

Rouyer, Y. Seldin, and N. Cesa-Bianchi. An algorithm for stochastic and adversarial bandits with switching costs. In International Conference on Machine Learning , pages 9127-9135. PMLR, 2021.

Rouyer, D. van der Hoeven, N. Cesa-Bianchi, and Y. Seldin. A near-optimal best-of-both-worlds algorithm for online learning with feedback graphs. Advances in Neural Information Processing Systems , 35, 2022.

Russo and B. Van Roy. Learning to optimize via information-directed sampling. Advances in Neural Information Processing Systems , 27, 2014.

Saha and P. Gaillard. Versatile dueling bandits: Best-of-both world analyses for learning from relative preferences. In International Conference on Machine Learning , pages 19011-19026. PMLR, 2022.

Schrijver. Theory of Linear and Integer Programming . John Wiley & Sons, 1998.

Tsuchiya, S. Ito, and J. Honda. Best-of-both-worlds algorithms for partial monitoring. In International Conference on Algorithmic Learning Theory , pages 1484-1515. PMLR, 2023.

C.-Y. Wei and H. Luo. More adaptive algorithms for adversarial bandits. In Conference On Learning Theory , pages 1263-1291. PMLR, 2018.

Zimmert and Y. Seldin. Tsallis-INF: An optimal algorithm for stochastic and adversarial bandits. Journal of Machine Learning Research , 22(28):1-49, 2021.

Zimmert, H. Luo, and C.-Y. Wei. Beating stochastic and adversarial semi-bandits optimally and simultaneously. In International Conference on Machine Learning , pages 7683-7692. PMLR, 2019.

A Notes on notation and assumption

For any two real vectors x = ( x 1 , . . . , x d ) ∈ R d and y = ( y 1 , . . . , y d ) ∈ R d , 〈 x, y 〉 denotes the the inner product of x and y , i.e., 〈 x, y 〉 = ∑ d i =1 x i y i . For any real vector x = ( x 1 , . . . , x d ) ∈ R d , let ‖ x ‖ 1 denote the L 1 -norm of x , i.e., ‖ x ‖ 1 = ∑ d i =1 | x i | . For any real symmetric matrix X , let tr( X ) denote the trace of X . For any two real symmetric matrices X and Y , we denote X glyph[precedesequal] Y (or Y glyph[followsequal] X ) if and only if Y -X is a positive semi-definite matrix. For any two real symmetric matrices X and Y , X · Y denotes the Frobenius inner product of X and Y , i.e., X · Y = tr( AB ) .

Without loss of generality, we may assume that A spans all vectors in R d . Indeed, if not, we can make A make full-dimensional by ignoring redundant dimensions. Under this assumption, distributions q given by exponential weight methods have non-singular variance-covariance matrices V ( q ) (and thus also H ( q ) ). This fact is used in some places in this paper without notice.

B Omitted proofs

B.1 Lemmas on log-concave distributions

Lemma 10 (Theorem 5.2 in [Lovász and Vempala, 2007]) . If X ∈ R n is a random variable following a log-concave distribution, for any constant matrix M ∈ R m × n , MX follows a logconcave distribution as well.

Lemma 11 (Lemmas 5.4 and 5.5 (a) in [Lovász and Vempala, 2007]) . If p is a one-dimensional log-concave distribution, we have

Further, if V ( p ) = 1 , we have for any a, b ∈ R .

Lemma 12. For any one dimensional log-concave distribution p over a one-dimensional segment [ L, R ] ⊆ R , we have

Proof. From Lemma 11, we have

where the first and the second inequalities follow from (16) and (17), respectively. We hence have V ( p ) ≤ e 2 ( µ ( p ) -L ) 2 . By considering -X instead of X , we can show V ( p ) ≤ e 2 ( R -µ ( p )) 2 in a similar way.

Lemma 13. Suppose φ is defined as in (7) . If y follows a log-concave distribution over R and if s = √ E [ y 2 ] ≤ √ 1 / 2 , we have

Proof. See, e.g., the proof of [Ito et al., 2020, Lemma 5].

B.2 Proof of Lemma 1

Proof. From the definition of the regret, we have

From the definition of bias given in (6), the part of E x ∼ q t [ 〈 glyph[lscript] t , x -a ∗ 〉 ] can be bounded as follows:

From this and (21), we have

By a standard analysis of the exponential weight method, we have

To show (24), we denote G t ( p ) = E a ∼ p [ g t ( a ; a t , f t )] in the following for notational simplicity. From (5), we have

where the first inequality follows from (5) and the second inequality follows from η T +1 ≤ η T and the fact that the values of the KL divergence are nonnegative. The second equality follows from the definition of q T ( x ) ; As q T ( x ) can be expressed as q T ( x ) = κ T exp( -η T ˜ g T ( x )) q 0 ( x ) for some

κ T > 0 , where we denote ˜ g T ( x ) = ∑ T -1 s =1 g s ( x ; a s , f s ) , we have

and

These equalities together imply that the second equality in (25). The last inequality of (25) can be obtained by applying similar transformations repeatedly. From (25), we have

which implies that (24) holds. By combining (23) and (24), we obtain the bound of (6).

B.3 Proof of Theorem 1

Proof. For any a ∗ ∈ X , let p ∗ be the uniform distribution over { (1 -1 T ) a ∗ + 1 T x | x ∈ X} . We then have R T ( a ∗ ) ≤ E a ∼ p ∗ [ R T ( a )] + 1 and D KL ( p ∗ , q 1 ) ≤ d log T . Define η t by η t = min { η 0 , √ d log T/ √ α + ∑ t -1 s =1 η -1 s Λ ∗ q s ,η s } . Suppose that p t and g t are chosen so that Λ q t ,η t ( p t , g t ) = Λ ∗ q t ,η t . As we have η t ≤ η 0 , from the assumption, we have η -1 t Λ q t ,η t ≤ α . Hence, from (8), we have

which implies the regret bound of R T = O ( √ αdT log T + d log T η 0 ) that holds in adversarial environments. We next show the improved regret bounds for stochastically constrained adversarial environments. From the definition of Λ , we have

where the last inequality holds since -glyph[lscript] ∗ ∈ L for any glyph[lscript] ∗ ∈ L , which follows from the definition of L . We then, hence, have

From this, we have

B.4 Proof of Lemma 2

Proof. We first see that Λ ∗ q,η = O (1) holds for any q ∈ P ( X ) and η > 0 . In fact, by setting p = q and letting g ∈ G to be the zero function ( g ( x ; a, f ) = 0 for all x, a and f ), we obtain Λ q,η ( p, g ) = sup glyph[lscript] ∈L bias( g, p, q, glyph[lscript] ) = sup glyph[lscript] ∈L ,x ∈X ,a ∗ ∈X 〈 glyph[lscript],x -a ∗ 〉 ≤ 2 .

Let us consider the case in which η ≤ 1 /d holds. As Λ ∗ q,η = O (1) for any η and q , we have Λ ∗ q,η = O ( ηω ( q )) if ηω ( q ) = Ω(1) . In the following, we assume ηω ( q ) ≤ 1 / 8 . Let ˜ p ∈ P ( X ) be a distribution such that µ (˜ p ) = µ ( q ) and w ( q ) = H (˜ p ) -1 · V ( q ) . Let π 0 ∈ P ( X ) be a distribution such that

holds for any a ∈ A . Note that such a distribution always exists (see, e.g., [Lattimore and Szepesvári, 2020a, Theorem 21.1]). Define p by

with γ = 4 ηω ( q ) ≤ 1 / 2 . With this p , we define g by (4). Then, as we have bias( g, p, q, glyph[lscript] ) = 0 , we have

As the definition of p implies µ ( p ) = (1 -γ ) µ (˜ p ) + γµ ( π 0 ) = (1 -γ ) µ ( q ) + γµ ( π 0 ) , it holds for any glyph[lscript] ∈ L that

Further, from (7) and (4), we have

for any fixed a ∈ A and f ∈ [ -1 , 1] . Let r represent the distribution of f · 〈 ( H ( p )) -1 a, x -µ ( q ) 〉 for x ∼ q with any fixed a ∈ A and f ∈ [ -1 , 1] . Then, as q is a log-concave distribution, from Lemma 10, r is a log-concave distribution as well. Further, we have

Since we have

we have we have

where I d represents the identity matrix of size d . Hence, we have

This implies that η 2 H ( r ) ≤ dη/ 2 ≤ 1 / 2 . Hence, from Lemma 13, we have

Hence, we have

Combining this with (34) and (35), we obtain

which implies that Λ ∗ q,η = O ( ηω ( q )) .

We next consider the case in which η ≤ 1 holds. As we have Λ ∗ q,η = O ( ηω ′ ( q )) if ηω ′ ( q ) > 1 / 8 (since Λ ∗ q,η = O (1) for any q and η ), we assume ηω ′ ( q ) ≤ 1 / 8 . We set p by (33) where ˜ p is the minimizer in the right-hand side of (12) and γ = 4 ηω ′ ( q ) ≤ 1 / 2 . We set g by (4). Then, (34) and (35) can be shown in a similar way to in the first half of this proof. As we have

From this, if we define r in a similar way to in the first half of this proof, we have H ( r ) ≤ 1 / (2 η ) . We hence have η 2 H ( r ) ≤ η/ 2 ≤ 1 / 2 under the assumption of η ≤ 1 . We can then apply Lemma 13 to obtain the following:

where the first inequality follows from a similar argument as in (42). Combining this with (34), (35), and γ = 4 ηω ′ ( q ) , we obtain Λ q,η ( p, g ) = O ( ηω ′ ( q )) , which implies Λ ∗ q,η = O ( ηω ′ ( q )) .

B.5 Proof of Lemma 3

Proof of Lemma 3. We will show that

with z = O (1 , ∆ glyph[lscript] ∗ ( µ ( q )) / ∆ glyph[lscript] ∗ , min ) . As we have H ( p ) = V ( p ) + µ ( p ) µ ( p ) glyph[latticetop] glyph[followsequal] V ( p ) , (47) leads to d · V ( q ) glyph[precedesequal] dz · V ( p ) glyph[precedesequal] dz · H ( p ) , which implies ω ′ ( q ) ≤ dz . Hence, to prove the lemma, it is sufficient to show that (47) holds with z = O (1 , ∆ glyph[lscript] ∗ ( µ ( q )) / ∆ glyph[lscript] ∗ , min ) . By considering the case of p = q , we can easily see that (47) holds with z = O (1) . for any q . In the following, we show that (47) holds with z = O ( ∆ glyph[lscript] ∗ ( µ ( q )) ∆ ∗ ) .

Without loss of generality, we may assume µ ( q ) = 0 , via a variable transformation x ← x -µ ( q ) . In fact, such a transformation preserves V ( q ) and V ( p ) . For any x ∈ X , there exists ξ x = ( ξ x ( a )) a ∈A ∈ P ( A ) such that µ ( ξ x ) = ∑ a ∈A ξ x ( a ) = x . For any ( ξ x ) x ∈X , define p ∈ P ( A ) by p ( a ) = E x ∼ q [ ξ x ( a )] . We then have µ ( p ) = E a ∼ p [ x ] = E x ∼ q [ E a ∼ ξ x [ a ]] = E x ∼ q [ x ] = µ ( q ) = 0 . Fix an arbitrary non-zero vector u ∈ R d \ { 0 } and an arbitrary a ∗ ∈ A . Denote ε = 1 -Pr a ∼ p [ a = a ∗ ] .

Set v = u/ √ u glyph[latticetop] V ( q ) u . Let p v and q v denote the distributions of 〈 v, a 〉 for a ∼ p and 〈 v, x 〉 for x ∼ q , respectively. We then have V ( q v ) = v glyph[latticetop] V ( q ) v = u glyph[latticetop] V ( q ) u/ ( u glyph[latticetop] V ( q ) u ) = 1 . Without loss of generality, we assume that 〈 v, a ∗ 〉 ≥ 〈 v, µ ( q ) 〉 = µ ( q v ) = 0 . (If not, we redefine v = -u/ √ u glyph[latticetop] V ( q ) u .) Since q is a log-concave distribution, from Lemma 10, q v is a log-concave distribution as well. Hence, from Lemma 11, we have

where the inequality follows from (16) and (17).

We hence have

where the last inequality follows from (48). Suppose 〈 v, x 〉 ≤ -1 2e . Since µ ( ξ x ) = x , we have

We hence have

where the second inequality comes from Jensen's inequality, and the last inequality follows from (50). From this and (49), we have

We further have

where the last inequality follows from (48). Combining this with (51), we obtain

As it follows from the definition of v that

Since this holds for all u ∈ R d \ { 0 } , we have

Letting a ∗ ∈ arg min a ∈A 〈 glyph[lscript] ∗ , a 〉 , we obtain

Combining this with (56), we obtain

we have which completes the proof.

B.6 Proof of Lemma 4

Proof. Let q be a log-concave distribution over X = { x ∈ [0 , 1] d | ‖ x ‖ 1 = 1 } . If x = ( x 1 , . . . , x d ) ∼ q , from Lemma 10, x i for each i also follows a log-concave distribution over [0 , 1] . Hence, from Lemma 12, the variance of x i ( i.e., the ( i, i ) entry of V ( q ) ) is bounded as

Consider the distribution p ∈ P ( A ) such that Pr a ∼ p [ a = e i ] = µ ( q ) . We then have µ i ( p ) = µ i ( q ) and that H ( p ) is a diagonal matrix with diagonal entries ( µ i ( q )) d i =1 . Hence, from (59), we have

which implies that ω ( q ) = O ( ∑ d i =1 µ i ( q )(1 -µ i ( q )) ) . Further, as we have ∑ d i =1 µ i ( q ) = 1 , it holds for any i ∗ ∈ { 1 , . . . , d } that glyph[negationslash]

which complete the proof.

B.7 Proof of Corollary 2

glyph[negationslash]

Proof. From Lemma 4 and the fact that ∑ d i =1 µ i ( q ) = 1 , it is clear that ω ( q ) = O (1) . Let glyph[lscript] ∗ ∈ L be an a loss vector such that e i ∗ ∈ arg min 1 ≤ i ≤ d glyph[lscript] ∗ i is unique. We then have ∆ glyph[lscript] ∗ ( µ ( q )) = ∑ i = i ∗ ∆ glyph[lscript] ∗ ( e i ) µ i ≥ ∆ glyph[lscript] ∗ , min (1 -µ i ∗ ( q )) . Hence, from Lemma 4, we have ω ( q ) = O (1 -µ i ∗ ( q )) ≤ O (∆ glyph[lscript] ∗ ( µ ( q )) / ∆ glyph[lscript] ∗ , min ) . From this and Lemma 2, (10) holds with α = O (1) , β = O ( 1 ∆ glyph[lscript] ∗ , min ) , and η 0 = O (1 /d ) .

glyph[negationslash]

B.8 Proof of Lemma 5

Proof. For each j ∈ { 1 , . . . , m } , let p ( j ) ∈ P ( X ( j ) ) be such that µ ( p ( j ) ) = µ ( q ( j ) ) and ω ( q ( j ) ) = V ( q ( j ) ) · ( H ( p ( j ) )) -1 . Let p ∈ P ( X ) be the product measure of p (1) , . . . , p ( m ) . As p is the product measure of q (1) , . . . , q ( m ) , V ( q ) ∈ R d × d is a block diagonal matrix with submatrices V ( q (1) ) , . . . , V ( q ( m ) ) . Similarly, H ( p ) ∈ R d × d is a block diagonal matrix with submatrices H ( p (1) ) , . . . , H ( p ( m ) ) . We hence have V ( q ) · ( H ( p )) -1 = ∑ m j =1 V ( q ( j ) ) · ( H ( p ( j ) )) -1 = ∑ m j =1 ω ( q ( j ) ) , which implies ω ( q ) ≤ ∑ m j =1 ω ( q ( j ) ) .

B.9 Proof of Corollary 3

Proof. Suppose A is given by A = A d 1 × · · · × A d m . For any glyph[lscript] = ( glyph[lscript] (1) , glyph[lscript] (2) , . . . , glyph[lscript] ( m ) ) ∈ L and x = ( x (1) , x (2) , . . . , x ( m ) ) ∈ X , where glyph[lscript] ( j ) ∈ R d j and x ( j ) ∈ X ( j ) for each j , denote ∆ ( j ) glyph[lscript] ( j ) ( x ( j ) ) = 〈 glyph[lscript] ( j ) , x ( j ) 〉 -min a ( j ) ∈A ( j ) 〈 glyph[lscript] ( j ) , a ( j ) 〉 = 〈 glyph[lscript] ( j ) , x ( j ) -a ∗ ( j ) 〉 . We then have ∆ glyph[lscript] ∗ ( x ) = ∑ m j =1 ∆ ( j ) glyph[lscript] ( j ) ( x ( j ) ) . Let glyph[lscript] be a loss vector such that a ∗ = ( a ∗ (1) , a ∗ (2) , . . . , a ∗ ( m ) ) ∈ arg min a ∈A 〈 glyph[lscript],a 〉 is unique. We then have ∆ glyph[lscript], min = min 1 ≤ j ≤ m ∆ ( j ) glyph[lscript] ( j ) , min .

If q ∈ X is a product measure of q j ∈ P ( X ( j ) ) , from Lemmas 4 and 5, we have

This means that (10) holds with α = O ( m ) and β = O ( 1 ∆ glyph[lscript] ∗ , min ) .

The case of A = { 0 , 1 } d can be interpreted as a problem instance in which A is a direct product of d copies of A 2 . Hence, we can see that (10) holds with α = O ( d ) and β = O ( 1 ∆ glyph[lscript] ∗ , min ) .

B.10 Proof of Lemma 6

Proof. From the definition of the regret, we have

From the definition of bias given in (14), the part of 〈 glyph[lscript] t , w t -a ∗ 〉 can be bounded as follows:

From this and (63), we have

By a standard analysis of the FTRL framework (see, e.g., [Lattimore and Szepesvári, 2020a, Exercise 28.12]), we have

By combining (65) and (66), we obtain the bound of (14).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of selecting an algorithm suited to the environment type, which is often unknown in real-world applications. This paper introduces the concept of best-of-both-worlds (BOBW) linear bandit algorithms that perform well in both stochastic and adversarial environments. Previous BOBW algorithms for linear bandit problems have achieved suboptimal regret bounds in stochastic environments due to an additional logarithmic factor. The authors use an existing approach called exploration by optimization (EXO), and prove that it achieves nearly optimal regret bounds in both regimes. EXO utilizes the exponential weight method to update the reference distribution and computes the sampling distribution and loss estimator to minimize an upper bound on regret. More precisely, the algorithm constructed using this approach achieves $O(d\\sqrt{T \\log T})$-regret in adversarial environments and $O(d^2 \\log T/\\Delta_{\\min})$-regret in stochastic environments. The paper establishes a connection between the EXO approach and the SCRiBLe algorithm, which uses a follow-the-regularized-leader (FTRL) method with self-concordant barrier regularization. The authors also propose a variant called mean-oriented EXO that encompasses the EXO approach and allows for interpretation of existing SCRiBLe-type algorithms as EXO-based methods.\n\nSTRENGTHS:\nThe paper addresses the challenge of constructing best-of-both-worlds linear bandit algorithms that perform well in both stochastic and adversarial environments. The distinction between these environments and the need for an optimal algorithm for both is considered as a quite significant open problem. The paper demonstrates the effectiveness of EXO in achieving nearly optimal regret bounds in both stochastic and adversarial environments by establishing regret bounds. \n\nThe paper also establishes a connection between the EXO approach and the SCRiBLe algorithm and its extensions. This connection helps in interpreting existing algorithms within the framework of exploration by optimization. \n\nThe paper suggests that the framework of exploration by optimization and best-of-both-worlds regret guarantees can be extended to other sequential decision-making problems beyond linear bandits. It mentions partial monitoring problems and episodic MDPs as potential areas for future research and highlights the broader implications of the EXO framework in various decision-making contexts.\n\nWEAKNESSES:\nWhile the paper presents an approach to constructing best-of-both-worlds linear bandit algorithms that perform well in both stochastic and adversarial environments, there are a few weak points that can be identified:\n\nThe paper assumes that the loss vectors are generated according to a specific model and that the environment falls under either a stochastic or adversarial setting. In real-world applications, the actual environment may not be strictly one or the other, and it might be challenging to accurately model the underlying process.\n\nThe paper primarily focuses on theoretical analysis and regret bounds, but it lacks an empirical evaluation or experimental results to validate the proposed algorithms. Without empirical evidence, it is challenging to assess the practical effectiveness and efficiency of the algorithms in real-world scenarios.\n\nImproving a problem or solution by a log(T) term may not always be considered a significant breakthrough, depending on the context and the magnitude of the improvement.\n\nThe paper mentions that the proposed algorithms involve optimization problems (minimization problem (9)) and self-bounding techniques. While these techniques can lead to improved performance, they may introduce complexity and implementation challenges. The paper does not provide detailed insights into the practical feasibility and implementation aspects of the algorithms.\n\nThe paper does not explicitly discuss the limitations and assumptions of the proposed algorithms. It would be beneficial to address the assumptions made about the underlying environment, potential limitations of the algorithms, and scenarios where the algorithms may not perform optimally.\n\nQUESTIONS:\nAre there any limitations or potential challenges in implementing the EXO-based algorithm in real-world applications?\n\nThe paper mentions the use of the self-bounding technique for analyzing regret bounds in stochastic environments. Can you explain how this technique is applied and what advantages it offers?\n\nHow do you think the algorithm interpolate between the two regimes? For example, How does it perform in corrupted stochastic environments or environments with adversarial corruptions?\n\nCan you elaborate on the potential extensions of the EXO framework to other sequential decision-making problems beyond linear bandits? How can this framework be applied to partial monitoring problems or episodic MDPs?\n\nAre there any plans for future empirical studies?\n\nLIMITATIONS:\nThe paper focuses solely on theoretical analysis.\n\nThe paper mentions that EXO can be complex in terms of structure and implementation. The details of how these algorithms can be efficiently implemented in practice are not provided, leaving open questions about their feasibility and scalability.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper considers the best-of-both-worlds problem for linear bandits. They investigate an Exploration-by-Optimization approach for this problem and show O(d^2\logT) regret for stochastic setting and O(d\sqrt{T}) regret for adversarial setting.

STRENGTHS:
This paper investigates a new approach for Best-of-both-worlds problem in linear bandits. 

In some special cases such as multi-armed bandits and hypercube, the authors provide improved results utilizing the problem structure. 

Previous BoBW results mainly depend on FTRL-type algorithms, this work uses a new type of approach, which may be hoped to solve the BoBW problems with other structure.

WEAKNESSES:
Given existing works on BoBW problem for linear bandits, the convergence results provided by this paper are not optimal. 

It is not clear what the advantage of EXO algorithm is compared with standard FTRL-type algorithms. Can you give more discussions?

QUESTIONS:
Please see the weakness part.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
The paper establishes $\log T$-style instance-dependent regret upper-bounds for linear bandit algorithms built upon the FTRL framework and appropriate exploration-by-optimization style update steps.

STRENGTHS:
The $\log T$-style instance-dependent regret upper-bounds can be achieved on general convex action sets and a number of typical important specific action sets. The paper establishes the claimed results by first identifying a sufficient condition for the so-called self-bounding property for FTRL with exploration-by-optimization style update steps and thus $O(\log T)$ regret bounds (Theorem 1), then showing this condition is indeed satisfied under the continuous exponential weight methods.  Overall, the paper is clearly written.

WEAKNESSES:
Similar to most existing works on linear bandits that mainly discusses sample complexity instead the actual computational cost, the paper by default assumes one can effectively and accurately perform the computation of EXO update steps. However, for most convex action sets excepts for some special ones, these steps may be computationally intractable on an ordinary RAM machine. I would happy to see some discussion about the actual computation model (e.g., with oracle access to separation hyperplanes) and how to implement the proposed EXO algorithms.

QUESTIONS:
See the weaknesses part.

LIMITATIONS:
See the weaknesses part.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
This paper studies the "best-of-both-world" problem for linear bandits, meaning that designing a single algorithm that simultaneously achieves a $O(\sqrt{T\log T})$ regret in the adversarial case and $O(\log T)$ regret in the stochastic case. They propose to use the Exploration-by-Optimization approach to obtain such bounds. The algorithm can be viewed as a continuous exponential weight algorithm with an optimally designed action sampling scheme.

STRENGTHS:
- The approach provides a unified and insightful way to obtain near-optimal best-of-both-worlds bounds for linear bandits and several of its special cases. The algorithm is conceptually simpler than previous work such as [Lee et al., 2021] and [Dann et al., 2023]. 
- The algorithm is based on a general framework of Exploration-by-Optimization, which has been proven to work for general adversarial decision making problems. Therefore, the proposed method has potential to be extended to more general cases, though the current analysis is only for linear bandits. 
- There are several novel elements in the analysis, especially Lemma 3. It shows that continuous exponential weights with a simple sampling scheme already works well.

WEAKNESSES:
- The analysis relies on the unique optimal action assumption, and the bound in the stochastic environment can be arbitrarily worse than the  instance-dependent bound by Lattimore and Szepesvari (2017). However, these are general open questions in the field but not the specific issues of this work.

QUESTIONS:
See the above sections.

LIMITATIONS:
There is no potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10