An Exploration-by-Optimization Approach to Best of Both Worlds in Linear Bandits
Authors: Shinji Ito
Kei Takemura
Venue: NeurIPS 2023 poster (2023)
Keywords: bandit
linear bandit
best of both worlds
exploration by optimization

Abstract: In this paper, we consider how to construct best-of-both-worlds linear bandit algorithms that achieve nearly optimal performance for both stochastic and adversarial environments.  For this purpose, we show that a natural approach referred to as exploration by optimization [Lattimore and Szepesvári, 2020] works well. Specifically, an algorithm constructed using this approach achieves $O(d \sqrt{ T \log{T}})$-regret in adversarial environments and $O(\frac{d^2 \log T}{\Delta_{\min}} )$-regret in stochastic environments.  Symbols $d$, $T$ and $\Delta_{\min}$ here represent the dimensionality of the action set, the time horizon, and the minimum sub-optimality gap, respectively.  We also show that this algorithm has even better theoretical guarantees for important special cases including the multi-armed bandit problem and multitask bandits.

An Explorationhy--Opimmiatooo Approach to Best of Both Worlds in Linear Bandits
Shinji Ito NEC Corporation, RIKEN AIP i 3-shinji@nec.com
Kei Takemura NEC Corporation rei_takkeura@nec.co com
Abstract
In this paper, we consider how construct best-of-both-wont linear bandit algo- rithms that achieve nearly optimal performance for both stochastic and adversarial environments. For this purpose, we show that natural approach referred to as exploration by optimization [Lattimore and Szepesvári (2200b works well. Specif- ically, an algorithm constructed using this approach achieves ood of log (
in adversarial environments and O Symbols d, T and Amin here represent the dimensionality of the action set, the time horizon, and the minimum sub-optimality gap, respectively. We also show that this algorithm has even better theoretical guarantees for important special cases including the multi-armed bandit problem and multitask bandits.
) Amin )-regret in stochastic environments.
Introduction
The linear bandit problem [Abernethy et al., 2008 is a fundamental sequential decision-making problem in which a player chooses d-dimensional vector a, from a given action set A C IR and incurs loss ft. The expectation of fa is assumed to be expressed as an inner product of & and or with an unknown loss vector l IRd in each round / 1,2, ...,.... T. This paper focuses on the (expected) regret defined as Rr = maxa E.A E (Le, a a²)). It is known that the order of achievable regret can vary greatly depending on the structure of the environment that generates losses. For example, in stochastic environments, i.c., if the loss vector E, is fixed at E" for all t, there exists an algorithm that achieves regret of Olog 7'). By way of contrast, for adversarial environments in which the loss vector changes arbitrarily, the optimal regret bound is ((VT) [Abernethy et al. 2008 Bubeck et al. 2012). While this clearly suggests the importance of choosing an algorithm that is well-suited to the environment, in real-world applications it is often quite difficult, if not virtually impossible, to know the type of environment in advance.
One promising approach to this difficulty is to apply best-of-both-worldds (BOBW) algorithms [Bubeck and Slivkins 2012] that work nearly optimally both for stochastic and adversarial environments. The first BOBW algorithm for linear bandit problems was proposed by Lee et al. [202T]: it achieves O(vdI df (T)) (Tg(T))A)) -reerre in adversarial environments and O(c" log(TT)AA log T') T')rrgrret in stochastic environments, where c* = c(AA, e*) represents an instancedeepnndent quantity characteriz- ing the instance-optimal asymptotic bound for stochastic environments [Lattimore and Szepesvari 2017]. This algorithm by [ee et al. [2021] also has the notable advantage that these regret guarantees hold with high probability. On the other hand, their regret bound for stochastic environments includes an extra log 7" factor[] This issue of extra log T' factors has been resolved in a recent study by Dann al. 2023b). They have proposed algorithms achieving regret bounds with optimal dependency on Their regret bounds for stochastic environments depend
T. which are summarized here in Table [
As shown in Appendix D by Lee et al. 2021]. this extra log T factor is inevitable as long as we pursue high- probability BOBW regret bounds.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Table 1: Regret bounds for linear bandits
Reference
Adversarial
Stochastic
Lattimore and Szepesvari 2017
c" log T + o(log T)
Bubeck et al.
2012]
0
dT (o))
Lee et al.
2021
0
dT log(T) log(T(AA)
0 (c* log(TT|A)) log
Dann et al.
2023b Cor.7]
O
v d log
de 2
Amin
Dann al.
2023b
Cor. 12]
0
v dT log((A])
0 dlogi[A] log
Aarin
[This work,
Theorem
0
/odd VadTlog?] log
0 (3dlog T)
Table 2: Bounds on parameters a and B
Setting
Action set
B
Corollary
General
A C Rd
0 (d)
0 (d/ Amin)
Corollary
Multi-armed
A Aa == (el
(1)
0 (1/ Amin
Corollary
Hypercube
A = {0, 1]d
(d)
0 (1/ Amin
Corollary
Multitask
A Ad, X x... Adm
0 (m)
0 (1/Amin
on the minimum sub sub-optimality gap Amin. Note that we have e² < O(d/Anm. min [Lee et al. 2021 1
Lemma 16] and that this inequality can be arbitrarily loose [Lattimore and Szepesvari [017]: The study by Dann et al.] (2033b) involves the proposal of a general reduction technique, which has the remarkable advantage of being applicable variety of sequential docision-making problems in
addition to linear bandit problems, though it does tend to complicate the structure and implementation of the algorithms.
The main contribution of this paper is to show that BOBW algorithms can be constructed via (conceptually) simple approach referred as exploration by optimization (EXO) [Latimore and Szepesvári 20206] Lattimore and Gyorgy 202T Foster et al. 2022 In this approach, we update the reference distribution q using the exponential weight method and then compute sampling distribution Ps for action 0₂ (by modifying ş moderately) and loss estimator g: by solving an optimization problem so that an upper bound on regret minimized. As shown by Lattimore and Gyorgy [202]) and Foster et al. [2022], this natural approach has connection to both the information ratio [Russo and Van Roy] 2014] and the decision-estimation coefficient [Foster et al.) 202T], and achieves nearly optimal worst-case regret bounds for some online decision-making problems. These existing
studies on EXO, however, focus on adversarial environments, and not much has yet been shown about the potential of its working for stochastic environments. In this paper, we show that the EXO approach also works for stochastic environments to achieve O(log regrrtt.
The regret bounds with the EXO approach and in existing studies are summarized in Table T This paper shows the regret bounds of stochastic environments, where a is parameter that depends on A and B is a parameter depending on A and ! Bounds on and 8 shown in this paper are summarized in Table For general action sets, the EXO approach reproduces the regret bound shown by Dann et al.] 2023b Corollary 7]. Further, for some special cases, such as multi-armed bandits and hypercube linear bandits, it achieves improved regret bounds that are tight in stochastic regimes parametrized by the sub-optimality gap Amin-
log T) in adversarial environments and of (gd T) in
To show BOBW regret bounds, we consider here the EXO approach combined with the continuons exponential weights method [Hoeven et al. 2018 in which we update reference distributions qe over a convex set. In our regret analysis of the exponential weight, the ratio of the variance of the sampling distribution Pe to the variance qe plays an important role; the larger the former, the better the guarantee obtained. To show O(log rrerrtt bounds for stochastic environments, we exploit the fact that q: is a log-concave distribution [Lovász and Vempala 2007]. Intuitively speaking, as any log-concave distribution has most of its weight concentrated around its mean (see, e.g., [Lovász] land Vempala. 2007 Lemma 5. 17],, we may choose sampling distribution Pt so that has a larger variance than gt which leads to improved regret bounds.

---

Our regret analysis uses the self- beffding technique (Wei and Luo 2018 Zimmert and Seldin 2021 to show regret bounds for stochastic environments. One advantage of this technique is that regret bound also leads to stochastic environments with adversarial corruptions or corrupted stochastic environments [Lykouris et al.) 2018 Li et al. 2019]. As with existing algorithms based on the self-bounding technique, our a algorithm achieves a regret bound of O(adT T + VCadT log T) for corrupted stochastic environments with the corruption level C 2 0. More generally, these regret bounds are achieved for environments in adversarial regimes with self-bounding constraints (Zimmert
and Seldin 2021] details of which are provided here in Section g
Another contribution of this paper is to establish a connection between the EXO approach and the SCRiBLe algorithm [Abernethy et al. 20088 [012] (and its extensions [Dann et al. 2023b Ito and Takemura 2.23). The SCRiBLe algorithm uses
(FTRL) method with self-concordant barrier regularization to select a point in the action set, rather than a probability
distribution. Abernethy et al. 20088) have shown that this approach achieves O( D(VdTT)-regrr bounds
for adversarial environments. Recently, Dann et al. (2023b) and Iio and Takemura 2023 have
shown that it can also achieve
Amin -regret in stochastic environments, through modification of the sampling distributions and loss estimators. To better understand the connection between these studies and the EXO approach, we propose variant that we refer to as mean-oriented EXO and which includes EXO with the continuous exponential weight method as a special case. Given this correspondence, existing SCRiBL algorithms can be interpreted as EXO-based methods in which the optimization is roughly solved with approximation ratios 2 (dd).
los" )
The results of this paper establish a connection between the framework of exploration by optimization and best-of-bothwwokde wooldw regret guarantees. We would like emphasize that there is potential to extend this result to a broader class of sequential decision-making beyond linear bandits, such as partial monitoring problems and episodic MDPs. In fact, the EXO framework can be applied to a general model of sequential decision- Decision-making with structured observation [Foster et al. [2022 and is known to have a connection to several key concepts in sequential decision-making, such as the Decision-EEsimationn Coefficients [Foster et al. [202]] and information-direeted sampling (Lattimore] and Gyorgy 202T] Russo and Van Roy| 2014)) which has been extensively applied to partial mon- itoring [Lattimore and vzepesvári, 2020b] and linear partial monitoring [Kirschner et al.) 2020). Building further relationships between these generic frameworks and BOBW regret analysis will be
an important topic to work on in the future.
Related work
For adversarial linear bandit problems, various approaches have been considered, including algo-
rithms based on self-concordant barriers (SCRiBLe) [Abernethy et al., 2008 2012 Rakhlin and
|ridharan| [2013]. discrete exponential weights (Geometric Hedge) Dann et al. 2008 Bubeck et al. 2012 Cesa-Bahhii and Lugosi| 20122]. J. and continuous exponential weights (continuous Geometric Hedge) (Hazan and Karnin 2016 Bubeck and Eldan 2019), all of which can be interpreted as methods in the FTRL framework. We can see that continuous exponential weight methods can be interpreted an FTRL approach with entropic barrier, d-self-concordant barrier [Chewi] 2021] which implies a close relationship between the SCRiBLe algorithm and the continuous Geometric Hedge, Bubeck and Eldan [019] have noted. One key difference lies in how the sampling distribution P: is chosen; the SCRiBLe algorithm employs a sampling scheme supported on the Dikin ellipsoid while the continuous Geometric Hedge designs pæ by mixing the reference distribution qe with some exploration basis (e.g.. a G-optimal design distribution). As a result of this difference, the latter can achieve an (d 1/2) -times better regret bound than the former. Recent studies by Dann ct al. 2023bb Theorem 3] and Ito and Takemura 22023) have proposed new sampling schemes based on Dikin ellipsoid sampling to achieve BOBW V regret bounds. This paper takes these approaches a step further, determining sampling distribution itself on the basis of an optimization problem for minimizing regret, which leads to improved performance.
The self self-bounding technique (Wei and Luo| 2018 Zimmert and Seldin 2021 is known as promising approach for designing and analyzing BOBW algorithms. The greatest impetus for the widespread use of this technique is considered to come from the Tsallis-INF algorithm (Zimmert et al. 2019 for the multi-armed bandit problem. Since the study on INN FTRL, methods with Tsallis- entropy regularizer (with some modification) have been used successfully in variety of sequential
3

---

docision-making, problems with limited (structured) observation, including combinatorial semi- bandits (Zimmert et al. 2019). bandits with switching costs [Rouyer et al., 2021 Amir et al., 2022].
bandits with delayed feedback [Masoudian et al.. 2022], online learning with feedback graphs [Frez
and Koren 2021 | Rouyer et al. 2022]. dueling bandits [Saha and Gaillard 2022]. and episodic MDPs
2021 Dann et al. 2023a]. On the other hand, a few (rather limited number of) studies have constructed BOBW algorithms using regularizer functions other than the Tsallis entropy. For example, [to [2021] proposed BOBW multi-armed bandit algorithm by using log-barrier regularization equipped with adaptive learning rates. More recently, Shannon-entropy regularization with some specific adjustment methods for learning rates has been shown to provide BOBW regret guarantees for several problems including online learning with feedback graphs [Ito
(Jin and Luo| 2020 Jin et al.,
et al. 2022b), partial monitoring (Tsuchiya et al. 2023], , and linear bandits [Kong et al., 2023 It
seems that these saaamon-entropy-basee methods have the strength of being versatile while these have the weakness that their regret bounds for stochastic environments include extra Oloo 7 -fcctrrs The approach in this paper may appear to be related to the previous studies using the Shannon entropy (Ito et al. 2022b Tsuchiya al. 2023 Kong et al. 2023] as it employs exponential weight methods, but it is more related to the studies using log-barrier regularization [Ito] [021]. In fact, both log barriers and continuous exponential weight can be captured in the category of self-concordant barriers we will discuss in Sections 5 and 6 These also have in common the type of regret bounds: Ollog T') in
stochastic environments and O[VT T) in adversarial environments.
3 Problem setting
In linear bandit problems, the player is given an action set A C Rd, which is a closed bounded set of d-dimensional vectors. Without loss of generality, we assume that A spans all vectors in IR In each round f, the environment chooses loss vector & C I while the player selects an action a E A without knowing & The player then gets feedback of ft E [-1, that is drawn from a distribution Ma., where the series of distributions M (Ma)ae. is in Me. == {(Ma))e. Va
A, [f-M,[f] (4,, a)}, }. The condition that the support of M,, is included in [-1, 1] for all a E A
implies that (fe, a) [-1,1] holds for all
A and [7]. In other words, it is assumed that
la L for any t, where we define L C Id by L {e gd Va A, ((f s 1}. The
performance of the player is evaluated by means of regret defined as
T
- Edhror)
T
RT(a") E
(le az)
sup lr(a"). a'EA
(1)
Regimes of environments In stochastic environments, l is assumed to be round-invariang i.c., there exists t E L such that E = 8 holds for all t. In adversarial environments, & can be chosen in an adversarial manner depending on the history so far a(L,,,))):= )- Stochastic environments with adversarial corruptions correspond to intermediate setting between these two types of environments. These environments are parametrized by the corruption level C' 2 0, in which the loss
vectors (4) are assumed to satisfy E EL= max (c+ L",a) I C. To capture these settings of environments, we define adversarial regime with (e", a", C, (eT---sll-bbundiei constraint Zimmert and Seldin 2021] in which the environments choose losses so that
Σ (e" 0²) 0r) -c
Ry(aa) 2 E
holds for any algorithms. This regime includes (comrupted) stochastic environments. In fact, any corrupted stochastic environments with the true loss 8 E and the corruption level C > are included in an adversarial regime with a (E*, a", C, C,TT)-sllf-buunding constraint. Our regret analysis for (corrupted) stochastic environments relies only on the condition of (2) as in some existing studies (Zimmert and Seldin 2021 Dann et al. 2023b).
Reduction to convex action set Let X conv(A) represent the convex hull of A. We may consider the action set X' instead of A, i.e., if we can construct an algorithm Alg, for the problem with action set X, we can convert it to algorithm Alg, for the problem with A. In fact, for any

---

output of E X of Alg we can pick a, E so that Ela.la]] = a When defining ae as the output of Al we have El (i, a.) |a4] (l,, af), which means that the expectation of the feedback ft given oj is indeed linear function in of and that the regret for Alg, coincides with that for Alg,x
4 Exploration by optimization
4.1 Algorithm framework and regret analysis
Let P(X) denote the set of all distributions over X. For any distribution p E P(R.). denote u(p) = Ex-pp [x]] TRd, (()) Ex=ppf>: E R.ddd and V(p) (p)) upp)p(p)²
R We consider algorithms that choose a, following a sampling distribution Pt E P(2) in each round (approximately) unbiased estimator 9r ge 0z, fi) of the loss function (4,, Let G be the set of all estimators that are linear in the first variable, i.e. 9 = {g-
To determine Pt, we compute
X' x X x [-1, 1] + R 3h : X' x [-1,1] + R g(æ; a. f) = (h(a, f), x)=) For any distribution p E P(X), let GD (p) S represent the set of all unbiased estimators:
[o(2; a, [ (2. ((.0))
(D) E g ve E V. X. E M
E
(3)
typical choice of given is the estimator defined as follows:
g(2; J) f) = (),a,)) where h(a, f (H(p))
(4)
where (p)) is assumed to be non-singular. This is an unbiased estimator, i.e., 9 defined by an element of gu. In fact, if M E &Me, we have Ea-p.f-, (1 []] (H))) aa
44) is
(H(D))-'(())e
Using {g.)-.. we set a reference distribution over X' by the continuous exponential weights method as follows: (x(z) ex)()>))(>>>)>>>>)))>))) Σ 9o (z; a. fa)), where go is arbitrary initial distribution over X' and 77 is a learning rate such that 171 2 T22 2 ... 2 Tf > O. We may choose 7/ depending on the historical actions and observations so far ((a,,
(z(z))
Let (x 4) represent the Heillback--eibble divergence of o' from q. Note that the exponential weights method can be interpreted as a bolloo-the-regulaiizee--ee
approach over P with the regularizer function q DKL. (q, on). i.e., the distribution de is solution to the following optimization problem:
Q: E arg min (EP(X))
Σ Ns (2; Ss)
+
1 1/t
DKI (o)
(5)
We may determine the sampling distribution pt on the basis of qe in an arbitrary way. The regret is then bounded as follows:
Lemma 1. For any distribution p" E EP(X) such that DKL(P", (0) < co, the expected value of the regret Rr(a") for a" p² is bounded by
T
stab(mgg(;aa DKL (p",
E
(Ce. - -((e)) + bias(gt, Pt, qt Le) + n
where (las(g.P.9..))
sup a M.M.
aa Ke,x a²) 100gggaa:aaa g(x;;
stab(g.g)
E 2~ d'EP(X)
sup
[g(x)]]
[o(x)] - Dkuld'o)) and
E
This follows from the standard analysis for exponential update methods or FTRL (sce, e.g., Lattimore
and Szepesvári 2020a Exercise 28. 12)). and all omitted proofs can be found in the appendix. For the case of the exponential weights, the stability term stab(g. q) can be bounded as
stab(g,g) q) < inf E [o(g(r) Ø)], where (0y)
+
gER 2r0
Such a distribution of given of can be computed efficiently e.g., by the algorithm given in (Schrijver 1998 Corollary 14.1g] or (Mirronni et al. 2017
5

---

The bias term bias(g, P. q, E) corresponds to the bias of the estimater g(:: a, f) for the fucntion (L. under the condition that a p. and Ejfla] (a, We can easily see that g ("(p) implies bias(g.P. = 0 for all q E ((2))) and E L.
For any reference distribution q E (()) and learning rate vj O, we define (P, by
Ag.(P.g)
sup BEC.MEM2
((mul)) vlg)) + bias(g, p. amp.frMaa
E
[stab(no(sa, A.al)
From Lemmal [ the regret is bounded as
E """pp"
[
S E
ggmm (pt: 96)
1
DKL (p" .00)
In the explortioon-hy-opiiiiaaaii (EXO) approach [Lattimore and Szepesvári 2020b Lattimore and Gyorgy|2023 2021). we choose St and Pt. so that the value of Aqun (9t.Pr) Pt) is as small as possible, i.e., we consider the Tollowing optimization problem:
Minimize
Ag..m. (p. g) subject to p ((X)),
(9)
We denote the optimal value of this problem by A Agu.
Remark 1. In most linear bandit algorithms based on exponential weights [Dani al.
2008 Bubeck 2019] Ps is
et al. 2012 Cesa-Bianchi and I. Lugosi| 2012 Hazan and Karnin 2 Besbes et al.
defined as q mixed with small component exploration basis To and gi given by
with p = Pt-
The exploration basis To here is distribution over X such that G(TT)) := su (TT is bounded. It is known that any action set X' C Id admits exploration basis ттp such that G(To) S d. which is called the G-optimal design or the John's ellipsoid exploration basis. Given such a mo. we set Pt (1 (1-7))9 + To with some "/2 (0, 1). Under the condition of m7. (2(1d) and % (dn)). we have Ag..m. (Pc. gt) = O(% + d)dd) = (()d), which implics that A
9.7h = ((nd) holds for any q E P(X). Hence, from (8, if DKL (p", qu) B, by setting 1/ 17 ( ((VBB(ddT) obtain Ea--p- [Rr (a*)] O(v BBT))
4.2 Sufficient condition for best-of-both-worlds regret guarantee
In this section, show that certain type of upper bounds on Ag that depend reference distribution will lead to ofst-of-both-woot both-worlds regret bounds. In the following, we set the initial disturibution 9o to be uniform distribution over X. Denote Ag- (x) (e* x a²) for any t E L. The following lemma provides a sufficient condition for
regret guarantees:
Theorem 1. Suppose that there exist > 0.3 3 O and 1jo > 0 such that
Na. S r min(a,8 3 A (r((())))
(10)
for all t and 7 S No- Then, algorithm based the EXO approach achieves the following regret
bounds simultaneously: (i) In adversarial environments, we have Rr = O
( (odd log T + dlog
11
(ii) In environments satisfying 2 we have Ryr = =0 (($+ 11) dlog T + + VCd log
In the proof of this theorem, consider the algorithm that chooses (pr. 9t) to be optimal solution to 9 and that sets learning rates v) by 7): min
doog 07
A complete proof is
Reine
given in the appendix.
Remark 2. In the implementation of the algorithm, we do not necessarily need to solve the opti- mization problem ( exactly. In fact, achieve the regret upper bounds in Theorem [ is sufficient to compute Pt. 9t. and 2,
that satisfy n, - "A... (Prs gt) I 2, min(a, 3 ()(g))))) for some a > 0 and B > 0, under the assumption of m S yo- In this case, setting
min(yo, // E 1 2s} will work.

---

4.3 Regret bounds for concrete examples
In this section, we will see how we can bound Ag... For q E P(X)) define w(g). by
min (q) pcP(
{H( V(g)},
(11)
w (4) pEP(X))(())=(())
min min > 0 d V(x) y H())).
(12)
Note that w(q)= S w'))) holds. In fact, from the definition of w/ (g),, there exists p (())) satisfying
((() - (4)) and d V4)) < w'()) ()) Hp). For such distribution p. we have Hpp) V(4)
tr( (r(H())-1/AV(()) H()) 1/2) S (/d=t -T) = w')), which implies that w(g) S w')
Using these, we can construct bound Nn, the following lemma provides:
Lemma 2. Suppose that q is log-concave distribution. In 1/d, we then have Agnn (g))
1f S 1, we then have Ag, = ((p (q)).
Bounds on Ay in this lemma can be achieved by p= is the minimizer on the right-hand side of (II) or (12). To is a G-optimal design for X, and some mixing weight parameter y (0, 1). The proof of this lemma given in the appendix.
--+ ===pp=
(1 7m and g defined by 44). where p
General action set We can provide an upper bound on w(g) by exploiting the property of log- concave distributions. A distribution q P(X) is called a log-concave distribution if has distribution function that is proportional to exp(-h(( for some convex function h. Note that the reference distribution qe is a log-concave function.
Lemma 3. Suppose that q a log-concave distribution. Let l" E C be an arbitrary loss vector such that a E arg a (E", a) is unique. Denote Dr min = minaeA)(a') De- We then have (4) = 0 (d min {1, Ae- (H(())/ (((a))/Arr,,iuu))
We here provide a proof sketch for this lemma. We consider a distribution p E P(A) of A generated by the following procedure: (i) Pick x E X following (ii) Let & be an arbitrary distribution over A such that uu{) = x. In other words, compute Ç A Rzo such that EVEeA((0)) = 1 and En'e (erexE(laee E(a')a' = (iii) Pick
following &. It is then clear that μ(p) ((g) holds. Further, follows from the assumption that q log-concave distribution that (1 T((')))TV((p))
V(p)u
N(u V (g)u) holds for any a* A and any u Rd (0), which implies d V(g) 3 O(1
p(a") pda"))d) - V (p) 3 (((1-1(11))))) p(aa))a))))))))))) (d) (p). Finally, since we have (1 - -p(aa))=))))))))))))>)>)))))-- < A.- (H(())) miin
the bound on w' w'(g) ()) in Lemma follows. A full proof can be found the appendix.
From Lemmas and 3 we obtain the following bounds on a and 3:
Corollary 1. For arbitrary action set A, the condition (T0p in Theorem 7 holds with = Od), = O(d/ Ae- min), and n/u- O(1)
Hence, there exists algorithm that achieves RT Odd/Tlog in adversarial environments and +
Rr = 0 A min
log T'
Cdda log 1") in stochastically constrained adversarial environments.
Aze min
Multi-armed bandits Suppose a problem setting in which the action set is the canonical bases:
A = Aa == fei, eal fa E 1) ||a]]] =}. For this action set, have the following bound on w(q):
Lemma 4. If A = As for any log-concave distribution q, w(q) defined in 1I is bounded as
w(g) < 0 Ed E (=(()(1-v((())) mi())) SO -
Let Et E L be an loss vector such that ex- arg mig is unique. We then have Ar-(w) = Eipre De- (ei)us > min (4-),, which, combined with Lemma implies
w(g) S (e-(u()))/A min- From this and Lemma 2 we have the following bounds and 3:
Corollary 2. For the multi-armer bandit problem A = Aa. the condition 10 Theorem holds with = O()), 3 O(1/ Ar- min) and 170 ((Idd).
7

---

Product-space action sets Suppose that A S Rd is expressed as product space of m sets, i.c., A = (1 x A x... x A(m) where A S R2 for each j and وز=ز d d. Denote x00 conv (A This setting is referred to the multi-task bandit problem (Cesa-Bianns Bianchi and Lugosi 2012]. in which the player chooses a. E A for each in parallel, and then gets only single feedback corresponding to the sum of the losses for all m actions. For such an action set, probability distribution 9: given by continuous exponential weights can be expressed as a product
th
measure of log-concave measures qi (5) E P(X()). In fact, if a probability density function is expressed as g(x:) cexp((nL,x))) for some > O7 > O, and L (L 3...,..., L(m) Rd. it can be decomposed as 7(x)) cexp(E, (nI. x01)) I c(t) exp((nI..))) 2(1))
I ((0())))) where x(5) E 2² and c(i) = SnexS) (np(()) ,z6)))d This fact leads to the following bounds on ((q) and An.
Lemma 5. AfA= = A ... A(m) and q E P(X) can be expressed as a product measure of g P(((()) for ,..... m, we have w(q) S Ej= w(gl))).
Corollary 3. If A Ad, x ... x Adm. the condition (10) in Theorem holds with a = (mm), ((1 min), and 11o = 0(1 1f A {0, 13, the condition in Theorem holds with
(d), 3 = O(1/Ae,,min and 1/0 (1/d).
4.4 Computational complexity
At this time, it is not known if there is an efficient algorithm to solve the optimization problems ( However, worth noting that the optimization problem corresponding to the right-hand side of (IT) is convex optimization problem. Further, as the minimum can be achieved by E (AA), we can reduce the problem into a convex optimization problem over [Al-dimensional space, which can be solved efficiently if [A|] is not very large.
Note that it is not always necessary solve the optimization problem ( exactly, as mentioned in
Remark For example, we can implement an algorithm achieving regret bounds of Corollary T
given a separation oracle (or equivalently, linear optimization oracle) for cA. without solving (9)
exactly. Indeed, to achieve the regret bounds of Corollary [ it suffices to find and such that
An.m (p. 9) bounded by the RHS of (T)). The construction of such p and g is provided in the proof of Lemmas 2 and [ which can be performed using a separation oracle for A. In fact, we can obtain samples from p by using the techniques for log-concave sampling (e.g.. [Lovász and Vempala 2007)) and for computing convex combination expression (cf. Carathéodory's 's theorem for convex hull and [Schrijver 1998 Corollary 14. Ig)). However, the analysis of log-concave sampling and calculations of (p) (which is required for constructing g) including consideration of calculation errors can be highly complicated, and the computational cost can be very large, although on the order
of polynomials (e.g., (Hazan and Karnin 2016 Corollary 6.2)).
5 Mean-oriented FTRL and SCRiBLe
For linear bandit problems, we have alternative approach that referred as mean-oriented FTRL, in which we compute a reference mean wy given as follows:
We arg min wEX
Σ ha fa), w)
1 _ (w(o) 17.
(13)
where (a). is an estimator of l, and z is convex regularization function over X' such that mingex (xx) 0. Let Dos x) denote the Bregman divergence associated with v: D (xxx) - ((x) (V)(x), x' - -z). This approach ensures the following regret bound:
8

---

Lemma 6. For any point x² E N. we have
(nash( Sc), se),
xb(x))
Rr(x*) s E
(Be, -we)
bias(he. Pt, We, (e)
7r+1
a''1
where
bias(h, w, (, eEX. Lampit-Ma
sup
E
lif h(a, --a
stab w: sup
{(: sowwwww" w/) Du((',w)) w'EN
(14)
One example of linear bandit algorithms based on mean-oriented FTRL
the SCRiBLe algorithm Abernethy et al. 2008 2012), which employs selffconcordant barriers as regularizer functions.
We can also consider an EXO approach for mean-oriented FTRL. For any reference point w X and learning rate > 0, define
(p,,((phh)
sup BEC.
(l,w(l) w) p,,
E II amp.f-aa
(1h(a
and Aw., inf pEP())))) Awn (p,
This achieves the following BOBW regret bounds, under assumptions Arr.
Lemma 7. Suppose that dp is u-self-concordant barrier. Suppose that there exist 0,B > 0 and T/o > such that
New. S 77- min B Dr- (w;)}
(15)
for all L and 1j S rjo- Then an algorithm based on the EXO approuch achieves the following regret bounds simultaneously: (i) In adversarial environments, have Ri O (VouT R
v vlog
710
1 To v log VC,Bvlog) 2 we have (a(aa*) ((+
(ii) In environments satisfying
5.1 Connection the SCRiBLe algorithm and variants it
Abernethy et al. 2008 2012 have proposed sampling scheme + pt supported the Dikin ellipsoid, which leads to Awi. (pr, /te) O(H,d)) with an appropriately defined unbiased estimator
hr This algorithm referred to as SCRiBLe hence achieves Rt = O d'vī log T"), the adversarial regret bound Lemma 7 with a = (d Recent studies by Dann et al. [2023b] Theorem 3]
and Ito and Takemura [2023] have shown that modified sampling scheme based on the Dikin ellipsoid sampling achieves Am,.n. (Pt, (1)) = O (1,d² min(1, (1, Ar- (A) /0////.mm)) Consequently,
these modified algorithms achieve BOBW regret bounds in Lemma 7 with = O(d)) and B
Odd² Ae. ,min). These results suggest that the following lemma holds:
Lemma 8 ([Dan et al. 2023b Ito and Takemura 2023). If 1/ is self-concordant barrier, 15 holds with = O(d), 3 Odf/Arr min) and 7)o- O(1dd).
5.2 Connection to EXO with continuous exponential weights
The entropic barrier is y-self-concordant barrier with v < d [Chewi 2021], for which definition and properties are given, e.g., in [Bubeck and Eldan 2019 We can see that mean-oriented EXO with entropic-barti barrier regularization coincides with the EXO approach with exponential weights. In fact, we compute using (3) with v the entropic barrier, from the definition of the entropic barrier, equal the mean of the distribution d. given by the continuous exponential weights. Given this
correspondence, we can see that bounds on A A follow immediately from Corollary [
Lemma 9, If d is the entropic barrier for X, 15 holds with a Od)), O(d/ (g/Ar mir. and To = O(1).
worth noting that these bounds are 0(1/)) -iimes better than the results for Lemma that follow from previous studies.

---

Limitations and future work
A limitation of this work is that the regret bounds for (corrupted) stochastic environments require the assumption that the optimal arm a* E arg min (e" a) is unique. While this assumption is common in analyses of best-of-both-worlds algorithms based on the self-bounding technique (Zimmert and Seldin 2021 Dann et al. 2023b]. a limited number of studies on the multi-armed bandit problem [Tro] 2021 Jin al. 2023] have removed this uniqueness assumption by careful analyses of regret bounds for follow- the teaee--guuaiidd-eeiteee leader methods. As these analyses rely on structures specific to the multi-ame arde bandit problems, extending them other problem settings does not seem trivial. Extending this analysis technique to linear bandits and removing the uniqueness assumptions will an important future challenge.
Another future direction is to work toward a tighter regret bound of O(c" log T) in stochastic environments while satisfying BOBW regret guarantee, where c" is instance cetaaceedeependet quantity that characterizes the optimal asymptotic bound (see, e.g., [Lattimore and Szepesvari 2017 Corollary 2D.. As c* < (dd/Amin) [Lee al. 2022 Lemma 16.] and the gap between these two quantities can be arbitrarily large [Lattimore and Szepesvari, 2017 Example 4], our bounds depending on 1 1/Amin can be far from tight. In the multi-armed bandit problem, special case of linear bandits, the quantity c* can be expressed as c* EuEA: (a)>0 [a-(a] and known FTRL-type BOBW algorithms with (c log -reggre bounds employs the Tsallis entropy with ((1/V)) -leaarnng rates: (w) = -vii Vws [Zimmert and Seldin 2021] logarithmic barriers with entry-wise
adaptive learning rates: (v(ww) = - log w. [Ito 2021 Ito et al. 2022a] The latter would be
more closely related to our results in this paper as logarithmic barrers are examples of self-concordant barriers. A major difference between the approach in this paper and the one by [to [221] is that the former only considers regularizer functions expressed as a scalar multiple of a fixed function,
while the latter even changes the shape of regularizer functions for each round. Such a more flexible adaptive regularization may be necessary for pursuing tighter regret bounds. Another promising approach is extend the Tsallis-entropy approach to linear bandits while the connection between the TNa (Zimmert and Seldin [021 and this paper may be somewhat tenuous as the Tsallis entropy is not self-concordant barrier.
References
J. Abernethy, E. Hazan, and A. Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. In Conference on Learning Theory, 2008
J. D. Abernethy, E. Hazan, and A. Rakhlin. Interior-point methods for full-information and bandit online learning. IEEE Transactions on Information Theory, 58(7))4164 4175, 2012.
I. Amir, Azov, T. Koren, and R. Livni. Better best of both worlds bounds for bandits with switching costs. Advances in Neural Information Processing Systems, 35, 2022.
O. Besbes, Y. Gur, and A. Zeevi. Optimal exploration-explootttion in a multi-armed bandit problem with non-stationary rewards. Stochastic Systems, 9(4):319-337, 2019.
S. Bubeck and R. Eldan. The entropic barrier: Exponential families, log-concave geometry, and self-concordance. Mathematics of Operations Research, 44(1):264-276, 2019.
S. Bubeck and A. Slivkins. The best of both worlds: Stochastic and adversarial bandits. In Conference
on Learning Theory, pages 42-1. 1. PMLR, 2012.
S. Bubeck, N. Cesa-Bianchi, and S. M. Kakade. Towards minimax policies for online linear optimization with bandit feedback. In Conference on Learning Theory, pages 41-1. JMLR Workshop and Conference Proceedings, 2012.
N. Cesa-Biannhhe and Lugosi. Combinatorial bandits. Journal of Computer and System Sciences, 78(5):1404-1422, 2012.
S. Chewi. The entropic barrier is n-self-conccrdant. arXiv preprint arXiv:2112.119%4 2021. V. Dani, S. M. Kakade, and T. P. Hayes. The price of bandit information for online optimization. In Advances in Neural Information Processing Systems, pages 345-352, 2008.
10

---

C. Dann, C.-Y. Wei, and J. Zimmert. Best of both worlds policy optimization. In International Conference on Machine Learning, pages 6968-7008. PMLR, 2023a.
C. Dann, C.-Y. Wei, and Zimmert. A blackbox approach to best of both worlds in bandits and beyond. In Conference on Learning Theory, pages 5503-5570 PMLR, 2023b.
L. Erez and T. Koren. Towards best-of-allwworlds online learning with feedback graphs Advances in Neural Information Processing Systems, 34, 2021.
D. J. Foster, M. Kakade, Qian, and A. Rakhlin. The statistical complexity of interactive decision making. arXiv preprint 1xXiv:2112.13444 2021.
D. J. Foster, A. Rakhlin, A. Sekhari, and K. Sridharan. On the complexity of adversarial decision making. Advances in Neural Information Processing Systems, 2022.
E. Hazan and Z. Karnin, Volumetric spanners: an efficient exploration basis for learning. The Journal of Machine Learning Research, 17(1):406244055, 2016.
D. Hoeven, T. Erven, and W. Kotlowski. The many faces of exponential weights in online learning. In Conference On Learning Theory, pages 2067-2092. PMLR. 2018.
Ito. Parameter-free multi-armed bandit algorithms with hybrid data-dependent regret bounds, In Conference on Learning Theory, pages 2552-2583. PMLR, 2021.
S. Ito and K. Takemura. Best-of-three-worlds linear handit algorithm with variance-adapten regret bounds. In Conference Learning Theory, pages (653-2677.. PMLR, 2023.
S. Ito, S. Hirahara, T. Soma, and Y. Yoshida. Tight first-and second-order regret bounds for adversarial linear bandits. In Advances in Neural Information Processing Systems, volume 33, 2020.
S. Ito, T. Tsuchiya, and J. Honda. Adversarially robust multi-armed bandit algorithm with variance- dependent regret bounds. In Conference on Learning Theory, pages 1421-1422. PMLR, 2022a.
S. Ito, T. Tsuchiya, and J. Honda. Nearly optimal best-of-both-worldd algorithms for online learning with feedback graphs. In Advances in Neural Information Processing Systems, volume 35, 2022b.
T. Jin and H. Luo. Simultaneously learning stochastic and adversarial episodic mdps with known transition. Advances in Neural Information Processing Systems, 33:16557-16566 2020.
T. Jin, L. Huang, and H. Luo. The best of both worlds: stochastic and adversarial episodic mdps with unknown transition. Advances in Neural Information Processing Systems, 34, 2021.
T. Jin, J. Liu, and H. Luo. Improved best-of-both -othoooh-woddds guarantees for multi-armed bandits: Ftrl with general regularizers and multiple optimal arms. arXiv preprint 233.v:2302.13534, 2023.
J. Kirschner. T. Lattimore, and A. Krause. Information directed sampling for linear partial monitoring.
In Conference on Learning Theory, pages 22282339.. PMLR, 2020.
Kong, C. Zhao, and Li. Bese-olfthree-worldss analysis for linear bandits with follow-the- regularized-leatn leader algorithm. In Proceedings of Thirty Sixth Conference Learning Theory, pages 657-673. PMLR, 2023.
T. Lattimore and A. Gyorgy. Mirror descent and the information ratio. Conference on Learning Theory, pages 2995-2992. PMLR, 2021.
T. Lattimore and C. Szepesvari. The end of optimism? an asymptotic analysis of aniee-armed linear bandits. In Artificial Intelligence and Statistics, pages 728-737. PMLR, 2017.
Lattimore and C. Szepesvári. Bandil Algorithms. Cambridge University Press, 2020a.
Lattimore and C. Szepesvári. Exploration by optimisation partial monitoring. In Conference on Learning Theory, pages 2488-2515 PMLR, 2020b.
C.WW. Lee, H. Luo, C.-Y. Wei, M. Zhang, and X. Zhang. Achieving near instance-optimality and minimax-oppimaliin in stochastic and adversarial linear bandits simultaneously. In International Conference Machine Learning, pages 6142-6151. PMLR, 2021.

---

Y. Li, E. Y. Lou, and L. Shan. Stochastic linear optimization with adversarial corruption. arXiv preprint UrXiv:1909.021000 2019.
Lovász and Vempala. The geometry of logconcave functions and sampling algorithms. Random Structures & Algorithms,, 30(3):307-358, 2007.
T. Lykouris, V. Mirrokni, and R. Paes Leme. Stochastic bandits robust to adversarial corruptions. In Proceedings of the 50th Anmual ACM SIGACT Symposium on Theory of Computing, pages 114 122, 2018.
Masoudian, Zimmert, and Y. Scldin. A best-of-both-worlds algorithm for bandits with delayed feedback. Advances in Neural Information Processing Systems, 35, 2022.
V. Mirrokni, R. P. Leme, A. Vladu, and S. C.-w. Wong. Tight bounds for approximate carathéodory and beyond. In International Conference on Machine Learning, pages 2440-2448. PMLR, 2017.
A. Rakhlin and K. Sridharan. Online learning with predictable sequences. Conference on Learning Theory, pages 993-1019. PMLR, 2013.
C. Rouyer, Y. Seldin, and N. Cesa-Bianchi An algorithm for stochastic and adversarial bandits with switching costs. In International Conference on Machine Learning, pages 9127-9135. PMLR, 2021.
C. Rouyer, D. van der Hoeven, N. Cesa-Bianchi, and Seldin. A near-optimal best-of-both-woote algorithm for online learning with feedback graphs. Advances in Neural Information Processing Systems, 35, 2022.
D. Russo and Van Roy. Learning to optimize via information-directel sampling. Advances in Neural Information Processing Systems, 27, 2014.
A. Saha and P. Gaillard. Versatile dueling bandits: Best-of-both world analyses for learning from relative preferences. In International Conference on Machine Learning, pages 19011-19026. PMLR, 2022.
A. Schrijver. Theory of Linear and Integer Programming. John Wiley Sons, 1998.
Tsuchiya, S. Ito, and J. Honda. Best-of-bothh-oorddd algorithms for partial monitoring. In International Conference on Algorithmic Learning Theory, pages 1484-1515 1515. PMLR, 2023.
C.-Y. Wei and H. Luo. More adaptive algorithms for adversarial bandits. In Conference On Learning Theory, pages 1263-1291. PMLR, 2018
Zimmert and Y. Seldin. IN:: An optimal algorithm for stochastic and adversarial bandits. Journal of Machine Learning Research, 22(28):1-499 2021.
Zimmert, H. Luo, and C.-Y. Wei. Beating stochastic and adversarial semi-bandits optimally and simultaneously. In International Conference on Machine Learning, pages 7683-7692, PMLR, 2019.
12

---

A Notes on notation and assumption
For any two real vectors = (II, 2dd) E R and (vi,...., ,...,.,,))))))..... Ud) E IR (2, denotes the the inner product of and y, i.c., (x, y) Σ Ziyi. For any real vector I (II,..., xd) Rd let ||ع|| denote the L1 -nor of r, i.e., || Ei= | For any real symmetric matrix X, let (X denote the trace of X. For any two real symmetric matrices and Y, we denote X I Y (or Y X) if and only if Y X is a positive semi-definit matrix. For any two real symmetric matrices X and Y, X Y denotes the Frobenius inner product of X and Y, i.e. X tr(AB).
Without loss of generality, we may assume that A spans all vectors in R.d. Indeed, if not, we make A make full-dimensional by ignoring redundant dimensions. Under this assumption, distributions given by exponential weight methods have non-sing -onngiau aariance-cvaarian matrices V V(g) (and thus also H (q)). This fact used in some places in this paper without notice.
B Omitted proofs
B.1 Lemmas log-concave distributions
Lemma 10 (Theorem 5.2 in [ováász and Vempala 2007 ). If X E Rn is a random variable
following a log-concave distribution, for any constant matrix M E Rmxx, MX follows a log- concave distribution well.
Lemma 11 (Lemmas 5.4 and 5.5 (a) in Lovász and Vempala 2007 If p is one-dimensional log-concave distribution, we have
Pr XX S u()) 2 ! N~a
(16)
Further, (p) = -, we have
Pr (a X S/ S Xxp
(17)
for any a, E R.
Lemma 12. For any one dimensional log-concave distribution over a one-dimensional segment [L, R] C R, we have
V e² min (p(p) L)3, (R=))))
(18)
Proof. From Lemma
11 we have
L
X
S u(p)
m(p) - (p))
<
Pr Pr S p(p]= X-p
=
Pr X-p (V
(19)
VV())
VV())
where the first and the second inequalities follow from (6) and 17 respectively. We hence have
V(p) S <x((p))) T.)². By considering -X instead of X, show V(p) S e2 u(p))² in
similar way.
Lemma 13, Suppose e is defined as in 7 If follows a log-concave distribution over R and if
E Ely < 11/2, we have
exp(3 111) 0 (5²). ex-exp(1 1-1)
Bio(y)] 5²
(20)
Proof. See, e.g., the proof of [Ito et al. 2020 Lemma 5].
13

---

B.2 Proof of Lemma 1
Proof. From the definition of the regret, we have
T
E
[Rr(a*)]
E
E aapp
[Ce,
((2,a-aa))
Sppp
[fa, a + (le, ((e, -((o))
E
E wouge
(21)
From the definition bias given in (6). the part Enary [Ce, it - a*)) can bounded as follows:
E z~qt
[[e, - a"}}
E au.i
E [le, a") ge des ft) + ge(2; ats f(]l xmgg.
E (z; -t)- ar fe)l
xr
< bias(gt, Pr. 2o, Le)
E ar./e
E xm
(1) st) ge(a" at St))
(22)
From this and we have
T
E pp
RT
S
Σ
(1e,, ((a)) m(p)) + bias(ge, Pc. qr, (:))
E
E
Σ
E wruge
[9 at, St) - (a"; at, Sc)l
(23)
ar
By standard analysis of the exponential weight method, we have
T
(9r( ae fi) - ge aa, fe)
stab(ng((1aa. +
DKL (pn. ,9)).
a rup"
mm
(24)
To show (24). have
we denote Gp(p) = Easo (a; at, [e)] in the following for notational simplicity. From
T Σ go ; fi)
DKL(p".
a p'p
m++1
T
+ 1 DKL(p",
Gi(D")+
1MT+1
Ge(gr+1)+ 1. (el(0111.g))
MT+1
1
Gelgr+1)+ + DKL (g1(9r11.g)) + Gr(qr+1) "
1 - DKL go) + Dkulqr+1.91) + Gy (gr+1)
Gelgr) + ar
L 1
G((at))
(25)
where the first inequality follows from 5 and the second inequality follows from 1/1 S Yr and the fact that the values of the KI. divergence are nonnegative. The second equality follows from the definition of (x);; As (I)) can be expressed as qr (x) Kr (xp((x19r(x))) (z))(((z) for some
14

---

NT > O, where we denote 0r(x) = =Ea=l T-
(2; az. ag: fs), we have
In 9r(a)
Gilar)
DKl 90)
E
9T (a)
"r
an-yy
go(a)
1
In KT
9T (T(
(In - -(rr(=))
(26)
ar
1/r
TT
and
Σ Ge(9++1)+ + Dka(gr+1,90)
E (r(a))
+ In (r+1(a)
7/r
nr (a
(r(a) + In (r+1(a) (r(a)
+ In rr (a(
E
nr
9o(a)
Tr In (
E amag++1
πr(a) +
+ (INNT Trār (x)) 1/r
(r(a)
In RT
(++1(a) (a)
In
In
+ DKu(fr+1,
(27)
7/
These equalities together imply that the second equality in (2)). The last inequality of 25 can be obtained by applying similar transformations repeatedly, From 25 we have
[LHS of 24)
(Grla)) - GG,()))
Gelge) - G.(g(+1) P + MT+1
DKI(Ø", .40)
stab(ma9((aa,, + Diu(p", -90).
(28)
t=1 n
4T+1
which implies that (24) holds. By combining (23 and 24). we obtain the bound of (6)
B.3 Proof of Theorem
Proof. For any a X, let p" be the uniform distribution over {{1 + )a fx X) We then have Rr(a") I Eaup' [Rr(a)] + and DKL(p", (p",4") Σ d dol T. Define by
N = min Noo vdlog T/ T/
v/vaa a E:=l 77 Agu. Suppose that pt and ge are chosen so that A mm. As we have S no. from the assumption, we have the "gu. S a. Hence,
Ag.mn (pr, 9t)
from we have
dlog T
/d log Tn, A
dlog T 11++1
Rí
E
A uome
+1<E
+1
M/+11
a + Σt=1 17
Ag Ag
dlog T
3
d dlogT
a Σ
T Aggm
+1
110
1=1
{ {
aT, B Σ Er Acllnae)))
d log T
< 3
dlog
a a min
Ry: (29)
which implies the regret bound of Rr = 0 (VadI log T' + d log vironments. We next show the improved regret bounds for stochastically constrained adversarial environments. From the definition of A, we have
that holds in adversarial en-
No
A mm An n. (pt, 9t) 2 sup (lep((p)) x((x)))) 2 (((e)) - p(p:)),
(30)
REC
15

---

where the last inequality holds since -t" E L for any l E L, which follows from the definition of L. We then, hence, have
T
R
E
Σ D.- (m(ur))
E
(Ar
(u(pi)) + (6" (((:)) u((x))))
T
T
I E
(m((1()))
+ E
Σ
A gi.
S R 12, s 212 +C
(31)
From this, we have
dlog
d log T
R I 3
dog T (a 3R.
+
+1 3E
V
dlogT B(2R' + +C])
+
+
17u
which implies that Ry Ry= 10
dlog T VCBdlog 1
holds.
no
B.4 Proof of Lemma 2
Proof. We first see that An. = O(1) holds for any g E P(x) and v > 0. In fact, by setting
p = q and letting g E 9 to be the zero function (g(2; a, f] 0 for all x, a and f), we obtain
(P. suPrec bias(g, P, q, E) s (f, - a²) I 2.
Let us consider the case which 77 S 1/d holds. As A = O(1) for any 7) and q, we have Ag., = (4)) if pw(()) = 22(). In the following, we assume mw(()) I 1/8. Let ý E P(x) be a
distribution such that p(p) sl() and w(g) = (p) V(q). Let To E P(X) be distribution such that
[1TT) S
(32)
holds for any A. Note that such distribution always exists (see, e.g., Lattimore and Szepesvári 2020a Theorem 21.1J). Define p by
p (1 -1- + y
(33)
with y = 4nw(g) $ 1/2. With this p. we define by (4. Then, as we have bias(g. p, = 0, we
have
(p. S sup p()) m()))) + sup E [stab(no(;;a "7 sEi-1.11 a>p
(34)
EEA
As the definition of p implies ppp) (1 ())((p) (M((T)) = (1 (1-))u(g) + (p(())) it holds for any E L that
(e, (()) (lg)) = (e, (1 - ()((1) + + - u(q)) = 77( - m(q)) s 27.
(35)
Further, from 7 and we have
stab(rg((5 f).9) S
E
[ongg( S ng((());
E zmn
(n)
(36)
for any fixed a E A and f E 1-) Let r represent the distribution of f ( aa, for x q with any fixed AA and f E [-1, Then, as g log-concave distribution, from
Lemma 10 r is a log-concave distribution as well. Further, we have
H(r)
gur | = f2
E z
-a,
Pa (a(() E (() (4()))
=
(37)
Since we have
(((16))11(V(())1)))) = V(g) H(p) v()) ((1 ((--))) +
-
V(()) (1- 1-7 w(g).
(38)
16

---

we have
w(g) 1-7 Id
H(p) ())
(39)
where la represents the identity matrix of size d. Hence, we have
(1-7 Ia (H(P)) 1/2 w(g) a (H(p)) -
((1)
H(r) s a (H(P)) 1/2
1-7
w(2) T T(1 Hi) +
w(9)
(T)( - la
1-7
1 1-7
w(1) T H(To) ( w(g)d (1-7)
< (2(()) d
(40)
(1-7)
?
2nj
This implies that (r) S dn/2 - 1/2. Hence, from Lemma
13 we have
stab(ng(=:a, f).9) = E
= o [d(m)] = ( (VV (41)
yoor
Hence, we have
sup
E
a, =0 =))
[a (V)))
EE|
1,1-
amop
anp
= 0
17 12
E
aa
(H(p)) (((())))V((((((())))))) = (1
(H(p))
arop
1
(M())) =0 ( = 00 ()72(()))
0
V(g)
(42)
Combining this with (34g and
35 we obtain
(p, 9) 27 sup
E =
= O ())(()))
(43)
7 re(-1.1
1.1] a<p
which implies that Ar = ()w(()))
We next consider the case in which 1j S 1 holds. As have Ar O(pe' (q)) if mpw > 1/8
(since A (1) for any and 17), assume my (q) S 1/8. We set by (33 where is the
minimizer in the right-hand side of (12) and = 4mp' (q) S 1/2. We set by Then, 34 and
35
can be shown in a similar way to in the first half of this proof. As we have
2(())
dV 2w =
1- H(p).
(44)
have
w'(g) (1 (1-7)d" la-
(45)
From this, if we define r in a similar way to in the first half of this proof, we have H(r) < 1/ We hence have (r) n/2 < 1/2 under the assumption of n < 1. We can then apply Lemma 1 to obtain the following:
( n7² (1- V()) (H(())))
sup
E (tabbng(::a,f)..ou 50
(1-
n² w'(9) 1-7 d
H( (H(())) O ()²'(a) (1-7
0
O (1)w((g))
(46)
where the first inequality follows from a similar argument as (42). Combining this with and 7 (2). we obtain (p, 9) O(p (g)), which implies A, O'pp' (g)).
17

---

B.5 Proof of Lemma 3
Proof of Lemma 3
We will show that
there exists
p P(2)) such that x(p) - =(4) and V(1) 3 2. vV))
(47)
with = = 011, A0,,Ae-(1 (u(())))/)/Aa min). As we have (p) - (p)) + u(p))(p) T = V()). (47 leads to d (g)) = dz (pp) 3 dz (p), which implies w/ (9) S da. Hence, to prove the lemma, it is
sufficient to show that (47) holds with & = Og, (p(g))/Ar -mi,))))). By considering the case of
p q. can easily see that (47) holds with = O(1). for any q. In the following, we show that [p(()(
447 holds with 0 ( .min
Without loss of generality, we may assume plq)) = 0. via a variable transformation x f- x - u(q). In fact, such transformation preserves (g)) and (p)). For any x E X, there exists {, = (a)))aEE.
P(A) such that f((z)) = EnEA (a(a) = z. For any (Ex)zEx, define E P(A) by p(a) =
Pew (a)]. We then have ju(p) = Ea-plai = Exzng [Eawe, [a]] = E Exnu [x] = p(g) = 0. Fix an
arbitrary non-zero vector u E Rdd \ {0} and an arbitrary a* E A. Denote E = Prappl la = a a'}.
Set 0 = u//u V(q)u. V(Q)). Let pv and qu denote the distributions of (r,a) for a ~ p and (v, z) for I ~ q. respectively. We then have V(g.) = v VQQ( u (V)uu/(u (g)u) = 1. Without ((())((TTV(()))
loss of generality, we assume that (v,a") 2 (c, ((a)) plg.) 0. (If not, we redefine = -u/ V(g)u.) Since q is log-concave distribution, from Lemma 10 4v is log-concave distribution as well. Hence, from Lemma [] we have
1
1
Pr x
Pr
X
0
Pr
x
<
2
(48)
X
2c
X--qu
2c
2c
20
where the inequality follows from (6 and
We hence have
V(p.)
E amp
[[(v,
{))}]=
E
E
[[(ve a;)}
20ng
1
1
E
E
[((e, a)) [9] (u, I
Pr X
X
<
z~gg
ariEz
20
20
E [((u, a) )"] (v, 5 aa
(49)
2c
20
where the last inequality follows from (48) Suppose (u. S 20
1
Since u(Ex) = x, we have
1
2v = Ex(aa) (v,""" +
Σ (a (v, >
Σ
Eo(a) (v,
(50)
2e
aEA)
aEA)(a>
We hence have
Σ -1-))))))(((((((0 ExEA) (a(a)((1,a))
E [((v,a))7]
2
ar
Enedy(ar)
aEAl(a')
E (e))) (EneAMar) Eo(a) (v. )² EoEA (a)
1
2 (a=)
1 (a*)
402 - Eo (a*)))
where the second inequality comes from Jensen's 's inequality, and the last inequality follows from (50).
From this and (49. we have
1
1
((P.))
E S ara
(v, <
>
(51)
(ar)
2e
Se
E Exng
[
Ex(ar)
(v. x) S
Ze.
We further have
1
1
E znq
[1 Ea(a"))
> E
1 - (x(a))
(v, S
Pr xag
(v, --
2e
2e
1
E
1 - (x((a)
(v.2) 5
(52)
2c xma
20
18

---

where the last inequality follows from (88
Combining this with (TD,, we obtain
1
V(De)
(53)
16848
As it follows from the definition of e that
V(p.) u V(D)o u Vp))u
(54)
u V(ggu'
we have
VV T V(quu.
(55)
Since this holds for all u E ( 1 {0}, we have
160 V()) = V(() Letting a" arg min (e", a), we obtain Ar. Dr. (n(p))
(56)
Ep(a)Ar
aEd
p(a) min
De (1 - --p(aa)) Aer,min
(57)
(EA((a")
Combining this with (56). we obtain
16e Ar (ulg)), V(p) V(o),
(58)
De min
which completes the proof.
B.6 Proof of Lemma
Proof. Let be a log-concave distribution over X = fx E [0, |||||
1]. If x
(Zi,..., ra) ~ q, from Lemma TO Iy for each also follows a log-concave distribution over
[0, 1]. Hence, from Lemma 12 the variance of x; (i.., the (i, entry of V i bounded as
(V(4),, S e min((; (1 (1))) < 4e (1) (1)(())))))
(59)
Consider the distribution p E P(A)) such that Pramp/a = es] = u(q). We then have (p) = (4) and that H (p) is a diagonal matrix with diagonal entries (4)( Hence, from (59). we have
V()) (H()))- = Σ (()laa())))) < 4e
Σ 1=1
-
< 4e
(1( ui(1)),
(60)
which implies that w(g) O Σ (1) -w(o))) Further, as we have Σ E pt(g) holds for any i {L...., d) that
(1)))1 p((g)) p.- (1) p4- (4) Σ
p1(1)(1 mx(g))
< uv (g) + Σ
mi(g) = 2 (1 - p4i (g)),
(61)
$
which complete the proof.
B.7 Proof of Corollary 2
Proof. From Lemma and the fact that Ed= (4(g) 1, is clear that w(g) = 0(1). Let
l E L be an a loss vector such that ex- E arg min E is unique. We then have A,- (m(g))
Exjre Ar- (e:)e: 2 Arimmin (1-p)- (q)). Hence, from Lemma we have w(g) = 0 14; (1)) <
( (p(g)) A mmm)). From this and Lemma
(T)p holds with 0(1), B 0 A,- ). and
7o O(Ld))
19

---

B.8 Proof of Lemmag 5
{l..... m) let pb) E P(X(i)) be such that u(p())) u(g()) and w(q()) ((( (H(b))) Let p E P(x) be the product measure of p(.,..... ...... (m)) As ladxd is a block diagonal matrix with subma- trices V(g("),.... ....,, (qg(m)). Similarly, Hp) E Rdxd is block diagonal matrix with submatri- ces H "....,),,(()).......... (p(m)) We hence have (gg) (H(p)) Σド= V(g()) (H(p)))
Proof. For each j
p is the product measure of q()) ,.... ........))))) (m),, V (gg)
Σ w(q which implies (())= - Σ== O1)
B.9 Proof of Corollary 3
Adm- For any (e()) ..... (lm)) L and (2(1), _x(2), _o(2)..... ...., x(()) X, where gi)) R& and x(5) E x(i) for each j. de- min
Proof. Suppose is given by A Ad,
= (e6), - a(6)). We then have A.. (2) = 2 (z!()). Let be a loss vector such that a = (a a (2) ...., a (m)) E
note 4 260 (x00) (160)2(01
arg min (e, a) is unique. We then have me minisism 200 min'
I E X' product measure of gj E (((())) from Lemmas 4 and we have
A AU) p-6) (u(a()))
w(1) 0
Σ w(gbi))
0
Σ min
1,
pmii),min .min
A (aG)
(m(a(t)))
})
(o6) (u(al)))
})
0
min
<
min
m,
g-ii),min min
De minm min
j=1
3 (p(g))
))
=0
min
m,
(62)
Ae Amimi min
This means that (T0) holds with a = (m)) and O (()))mmm)
The case of A {0, 13d can be interpreted problem instance in which is direct product of copies of A2. Hence, we can see that (TO) holds with O(d) and 3 = Ol A man
B.10 Proof of Lemma 6
Proof. From the definition of the regret, we have
Rr(a*)
Σ E
[1 a²)l
ampp
(le, wi - a²) + (le, we - p(p))))
(63)
1=1
From the definition of bias given in (14). the part of (le, we a²) can be bounded as follows:
(Ce, w. a²)
E
[ee, hr fi), wi aa)) + (hr(ar, fe), w. a*)]
azch
< bbass((h)) h,, Pt, we, le) + E ar.ss
[he(ars ft), fi),w, We - a").
(64)
From this and 63 we have
T
(a S E
Σ ((l,, wi - p(p))) + bias(h,, Pe, we, 6)) +
Σ (ni(ar, fi), wr - a*)
(65)
By standard analysis of the FTRL framework (see, e.g. Lattimore and Szepesvári
2020a Exercise
28. 12]), we have
T
T
(hrlar, ft), -a)) x²)= < Σ Σ
1 _
(1,h, w.).w)) + "baa).
(66)
7
1/-
t=1
20

---

By combining (65 and 66
we obtain the bound of 14
21


Reviews:
[{"review_id": "qqeLGmHvoy", "reviewer": "Reviewer_JH8B", "summary": "This paper studies the \"best-of-both-world\" problem for linear bandits, meaning that designing a single algorithm that simultaneously achieves a $O(\\sqrt{T\\log T})$ regret in the adversarial case and $O(\\log T)$ regret in the stochastic case. They propose to use the Exploration-by-Optimization approach to obtain such bounds. The algorithm can be viewed as a continuous exponential weight algorithm with an optimally designed action sampling scheme. ", "questions": "See the above sections. ", "limitations": "There is no potential negative societal impact. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The approach provides a unified and insightful way to obtain near-optimal best-of-both-worlds bounds for linear bandits and several of its special cases. The algorithm is conceptually simpler than previous work such as [Lee et al., 2021] and [Dann et al., 2023]. \n- The algorithm is based on a general framework of Exploration-by-Optimization, which has been proven to work for general adversarial decision making problems. Therefore, the proposed method has potential to be extended to more general cases, though the current analysis is only for linear bandits. \n- There are several novel elements in the analysis, especially Lemma 3. It shows that continuous exponential weights with a simple sampling scheme already works well.  ", "weaknesses": "- The analysis relies on the unique optimal action assumption, and the bound in the stochastic environment can be arbitrarily worse than the  instance-dependent bound by Lattimore and Szepesvari (2017). However, these are general open questions in the field but not the specific issues of this work. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort you have invested in this review.\n\n> The analysis relies on the unique optimal action assumption, and the bound in the stochastic environment can be arbitrarily worse than the instance-dependent bound by Lattimore and Szepesvari (2017). However, these are general open questions in the field but not the specific issues of this work.\n\nAs discussed in Section 6, we recognize the two points you raised as important issues.\nSection 6 also discusses possible approaches to these issues,\nand we intend to investigate the effectiveness of these approaches in future research."}, {"review_id": "gXbQ40dh0s", "reviewer": "Reviewer_mre2", "summary": "The paper establishes $\\log T$-style instance-dependent regret upper-bounds for linear bandit algorithms built upon the FTRL framework and appropriate exploration-by-optimization style update steps.", "questions": "See the weaknesses part.\n", "limitations": "See the weaknesses part.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The $\\log T$-style instance-dependent regret upper-bounds can be achieved on general convex action sets and a number of typical important specific action sets. The paper establishes the claimed results by first identifying a sufficient condition for the so-called self-bounding property for FTRL with exploration-by-optimization style update steps and thus $O(\\log T)$ regret bounds (Theorem 1), then showing this condition is indeed satisfied under the continuous exponential weight methods.  Overall, the paper is clearly written. ", "weaknesses": "Similar to most existing works on linear bandits that mainly discusses sample complexity instead the actual computational cost, the paper by default assumes one can effectively and accurately perform the computation of EXO update steps. However, for most convex action sets excepts for some special ones, these steps may be computationally intractable on an ordinary RAM machine. I would happy to see some discussion about the actual computation model (e.g., with oracle access to separation hyperplanes) and how to implement the proposed EXO algorithms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper.\nWe hope that our response below addresses your concerns.\n\n> Similar to most existing works on linear bandits that mainly discusses sample complexity instead the actual computational cost, the paper by default assumes one can effectively and accurately perform the computation of EXO update steps. However, for most convex action sets excepts for some special ones, these steps may be computationally intractable on an ordinary RAM machine. I would happy to see some discussion about the actual computation model (e.g., with oracle access to separation hyperplanes) and how to implement the proposed EXO algorithms.\n\nBrief notes on the computational complexity and on the implementation of the algorithm can be found in Appendix E of the supplementary material and in Remark 2.\nWe consider that one reasonable computation model would be the Real-RAM model equipped with a separation oracle for (the convex hull of) the action set $\\mathcal{A}$ as the reviewer suggested.\nUnfortunately, it is not known if there is an efficient algorithm to solve the optimization problems (9) as we discussed in Appendix E.\nHowever,\nwe do not necessarily need to solve the optimization problem (9) exactly as noted in Remark 2.\nFor example,\nit is expected that a polynomial-time algorithm that achieves regret bounds of Corollary 1 can be implemented given a separation oracle (or equivalently, a linear optimization oracle) for $\\mathcal{A}$.\nIndeed,\nfor achieving the regret bounds of Corollary 1,\nit suffices to find $p$ and $g$ such that $\\Lambda_{q_t, \\eta_t} (p, g)$ is bounded by the RHS of (10).\nThe construction of such $p$ and $g$ is provided in the proof of Lemmas 2 and 3 (Lines 242--261),\nwhich can be performed using a separation oracle for $\\mathcal{A}$.\nIn fact,\nwe can obtain samples from $p$ by using the techniques for log-concave sampling (e.g., [Lovász and Vempala, 2007]) and for computing convex combination expression (cf. Carathéodory's theorem for convex hull and [Schrijver, 1998, Corollary 14.1g]).\nHowever, the analysis of log-concave sampling and calculations of $H(p)$ (which is required for constructing $g$)\nincluding consideration of calculation errors can be highly complicated,\nand the computational cost can be very large, although on the order of polynomials (cf. [Hazan and Karnin, 2016, Corollary 6.2]).\nA more detailed discussion will be added in the revised version.\n\nReferences:\n- E. Hazan and Z. Karnin. Volumetric spanners: an efficient exploration basis for learning. The Journal of Machine Learning Research, 17(1):4062–4095, 2016.\n- L. Lovász and S. Vempala. The geometry of logconcave functions and sampling algorithms. Random Structures & Algorithms, 30(3):307–358, 2007.\n- A. Schrijver. Theory of Linear and Integer Programming. John Wiley & Sons, 1998.\n\n"}, {"review_id": "mLUbkbpd7u", "reviewer": "Reviewer_KqE4", "summary": "This paper considers the best-of-both-worlds problem for linear bandits. They investigate an Exploration-by-Optimization approach for this problem and show O(d^2\\logT) regret for stochastic setting and O(d\\sqrt{T}) regret for adversarial setting. ", "questions": "Please see the weakness part. ", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper investigates a new approach for Best-of-both-worlds problem in linear bandits. \n\nIn some special cases such as multi-armed bandits and hypercube, the authors provide improved results utilizing the problem structure. \n\nPrevious BoBW results mainly depend on FTRL-type algorithms, this work uses a new type of approach, which may be hoped to solve the BoBW problems with other structure. \n", "weaknesses": "Given existing works on BoBW problem for linear bandits, the convergence results provided by this paper are not optimal. \n\nIt is not clear what the advantage of EXO algorithm is compared with standard FTRL-type algorithms. Can you give more discussions? \n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to read and review our paper.\nWe hope the following answers address your questions.\n\n> Given existing works on BoBW problem for linear bandits, the convergence results provided by this paper are not optimal.\n\nAs the reviewer pointed out, our bounds are not tight and we have positioned this as an important issue.\nSection 6 discusses possible approaches to these issues,\nand we intend to investigate the effectiveness of these approaches in future research.\n\n> It is not clear what the advantage of EXO algorithm is compared with standard FTRL-type algorithms. Can you give more discussions?\n\nConventional FTRL-type algorithms for linear bandits,\nsuch as exponential weight algorithms with $p_t \\approx q_t$ [Bubeck et al. 2012] and SCRiBLe with Dikin ellipsoid sampling [Abernethy et al., 2008, 2012],\nhave been originally designed for adversarial environments,\nand do not achieve best-of-both-worlds regret bounds,\ni.e.,\nthey do not have better bounds than $O(\\sqrt{T})$ even in stochastic environments.\nThe advantage of EXO algorithms compared to this is that they achieve best-of-both-worlds regret bounds,\ni.e.,\nthey have $O(\\log T)$-regret bounds for stochastic environments.\n\nReferences:\n- J. Abernethy, E. Hazan, and A. Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. In 21st Annual Conference on Learning Theory, 2008.\n- J. Abernethy, E. Hazan, and A. Rakhlin. Interior-point methods for full-information and bandit online learning. IEEE Transactions on Information Theory, 58(7):4164–4175, 2012.\n- S. Bubeck, N. Cesa-Bianchi, and S. M. Kakade. Towards minimax policies for online linear optimization with bandit feedback. In Conference on Learning Theory, pages 41–1. JMLR Workshop and Conference Proceedings, 2012."}, {"review_id": "lw990g8WbA", "reviewer": "Reviewer_X2KX", "summary": "The paper addresses the challenge of selecting an algorithm suited to the environment type, which is often unknown in real-world applications. This paper introduces the concept of best-of-both-worlds (BOBW) linear bandit algorithms that perform well in both stochastic and adversarial environments. Previous BOBW algorithms for linear bandit problems have achieved suboptimal regret bounds in stochastic environments due to an additional logarithmic factor. The authors use an existing approach called exploration by optimization (EXO), and prove that it achieves nearly optimal regret bounds in both regimes. EXO utilizes the exponential weight method to update the reference distribution and computes the sampling distribution and loss estimator to minimize an upper bound on regret. More precisely, the algorithm constructed using this approach achieves $O(d\\sqrt{T \\log T})$-regret in adversarial environments and $O(d^2 \\log T/\\Delta_{\\min})$-regret in stochastic environments. The paper establishes a connection between the EXO approach and the SCRiBLe algorithm, which uses a follow-the-regularized-leader (FTRL) method with self-concordant barrier regularization. The authors also propose a variant called mean-oriented EXO that encompasses the EXO approach and allows for interpretation of existing SCRiBLe-type algorithms as EXO-based methods.", "questions": "Are there any limitations or potential challenges in implementing the EXO-based algorithm in real-world applications?\n\nThe paper mentions the use of the self-bounding technique for analyzing regret bounds in stochastic environments. Can you explain how this technique is applied and what advantages it offers?\n\nHow do you think the algorithm interpolate between the two regimes? For example, How does it perform in corrupted stochastic environments or environments with adversarial corruptions?\n\nCan you elaborate on the potential extensions of the EXO framework to other sequential decision-making problems beyond linear bandits? How can this framework be applied to partial monitoring problems or episodic MDPs?\n\nAre there any plans for future empirical studies?\n", "limitations": "The paper focuses solely on theoretical analysis.\n\nThe paper mentions that EXO can be complex in terms of structure and implementation. The details of how these algorithms can be efficiently implemented in practice are not provided, leaving open questions about their feasibility and scalability.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses the challenge of constructing best-of-both-worlds linear bandit algorithms that perform well in both stochastic and adversarial environments. The distinction between these environments and the need for an optimal algorithm for both is considered as a quite significant open problem. The paper demonstrates the effectiveness of EXO in achieving nearly optimal regret bounds in both stochastic and adversarial environments by establishing regret bounds. \n\nThe paper also establishes a connection between the EXO approach and the SCRiBLe algorithm and its extensions. This connection helps in interpreting existing algorithms within the framework of exploration by optimization. \n\nThe paper suggests that the framework of exploration by optimization and best-of-both-worlds regret guarantees can be extended to other sequential decision-making problems beyond linear bandits. It mentions partial monitoring problems and episodic MDPs as potential areas for future research and highlights the broader implications of the EXO framework in various decision-making contexts.\n", "weaknesses": "While the paper presents an approach to constructing best-of-both-worlds linear bandit algorithms that perform well in both stochastic and adversarial environments, there are a few weak points that can be identified:\n\nThe paper assumes that the loss vectors are generated according to a specific model and that the environment falls under either a stochastic or adversarial setting. In real-world applications, the actual environment may not be strictly one or the other, and it might be challenging to accurately model the underlying process.\n\nThe paper primarily focuses on theoretical analysis and regret bounds, but it lacks an empirical evaluation or experimental results to validate the proposed algorithms. Without empirical evidence, it is challenging to assess the practical effectiveness and efficiency of the algorithms in real-world scenarios.\n\nImproving a problem or solution by a log(T) term may not always be considered a significant breakthrough, depending on the context and the magnitude of the improvement.\n\nThe paper mentions that the proposed algorithms involve optimization problems (minimization problem (9)) and self-bounding techniques. While these techniques can lead to improved performance, they may introduce complexity and implementation challenges. The paper does not provide detailed insights into the practical feasibility and implementation aspects of the algorithms.\n\nThe paper does not explicitly discuss the limitations and assumptions of the proposed algorithms. It would be beneficial to address the assumptions made about the underlying environment, potential limitations of the algorithms, and scenarios where the algorithms may not perform optimally.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewers' thoughtful and comprehensive comments and feedback.\nThe reviewers' insights have greatly contributed to the improvement of our work, and we sincerely appreciate the time and effort you have invested in this review.\nWe hope our response below addresses your concerns.\n\n> Are there any limitations or potential challenges in implementing the EXO-based algorithm in real-world applications?\n\nBrief notes on the computational complexity and on the implementation of the algorithm can be found in Appendix E of the supplementary materials and in Remark 2.\nUnfortunately, it is not known if there is an efficient algorithm to solve the optimization problems (9) as we discussed in Appendix E,\nwhich is a major limitation in the implementation.\nHowever,\nwe do not necessarily need to solve the optimization problem (9) exactly as noted in Remark 2.\nFor example,\nit is expected that a polynomial-time algorithm that achieves regret bounds of Corollary 1 can be implemented given a separation oracle (or equivalently, a linear optimization oracle) for $\\mathcal{A}$.\nIndeed,\nfor achieving the regret bounds of Corollary 1,\nit suffices to find $p$ and $g$ such that $\\Lambda_{q_t, \\eta_t} (p, g)$ is bounded by the RHS of (10).\nThe construction of such $p$ and $g$ is provided in the proof of Lemmas 2 and 3 (Lines 242--261),\nwhich can be performed using a separation oracle for $\\mathcal{A}$.\nIn fact,\nwe can obtain samples from $p$ by using the techniques for log-concave sampling (e.g., [Lovász and Vempala, 2007]) and for computing convex combination expression (cf. Carathéodory's theorem for convex hull and [Schrijver, 1998, Corollary 14.1g]).\nHowever, the analysis of log-concave sampling and calculations of $H(p)$ (which is required for constructing $g$)\nincluding consideration of calculation errors can be highly complicated,\nand the computational cost can be very large, although on the order of polynomials (cf. [Hazan and Karnin, 2016, Corollary 6.2]).\nTherefore, it is also an important challenge to consider more efficient implementation methods.\nA more detailed discussion will be added in the revised version.\n\n\n> How do you think the algorithm interpolate between the two regimes? For example, How does it perform in corrupted stochastic environments or environments with adversarial corruptions?\n\nThe proposed algorithm interpolates between the two regimes well.\nIn fact,\nregret bounds in this paper can be applied to a class of stochastic environment with adversarial corruptions as noted on Lines 153--165.\nDiscussion on Lines 153--165 and Theorem 1 together imply that\nthe proposed approach achieves $O( \\log T + \\sqrt{C \\log T} )$-regret\n(ignoring factors w.r.t. parameters other than $C$ and $T$)\nin stochastic environments with corruptions,\nwhere $C$ represents the total amount of corruption.\n\n> The paper mentions the use of the self-bounding technique for analyzing regret bounds in stochastic environments. Can you explain how this technique is applied and what advantages it offers?\n\nIn the analysis based on the self-bounding technique,\nwe exploit self-bounding constraints (2) to obtain regret bounds such that the regret itself appears on the right-hand side,\ne.g.,\n\\\\[\n\tR_T(a^*) \\leq \\sqrt{ \\beta (R_T(a^*) + C) \\log T } .\n\\\\]\nA similar expression can be found in Lines 524--525 in the supplementary material.\nThis expression can be interpreted as a quadratic inequality for $R_T(a^*)$,\nand solving it yields $R_T(a^*) = O( \\beta \\log T + \\sqrt{C \\beta \\log T}   )$.\nOne advantage of this technique is that it also provides good bounds for stochastic environments with adversarial corruptions,\ni.e.,\nfor the environments with $C > 0$.\n\n> Can you elaborate on the potential extensions of the EXO framework to other sequential decision-making problems beyond linear bandits? How can this framework be applied to partial monitoring problems or episodic MDPs?\n\nWe believe that the algorithmic framework and Theorem 1 can be extended to other problems,\nsuch as partial monitoring problems or episodic MDPs,\nrelatively easily.\nHowever, the analysis of $\\Lambda^*$ provided in Section 4.3,\nwhich is essential for exploiting Theorem 1,\nheavily relies on a structure specific to linear bandits,\nand we believe that extending this part to other problems would require a great deal of nontrivial thought.\n\n> Are there any plans for future empirical studies?\n\nWhile we agree with the importance of empirical studies,\nwe have no plans to conduct experimental studies in the near future,\nat this time.\nAs we answered your first question,\nwe need to consider how to improve the implementation of the proposed algorithm in order to make it work in a realistic computation time for high-dimensional problems.\nWe believe that full-scale numerical experiments will become possible only after an efficient implementation method is found.\n\n\nReferences:\n- E. Hazan and Z. Karnin. Volumetric spanners: an efficient exploration basis for learning. The Journal of Machine Learning Research, 17(1):4062–4095, 2016.\n- L. Lovász and S. Vempala. The geometry of logconcave functions and sampling algorithms. Random Structures & Algorithms, 30(3):307–358, 2007.\n- A. Schrijver. Theory of Linear and Integer Programming. John Wiley & Sons, 1998."}]

Author rebuttal:

Meta-review: This paper provides a novel solution to the Best of both worlds problem for linear bandits. It is the first paper using a recent exploration by optimization approach for BoB problems and the techniques are non-trivial and potentially useful for future work.
The algorithm does not obtain better bounds than prior work, but is simpler and arguably more elegant. The work has the same weaknesses, i.e. uniqueness assumption of the best arm and sub-optimal asymptotic instance dependent bound, as the literature.
Unlike the state-of-the-art solution for this problem, the algorithm does not rely on restarts and doubling and the uniqueness assumption might be an artefact of the analysis.

There is a consensus among the reviewer that this paper is above the acceptance threshold, but the lack of improvement over the sota and the lack of an implementable algorithm do not allow for a higher than poster rating.
Decision: Accept (poster)