Title: First- and Second-Order Bounds for Adversarial Linear Contextual Bandits
Venue: NeurIPS 2023
Accepted: yes
Keywords: contextual bandits

bandits

sequential learning

regret bounds
Abstract: We consider the adversarial linear contextual bandit setting, which
allows for the loss functions associated with each of $K$ arms to change
over time without restriction. Assuming the $d$-dimensional contexts are
drawn from a fixed known distribution, the worst-case expected regret
over the course of $T$ rounds is known to scale as $\tilde O(\sqrt{Kd
T})$. Under the additional assumption that the density of the contexts
is log-concave, we obtain a second-order bound of order $\tilde
O(K\sqrt{d V_T})$ in terms of the cumulative second moment of the
learner's losses $V_T$, and a closely related first-order bound of order
$\tilde O(K\sqrt{d L_T^*})$ in terms of the cumulative loss of the best
policy $L_T^*$. Since $V_T$ or $L_T^*$ may be significantly smaller than
$T$, these improve over the worst-case regret whenever the environment
is relatively benign. Our results are obtained using a truncated version
of the continuous exponential weights algorithm over the probability
simplex, which we analyse by exploiting a novel connection to the linear
bandit setting without contexts.

First- and Second-Order Bounds for Adversarial Linear Contextual Bandits

Abstract

We consider the adversarial linear contextual bandit setting, which allows for the loss functions associated with each of K arms to change over time without restriction. Assuming the d -dimensional contexts are drawn from a fixed known distribution, the worst-case expected regret over the course of T rounds is known to scale as ˜ O ( √ KdT ) . Under the additional assumption that the density of the contexts is log-concave, we obtain a second-order bound of order ˜ O ( K √ dV T ) in terms of the cumulative second moment of the learner's losses V T , and a closely related first-order bound of order ˜ O ( K √ dL ∗ T ) in terms of the cumulative loss of the best policy L ∗ T . Since V T or L ∗ T may be significantly smaller than T , these improve over the worst-case regret whenever the environment is relatively benign. Our results are obtained using a truncated version of the continuous exponential weights algorithm over the probability simplex, which we analyse by exploiting a novel connection to the linear bandit setting without contexts.

1 Introduction

The contextual bandit problem is a generalization of the multi-armed bandit setting in which a learner observes relevant contextual information before choosing an arm. The goal of the learner is to minimize the excess cumulative loss of the chosen arms compared to the best fixed policy for mapping contexts to arms. This framework addresses a broad range of important real-world problems like sequential treatment allocation (Tewari and Murphy, 2017), online recommendation (Beygelzimer et al., 2011) or online advertising (Li et al., 2010), and is actively used in practice (Agarwal et al., 2016). Numerous variants of the setting have been studied, which differ in the assumptions they make about the losses and the contexts. In this paper, we focus on the recently introduced setting of Neu and Olkhovskaya (2020) where the contexts are finite-dimensional i.i.d. random vectors, and the losses are time-varying linear functions of the context that may potentially be generated by an adversary. In this setting, the worst-case rate for the expected regret is known to be ˜ O ( √ T ) for time horizon T (Neu and Olkhovskaya, 2020).

Our main contribution is to replace the worst-case rate by adaptive bounds. Specifically, we obtain a bound of ˜ O ( √ V T ) in terms of a quadratic measure of variance V T for the losses of the algorithm, and a bound of ˜ O ( √ L ∗ T ) , where L ∗ T is the cumulative loss incurred by the optimal policy. Such bounds in terms of L ∗ T or V T are generally referred to as first-order and second-order bounds , respectively,

and have been extensively studied in the bandit literature. They can lead to much stronger guarantees in the often realistic case when T is large, but the losses vary little or when there exists a policy with very low cumulative loss.

Worst-case guarantees in terms of T have first been proved for the contextual bandit problem with finite policy classes by Auer et al. (2002b), with further improvements by Beygelzimer et al. (2011). These methods can deal with adversarial losses and contexts, but only work for finite policy classes and have run-time scaling linearly with the size of the class-which is generally unacceptable in practice. This latter challenge has been addressed by a line of work culminating in Agarwal et al. (2014), which only requires access to an optimization oracle over the policy class. Their results, however, remain restricted to i.i.d. contexts and losses. An alternative line of work has been initiated by Auer (2002); Chu et al. (2011); Abbasi-Yadkori et al. (2011), who studied the special case of i.i.d. linear loss functions with changing decision sets. The case of i.i.d. contexts and adversarial linear losses has first been studied by Neu and Olkhovskaya (2020).

Improvements of worst-case guarantees of order √ T to first-order bounds scaling with √ L ∗ T have been known for a variety of bandit settings since the works of Stoltz (2005); Allenberg et al. (2006), and Neu (2015). Regarding contextual bandits, the COLT 2017 open problem of Agarwal et al. (2017) asks for efficient algorithms that achieve first-order bounds for large, but finite, policy classes, either when both contexts and losses are i.i.d. or when both are fully adversarial. First to answer the open problem were Allen-Zhu et al. (2018), who obtained an optimal first-order regret guarantee for adversarial losses and contexts, but with an algorithm that is inefficient for large policy classes. Foster and Krishnamurthy (2021) provide the first efficient algorithm for the non-adversarial setting where the loss function is fixed over time and one has access to an oracle that can solve various optimization tasks over the policy class. We improve on these works in terms of the computational efficiency of our algorithm and by allowing the loss function to vary adversarially over time, although we do rely on the extra assumption that the loss functions are linear.

Another relevant framework is the adversarial linear bandit setting (without contexts), where there also exist adaptive results (Bubeck et al., 2019; Lee et al., 2020; Ito et al., 2020). While conceptually related, an important distinction is that the linear bandit setting assumes a fixed decision set, whereas reducing the linear contextual bandit problem to a linear bandit problem requires the use of decision sets that change as a function of the contexts.

Main Contributions. We consider a K -armed linear contextual bandit problem with d -dimensional contexts over T rounds. The contexts are assumed to be drawn i.i.d., but the linear loss functions mapping contexts to losses for the arms are chosen by an adaptive adversary. The aim of the learner is to minimize their regret, which is the gap between the expected cumulative loss of the learner and the expected cumulative loss of the best fixed policy π ∗ T chosen in full knowledge of the sequence of losses. In this setting, π ∗ T is known to be a linear classifier, i.e. it chooses the arm with smallest predicted loss, where the predictions are fixed linear functions of the context (see Section 2). The goal is therefore to compete with all linear classifiers. We first obtain the following second-order bound on the expected regret

where V T is defined in (5) as a measure of the cumulative second moments of the losses for the arms played by the algorithm. Following Ito et al. (2020), we allow these moments to be centered around optimistic estimates that can further improve the bound when available or can simply be set to zero when they are not. We further obtain a first order bound of the form

The second-order bound is obtained using a truncated version of the continuous exponential weights algorithm over the probability simplex, similar to the algorithm for linear non-contextual bandits of Ito et al. (2020), and the first-order bound may be obtained as a corollary. As discussed in Section 3.3, the computational complexity of this method is dominated by two steps that together require ˜ O ( K 5 )+ ( d/ϵ ) O (1) per round for approximation up to precision ϵ > 0 , which is computationally feasible for moderate K and ϵ . Both results are not strict improvements on the worst-case rate of ˜ O ( √ KdT ) by Neu and Olkhovskaya (2020): first, they have a slightly worse dependence on K . We consider this a price worth paying for the first adaptive bounds in this setting. Second, they require the extra

assumption that the distribution of the contexts is log-concave . Although log-concavity is weaker than assuming the contexts follow e.g. (truncated) Gaussian distributions, we conjecture that it may not be necessary to obtain a computationally efficient algorithm. This conjecture is based on the observation that there exists in fact an easy way to obtain at least the first-order bound (2) without the log-concavity assumption, but with an algorithm that has no hope of being efficiently implemented. As described in Section 2.2, this is possible by running the MYGA algorithm (Allen-Zhu et al., 2018) on O ( T dK 2 ) Kd experts that cover the set of linear classifiers to sufficient precision. The run-time of this approach is prohibitive, because it scales linearly with the number of experts, which is a large polynomial in T .

Techniques. The LinExp3 method of Neu and Olkhovskaya (2020) is based on an adaptation of the classic Exp3 algorithm for regular multi-armed bandits (Auer et al., 2002a). A natural approach would therefore be to replace the Exp3 component in LinExp3 by a method with first-order guarantees for the multi-armed bandit setting, but, as discussed in Section D, this leads to difficulties controlling the variance. Instead of building on Exp3, we therefore follow the perhaps surprising approach of building our algorithm on continuous exponential weights over the probability simplex (van der Hoeven et al., 2018). In particular, our approach is based on a combination of the recently proposed techniques of Ito et al. (2020) for linear bandits with tools designed by Neu and Olkhovskaya (2020) to deal with the contextual case.

Outline. The rest of the paper is organized as follows. After describing the setting in the next section, we state a formal version of the simple first-order bound that can be obtained using the MYGA algorithm (Theorem 2.1). This is followed by Section 3, which states our main results corresponding to the regret bounds in Equations 1 and 2. Section 4 then gives a high-level overview of the proofs, with pointers provided to the details in the appendix. Finally, Section 5 concludes with discussion.

2 Preliminaries

Notation Let ∆ K = { w ∈ R K | w 1 ≥ 0 , . . . , w K ≥ 0 , ∑ K a =1 w a = 1 } denote the ( K -1) -dimensional probability simplex. For any positive semi-definite matrix M ∈ R d × d , ∥ v ∥ M = √ v T Mv denotes the corresponding Mahalanobis norm, and for any positive integer n , we abbreviate [ n ] = { 1 , . . . , n } .

2.1 Setting

We consider the setting of (Neu and Olkhovskaya, 2020), in which there is an interaction between a learner and an unknown environment. This interaction proceeds in rounds indexed by t ∈ [ T ] , such that for each t :

The environment commits to [ K ] parameter vectors θ t, 1 , . . . , θ t,K ∈ R d without revealing any to the learner.

A context vector X t ∈ R d is drawn i.i.d. from some fixed distribution D according to X t ∼ D , and revealed to the learner.

The learner commits to an action A t ∈ [ K ] , and incurs the loss ℓ t ( X t , A t ) , where ℓ t ( X,a ) = ⟨ X,θ t,a ⟩ .

The environment is allowed to randomize its choices of θ t,a . These must be independent from the context X t in round t , but they may depend on previous contexts X s and actions A s for s < t .

We write π t ( a | X t ) for the policy of the learner in round t conditional on observing context X t , so that A t ∼ π t ( X t ) , and we use the following notation for the expected cumulative losses of the algorithm and policy π , respectively:

Let Π be the set of all all stationary deterministic policies π : R d → [ K ] , we define the optimal policy π ∗ as π ∗ = arg min π ∈ Π L π T . Then the learner's goal is to compete with policy π ∗ , as measured by the expected regret:

where the expectation is taken over each X t ∼ D , and any randomness applied by the learner or environment in their respective choices. Using the linearity of the loss functions it can be shown that the optimal policy is always a linear classifier (Neu and Olkhovskaya, 2020):

We may therefore restrict attention to competing with policies of the form

For deriving our technical results, it will be useful to define the filtration F t = σ ( { X s , A s : s ≤ t } ) , and the notations E t [ · ] = E [ ·| F t -1 ] and P t [ · ] = P [ ·| F t -1 ] .

Assumptions Following Neu and Olkhovskaya (2020), we assume that ∥ X t ∥ ≤ σ , ∥ θ t,a ∥ ≤ R and ℓ t ( x, a ) ∈ [ -1 , 1] almost surely. In addition, the covariance matrix Σ = E [ XX T ] of the context distribution is assumed to be positive definite, with smallest eigenvalue λ min (Σ) > 0 .

2.2 An Inefficient Algorithm

A first order bound for our problem can be obtained by instantiating the MYGA algorithm of AllenZhu et al. (2018) for a set of Θ( T K 2 d ) Kd experts that cover the parameter space of policies of the form (3), which is guaranteed to contain the optimal policy π ∗ T :

Theorem 2.1. Suppose that 0 ≤ ℓ t ( a, X t ) ≤ 1 almost surely for all a ∈ [ K ] . Then, by instantiating MYGA with Θ( T K 2 d ) Kd experts, it obtains the following first-order bound for the adversarial linear contextual bandit problem:

Although this provides a quick way to see that first-order bounds are possible, the resulting algorithm is completely impractical, because its run-time is proportional to the number of experts, which grows as a large polynomial in T . The proof, including a more detailed description of the experts, can be found in Appendix A.

3 First- and Second-Order Bounds

In this section we present an algorithm using a novel adaptation of a method developed for the adversarial linear bandit to be suitable for use in the adversarial linear contextual bandit setting. The method proposed is based on a form of continuous exponential weights that has been shown to lead to a first-order bound in the former (Ito et al., 2020). The algorithm allows for optimistic estimates m t,a ∈ R d for the environment's choices θ t,a , which can always be set to 0 when they are not available. We show two types of guarantees. First, in Theorem 3.1, we obtain a second-order regret bound in terms of the cumulative squared error of the estimates m t,a :

Taking m t,a = 0 , this provides a second-order regret bound in terms of the squared losses. Alternatively, m t,a may be estimated using an online regression algorithm, as described by Ito et al. (2020). As our second result, we show in Theorem 3.2 that a first-order bound can be derived for the same algorithm with a different choice of hyperparameters and the assumption that the losses are non-negative.

Algorithm 1 CONTEXTEW

Parameters: γ > 0 , η 1 ≥ . . . ≥ η T > 0 , m 1 , . . . , m T For t = 1 , . . . , T :

Observe X t .

2. Repeat:

Pick Q t from the distribution p t defined in (8), until

where Σ t,a is defined in (9).

Set ˜ Q t = Q t equal to the last sample of Q t , which caused the loop to exit, and choose an arm according to A t ∼ ˜ Q t .

Observe the loss ℓ t ( X t , A t ) and estimate ̂ θ t,a for all a according to (12).

3.1 Algorithm Description

Our full algorithm is shown in Algorithm 1. As it is an adaptation of continuous exponential weights for the contextual bandits setting, we refer to it as CONTEXTEW. It runs a two-stage sampling procedure: after observing context X t , the first stage of the algorithm samples a random policy ˜ Q t ∈ ∆ K , and then the second stage consists of drawing an arm A t randomly from ˜ Q t . The distribution of ˜ Q t is constructed as follows: first we sample a different policy Q t from the exponential weights distribution over the probability simplex with density proportional to

where m s,a is a function that is measurable with respect to F s -1 . The sum ∑ K a =1 q a 〈 X t , ∑ t -1 s =1 ̂ θ s,a 〉 estimates the cumulative loss that the policy q would have incurred if it had been played in all previous rounds. It relies on estimates ̂ θ s,a of the loss vectors θ s,a , which will be defined below, and a time-varying learning rate η t > 0 , which is hyperparameter of the algorithm. The normalized density function corresponding to the weights in (7) is:

Following Ito et al. (2020), we then introduce a rejection sampling step (6) to reduce the variance, which is based on the following covariance matrices Σ t,a corresponding to Q t :

so that ˜ Q t ends up being sampled according to the following truncated exponential weights density:

with truncation level hyperparameter γ > 0 . We will show that all Σ t,a are invertible, as are their analogues in which Q t is replaced by ˜ Q t :

It remains to specify our estimators for θ t,a , which are defined as follows:

These estimates can be shown to be unbiased:

3.2 Results

We instantiate CONTEXTEW with adaptive learning rates η t . For our second-order result, these are defined in terms of the empirical counterpart to V t : ̂ V t = ∑ t s =1 ⟨ X s , θ s,A s -m s,A s ⟩ 2 , and we abbreviate G t = 8 √ ̂ V t -1 ln(2 T 2 ) + 144 ln 2 T +176ln T. Then we set

This leads to the following second-order bound:

Theorem 3.1 (Second-Order) . Suppose D has a log-concave density. Then, for γ = 4log(10 dKT ) , η t as in (13) and any F t -1 -measurable estimates m t , the expected regret of CONTEXTEW is at most R T = ˜ O ( K √ dV T ) .

To tune η t adaptively for our first-order bound, we define it using the algorithm's empirical cumulative loss ̂ L t = ∑ t s =1 ℓ t ( X s , A s ) , which acts as a self-confident empirical estimate of L ∗ T . We further abbreviate and then set

This leads to the following first-order bound:

Theorem 3.2 (First-Order) . Suppose that D has a log-concave density and that 0 ≤ ℓ t ( a, X t ) ≤ 1 almost surely for all a ∈ [ K ] . Then, for γ = 4log(10 dKT ) , η t as in (15) and m t = 0 , the expected regret of CONTEXTEW is at most R T = ˜ O ( K √ dL ∗ T ) .

3.3 Computational Efficiency

The two computational bottlenecks in the algorithm are the cost of sampling from the output distribution p t ( q | X t ) and computation of the covariance matrices Σ t,a in each round.

Due to the log-linearity of our method, there exists several practical methods of sampling. As mentioned in Ito et al. (2020), one can employ the methods of Lovász and Vempala (2007), which was shown in Lovász and Vempala (2006) to enjoy a bound of O ( K 4 log(1 /ϵ )) (where ϵ is a bound on the total variation distance between the output distribution and the target), but this still requires knowledge of a density dominating the target distribution on all but a set with total starting mass ≤ ϵ/ 2 . In Narayanan and Rakhlin (2017), a method is developed for general log-concave distributions which, specialized to log-linear distributions (and without additional assumptions on the initial distribution) yields an O ( K 3 ν 2 +log(1 /ϵ )) method when the geometry admits a ν -self concordant barrier. Since there always exists a K -self-concordant barrier for a K -dimensional convex body, and thus the running time of this method for our problem is O ( K 5 +log T ) up to a precision ϵ ∼ 1 T β for some

β > 0 . As referred to in Ito et al. (2020), the covariance matrix Σ t,a is computable in O (( d/ϵ ) O (1) ) sampling steps drawing upon the results of Lovász and Vempala (2007).

4 Analysis

In this section we provide the analysis of CONTEXTEW from which Theorems 3.1 and 3.2 follow. Throughout the analysis, we will be extensively using the following property of log-concave distributions:

Lemma 4.1. If x follows a log-concave distribution p over R d and E [ xx T ] ≼ I , we have, for any α ≥ 0 :

This result was proven in Lemma 1 in Ito et al. (2020), and also follows from Lemma 5.7 in Lovász and Vempala (2007).

First, we need to introduce some notation which will be useful for the reduction to the linear bandit setting and for the accompanying proofs. We denote z a ( q, x ) = q a x and z ( q, x ) = ( z 1 ( q, x ) , . . . , z K ( q, x )) T . We also define Σ t = diag a ∈ [ K ] (Σ a,t ) as a block diagonal arrangement of the covariance matrices per arm. Using this notation, the distribution of the sampling algorithm (10) may be rewritten as

Let ˜ Q t ( x ) ∼ ˜ p t ( q | x ) , Q t ( x ) ∼ p t ( q | x ) and ˜ Z t ( x ) = z ( ˜ Q t ( x ) , x ) , Z t ( x ) = z ( Q t ( x ) , x ) , Z ∗ ( x ) = z ( π ∗ ( x ) , x ) . And we denote the aggregated loss parameter θ t = ( θ 1 , . . . , θ K ) T and its estimate ̂ θ t = ( ̂ θ 1 , . . . , ̂ θ K ) T . Then we can express the regret as follows:

The crucial observation is that the log-concavity of the distribution of Z t ( X t ) follows from that of the distribution of X t :

Lemma 4.2. Suppose z ( q, x ) = ∑ a q a φ ( x, a ) for φ ( x, a ) = ( ¯ 0 ⊺ , . . . , x ⊺ , · · · ) such that x is on the da 'th co-ordinate and Q ( x ) ∼ p t ( ·| x ) for p t ( ·| x ) defined in (8). If X ∼ p X ( · ) and p X ( · ) is log-concave and Z ( x ) = z ( Q t ( x ) , x ) , then Z ( X ) also follows a log-concave distribution.

The proof of this result is a rather straightforward computation of the density of Z t ( X t ) and can be found in Appendix C. To proceed, we write regret as a sum of two terms

Having shown that Z t ( X t ) is log-concave, and since the log-concavity is preserved under linear transformations, for y = Σ -1 / 2 T t Z t ( X t ) we can see that E [ yy T ] = I , and thus by Lemma 4.1 it immediately follows that the probability that (6) is not satisfied is small for a proposed choice of γ = 4log(10 dKT ) :

Using this observation, we show that the first term of (19) is just O (1) , which is formally proved in Lemma C.2 in the appendix.

To control the second term of the regret decomposition (19), consider the reduction of the contextual bandit problem to a combination of auxiliary online learning problems that are defined separately for each context, as proposed in Neu and Olkhovskaya (2020), Lemma 3. More details and a full proof can be found in Appendix C.

Lemma 4.3. Let π ∗ be any fixed stochastic policy and let X 0 ∼ D be a sample from the context distribution independent from F T . Suppose that p t ∈ F t -1 , such that p t ( ·| x ) is a probability density with respect to Lebesgue measure with support ∆ K and let Q t ( x ) ∼ p t ( ·| x ) . Then,

To see why this would be useful further in the proof, we interpret the right-hand side of (20) as follows. Consider the online learning problem for a fixed x with the decision set to be ∆ K and losses ℓ t ( x, q ) = ⟨ z ( q, x ) , ̂ θ t ⟩ and consider running a version of a contextual bandit problem with a fixed context x , such that the probability of an action q defined as in Equation 8, so p t ( q | x ) ∝ exp ( -η t ∑ K a =1 q a 〈 x, ∑ t -1 s =1 ̂ θ s,a 〉) . Then, the regret for the fixed x against π ∗ ( x ) can be written as:

Then it is easy to see that the right-hand side of (20) is equal to E [ ̂ R T ( X 0 ) ] . Thus, we first show a bound on ̂ R T ( x ) that holds almost surely for any x and then take an expectation with respect to X 0 . We control the regret ̂ R T ( x ) by following the general schema of the optimistic mirror descent analysis developed in (Rakhlin and Sridharan, 2013; Ito et al., 2020). With this analysis, we get the following bound for any x ∈ X :

Lemma 4.4. Assume that η t +1 ≤ η t for all t , let q 0 be a uniform distribution over [ K ] and ψ ( y ) = exp( y ) -y -1 . Then, the regret ̂ R T ( x ) of CONTEXTEW almost surely satisfies

We place the derivation of the this bound in the appendix. The crucial ingredient is to show that the square of the estimated loss can be bounded by the square of the true loss. Using the definition of θ t , denoting V ar t = tr ( ˜ Σ -1 t Z t ( X 0 ) Z t ( X 0 ) T ˜ Σ -1 t Z t ( X t ) Z t ( X t ) T ) , we get

As additional corollary of the concentration result for log-concave random variables, we can show the following relation between matrices Σ t and ˜ Σ t :

which we prove in Lemma C.2 in the appendix. Then we can show that, almost surely:

where the first inequality follows from (23) and the second inequality is immediate from (23) and the fact that for symmetric positive definite matrices A ⪰ B follows from B -1 ⪰ A -1 . The last inequality follows from (6) in the CONTEXTEW. So, from (22) and (35), we get

which, as we stated above, is the key step to prove Theorem 3.1.

First-order regret bound To prove result of Theorem 3.2, we show that the bound in the Theorem 3.1 can instantiated to obtain a first-order regret bound with a different choice of the learning rate η t . Going along the same lines with regard to the concentration of ̂ L t as for ̂ V t , by setting m t = ¯ 0 and noticing that then V T ≤ L T we get

Since R T = L t -L ∗ T , by solving the quadratic inequality with respect to L ∗ T , we get that L T ≤ L ∗ T + ˜ O ( K √ d ) , yielding the final bound.

5 Discussion

In conclusion, by applying the approach of (Ito et al., 2020) we have constructed the first scheme achieving ˜ O ( K √ dL ∗ T ) regret with a runtime of O ( ( K 5 +log T ) · g Σ ) , where g Σ is the time taken to construct the covariance matrix per round - a potentially large polynomial improvement over the O ( T Kd ) runtime of MYGA. The application of linear bandit algorithms to the contextual bandit problem constitutes, to the best of our knowledge, a novel approach. In doing so we've found a number of positive aspects, including efficiency, but also the direct applicability of other properties enjoyed by the algorithm such as second order bounds (Ito et al., 2020).

Our approach is based on reducing the linear contextual bandit problem to a linear bandit problem, as opposed to a multi-armed bandit problem as in (Neu and Olkhovskaya, 2020). While the specifics of this reduction heavily relied on the joint log-concavity of the context distributions and the exponentialweights posterior over the simplex of actions, we wonder if such approaches can be successfully applied to achieve other types of improvements for linear contextual bandits. In particular, it is curious to what extent other recent advances in the linear bandit problem can be translated to the linear contextual bandit setting. Note that, while the truncation step in Algorithm 1 has an insignificant computational cost as the condition is satisfied with probability O (1 -1 /T ) , it can be removed by paying a log(1 /λ min (Σ)) multiplicative term in the regret by implementing additional exploration with probability 1 /T . It is natural to ask whether or not approaches based on other instantiations of online mirror descent would also yield first-order bounds, and possibly improve the dependence on K . The answer is not obvious: for an example of how a naive application of an instantiation of FTRL fails to achieve a first-order bound, see Appendix D.

A relevant question pertains to whether or not such an application of algorithms for linear bandits is necessary at all, but standard approaches such as direct adaptation of Exp3, and first-order adaptations thereof such as GREEN Allenberg et al. (2006) do not seem to give the desired result.In addition, thresholding the worst performing arms inevitably biases the loss estimator due to undersampling of those arms for which the threshold has been applied, and the resulting additional bias term picked up in the regret scales with 1 /λ min (Σ t,a ) , which may be arbitrarily large. Another standard approach of finding an optimistic estimator yielded no fruit during the course of this study due to the lack of the existence of such an estimator without saving all previous losses explicitly.

Our algorithm achieves the regret bound O ( K √ dV T ) , while the worst case guarantee of LINEXP3 of Neu and Olkhovskaya (2020) is O ( √ dKT ) . This discrepancy is not surprising as the Algorithm 1 of Ito et al. (2020) scales as O ( n √ T ) ( n being the dimension of the action space for the linear bandit), which arises from the deployment of continuous exponential weights. MYGA achieves the same O ( K √ dL ∗ T ) bound due to the number of experts needed to cover the joint set of additive loss parameters. It is worth here emphasising that no known algorithm achieves a better dependence on K than O ( K √ dL ∗ T ) for the linear adversarial contextual bandit problem. Meanwhile, if the linear bandit is played on the n -simplex, an improvement to √ nT is possible. For further discussion of this point, see Section 28.5 of Lattimore and Szepesvári (2020). It is thus still unclear whether or not the extra factor of √ K is necessary if one aims for a first-order bound.

An additional point is that while the MYGA algorithm Allen-Zhu et al. (2018) allows for adversarially chosen contexts, the analysis of MYGA for our setting relies heavily on the assumption that contexts are drawn i.i.d. at each iteration. A natural question is then whether or not a similar result is achievable in the adversarial context case. It is known that achieving sub-linear regret is not possible even for full-information online learning of one-dimensional threshold classifiers when both contexts and losses are adversarial (Ben-David et al., 2009; Syrgkanis et al., 2016), which renders sub-linear regret similarly impossible to guarantee for the even harder setting that we consider in this paper. However, we do conjecture that we could overcome the assumption that the distribution is known or that we can sample from it by employing a more elaborate algorithm to estimate the distribution from the data. Indeed, it is not obvious if the distributional assumption of a lower bound to the covariance matrix eigenvalues is entirely necessary, since the regret does not depend on this.

Lastly, it would be an interesting challenge to see if a high-probability regret bound could be obtained in the form stated in the COLT 2017 open problem Agarwal et al. (2017) for this setting, but since a high-probability O ( √ T ) has not yet been proved for the problem here considered, the latter may be more worthy of focus in the short term.

6 Acknowledgments

Tim van Erven and Jack Mayo were supported by the Netherlands Organization for Scientific Research (NWO) under grant number VI.Vidi.192.095. Gergely Neu was supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (Grant agreement No. 950180). Chen-Yu Wei would like to acknowledge the support from Simons-Berkeley Research Fellowship.

References

Abbasi-Yadkori, Y., Pal, D., and Szepesvári, Cs. (2011). Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems (NIPS) .

Agarwal, A., Bird, S., Cozowicz, M., Hoang, L., Langford, J., Lee, S., Li, J., Melamed, D., Oshri, G., Ribas, O., Sen, S., and Slivkins, A. (2016). Making contextual decisions with low technical debt.

Agarwal, A., Hsu, D., Kale, S., Langford, J., Li, L., and Schapire, R. (2014). Taming the monster: A fast and simple algorithm for contextual bandits. In Xing, E. P. and Jebara, T., editors, Proceedings of the 31st International Conference on Machine Learning , volume 32 of Proceedings of Machine Learning Research , pages 1638-1646, Bejing, China. PMLR.

Agarwal, A., Krishnamurthy, A., Langford, J., Luo, H., and Schapire, R. E. (2017). Open problem: First-order regret bounds for contextual bandits. In Kale, S. and Shamir, O., editors, Proceedings of the 2017 Conference on Learning Theory , volume 65 of Proceedings of Machine Learning Research , pages 4-7. PMLR.

Allen-Zhu, Z., Bubeck, S., and Li, Y. (2018). Make the minority great again: First-order regret bound for contextual bandits. In Dy, J. and Krause, A., editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 186-194. PMLR.

Allenberg, C., Auer, P., Györfi, L., and Ottucsák, G. (2006). Hannan consistency in on-line learning in case of unbounded losses under partial monitoring. In Balcázar, J. L., Long, P. M., and Stephan, F., editors, Algorithmic Learning Theory , pages 229-243, Berlin, Heidelberg. Springer Berlin Heidelberg.

Auer, P. (2002). Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3:397-422.

Auer, P., Cesa-Bianchi, N., and Fischer, P. (2002a). Finite-time analysis of the multiarmed bandit problem. Machine learning , 47(2):235-256.

Auer, P., Cesa-Bianchi, N., Freund, Y., and Schapire, R. E. (2002b). The nonstochastic multiarmed bandit problem. SIAM J. Comput. , 32(1):48-77.

Bartlett, P., Dani, V ., Hayes, T., Kakade, S., Rakhlin, A., and Tewari, A. (2008). High-probability regret bounds for bandit online linear optimization. pages 335-342.

Ben-David, S., Pál, D., and Shalev-Shwartz, S. (2009). Agnostic online learning.

Beygelzimer, A., Langford, J., Li, L., Reyzin, L., and Schapire, R. (2011). Contextual bandit algorithms with supervised learning guarantees. In Gordon, G., Dunson, D., and Dudík, M., editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , volume 15 of Proceedings of Machine Learning Research , pages 19-26, Fort Lauderdale, FL, USA. PMLR.

Bubeck, S., Li, Y., Luo, H., and Wei, C.-Y. (2019). Improved path-length regret bounds for bandits. In Beygelzimer, A. and Hsu, D., editors, Proceedings of the Thirty-Second Conference on Learning Theory , volume 99 of Proceedings of Machine Learning Research , pages 508-528. PMLR.

Chu, W., Li, L., Reyzin, L., and Schapire, R. (2011). Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 208-214.

Foster, D. J. and Krishnamurthy, A. (2021). Efficient first-order contextual bandits: Prediction, allocation, and triangular discrimination. volume 23, page 18907 - 18919.

Freedman, D. A. (1975). On Tail Probabilities for Martingales. The Annals of Probability , 3(1):100 118.

Ito, S., Hirahara, S., Soma, T., and Yoshida, Y. (2020). Tight first- and second-order regret bounds for adversarial linear bandits. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems , volume 33, pages 2028-2038. Curran Associates, Inc.

Lattimore, T. and Szepesvári, C. (2020). Bandit Algorithms . Cambridge University Press.

Lee, C.-W., Luo, H., Wei, C.-Y., and Zhang, M. (2020). Bias no more: high-probability datadependent regret bounds for adversarial bandits and mdps. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems , volume 33, pages 15522-15533. Curran Associates, Inc.

Li, L., Chu, W., Langford, J., and Schapire, R. E. (2010). A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web , pages 661-670. ACM.

Lovász, L. and Vempala, S. (2006). Hit-and-run from a corner. SIAM Journal on Computing , 35(4):985-1005.

Lovász, L. and Vempala, S. (2007). The geometry of logconcave functions and sampling algorithms. Random Structures & Algorithms , 30(3):307-358.

Narayanan, H. and Rakhlin, A. (2017). Efficient sampling from time-varying log-concave distributions. Journal of Machine Learning Research , 18(112):1-29.

Neu, G. (2015). First-order regret bounds for combinatorial semi-bandits. In Conference on Learning Theory , pages 1360-1375. PMLR.

Neu, G. and Olkhovskaya, J. (2020). Efficient and robust algorithms for adversarial linear contextual bandits. In Proceedings of the 33rd Annual Conference on Learning Theory (COLT 2020) , pages 3049-3068.

Rakhlin, A. and Sridharan, K. (2013). Optimization, learning, and games with predictable sequences. In Proceedings of the 26th International Conference on Neural Information Processing Systems Volume 2 , NIPS'13, page 3066-3074, Red Hook, NY, USA. Curran Associates Inc.

Stoltz, G. (2005). Incomplete information and internal regret in prediction of individual sequences . PhD thesis, Université Paris Sud-Paris XI.

Syrgkanis, V., Krishnamurthy, A., and Schapire, R. (2016). Efficient algorithms for adversarial contextual learning. In Balcan, M. F. and Weinberger, K. Q., editors, Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pages 2159-2168, New York, New York, USA. PMLR.

Tewari, A. and Murphy, S. A. (2017). From ads to interventions: Contextual bandits in mobile health. In Mobile Health - Sensors, Analytic Methods, and Applications .

van der Hoeven, D., van Erven, T., and Kotłowski, W. (2018). The many faces of exponential weights in online learning. In Conference On Learning Theory , pages 2067-2092.

A First-order Bound by Reduction to MYGA

Proof. The MYGA algorithm of Allen-Zhu et al. (2018) competes with a class of experts E , where each expert e ∈ E provides a stochastic prediction ξ e t ∈ ∆ K in each round t . It provides the following expected regret bound with respect to the best expert:

Losses for the arms can be adversarial, and are assumed to take values in [0 , 1] .

We will instantiate the experts to cover the parameter space { β ∈ R K × d : max a ∥ β a ∥ ≤ RT } of potentially optimal parameters for deterministic policies of the form (3), which we know must contain the optimal policy π ∗ T with corresponding parameters β ∗ = E [ ∑ T t =1 θ t ] . The covering number for a ball of radius RT at precision ϵ > 0 is between ( RT ϵ ) d and ( 3 RT ϵ ) d , so by taking the Cartesian product of this covering with itself K times we can cover all β with ( RT ϵ ) Kd ≤ | E | ≤ ( 3 RT ϵ ) Kd points β 1 , . . . , β | E | . Let ¨ β ∈ { β 1 , . . . , β | E | } be the closest point in the covering to the optimal parameters β ∗ . Then its expected approximation error can be upper bounded as follows:

Adding this to (25), instantiated with | E | ≤ ( 3 RT ϵ ) Kd , and choosing ϵ = dK 2 2 completes the proof.

B Auxiliary lemmas

To ensure that step 2 in CONTEXTEW is defined correctly, we show that the matrix Σ t is full rank: Lemma B.1. Let the distribution of X t be such that λ min ( E [ X t X T t ]) > 0 . Then, we can show

Proof. To show that Σ t,a is full rank, it suffices to show that there is no v ∈ R d such that v T Σ t,a v = 0 . Suppose, to the contrary, that such a v does exist. Then 0 = v T E t [ Q 2 t,a ( X t ) X t X T t ] v = E t [ Q 2 t,a ( v T X t ) 2 ] , which implies that Q t,a v T X t = 0 almost surely. Since Q t,a > 0 almost surely, it follows that in fact v T X t = 0 almost surely and therefore 0 = E t [ ( v T X t ) 2 ] = v T E t [ X t X T t ] v . But this contradicts our assumption that λ min ( E t [ X t X T t ]) > 0 .

We will use a simple corollary of Freedman's inequality Freedman (1975) that was introduced in Lemma 2 in Bartlett et al. (2008):

Lemma B.2. Let Y 1 , . . . , Y t be a martingale difference sequence with respect to a filtration F 1 ⊂ · · · ⊂ F t such that E [ Y s |F s ] = 0 . Suppose that Y s ≤ b holds almost surely. Then with probability at least 1 -δ we have ∑ t s =1 Y s ≤ 2 max { 2 √ ∑ t s =1 E [ Y 2 s |F s ] , b √ ln(1 /δ ) } √ ln(1 /δ ) .

for any a ∈ [ K ] , and consequently

In our analysis, we will use some results from Ito et al. (2020). We will make use of the following concentration property of log-concave distributions, Lemma 1 from Ito et al. (2020):

Lemma B.3. If y follows a log-concave distribution p over R d and E y ∼ p [ yy T ] ≼ I , we have

for arbitrary α ≥ 0 .

The lemma presented below introduces a property that will be instrumental in analyzing the variance of loss estimates, presented as Lemma 6 in Ito et al. (2020):

Lemma B.4. If y follows a log-concave distribution over R , and if E [ y 2 ] ≤ 1 / 100 , we have

C Proof of Theorem 3.1

The proof of Theorem 3.1 proceeds in a sequence of lemmas. First, we need to show that the distribution of Z t ( X t ) is log-concave for all t ∈ [ T ] , and after we follow the analysis of Algorithm 1 of Ito et al. (2020), bounding both components of (19) taking into account the required alterations to incorporate contextual structure

Lemma C.1. Suppose z ( q, x ) = ∑ a q a φ ( x, a ) for φ ( x, a ) = ( ¯ 0 ⊺ , . . . , x ⊺ , · · · ) such that x is on the da 'th co-ordinate and Q ( x ) ∼ p ( ·| x ) for p ( ·| x ) log-concave. If X ∼ p X ( · ) and p X ( · ) is log-concave and Z ( x ) = z ( Q ( x ) , x ) , then Z ( X ) also follows a log-concave distribution.

Proof. Assume that | x i | > 0 for all i ∈ [ d ] . Set ( z 1 , . . . , z K -1 , x ) = h ( q 1 ¯ 1 , . . . , q K -1 ¯ 1 , x ) , where h : R dK → R dK and z i = ( . . . , z j i , . . . ) T for each i ∈ { 1 , . . . , K -1 } . Thus z j i = h i ( q i , x j ) = q i ( x j ) and h K ( x ) = x . The Jacobian of h -1 ( z 1 , . . . , z K -1 , x ) can be expressed as the block matrix

where Λ x ∈ R ( K -1) × ( K -1) is diagonal with (Λ x ) ii = 1 x i and Γ z,x ∈ R ( K -1) × d with (Γ z,x ) ij = -z j i ( x j ) 2 . Since J ( h -1 ( z 1 , . . . , z K -1 , x )) is upper-triangular, det( J ( h -1 ( z 1 , . . . , z K -1 , x ))) = ( ∏ d i =1 1 x i ) K -1 . The joint distribution of Z i and X can thus be written

with the joint distribution between Q and X of the form

where ( ψ ( z, x, φ )) i = ∑ K -1 a =1 z i a x i φ ( x, a ) i + ( 1 -∑ K -1 a =1 z i a x i ) φ ( x, K ) i has been defined for readK -1

ability. We can reabsorb the factor ( ∏ d i =1 1 x i ) in the denominator to rewrite the normalization constant as a in terms of the random variable Z ( x ) , and so

Define a new function g : R d × K → R d × K such that y = g ( z 1 , . . . , z K -1 , x ) = ( . . . , g i ( z 1 , . . . , z K -1 , x ) , . . . ) T , where for i ∈ [1 , K -1] , g i ( z 1 , . . . , z K -1 , x ) = z i and

g K ( z 1 , . . . , z K -1 , x ) = ( . . . , ( 1 -1 x i ∑ K -1 a =1 z i a ) x i , . . . ) . Then for i ∈ { 1 , . . . , K -1 } , g -1 i ( y ) = y i and g -1 K ( y ) = ∑ K a =1 y a . The determinant det( J ( g -1 ( y ))) = 1 , so

Since both p X and e -η ⟨ y, ̂ Θ t -1 ⟩ ∫ y ′ ∈ Y ( y ) e -η ⟨ y ′ , ̂ Θ t -1 ⟩ dy ′ are both log-concave, the lemma follows.

Having shown the log-concavity of Z ( X t ) , we may safely proceed.

We state the analog of Lemma 4 in Ito et al. (2020) adapted to our setting, leading to a bound on the first term of (19) as well as providing a useful relation between Σ t and ˜ Σ t .

Lemma C.2.

and we have

Proof. From definition of ˜ p t , for any x ∈ X , θ ∈ Θ , we have

where δ = P t [ ∥ Z t ( X t ) ∥ 2 Σ -1 t > dKγ 2 ] . Plugging this into the l.h.s. of (28) yields

Since the distribution of Z t ( X t ) is log-concave (Lemma C.1), we can apply Lemma 1 of Ito et al. (2020) to x = Σ -1 / 2 t Z t ( X t ) . The assumptions of Lemma 1 of Ito et al. (2020) hold since we have E [ xx T ] = I and since log-concavity is preserved under linear maps. Using Lemma 1 of Ito et al. (2020), we have

where the last inequality follows from γ ≥ 4 log(10 dKT ) , which obtains (28). We proceed to showing (29). For any y ∈ R dK , we have

Since this holds for all y ∈ R dK and 1 1 -δ ≤ 4 3 , the second inequality in (29) holds. Furthermore, we have

where the last inequality follows from Cauchy-Schwarz:

The right-hand side of (30) can be bounded using Lemma B.3 as follows:

Combining (31) and (30) we get the first inequality of (29).

Lemma C.3. Let π ∗ be any fixed stochastic policy and let X 0 ∼ D be a sample from the context distribution independent from F T . Suppose that p t ∈ F t -1 , such that p t ( ·| x ) is a probability density with respect to Lebesgue measure with support ∆ K and let Q t ( x ) ∼ p t ( ·| x ) . Then,

Proof. For any t , we have

Then, we prove the almost sure regret bound for any x and then take an expectation over X 0 . We further proceed with an adaptation of the analysis of the continuous exponential weights algorithm, which was stated in Ito et al. (2020) as Lemma 16, but we include it here for the clarity. Let ψ ( y ) = exp( y ) -y -1 . For any x ∈ X , we show the following :

Lemma C.4. Assume that η t +1 ≤ η t for all t , let q 0 be a uniform distribution over [ K ] and ψ ( y ) = exp( y ) -y -1 . Then, the regret ̂ R T ( x ) for any x ∈ X of CONTEXTEW almost surely satisfies

Proof. Note that we can write ̂ R T ( x ) as

Define W t ( x ) = ∫ ∆ K w t ( q | x ) dq , u t ( q | x ) = exp ( -η t ∑ a q a 〈 x, ∑ t s =1 ̂ θ s,a 〉) , U t ( x ) = ∫ ∆ K u t ( q | x ) dq and v t ( q | x ) = exp ( -η t +1 ∑ a q a 〈 x, ∑ t s =1 ̂ θ s,a 〉) , V t ( x ) = ∫ ∆ K v t ( q | x ) dq . We have

Taking the logarithm of both sides, we get

where we used the inequality log(1 + x ) ≤ x for x > -1 .

using Jensen's inequality. It holds that

Then, we get

T ∑ t =1 ∫ ∆ K p t ( q | x ) 〈 z ( q, x ) , ̂ θ t 〉 dq ≤ T ∑ t =1 1 η t ( log V t -1 ( x ) U t ( x ) + ∫ ∆ K p t ( q | x ) ψ ( -η t 〈 z ( q, x ) , ̂ θ t -m t 〉 ) dq ) . Noting that V 0 = U 0 , we have

To bound the first term, we use that η t +1 ≤ η t and an additional application of Jensen's inequality:

Set Q π ∗ ( x ) := { (1 -1 T ) π ∗ ( x ) + 1 T q | q ∈ ∆ K } , and denote q 0 as the uniform distribution over K arms. We then have

where the first inequality constitutes a change of variables and the second follows from Jensen's bound. After rearranging and taking the logarithm, we get

Combining everything together, we get

From Lemma 4.3 and Lemma 4.4, we get a bound on the second term of (19):

where the first inequality follows from ℓ t ≤ 1 and (29), the second is immediate from (29) and the fact that for symmetric positive definite matrices A ⪰ B follows from B -1 ⪰ A -1 . The third

inequality follows from the truncation in the algorithm and the last is immediate from plugging in the definition of η t . So, by applying Lemma B.4 and (35), we get:

For the second term of (34), we simply get from -1 ≤ ℓ t ≤ 1 and ̂ θ t is unbiased:

The expression that we use for the learning rate is the following:

where G t = 8 √ 2 ̂ V t -1 ln T +144ln 2 T +176ln T . We show that with probability at least 1 -δ the following holds for all t ∈ [ T ] :

Let Y s = E s [ ( ℓ t ( A t , X t ) -X T t m t,A t ) 2 ] -( ℓ t ( A t , X t ) -X T t m t,A t ) 2 . Then, Y s ≤ 4 almost surely, since ℓ t ( A t , X t ) -X T t m t,A t ≤ 2 . Similarly we bound the second moment of Y s , using Jensen's inequality:

By Lemma B.2, the following holds for some δ ′ ∈ (0 , 1) :

Note that this inequality i can be rearranged as

Then, taking a union bound over t ∈ [ T ] and taking δ = δ ′ /T , we get that (39) holds for all t ∈ [ T ] . Let E T be an event that for all t ∈ [1 , T ] , (39) holds with δ = 1 /T . From (37), (38), and the choice of η t , we get:

T

which implies the result of Theorem 3.1. In the equation above, η ′ T = (100 dKγ 2 + d ( V t -1 +1+ G ′ t ))) -1 / 2 and G ′ t = 8 √ 2 V t -1 ln T +144ln 2 T + 176ln T . In line 4 we used that E [1 /η T ] ≤ E [1 /η ′ ] by Jensen's inequality to show that

Proof of Theorem 3.2 As it was done in the proof of Theorem 3.1 we control the deviation of the learning rate

where H t is as defined in (14). Using Lemma B.2, we show that with probability at least 1 -δ the following holds for all t ∈ [ T ] :

D

E

s

D

. Then, s,A

, θ

X

-⟨

]

s

⟩

⟨

)

2

almost surely and by Jensen's inequality

]

+2

⟩

2

E

2

s

s

s

=

[

D

E

s

2

s

[

⟨

=

X

s

E

, θ

s,A

[

s

(

E

s

[

X

s

, θ

s

s,A

s

⟩

]

s

⟩

-⟨

X

s

, θ

s,A

s

⟩

By Lemma B.2, the following holds for some δ ′ ∈ (0 , 1) :

which can be rearranged as

Then, taking a union bound over t ∈ [ T ] and taking δ = δ ′ /T , we get that (42) holds for all t ∈ [ T ] . Let E T be an event that for all t ∈ [1 , T ] , (42) holds with δ = 1 /T . From (37), (38), the choice of η t , m t = ¯ 0 and since 0 ≤ ℓ t ≤ 1 , we get:

where in the equation above, η ′ t = (100 dKγ 2 + d ( L t -1 + 1 + H ′ t -1 ))) -1 / 2 and H ′ t = 8 √ 2 L t -1 ln T +40ln T + 72ln T . By solving the quadratic equation over L ∗ T , we obtain the statement of the theorem.

]

s

s,A

s

s

s

s,A

s

[

⟨

X

, θ

[

(

⟨

X

, θ

⟩

2

]

E

)

2

]

4

E

t

[

ℓ

t

(

X

t

, A

t

)]

.

D On the difference between LINEXP3 and CONTEXTEW

Consider the LINEXP3 algorithm of Neu and Olkhovskaya (2020), that draws actions after observing the context X t with probability

where w t ( X t , a ) = exp ( -η ∑ t -1 s =0 ⟨ X t , ̂ θ s,a ⟩ ) and using the estimator

where S t,a = E t [ π t ( a | X t ) X t X T t ] . Since LINEXP3 uses implicit exploration with probability γ , λ min ( S t,a ) ≥ λ min (Σ) γ K . But then, setting γ = 0 , S t,a is still invertible as no actions have π t ( a | X t ) = 0 . But still, the smallest eigenvalue λ min ( S t,a ) can be arbitrary small. Then, the analysis of the variance term in LINEXP3 looks as:

We can define V ar ′ t for LINEXP3 in direct analogy to V ar t for CONTEXTEW above, which gives (almost surely):

which can be arbitrary large.

Meanwhile, the smallest eigenvalue λ min (Σ t,a ) can be arbitrary small too. But, as we showed above in the analysis of CONTEXTEW, V ar t is bounded by dKγ 2 because of the log-concavity of Z t ( X t ) and step (6) of CONTEXTEW.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the adversarial linear contextual bandit setting, an online learning problem with $K$ arms, stochastic contexts in $\\mathbb{R}^d$, and adversarial rewards (which are assumed to be a linear function of the context for each arm). Previous work proves a worst-case regret guarantee that is $\\sqrt{KdT}$. They obtain regret guarantees which instead of scaling with $\\sqrt{T}$ can scale with other quantities that are smaller: a certain \"second-order\" quantity called $V_T$, and a first-order or (small loss) bound of $L_T^\\star$.\n\nThis is accomplished using a new algorithm called ContextEW which is a continuous exponential weight style algorithm, inspired by previous work of [Ito et al. '20]. They show that it achieves the aforementioned regret guarantees and can be efficiently implemented.\n\nSTRENGTHS:\n- The paper is generally well written and explains the problem statement and results well.\n- The paper provides a new algorithm for adversarial linear contextual bandits which achieves first and second order bounds, a standard object of study in bandit literature.\n\nWEAKNESSES:\n- It is a bit unclear from me how much of the work is original, or whether it is a straightforward exercise to modify the results from [Ito et al. '20] and [Neu-Olkhovskaya '20]. Perhaps this can be made clearer in the proof sketch and technical sections to delineate what techniques/results are new.\n- The discussion on computational complexity could be improved (see questions below).\n- The key intuitions for the algorithm and analysis are unclear to me (see questions below).\n- The results rely on the assumption that the distribution of the contexts is log-concave (which is a limitation that comes from building upon previous work).\n\nQUESTIONS:\n1. In the equation after line 131, what exactly is $\\mathbb{E} \\theta_{t,a}$? The $\\theta$ are assumed to be adversarially selected, under some choices of actions selected by a learner, so what does it mean to take the expectation here?\n2. What is the intuition for the truncation/rejection sampling step? It is mentioned that it is used to reduce the variance. What would the final bound look like if there was no truncation/rejection sampling step?\n3. Typo: in (10), $P_t$ should be $\\mathbb{P}_t$ to be consistent with the rest of the notation.\n4. Computational complexity: \n    - I don't really understand how one should calculate $\\tilde{\\Sigma}$. This requires a sampling oracle to $\\mathcal{D}$ in order to use the [Lovasz-Vempala '07] results right?  My understanding is that one can try to approximate (9) using draws from $\\mathcal{D}$. Can the authors explain how to (approximately) calculate  $\\tilde{\\Sigma}_{t,a}$? \n    - A follow up to the previous question is how does the error in sampling/computing the covariances propagate to the final error bound? \n    - One can try to pick $\\epsilon = 1/\\mathrm{poly}(T)$. Is computing the covariance still \"computationally efficient\" since the computational complexity seems to scale polynomially with the total length of the problem $T$?\n5. At a high level, what is the motivation for studying second-order bounds? When would we expect these to be small in practice?\n6. Questions about proof details:\n    - typo: I believe in Lemma C.1 there should not be a subscript of $Q_t$ in line 447.\n    - In the last inequality after line 468, where did the integration over $x$ go?\n    - I'm a bit confused about why the quantity $m_t$ appears in the Proof of Lemma C.4. For example, in the first equality after 495, how does the $-m_t$ appear? I'm not able to follow this. Is there some relationship between $\\hat{\\theta}_t$ and $m_t$ that is being exploited in this proof?\n    - I didn't follow (38), how is the unbiasedness of $\\hat{\\theta}_t$ used?\n7. It would be nice if the relevant lemmas from [Ito et al. '20] were restated in the appendix, to make the proof self contained. For example, Lemma 6 of their paper is used in the proof.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission considers semi-adversarial contextual bandit setting --- where d-dimensional contexts x_t are drawn iid, where some opponent picks K loss functions that are linear in the contexts at the beginning of each round, before the actual context x_t is drawn, and where he learner picks one of the K loss functions and suffers the corresponding linear loss. This setting was introduced by Neu and Olkhovskaya (COLT 2020), who provided 'zero-order' regret bounds of order \\sqrt{K d T}. The present submission refines the regret bounds into first-order (for non-ngative losses, as usual) and second-order regret bounds, involving respectively the cumulative loss of the best static policy and the expected sum of instantaneous variances of the rewards obtained by the algorithms (and generalizations thereof). This is achieved by considering a continuous version of exponential weights, à la Cover (1991), and by leveraging techniques introduced by Ito, Hirahara, Soma, and Yoshida (Neurips 2020) for such refined first-order and second-order bounds in the setting of non-contextual adversarial linear bandits. \n\nThe key to the results is the reduction (18) of this semi-adversarial contextual bandit problem into a linear bandit problem, as underlined in lines 285-288.\n\nSTRENGTHS:\nThe submission has a clear technical focus, and combines in a neat way techniques from Neu and Olkhovskaya (COLT 2020) and Ito, Hirahara, Soma, and Yoshida (Neurips 2020). The proof sketch provided on pages 6--8 is well structured. Up to some typos and some unclear pieces of notation, I can only praise the writing.\n\nWEAKNESSES:\nMy main issue is with the algorithm, which relies on distributions that are difficult to compute exactly---as always with continuous exponential weights (see articles from Vempala and co-authors at the end of the 90s, commenting on Cover's original contributions). Allowing \\epsilon--approximations reduces the computational burden (see Section 3.3) but the effect of using these \\epsilon--approximations on the regret bounds is not discussed. Would this add some \\epsilon T terms? Then, it would probably be wise to pick \\epsilon = 1/T, not to kill with this extra term the possible fast rates of convergence yielded by first- and second-order bounds. (Also, it is not clear to me from Section 3.3 whether these \\epsilon--errors propagate and get worse and worse as t increases, or if for each t, we may and should compute an \\epsilon--approximation from scratch, not based on previous probability distributions.)\n\nAnother issue would be that I am unsure of what the second-order bound of Theorem 3.2 brings: it is highly general, and no corollary or example for the functions m_t are provided (except, I guess, the choices m_t \\equiv 0). More generally, I am unsure of how important it is to get refined bounds with a rather non-practical algorithm based on continuous exponential weights. \n\nA more minor issue is about the assumption of log-concavity. It is satisfied by many distributions (see https://en.wikipedia.org/wiki/Logarithmically_concave_function#Log-concave_distributions, and the submission could give examples) but is also not satisfied by other important ones.\n\nTypos / Remarks\n- Lines 96-97: rather cite Cover (1991) or to the very least the monograpu by Cesa-Bianchi and Lugosi (2006), instead of a 2018 reference \n- Line 123: I believe you refer to it as \\pi(X_t) and never use \\pi(a|X_t)\n- Line 131: Yes, this seems clear indeed\n- Line 134: I would rather have written E_{t-1} instead of E_t \n- Line 144 / Eq. (4); see also later occurrences like in Th.3.2: Is L^*_T the same as L_T^{\\pi^*}?\n- Algorithm 1: Say that each m_s is a function that is measurable w.r.t. \\mathcal{F}_{s-1}\n- Eq.(6): Somewhere earlier than in lines 291 and following, discuss the number of steps needed to achieve (6), i.e., geometric distribution with a parameter that you anyway bound in the proof---but mention it early, at latest on page 5 or in Section 3.3\n-  Line 185: Define somewhare what log-concavity of a distribution is, and provide examples\n- Eq. (18) and comments in lines 285-288: This is the key step and it's somewhat hidden, I'd suggest to put it more in front\n- Lemma 4.2: Notation is heavy and the result is extremely difficult to read, is there any way to make it smoother? That part and Eq. (19) should also be put more in front. \n- Line 291: Missing word before 1 ('Algorithm' I guess) \n- Appendices: Cauchy-Schwarz without a 't'\n\nQUESTIONS:\nPlease comment on the main two weaknesses that I listed.\n\nLIMITATIONS:\nThe authors are clear about the limitations, and in particular, on dependency of their regret bounds on K, which does not match, in the worst-case, the known \\sqrt{d K T} regret bound. The discussion on page 9 is complete and upfront about the scope of their contribution.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors explore the adversarial linear contextual bandit problem, where  the context is i.i.d. drawn from a known fixed log-concave distribution, and the linear loss functions are adaptive adversaries. \nThey provide first adaptive bounds for this setting, including a first-order bound based on the cumulative loss of the optimal policy and a second-order bound based on the squared losses of the algorithm. These bounds demonstrate improvements over the worst-case bound in scenarios where the environment is  'nice'.\n\nSTRENGTHS:\nThe authors first provide the adaptive bound for adversarial linear contextual bandits, which improves the worst case bound in benign environments. \nThe computational complexity of the algorithm is commendable, particularly in moderate environments. \nThe results and the addressed problem hold substantial significance and are likely to capture the interest of a broad audience in Neurips. \nFurthermore, the authors thoroughly discuss future directions, which have the potential to inspire further research aimed at improved adaptive bounds in linear bandits setting.\n\nWEAKNESSES:\nI have two primary concerns regarding this work and would like to request the authors to provide additional clarifications. \n\n==Weakness 1==\n\nIt appears that the results and techniques heavily rely on previous work, such as\n\nIto, S., Hirahara, S., Soma, T., and Yoshida, Y. (2020). Tight first- and second-order regret bounds for adversarial linear bandits,\n\nwhich first introduce adaptive bounds for adversarial linear bandits. \nIn comparison to the adversarial linear bandits setting, the authors in this paper consider the case with i.i.d. contexts and introduce some additional assumptions, such as known context distribution and log-concavity. These assumptions may simplify the problem and restrict the practical application of the proposed algorithms. I suggest the authors emphasize the technical contribution of their paper, as this aspect is not clearly highlighted in the \"Techniques\" subsection in the Introduction.\n\n==Weakness 2==\n\nThere does not seem to be any experiments of any type. \nNeither the first-order bound nor the second-order bound demonstrate strict improvements over the worst-case bound.\nIn other words, they may not be tight enough. \nIn this case, I kindly recommend that the authors provide numerical results, which would enable readers to gain a better understanding of the algorithm's behavior.  While numerical evaluations may not significantly enhance the primary theoretical message of the paper, they would improve the overall completeness of the work, considering that linear bandits are closely related to practical applications.\n\nQUESTIONS:\nPlease see the points mentioned in the weaknesses above and provide comments on them.\n\nLIMITATIONS:\nI don't see any issue on societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

comments: SUMMARY:
Addressing to the K-armed linear contextual adaptive adversary bandit problem with d-dimensional contexts over T rounds, using a truncated version of the continuous exponential weights algorithm over the probability simplex, the expected regret about the cumulative second moments of the losses is obtained. and then the first order bound is derived.

STRENGTHS:
generalizing the results of tight First- and Second-Order Regret Bounds for Adversarial Linear Bandits to First- and Second-Order Bounds for Adversarial Linear Contextual Bandits.

WEAKNESSES:
1 The results of the paper heavily rely on a combination of the recently proposed techniques of Ito et al. (2020) for linear bandits with tools designed by Neu and Olkhovskaya (2020) to deal with the contextual case.
2 The conclusion only applies to the scenario of the joint log-concavity of the context distributions and the exponential weights posterior over the simplex of actions, results may not be generalizable.

QUESTIONS:
1 if the distribution for contexts is time-vary or need to be estimated from data, what will the situation be like? 
2 the analysis is all based on the assumption that they can access the distribution for contexts, which is not the case for practical case.  How to handle this difficulty, please give further discussion.

LIMITATIONS:
the authors adequately addressed the limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06