Title: Bypassing the Simulator: Near-Optimal Adversarial Linear Contextual Bandits
Venue: NeurIPS 2023
Accepted: yes
Keywords: adversarial linear contextual bandits

log-determinant barrier
Abstract: We consider the adversarial linear contextual bandit problem, 
where the loss vectors are selected fully adversarially and the per-round action set (i.e. the context) is drawn from a fixed distribution. Existing methods for this problem either require access to a simulator to generate free i.i.d. contexts, achieve a sub-optimal regret no better than $\tilde{\mathcal{O}}(T^{\frac{5}{6}})$, or are computationally inefficient. 
We greatly improve these results by achieving a regret of $\tilde{\mathcal{O}}(\sqrt{T})$ without a simulator, while maintaining computational efficiency when the action set in each round is small. 
In the special case of sleeping bandits with adversarial loss and stochastic arm availability, our result answers affirmatively the open question by [SGV20]  on whether there exists a polynomial-time algorithm with  $poly(d)\sqrt{T}$ regret. Our approach naturally handles the case where the loss is linear up to an additive misspecification error, and our regret shows near-optimal dependence on the magnitude of the error.

Bypassing the Simulator: Near-Optimal Adversarial Linear Contextual Bandits

Abstract

Weconsider the adversarial linear contextual bandit problem, where the loss vectors are selected fully adversarially and the per-round action set (i.e. the context) is drawn from a fixed distribution. Existing methods for this problem either require access to a simulator to generate free i.i.d. contexts, achieve a sub-optimal regret no better than ˜ O ( T 5 / 6 ) , or are computationally inefficient. We greatly improve these results by achieving a regret of ˜ O ( √ T ) without a simulator, while maintaining computational efficiency when the action set in each round is small. In the special case of sleeping bandits with adversarial loss and stochastic arm availability, our result answers affirmatively the open question by Saha et al. [2020] on whether there exists a polynomial-time algorithm with poly( d ) √ T regret. Our approach naturally handles the case where the loss is linear up to an additive misspecification error, and our regret shows near-optimal dependence on the magnitude of the error.

1 Introduction

Contextual bandit is a widely used model for sequential decision making. The interaction between the learner and the environment proceeds in rounds: in each round, the environment provides a context; based on it, the learner chooses an action and receive a reward. The goal is to maximize the total reward across multiple rounds. This model has found extensive applications in fields such as medical treatment [Tewari and Murphy, 2017], personalized recommendations [Beygelzimer et al., 2011], and online advertising [Chu et al., 2011].

Algorithms for contextual bandits with provable guarantees have been developed under various assumptions. In the linear regime, the most extensively studied model is the stochastic linear contextual bandit , in which the context can be arbitrarily distributed in each round, while the reward is determined by a fixed linear function of the context-action pair. Near-optimal algorithms for this setting have been established in, e.g., [Chu et al., 2011, Abbasi-Yadkori et al., 2011, Li et al., 2019, Foster et al., 2020]. Another model, which is the focus of this paper, is the adversarial linear contextual bandit , in which the context is drawn from a fixed distribution, while the reward is determined by a time-varying linear function of the context-action pair. 3 A computationally efficient algorithm for this setting is first proposed by Neu and Olkhovskaya [2020]. However, existing research for this setting still faces challenges in achieving near-optimal regret and sample complexity when the context distribution is unknown.

The algorithm by Neu and Olkhovskaya [2020] requires the learner to have full knowledge on the context distribution, and access to an exploratory policy that induces a feature covariance matrix with a smallest eigenvalue at least λ . Under these assumptions, their algorithm provides a regret guarantee of ˜ O ( √ d log( |A| ) T/λ ) 4 , where d is the feature dimension, |A| is the maximum size of the action set, and T is the number of rounds. These assumptions are relaxed in the work of Luo et al. [2021], who studied a more general linear MDP setting. When specialized to linear contextual bandits, Luo et al. [2021] only requires access to a simulator from which the learner can draw free i.i.d. contexts. Their algorithm achieves a ˜ O (( d log( |A| ) T 2 ) 1 / 3 )) regret. The regret is further improved to the near-optimal one ˜ O ( √ d log( |A| ) T ) by Dai et al. [2023] through refined loss estimator construction.

All results that attain ˜ O ( T 2 / 3 ) or ˜ O ( √ T ) regret bound discussed above rely on access to the simulator. In their algorithms, the number of calls to the simulator significantly exceeds the number of interactions between the environment and the learner, but this is concealed from the regret bound. Therefore, their regret bounds do not accurately reflect the sample complexity of their algorithms. Another set of results for linear MDPs [Luo et al., 2021, Dai et al., 2023, Sherman et al., 2023, Kong et al., 2023] also consider the simulator-free scenario, essentially using interactions with the environment to fulfill the original purpose of the simulator. When applying their techniques to linear contextual bandits, their algorithms only achieve a regret bound of ˜ O ( T 5 / 6 ) at best (see detailed analysis and comparison in Appendix G).

Our result significantly improves the previous ones: without simulators, we develop an algorithm that ensures a regret bound of order ˜ O ( d 2 √ T ) , and it is computationally efficient as long as the size of the action set is small in each round (similar to all previous work). Unlike previous algorithms which always collect new contexts (through simulators or interactions with the environment) to estimate the feature covariance matrix, we leverage the context samples the learner received in the past to do this. Although natural, establishing a near-tight regret requires highly efficient use of context samples, necessitating a novel way to construct the estimator of feature covariance matrix and a tighter concentration bound for it. Additionally, to address the potentially large magnitude and the bias of the loss estimator, we turn to the use of log-determinant (logdet) barrier in the follow-theregularized-leader (FTRL) framework. Logdet accommodates larger loss estimators and induces a larger bonus term to cancel the bias of the loss estimator, both of which are crucial for our result.

Our setting subsumes sleeping bandits with stochastic arm availability [Kanade et al., 2009, Saha et al., 2020] and combinatorial semi-bandits with stochastic action sets [Neu and Valko, 2014]. Our result answers affirmatively the main open question left by Saha et al. [2020] on whether there exists a polynomial-time algorithm with poly( d ) √ T regret for sleeping bandits with adversarial loss and stochastic availability.

As a side result, we give a computationally inefficient algorithm that achieves an improved ˜ O ( d √ T ) regret without a simulator. While this is a direct extension from the EXP4 algorithm [Auer et al., 2002], such a result has not been established to our knowledge, so we include it for completeness.

1.1 Related work

We review the literature of various contextual bandit problems, classifying them based on the nature of the context and the reward function, specifically whether they are stochastic/fixed or adversarial.

Contextual bandits with i.i.d. contexts and fixed reward functions (S-S) Significant progress has been made in contextual bandits with i.i.d. contexts and fixed reward functions, under general reward function classes or policy classes [Langford and Zhang, 2007, Dudik et al., 2011, Agarwal et al., 2012, 2014, Simchi-Levi and Xu, 2022, Xu and Zeevi, 2020]. In the work by Dudik et al. [2011], Agarwal et al. [2012, 2014], the algorithms also use previously collected contexts to estimate the inverse probability of selecting actions under the current policy. However, these results only obtain regret bounds that polynomially depend on the number of actions. Furthermore, these results rely on having a fixed reward function, making their techniques not directly applicable to our case.

Contextual bandits with adversarial contexts and fixed reward functions (A-S) In this category, the most well-known results are in the linear setting [Chu et al., 2011, Abbasi-Yadkori et al., 2011, Zhao et al., 2023]. Besides the linear case, previous work has investigated specific reward function classes [Russo and Van Roy, 2013, Li et al., 2022, Foster et al., 2018]. Recently, Foster and Rakhlin [2020] introduced a general approach to deal with general function classes with a finite number of actions, which has since been improved or extended by Foster and Krishnamurthy [2021], Foster et al. [2021], Zhang [2022]. This category of problems is not directly comparable to the setting studied in this paper, but both capture a certain degree of non-stationarity of the environment.

Contextual bandits with i.i.d. contexts and adversarial reward functions (S-A) This is the category which our work falls into. Several oracle efficient algorithms that require simulators have been proposed for general policy classes [Rakhlin and Sridharan, 2016, Syrgkanis et al., 2016b]. The oracle they use (i.e., the empirical risk minimization, or ERM oracle), however, is not generally implementable in an efficient manner. For the linear case, the first computationally efficient algorithm is by Neu and Olkhovskaya [2020], under the assumption that the context distribution is known. This is followed by Olkhovskaya et al. [2023] to obtain refined data-dependent bounds. A series of works [Neu and Olkhovskaya, 2021, Luo et al., 2021, Dai et al., 2023, Sherman et al., 2023] apply similar techniques to linear MDPs, but when specialized to linear contextual bandits, they all assume known context distribution, or access to a simulator, or only achieves a regret no better than ˜ O ( T 5 / 6 ) . The work of Kong et al. [2023] also studies linear MDPs; when specialized to contextual bandits, they obtain a regret bound of ˜ O ( T 4 / 5 +poly( 1 λ )) without a simulator but with a computationally inefficient algorithm and an undesired inverse dependence on the smallest eigenvalue of the covariance matrix. Related but simpler settings have also been studied. The sleeping bandit problem with stochastic arm availability and adversarial reward [Kleinberg et al., 2010, Kanade et al., 2009, Saha et al., 2020] is a special case of our problem where the context is always a subset of standard unit vectors. Another special case is the combinatorial semi-bandit problem with stochastic action sets and adversarial reward [Neu and Valko, 2014]. While these are special cases, the regret bounds in these works are all worse than ˜ O (poly( d ) √ T ) . Therefore, our result also improves upon theirs. 5

Contextual bandits with adversarial contexts and adversarial reward functions (A-A) When both contexts and reward functions are adversarial, there are computational [Kanade and Steinke, 2014] and oracle-call [Hazan and Koren, 2016] lower bounds showing that no sublinear regret is achievable unless the computational cost scales polynomially with the size of the policy set. Even for the linear case, Neu and Olkhovskaya [2020] argued that the problem is at least as hard as online learning a one-dimensional threshold function, for which sublinear regret is impossible. For this

challenging category, besides using the inefficient EXP4 algorithm, previous work makes stronger assumptions on the contexts [Syrgkanis et al., 2016a] or resorts to alternative benchmarks such as dynamic regret [Luo et al., 2018, Chen et al., 2019] and approximate regret [Emamjomeh-Zadeh et al., 2021].

Lifting and exploration bonus for high-probability adversarial linear bandits Our technique is related to those obtaining high-probability bounds for linear bandits. Early development in this line of research only achieves computational efficiency when the action set size is small [Bartlett et al., 2008] or only applies to special action sets such as two-norm balls [Abernethy and Rakhlin, 2009]. Recently, near-optimal high-probability bounds for general convex action sets have been obtained by lifting the problem to a higher dimensional one, which allows for a computationally efficient way to impose bonuses [Lee et al., 2020, Zimmert and Lattimore, 2022]. The lifting and the bonus ideas we use are inspired by them, though for different purposes. However, due to the extra difficulty arising in the contextual case, currently we only obtain a computationally efficient algorithm when the action set size is small.

1.2 Computational Complexity

Our main algorithm is based on log-determinant barrier optimization similar to Foster et al. [2020], Zimmert and Lattimore [2022]. Computing its action distribution is closely related to computing the D-optimal experimental design [Khachiyan and Todd, 1990]. Per step, this is shown to require ˜ O ( |A t | poly( d )) computational and ˜ O (log( |A t | ) poly( d )) memory complexity [Foster et al., 2020, Proposition 1], where |A t | is the action set size at round t . The computational bottleneck comes from (approximately) maximizing a quadratic function over the action set. It is an open question whether linear optimization oracles or other type of oracles can lead to efficient implementation of our algorithm for continuous action sets.

In the literature, there are few linear context bandit algorithms that provably avoid |A| computation per round. The LinUCB algorithm [Chu et al., 2011, Abbasi-Yadkori et al., 2011] suffers from the same quadratic function maximization issue, and therefore is computationally comparable to our algorithm. The SquareCB.Lin algorithm by Foster et al. [2020] is based on the same log-determinant barrier optimization. Another recent algorithm by Zhang [2022] only admits an efficient implementation for continuous action sets in the Bayesian setting but not in the frequentist setting (though they provided an efficient heuristic implementation in their experiments). The Thompson sampling algorithm by Agrawal and Goyal [2013], which has efficient implementation, also relies on well-specified Gaussian prior. The only work that we know can avoids |A| computation in the frequentist setting is Zhu et al. [2022], but their technique is only known to handle the A-S setting.

2 Preliminaries

We study the adversarial linear contextual bandit problem where the loss vectors are selected fully adversarially and the per-round action set (i.e. the context) is drawn from a fixed distribution. The learner and the environment interact in the following way. Let B d 2 be the L2-norm unit ball in R d .

The environment decides an adversarial loss vector y t ∈ B d 2 , and generates a random action set (i.e., context) A t ⊂ B d 2 from a fixed distribution D independent from anything else.

The learner observes A t , and (randomly) chooses an action a t ∈ A t .

The learner receives the loss ℓ t ∈ [ -1 , 1] with E [ ℓ t ] = ⟨ a t , y t ⟩ .

A policy π is a mapping which, given any action set A ⊂ R d , maps it to an element in the convex hull of A . We use π ( A ) to refer to the element that it maps A to. The learner's regret with respect to policy π is defined as the expected performance difference between the learner and policy π :

where the expectation is taken over all randomness from the environment ( y t and A t ) and from the learner ( a t ). The pseudo-regret (or just regret ) is defined as Reg = max π Reg ( π ) , where the maximization is taken over all possible policies.

Notations For any matrix A , we use λ max ( A ) and λ min ( A ) to denote the maximum and minimum eigenvalues of A , respectively. We use Tr( A ) to denote the trace of matrix A . For any action set A , let ∆( A ) be the space of probability measures on A . Let F t = σ ( A s , a s , ∀ s ≤ t ) be the σ -algebra at round t . Define E t [ · ] = E [ ·|F t -1 ] . Given a differentiable convex function F : R d → R ∪{∞} , the Bregman divergence with respect to F is defined as D F ( x, y ) = F ( x ) -F ( y ) -⟨∇ F ( y ) , x -y ⟩ . Given a positive semi-definite (PSD) matrix A , for any vector x , define the norm generated by A as ∥ x ∥ A = √ x ⊤ Ax . For any context A ⊂ R d and p ∈ ∆( A ) , define µ ( p ) = E a ∼ p [ a ] and Cov( p ) = E a ∼ p [( a -µ ( p ))( a -µ ( p )) ⊤ ] . For any a , define the lifted action a = ( a, 1) ⊤ and the lifted

We use bold matrices to denote matrices in the lifted space (e.g., in Algorithm 1 and Definition 1).

3 Follow-the-Regularized-Leader with the Log-Determinant Barrier

In this section, we present our main algorithm, Algorithm 1. This algorithm can be viewed as instantiating an individual Follow-The-Regularized-Leader (FTRL) algorithm on each action set (Line 2), with all FTRLs sharing the same loss vectors. This perspective has been taken by previous works Neu and Olkhovskaya [2020], Olkhovskaya et al. [2023] and simplifies the understanding of the problem. The rationale comes from the following calculation due to Neu and Olkhovskaya [2020]: for any policy π that may depend on F t -1 ,

E t [ ⟨ π ( A t ) , y t ⟩ ] = E A t [ E y t [ ⟨ π ( A t ) , y t ⟩ | F t -1 ]] = E A 0 [ E y t [ ⟨ π ( A 0 ) , y t ⟩ | F t -1 ]] = E t [ ⟨ π ( A 0 ) , y t ⟩ ] where A 0 is a sample drawn from D independent of all interaction history. This allows us to calculate the regret as

where π t is the policy used by the learner at time t . Note that this view does not require the learner to simultaneously 'run' an algorithm on every action set since the learner only needs to calculate the policy on A whenever A t = A . In the regret analysis, in view of Eq. (1), it suffices to consider a single fixed action set A 0 drawn from D and bound the regret on it, even though the learner may never execute the policy on it. This A 0 is called a 'ghost sample' in Neu and Olkhovskaya [2020].

3.1 The lifting idea and the execution of Algorithm 1

Our algorithm is built on the logdet-FTRL algorithm developed by Zimmert and Lattimore [2022] for high-probability adversarial linear bandits, which lifts the original d -dimensional problem over the feature space to a ( d +1) × ( d +1) one over the covariance matrix space, with the regularizer being the negative log-determinant function. In our case, we instantiate an individual logdet-FTRL on each action set. The motivation behind Zimmert and Lattimore [2022] to lift the problem to the space of covariance matrix is that it casts the problem to one in the positive orthant, which allows for an easier way to construct the bonus term that is crucial to compensate the variance of the losses, enabling a high-probability bound in their case. In our case, we use the same technique to introduce the bonus term, but the goal is to compensate the bias resulting from the estimation error in the covariance matrix (see Section 3.4). This bias only appears in our contextual case but not in the linear bandit problem originally considered in Zimmert and Lattimore [2022].

As argued previously, we can focus on the learning problem over a fixed action set A , and our algorithm operates in the lifted space of covariance matrices H A = { ̂ Cov( p ) : p ∈ ∆( A ) } ⊂

that ⟨ ̂ Cov( p ) , γ t ⟩ = E a ∼ p [ a ⊤ y t ] = ⟨ µ ( p ) , y t ⟩ and thus the loss value in the lifted space (i.e., ⟨ ̂ Cov( p ) , γ t ⟩ ) is the same as that in the original space (i.e., ⟨ µ ( p ) , y t ⟩ ).

Algorithm 1 Logdet-FTRL for linear contextual bandits

In each round t , the FTRL on A outputs a lifted covariance matrix H A t ∈ H A that corresponds to a probability distribution p A t ∈ ∆( A ) such that ̂ Cov( p A t ) = H A t (Line 2 and Line 3). Upon receiving A t , the learner samples an action from p A t t and the agent constructs the loss estimator ˆ y t (Line 5).

makes ⟨ ̂ Cov( p ) , ˆ γ t ⟩ = E a ∼ p [ a ⊤ ˆ y t ] = ⟨ µ ( p ) , ˆ y t ⟩ . The lifted loss estimator, along with the bonus term -α t ˆ Σ t ˆ Σ t ˆ Σ t -1 , is then fed to the FTRL on all A 's. The purpose of the bonus term will be clear in Section 3.4.

In the rest of this section, we use the following notation in addition to those defined in Algorithm 1. Definition 1. Define x A t = E a ∼ p A t [ a ] , x t = E A∼ D [ x A t ] , H A t = E a ∼ p A t [( a -ˆ x t )( a -ˆ x t ) ⊤ ] , H t = E A∼ D [ H A t ] , H t = E A∼ D [ H A t ] . Let p A ⋆ ∈ ∆( A ) be the action distribution used by the benchmark policy on A , and define u A = E a ∼ p A ⋆ [ a ] , u = E A∼ D [ u A ] , U A = E a ∼ p A ⋆ [ aa ⊤ ] , U = E A∼ D [ U A ] . Notice that the x A t and u A defined here is equivalent to the π t ( A ) and π ( A ) in Eq. (1) , respectively.

3.2 The construction of loss estimators and feature covariance matrix estimators

Our goal is to make ˆ y t in Line 5 an estimator of y t with controllable bias and variance. If the context distribution is known (as in Neu and Olkhovskaya [2020]), then a standard unbiased estimator of y t is

To see its unbiasedness, notice that E [ a t ℓ t ] = E A∼ D E a ∼ p A t [ aa ⊤ y t ] and thus E [ˆ y t ] = y t . This ˆ y t , however, can have a variance that is inversely related to the smallest eigenvalue of the covariance matrix ˆ Σ t , which can be unbounded in the worst case. This is the main reason why Neu and Olkhovskaya [2020] does not achieve the optimal bound, and requires the bias-variance-tradeoff techniques in Dai et al. [2023] to close the gap. When the context distribution is unknown but the learner has access to a simulator [Luo et al., 2021, Dai et al., 2023, Sherman et al., 2023, Kong et al., 2023], the learner can draw free contexts to estimate the covariance matrix ˆ Σ t up to a very high accuracy without interacting with the environment, making the problem close to the case of known context distribution.

Challenges arise when the learner has no knowledge about the context distribution and there is no simulator. In this case, there are two natural ways to estimate the covariance matrix under the current policy. One is to draw new samples from the environment, treating the environment like a simulator. This approach is essentially taken by all previous work studying linear models in the 'S-A' category. However, this is very expensive, and it causes the simulator-equipped bound √ T in Dai et al. [2023]

to deteriorate to the simulator-free bound T 5 / 6 at best (see Appendix G for details). The other is to use the contexts received in time 1 to t to estimate the covariance matrix under the policy at time t . This demands a very high efficiency in reusing the contexts samples, and existing ways of constructing the covariance matrix and the accompanied analysis by Dai et al. [2023], Sherman et al. [2023] are insufficient to achieve the near-optimal bound even with context reuse. This necessitates our tighter construction of the covariance matrix estimator and tighter concentration bounds for it.

Our construction of the loss estimator (Line 5) is

where ˆ D t = Uniform {A 1 , A 2 , . . . , A t -1 } , ˆ x t = E A∼ ˆ D t , E a ∼ p A t [ a ] , and β t = ˜ O ( d 3 /t ) . Comparing Eq. (3) with Eq. (2), we see that besides using the empirical context distribution ˆ D t in place of the ground truth D and adding a small term β t I to control the smallest eigenvalue of the covariance matrix, we also centralize the features by ˆ x t , an estimation of the mean features under the current policy. The centralization is important in making the bias y t -ˆ y t appear in a nice form that can be compensated by a bonus term. The estimator might seem problematic on first sight, because p A t is strongly dependent on ˆ D t , which rules out canonical concentration bounds. We circumvent this issue by leveraging the special structure of p t in Algorithm 1, which allows for a union bound over a sufficient covering of all potential policies (Appendix C.3). The analysis on the bias of this loss estimator is also non-standard, which is the key to achieve the near-optimal bound . In the next two subsections, we explain how to bound the bias of this loss estimator (Section 3.3), and how the bonus term can be used to compensate the bias (Section 3.4).

3.3 The bias of the loss estimator

Since the true loss vector is y t and we use the loss estimator ˆ y t in the update, there is a bias term emerging in the regret bound at time t :

where definitions of x A t , u A , x t , u can be found in Definition 1, and we use the definition of ˆ y t in Eq. (3) in the last equality. Now taking expectation over A t and a t conditioned on F t -1 , we can further bound the expectation in the last expression by

(see Definition 1 for the definition of H t ). The two terms ∥ ( ˆ Σ t -H t ) y t ∥ ˆ Σ -1 t and ∥ x t -ˆ x t ∥ ˆ Σ -1 t in Eq. (4) are related to the error between the empirical context distribution ˆ D t = Uniform {A 1 , . . . , A t -1 } and the true distribution D . We handle them through novel analysis and bound both of them by ˜ O (√ d 3 /t ) . See Lemma 13, Lemma 14, Lemma 18, and Lemma 19 for details. The techniques we use in these lemmas surpass those in Dai et al. [2023], Sherman et al. [2023]. As a comparison, a similar term as ∥ ( ˆ Σ t -H t ) y t ∥ ˆ Σ -1 t is also presented in Eq. (16) of Dai et al. [2023] and Lemma B.5 of Sherman et al. [2023] when bounding the bias. While their analysis uses off-the-shelf matrix concentration inequalities, our analysis expands this expression by its definition, and applies concentration inequalities for scalars on individual entries. Overall, our analysis is more tailored for this specific expression. Previous works ensure that this term can be bounded by O ( √ β ) after collecting O ( β -2 ) new samples (Lemma 5.1 of Dai et al. [2023] and Lemma B.1 of Sherman et al. [2023]), we are able to bound it by O (1 / √ t ) only using t samples that the learner received up to time t . This essentially improves their O ( β -2 ) sample complexity bound to O ( β -1 ) . See Appendix G for detailed comparison with Dai et al. [2023] and Sherman et al. [2023].

Now we have bounded the regret due to bias of ˆ y t by the order of √ d 3 /t ∥ x t -u ∥ ˆ Σ -1 t . The next problem is how to mitigate this term. This is also a problem in previous work [Luo et al., 2021, Dai et al., 2023, Sherman et al., 2023], and it has become clear that this can be handled by incorporating bonus in the algorithm.

3.4 The bonus term

To handle a bias term in the form of ∥ x t -u ∥ ˆ Σ -1 t , we resort to the idea of bonus . To illustrate this, suppose that instead of feeding ˆ y t to the FTRLs, we feed ˆ y t -b t for some b t . Then this would give us a regret bound of the following form:

where we assume that FTRL can give us ˜ O ( d 2 √ T ) bound for the loss sequence ˆ y t -b t . Our hope here is to design a b t such that ⟨ x t -u, b t ⟩ provides a negative term that can be used to cancel the bias term √ d 3 /t ∥ x t -u ∥ ˆ Σ -1 t in the following manner:

which gives us a ˜ O ( d 2 √ T ) overall regret by Eq. (5). This approach relies on two conditions to be satisfied. First, we have to find a b t that makes Eq. (6) hold. Second, we have to ensure that the FTRL algorithm achieves a ˜ O ( d 2 √ T ) bound under the loss sequence ˆ y t -b t .

To meet the first condition, we take inspiration from Zimmert and Lattimore [2022] and lift the problem to the space of covariance matrix in R ( d +1) × ( d +1) . Considering the bonus term α t ˆ Σ ˆ Σ ˆ Σ -1 t in the lifted space, we have

Using Lemma 17 and Corollary 22, we can upper bound Eq. (7) by O ( dα t ) -α t 4 ∥ u -ˆ x t ∥ 2 ˆ Σ -1 t . This gives

Using Lemma 18 to bound the second term above by ˜ O ( ∑ t d 3 /t ) = ˜ O ( d 3 ) , and AM-GM to bound the third term by ˜ O ( ∑ t d 3 / ( tα t )) = ˜ O ( d 2 √ T ) , we get Eq. (6), through the help of lifting.

To meet the second condition, we have to analyze the regret of FTRL under the loss ˆ y t -b t . The key is to show that the bonus α t ˆ Σ ˆ Σ ˆ Σ -1 t introduces small stability term overhead. Thanks to the use of the logdet regularizer and its self-concordance property, the extra stability term introduced by the bonus can indeed be controlled by the order √ T . The key analysis is in Lemma 27.

Previous works rely on exponential weights [Luo et al., 2021, Dai et al., 2023, Sherman et al., 2023] rather than logdet-FTRL, which comes with the following drawbacks. 1) In Luo et al. [2021], Sherman et al. [2023] where exponential weights is combined with standard loss estimators, the bonus introduces large stability term overhead. Therefore, their bound can only be T 2 / 3 at best even with simulators. 2) In Dai et al. [2023] where exponential weights is combined with magnitude-reduced loss estimators, the loss estimator for action a can no longer be represented as a simple linear function a ⊤ ˆ y t . Instead, it becomes a complex non-linear function. This restricts the algorithm's potential to leverage linear optimization oracle over the action set and achieve computational efficiency.

3.5 Overall regret analysis

With all the algorithmic elements discussed above, now we give a formal statement for our regret guarantee and perform a complete regret analysis. Our main theorem is the following.

Theorem 2. Algorithm 1 ensures Reg ≤ O ( d 2 √ T log T ) .

Proof sketch. Let A 0 be drawn from D independently from all the interaction history between the learner and the environment. Recalling the definitions in Definition 1, we have

Each term can be bounded as follows:

Bonus ≤ O ( d 2 √ T log T ) -1 4 ∑ T t =1 α t ∥ u -x t ∥ 2 ˆ Σ -1 t (discussed in Section 3.4). √

Bias ≤ O ( d 2 √ T log T ) + 1 4 ∑ T t =1 α t ∥ u -x t ∥ 2 ˆ Σ -1 t (discussed in Section 3.3).

FTRL-Reg ≤ O ( d 2 T log T ) .

Combining all terms gives the desired bound. The complete proof is provided in Appendix D.

3.6 Handling Misspecification

In this subsection, we show how our approach naturally handles the case when the expectation of the loss cannot be exactly realized by a linear function but with a misspecification error. In this case, we assume that the expectation of the loss is given by E [ ℓ t | a t = a ] = f t ( a ) for some f t : R d → [ -1 , 1] , and the realized loss ℓ t still lies in [ -1 , 1] . We define the following notion of misspecification (slightly more refined than that in Neu and Olkhovskaya [2020]):

Based on previous discussions, the design idea of Algorithm 1 is to 1) identify the bias of the loss estimator, and 2) add necessary bonus to compensate the bias. When there is misspecification, this design idea still applies. The difference is that now the loss estimator ˆ y t potentially has more bias due to misspecification. Therefore, the bias becomes larger by an amount related to ε . Consequently, we need to enlarge bonus (raising α t ) to compensate it. Due to the larger bonus, we further need to tune down the learning rate η t to make the algorithm stable. Overall, to handle misspecification, when ε is known, it boils down to using the same algorithm (Algorithm 1) with adjusted α t and η t . The case of unknown ε can be handled by the standard meta-learning technique Corral [Agarwal et al., 2017, Foster et al., 2020, Luo et al., 2022]. We defer all details to Appendix E and only state the final bound here.

Theorem 3. Under misspecification, there is an algorithm ensuring Reg ≤ ˜ O ( d 2 √ T + √ dεT ) , without knowing ε in advance.

4 Linear EXP4

To tighten the d -dependence in the regret bound, we can use the computationally inefficient algorithm EXP4 [Auer et al., 2002]. The original regret bound for EXP4 has a polynomial dependence on the number of actions, but here we take the advantage of the linear structure to show a bound that only depends on the feature dimension d . The algorithm is presented in Algorithm 2.

Algorithm 2 Linear EXP4

input : Π , η, γ .

Receive A t ⊂ R d .

Construct ν t ∈ ∆( A t ) such that max a ∈A t ∥ a ∥ 2 G -1 t ≤ d , where G t = E a ∼ ν t [ aa ⊤ ] . Set

To run Algorithm 2, we restrict ourselves to a finite policy class. The policy class we use in the algorithm is the set of linear policies defined as

where Θ is an 1 -net of [ -T, T ] d . The next theorem shows that this suffices to give us near-optimal bounds for our problem. The proof is given in Appendix F.

Theorem 4. With γ = 2 d √ (log T ) /T and η = √ (log T ) /T , Algorithm 2 with the policy class defined in Eq. (8) guarantees Reg = O ( d √ T log T ) .

Note that this result technically also holds in the 'A-A' category with respect to the policy class defined in Eq. (8). However, this policy class is not necessarily a sufficient cover of all policies of interest when the contexts and losses are adversarial.

5 Conclusions

We derived the first algorithm that obtains √ T regret in contextual linear bandits with stochastic action sets in the absence of a simulator or prior knowledge on the distribution. As a side result, we obtained the first computationally efficient poly( d ) √ T algorithm for adversarial sleeping bandits with general stochastic arm availabilities. We believe the techniques in this paper will be useful for improving results for simulator-free linear MDPs as well.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24:2312-2320, 2011.

Jacob Abernethy and Alexander Rakhlin. Beating the adaptive bandit with high probability. In 2009 Information Theory and Applications Workshop , pages 280-289. IEEE, 2009.

Jacob D Abernethy, Elad Hazan, and Alexander Rakhlin. Competing in the dark: An efficient algorithm for bandit linear optimization. In Conference on Learning Theory , 2009.

Alekh Agarwal, Miroslav Dudík, Satyen Kale, John Langford, and Robert Schapire. Contextual bandit learning with predictable rewards. In Artificial Intelligence and Statistics , pages 19-26. PMLR, 2012.

Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In International Conference on Machine Learning , pages 1638-1646. PMLR, 2014.

Alekh Agarwal, Haipeng Luo, Behnam Neyshabur, and Robert E Schapire. Corralling a band of bandit algorithms. In Conference on Learning Theory , pages 12-38. PMLR, 2017.

Shipra Agrawal and Navin Goyal. Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning , pages 127-135. PMLR, 2013.

Peter Auer, Nicolo Cesa-Bianchi, Yoav Freund, and Robert E Schapire. The nonstochastic multiarmed bandit problem. SIAM journal on computing , 32(1):48-77, 2002.

Peter L Bartlett, Varsha Dani, Thomas P Hayes, Sham M Kakade, Alexander Rakhlin, and Ambuj Tewari. High-probability regret bounds for bandit online linear optimization. In COLT , pages 335-342, 2008.

Alina Beygelzimer, John Langford, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandit algorithms with supervised learning guarantees. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 19-26. JMLR Workshop and Conference Proceedings, 2011.

Yifang Chen, Chung-Wei Lee, Haipeng Luo, and Chen-Yu Wei. A new algorithm for non-stationary contextual bandits: Efficient, optimal and parameter-free. In Conference on Learning Theory , pages 696-726. PMLR, 2019.

Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 208-214. JMLR Workshop and Conference Proceedings, 2011.

Yan Dai, Haipeng Luo, Chen-Yu Wei, and Julian Zimmert. Refined regret for adversarial mdps with linear function approximation. In International Conference on Machine Learning , 2023.

Christoph Dann, Chen-Yu Wei, and Julian Zimmert. Best of both worlds policy optimization. In International Conference on Machine Learning , 2023a.

Christoph Dann, Chen-Yu Wei, and Julian Zimmert. A blackbox approach to best of both worlds in bandits and beyond. In Conference on Learning Theory , 2023b.

MDudik, D Hsu, S Kale, N Karampatziakis, J Langford, L Reyzin, and T Zhang. Efficient optimal learning for contextual bandits. In Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence, UAI 2011 , page 169, 2011.

Ehsan Emamjomeh-Zadeh, Chen-Yu Wei, Haipeng Luo, and David Kempe. Adversarial online learning with changing action sets: Efficient algorithms with approximate regret bounds. In Algorithmic Learning Theory , pages 599-618. PMLR, 2021.

Dylan Foster and Alexander Rakhlin. Beyond ucb: Optimal and efficient contextual bandits with regression oracles. In International Conference on Machine Learning , pages 3199-3210. PMLR, 2020.

Dylan Foster, Alekh Agarwal, Miroslav Dudík, Haipeng Luo, and Robert Schapire. Practical contextual bandits with regression oracles. In International Conference on Machine Learning , pages 1539-1548. PMLR, 2018.

Dylan Foster, Alexander Rakhlin, David Simchi-Levi, and Yunzong Xu. Instance-dependent complexity of contextual bandits and reinforcement learning: A disagreement-based perspective. In Conference on Learning Theory , pages 2059-2059. PMLR, 2021.

Dylan J Foster and Akshay Krishnamurthy. Efficient first-order contextual bandits: Prediction, allocation, and triangular discrimination. In Advances in Neural Information Processing Systems , 2021.

Dylan J Foster, Claudio Gentile, Mehryar Mohri, and Julian Zimmert. Adapting to misspecification in contextual bandits. Advances in Neural Information Processing Systems , 33:11478-11489, 2020.

Elad Hazan and Tomer Koren. The computational power of optimization in online learning. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing , pages 128-141, 2016.

Tiancheng Jin, Junyan Liu, Chloé Rouyer, William Chang, Chen-Yu Wei, and Haipeng Luo. Noregret online reinforcement learning with adversarial losses and transitions. arXiv preprint arXiv:2305.17380 , 2023.

Varun Kanade and Thomas Steinke. Learning hurdles for sleeping experts. ACM Transactions on Computation Theory (TOCT) , 6(3):1-16, 2014.

Varun Kanade, H Brendan McMahan, and Brent Bryan. Sleeping experts and bandits with stochastic action availability and adversarial rewards. In Artificial Intelligence and Statistics , pages 272-279. PMLR, 2009.

Leonid G Khachiyan and Michael J Todd. On the complexity of approximating the maximal inscribed ellipsoid for a polytope. Technical report, Cornell University Operations Research and Industrial Engineering, 1990.

Robert Kleinberg, Alexandru Niculescu-Mizil, and Yogeshwer Sharma. Regret bounds for sleeping experts and bandits. Machine learning , 80(2-3):245-272, 2010.

Fang Kong, Xiangcheng Zhang, Baoxiang Wang, and Shuai Li. Improved regret bounds for linear adversarial mdps via linear optimization. arXiv preprint arXiv:2302.06834 , 2023.

John Langford and Tong Zhang. The epoch-greedy algorithm for contextual multi-armed bandits. Advances in neural information processing systems , 20(1):96-1, 2007.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Chung-Wei Lee, Haipeng Luo, Chen-Yu Wei, and Mengxiao Zhang. Bias no more: high-probability data-dependent regret bounds for adversarial bandits and mdps. In Advances in Neural Information Processing Systems , 2020.

Gene Li, Pritish Kamath, Dylan J Foster, and Nati Srebro. Understanding the eluder dimension. Advances in Neural Information Processing Systems , 35:23737-23750, 2022.

Yingkai Li, Yining Wang, and Yuan Zhou. Nearly minimax-optimal regret for linearly parameterized bandits. In Conference on Learning Theory , pages 2173-2174. PMLR, 2019.

Tzon-Tzer Lu and Sheng-Hua Shiou. Inverses of 2 × 2 block matrices. Computers & Mathematics with Applications , 43(1-2):119-129, 2002.

Haipeng Luo, Chen-Yu Wei, Alekh Agarwal, and John Langford. Efficient contextual bandits in non-stationary worlds. In Conference On Learning Theory , pages 1739-1776. PMLR, 2018.

Haipeng Luo, Chen-Yu Wei, and Chung-Wei Lee. Policy optimization in adversarial mdps: Improved exploration via dilated bonuses. Advances in Neural Information Processing Systems , 34:2293122942, 2021.

Haipeng Luo, Mengxiao Zhang, Peng Zhao, and Zhi-Hua Zhou. Corralling a larger band of bandits: A case study on switching regret for linear bandits. In Conference on Learning Theory , pages 3635-3684. PMLR, 2022.

Arkadi Nemirovski. Interior point polynomial time methods in convex programming. Lecture notes , 42(16):3215-3224, 2004.

Gergely Neu and Julia Olkhovskaya. Efficient and robust algorithms for adversarial linear contextual bandits. In Conference on Learning Theory , pages 3049-3068. PMLR, 2020.

Gergely Neu and Julia Olkhovskaya. Online learning in mdps with linear function approximation and bandit feedback. Advances in Neural Information Processing Systems , 34:10407-10417, 2021.

Gergely Neu and Michal Valko. Online combinatorial optimization with stochastic decision sets and adversarial losses. Advances in Neural Information Processing Systems , 27, 2014.

Julia Olkhovskaya, Jack Mayo, Tim van Erven, Gergely Neu, and Chen-Yu Wei. First-and secondorder bounds for adversarial linear contextual bandits. arXiv preprint arXiv:2305.00832 , 2023.

Alexander Rakhlin and Karthik Sridharan. Bistro: An efficient relaxation-based method for contextual bandits. In International Conference on Machine Learning , pages 1977-1985. PMLR, 2016.

Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems , 26, 2013.

Aadirupa Saha, Pierre Gaillard, and Michal Valko. Improved sleeping bandits with stochastic action sets and adversarial rewards. In International Conference on Machine Learning , pages 8357-8366. PMLR, 2020.

Uri Sherman, Tomer Koren, and Yishay Mansour. Improved regret for efficient online reinforcement learning with linear function approximation. In International Conference on Machine Learning , 2023.

David Simchi-Levi and Yunzong Xu. Bypassing the monster: A faster and simpler optimal algorithm for contextual bandits under realizability. Mathematics of Operations Research , 47(3):1904-1931, 2022.

Vasilis Syrgkanis, Akshay Krishnamurthy, and Robert Schapire. Efficient algorithms for adversarial contextual learning. In International Conference on Machine Learning , pages 2159-2168. PMLR, 2016a.

Vasilis Syrgkanis, Haipeng Luo, Akshay Krishnamurthy, and Robert E Schapire. Improved regret bounds for oracle-based adversarial contextual bandits. Advances in Neural Information Processing Systems , 29, 2016b.

Ambuj Tewari and Susan A Murphy. From ads to interventions: Contextual bandits in mobile health. Mobile Health: Sensors, Analytic Methods, and Applications , pages 495-517, 2017.

Chen-Yu Wei and Haipeng Luo. More adaptive algorithms for adversarial bandits. In Conference on Learning Theory , pages 1263-1291. PMLR, 2018.

Chen-Yu Wei, Christoph Dann, and Julian Zimmert. A model selection approach for corruption robust reinforcement learning. In International Conference on Algorithmic Learning Theory , pages 1043-1096. PMLR, 2022.

Yunbei Xu and Assaf Zeevi. Upper counterfactual confidence bounds: a new optimism principle for contextual bandits. arXiv preprint arXiv:2007.07876 , 2020.

Tong Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science , 4(2):834-857, 2022.

Heyang Zhao, Jiafan He, Dongruo Zhou, Tong Zhang, and Quanquan Gu. Variance-dependent regret bounds for linear bandits and reinforcement learning: Adaptivity and computational efficiency. In Conference on Learning Theory , 2023.

Yinglun Zhu, Dylan J Foster, John Langford, and Paul Mineiro. Contextual bandits with large action spaces: Made practical. In International Conference on Machine Learning , pages 27428-27453. PMLR, 2022.

Julian Zimmert and Tor Lattimore. Return of the bias: Almost minimax optimal high probability bounds for adversarial linear bandits. In Conference on Learning Theory , pages 3285-3312. PMLR, 2022.

Julian Zimmert, Naman Agarwal, and Satyen Kale. Pushing the efficiency-regret pareto frontier for online learning of portfolios and quantum states. In Conference on Learning Theory , pages 182-226. PMLR, 2022.

Appendices

A Summary of Notation

We summarize the notations that have been defined in Algorithm 1 and Definition 1.

B Auxiliary Lemmas

Lemma 5 (FTRL regret bound, Lemma 18 of Dann et al. [2023a]) . Let Ω ⊂ R d be a convex set, g 1 , . . . , g T ∈ R d , and η 1 , . . . , η T > 0 . Then the FTRL update

ensures for any u ∈ Ω and η 0 > 0 ,

When η 0 , η 1 , . . . , η T is non-increasing, the penalty term can further be upper bounded by

Lemma 6 (Bernstein's inequality) . Let X 1 , · · · , X n be iid random variables; let E [ X ] be the expectation and Var( X ) be the variance of these random variables. If for any i , | X i -E [ X i ] | ≤ R , then with probability of at least 1 -δ ,

Lemma 7 (Hoeffding's inequality) . Let X 1 , · · · , X n be iid random variables; let a ≤ X i ≤ b and let E [ X ] be the expectation. Then with probability of at least 1 -δ ,

Given F ( X ) = -log det( X ) , D 2 F ( X ) = X -1 ⊗ X -1 where ⊗ is the Kronecker prod-

 

A to a column vector by stacking the columns A . The second order directional derivative for F is D 2 F ( X )[ A,A ] = vec ( A ) T ( X -1 ⊗ X -1 ) vec ( A ) = Tr( A ⊤ X -1 AX -1 ) . We define ∥ A ∥ ∇ 2 F ( X ) = √ Tr( A ⊤ X -1 AX -1 ) and ∥ A ∥ ∇ -2 F ( X ) = √ Tr( A ⊤ XAX ) . It is a pseudo-norm and more discussion can be found in Appendix D of Zimmert et al. [2022]. In the following analysis, we will only use one property of this pseudo-norm shown below which is similar to the Holder inequality.

Lemma 8. For any two symmetric matrices A,B and positive definite matrix X ,

Proof. Since ( X ⊗ X ) -1 = X -1 ⊗ X -1 , from Holder inequality, we have

C Concentration Inequalities

The goal of this section is to show Lemma 18 and Lemma 19, which are key to bound the bias term. We first introduce a useful lemma from Dai et al. [2023], which will be used later to prove our concentration bounds.

C.1 General Concentration Inequalities

Lemma 9 (Lemma A.4 in Dai et al. [2023]) . Let H 1 , H 2 , . . . , H n be i.i.d. PSD matrices such that E [ H i ] = H , H i ⪯ I almost surely and H ⪰ 1 dn log d δ I . Then with probability 1 -δ ,

Corollary 10. Let H 1 , H 2 , . . . , H n be i.i.d. PSD matrices such that E [ H i ] = H and H i ⪯ cI almost surely for some positive constant c . Let ˆ H = 1 n ∑ n i =1 H i , then with probability 1 -δ ,

Proof. A simple corollary of Lemma 9 under the condition of Lemma 9 is that

where we use that H 1 2 ⪯ k 2 H + 1 2 k for any k > 0 .

Now consider the condition of this corollary. We first consider the case where d n log( d δ ) ≤ 1 . In this case, we apply Eq. (10) with H ′ i = 1 2 c H i + d 2 n log( d δ ) I , which satisfies the condition for Eq. (10) to hold. This gives

with probability at least 1 -δ . When d n log( d δ ) > 1 . Eq. (9) is trivial because 1 2 H ⪯ c 2 I ⪯ c 2 · d n log( d δ ) I .

C.2 Concentration Inequalities under a Fixed Policy p

In this subsection, we establish concentration bounds for a fixed policy p (with p A ∈ ∆( A ) denoting the action distribution it uses over A ) over i.i.d. contexts. The results in this subsection are preparation for Appendix C.3 where we take union bounds over policies.

The setting and notation to be used in this subsection are defined in Definition 11.

Definition 11. Let {A 1 , . . . , A n } be i.i.d. context samples drawn from D . Let ˆ D be the uniform distribution over {A 1 , . . . , A n } .

Over this set of context samples, define for any policy p ,

where β = 5 d log(6 d/δ ) n .

Lemma 12. Under the setting of Definition 11, for any fixed p , with probability at least 1 -δ ,

Proof. In this proof, we use ˆ x, x, ˆ H,H, ˆ H ˆ H ˆ H,H to denote ˆ x ( p ) , x ( p ) , ˆ H ( p ) , H ( p ) , ˆ H ˆ H ˆ H ( p ) , H ( p ) since p is fixed throughout the proof.

Since ∥ a ∥ ≤ 1 , H ⪯ 2 I and ˆ H ˆ H ˆ H ⪯ 2 I . Thus, we can directly apply Corollary 10 with c = 2 to get with probability 1 -δ 3

To prove the first inequality, we first decompose H and ˆ H

From Hoeffding inequality (Lemma 7) and union bound, with probability 1 -δ 3 , for all k ∈ [ d ] , we have

which implies that e ⊤ k ( x -ˆ x )( x -ˆ x ) ⊤ e k ≤ 1 2 n log( 6 d δ ) for all k , and thus

By directly applying Corollary 10 with c = 2 , we get with probability at least 1 -δ 3 ,

Further using Eq. (11), Eq. (12) and Eq. (13), we get with probability at least 1 -2 δ 3 ,

Taking union bound for both inequality finishes the proof.

Lemma 13. Under the setting of Definition 11, for any fixed policy p , with probability at least 1 -O ( δ ) ,

Proof. In this proof, we use ˆ x, x, ˆ H,H, ˆ H ˆ H ˆ H,H, ˆ Σ , ˆ Σ ˆ Σ ˆ Σ to denote ˆ x ( p ) , x ( p ) , ˆ H ( p ) , H ( p ) , ˆ H ˆ H ˆ H ( p ) , H ( p ) , ˆ Σ( p ) , ˆ Σ ˆ Σ( p ) since p is fixed throughout the proof.

We first rewrite H .

To simplify analysis, we perform diagonalization. Suppose that E A∼ D E a ∼ p A [( a -x )( a -x ) ⊤ ] admits the following eigen-decomposition:

where V is an orthogonal matrix and Λ is a diagonal matrix. By Lemma 12 and the definition of β in Definition 11, we have with probability 1 -δ ,

with some ρ = Θ ( d log( d/δ ) n ) , where the second inequality is by Eq. (14). Thus,

Define

Since E A i ∼ D [ Z ( i ) k ] = Z k , by Bernstein's inequality, with probability at least 1 -δ , we have

for all k , where

On the other hand,

From Jensen's inequality,

Thus,

Lemma 14. Under the setting of Definition 11, for any fixed policy p , with probability at least 1 -O ( δ ) ,

for any y ∈ B d 2 .

Proof. In this proof, we use ˆ x, x, ˆ H,H, ˆ H ˆ H ˆ H,H, ˆ Σ , ˆ Σ ˆ Σ ˆ Σ to denote ˆ x ( p ) , x ( p ) , ˆ H ( p ) , H ( p ) , ˆ H ˆ H ˆ H ( p ) , H ( p ) , ˆ Σ( p ) , ˆ Σ ˆ Σ( p ) since p is fixed throughout the proof.

First, we re-write H and ˆ H :

Then, by definition (in Definition 11) and the calculation above,

define this as

Γ

Using ∥ a + b + c ∥ 2 ≤ 3 ∥ a ∥ 2 +3 ∥ b ∥ 2 +3 ∥ c ∥ 2 , we have

The second and third term are bounded by O ( d log( d/δ ) n ) using Lemma 13 and the definition of β , with probability at least 1 -O ( δ ) . Below, we further deal with the first term. To simplify analysis, we perform diagonalization. Suppose that E A∼ D E a ∼ p A [( a -x )( a -x ) ⊤ ] admits the following eigen-decomposition:

where V is an orthogonal matrix and Λ is a diagonal matrix. Then

Below, we further deal with the V ⊤ Γ V and V ⊤ Λ V terms in Eq. (18). By Lemma 12, with probability at least 1 -δ ,

for some ρ = Θ ( d log( d/δ ) n ) , where we use Eq. (16) in the second inequality. Therefore,

Next, denote ∆ = V ⊤ Γ V . By definition, it can be written as the following:

with Λ ( i ) being i.i.d. samples with mean E [Λ ( i ) ] = Λ . While these are d × d matrices, we will apply concentration inequalities to individual entries.

Let λ ikh = e ⊤ k Λ ( i ) e h be the ( k, h ) -th entry of Λ ( i ) . Notice that E [ λ ikh ] = e ⊤ k Λe h = Λ kh , the ( k, h ) -th entry of Λ .

By Bernstein's inequality, with probability at least 1 -δ , we have

With the manipulations and notations above, we continue to bound Eq. (18) by

By direct expansion and the fact that Λ is diagonal,

By definition, and thus

and ∑ d k =1 E [ λ 2 ikh ] ≤ E [ λ ihh ] = Λ hh . Continuing from Eq. (21) and using that ρ = Θ ( d log( d/δ ) n ) ,

This gives a bound on ∥ Γ y ∥ 2 ˆ Σ -1 and finishes the proof after combining Eq. (17).

C.3 Union Bound over Policies

In Lemma 12, Lemma 13, and Lemma 14, we have obtained the desired concentration inequalities under a fixed policy p . In this subsection, we proceed to take union bound over all policies that are possibly used by Algorithm 1.

The set of policies that could be generated by Algorithm 1 is the following:

where Z = [ -T 2 , T 2 ] ( d +1) × ( d +1) ∩ S with S denoting the set of symmetric matrices. To see this, notice that Algorithm 1 at round t corresponds to the policy defined above with Z = η t ∑ t -1 s =1 (ˆ γ s -α s ˆ Σ ˆ Σ ˆ Σ -1 s ) .

Our goal is to construct a ϵ -cover P ′ so that every policy p ∈ P can find a policy p ′ ∈ P ′ making -ϵI ⪯ ̂ Cov( p A ) -̂ Cov( p ′A ) ⪯ ϵI on every action set A . The size of such a cover is bounded in the Proposition below.

Lemma 15. There exists an ϵ -cover P ′ of P with size log | P ′ | = O ( d 2 log d ϵ ) such that for any p ∈ P , there exists an p ′ ∈ P ′ satisfying

for all A .

Proof. It is straightforward to construct an ϵ 4 -cover C for Z = [ -T 2 , T 2 ] ( d +1) × ( d +1) ∩ S in Frobenius norm with size |C| = ( 24( d +1) 2 ϵ ) ( d +1) 2 (Exercise 27.6 of Lattimore and Szepesvári [2020]). Now define P ′ as

Below, we show that this is a ϵ -cover for P .

Consider two policies p 1 and p 2 defined as the following:

with ∥ Z 1 -Z 2 ∥ F ≤ ϵ 4 . Consider an arbitrary A and define H 1 = ̂ Cov( p A 1 ) , H 2 = ̂ Cov( p A 2 ) . Below we show ∥ H 1 -H 2 ∥ F ≤ ϵ .

Since F ( H ) is convex for H , from the first-order optimality condition for convex function, we have

Adding up these the two inequalities, we get

Since the second order directional derivative for F is D 2 F ( H )[ X,X ] = Tr( XH -1 XH -1 ) for any symmetric matrix X , from the Taylor series, there exists H ′ that is a line segment between H 1 and H 2 such that

Thus we have ∥ H 1 -H 2 ∥ ∇ 2 F ( H ′ ) ≤ ∥ Z 1 -Z 2 ∥ ∇ -2 F ( H ′ ) . Since ∥ a ∥ 2 ≤ 1 , H ′ ⪯ 2 I . The left-hand side and right-hand side can be bounded as follows,

Combining the three inequalities above, we conclude that

Lemma 16. Suppose that p, p ′ are two policies such that for all action set A ,

Then all quantities defined in Definition 11 under p and p ′ are close. That is,

Proof. Eq. (28) and Eq. (29) are direct consequences of Eq. (23) since H ( p ) and ˆ H ˆ H ˆ H ( p ) are expectations of ̂ Cov( p A ) over distributions over A . Eq. (31) is directly implied by Eq. (29) because ˆ Σ ˆ Σ( p ) = ˆ H ˆ H ˆ H ( p ) + βI .

To show Eq. (24) and Eq. (25), observe that by the definition of x ( p ) and H ( p ) ,

Therefore, ∥ x ( p ) -x ( p ′ ) ∥ ≤ ∥ H ( p ) -H ( p ′ ) ∥ F ≤ ϵ . Similarly, ∥ ˆ x ( p ) -ˆ x ( p ′ ) ∥ ≤ ∥ ˆ H ˆ H ˆ H ( p ) -ˆ H ˆ H ˆ H ( p ′ ) ∥ F ≤ ϵ .

If remains to show Eq. (26), Eq. (27) and Eq. (30). Next, we show Eq. (26):

Using the property

∥

we continue from Eq. (32) and bound

Eq. (27) can be shown in the same manner, which further implies Eq. (30) by the definition of ˆ Σ( p ) .

Lemma 17. With probability 1 -δ , for all t = 1 , · · · , T ,

Proof. Notice that ˆ H t , ˆ H ˆ H ˆ H t , H t , H t corresponds to ˆ H ( p t ) , ˆ H ˆ H ˆ H ( p t ) , H ( p t ) , H ( p t ) defined in Definition 11 with n = t -1 . To show the lemma, our strategy is to argue the following two facts: 1) the two desired inequalities hold for all policies in the cover P ′ (defined in Eq. (22)) with high probability. This is simply by applying Lemma 12 with an union bound over policies in P ′ . 2) p t is sufficiently close to the nearest element in P ′ so the desired inequalities still approximately hold.

By Lemma 15, we can find p ′ ∈ P ′ such that for all A ,

On the other hand, using Lemma 12 and union bound, with probability 1 -δ , we have

Combining Eq. (35) and Eq. (33), we get

which implies the first inequality in the lemma by plugging in the choice of ϵ = 1 T 3 and the upper bound of log | P ′ | in Lemma 16. The second inequality in the lemma can be obtained similarly by combining Eq. (34) and Eq. (36).

Lemma 18. With probability of at least 1 -δ , for all t = 1 , · · · , T ,

Proof. Notice that x t , ˆ x t , ˆ Σ t corresponds to x ( p t ) , ˆ x ( p t ) , ˆ Σ( p t ) defined in Definition 11 with n = t -1 . To show the lemma, our strategy is to argue the following two facts: 1) the two desired inequalities hold for all policies in the cover P ′ with high probability. This is simply by applying Lemma 13 with an union bound over policies in P ′ . 2) p t is sufficiently close to the nearest element in P ′ so the desired inequalities still approximately hold.

By Lemma 15, we can find p ′ ∈ P ′ such that for all A ,

Thus,

The first two terms above can be bounded by the order of O ( ϵt 2 ) by Eq. (37). Using the choice ϵ = 1 T 3 and recalling that log | P ′ | = O ( d 2 log( d/ϵ )) finishes the proof.

Lemma 19. With probability of at least 1 -δ , for all t = 1 , 2 , . . . , T ,

Proof. Notice that x t , ˆ x t , ˆ Σ t corresponds to x ( p t ) , ˆ x ( p t ) , ˆ Σ( p t ) defined in Definition 11 with n = t -1 . To show the lemma, our strategy is to argue the following two facts: 1) the two desired inequalities hold for all policies in the cover P ′ with high probability. This is simply by applying Lemma 13 with an union bound over policies in P ′ . 2) p t is sufficiently close to the nearest element in P ′ so the desired inequalities still approximately hold.

By Lemma 15, we can find p ′ ∈ P ′ such that for all A ,

∥ ∥ ∥ ̂ Cov( p A t ) -̂ Cov( p ′A ) ∥ ∥ ∥ F ≤ ϵ. By Lemma 16, we have ∥ x ( p ′ ) -x ( p t ) ∥ ≤ ϵ, ∥ ˆ x ( p ′ ) -ˆ x ( p t ) ∥ ≤ ϵ, ∥ ˆ Σ( p ′ ) -ˆ Σ( p t ) ∥ F ≤ 7 ϵ (38) Thus, for any ∥ y t ∥ 2 ≤ 1 , ∥ ( ˆ Σ( p t ) -H ( p t )) y t ∥ 2 ˆ Σ( p t ) -1 = ( ∥ ( ˆ Σ( p t ) -H ( p t )) y t ∥ 2 ˆ Σ( p t ) -1 -∥ ( ˆ Σ( p ′ ) -H ( p ′ )) y t ∥ 2 ˆ Σ( p ′ ) -1 ) + ∥ ( ˆ Σ( p ′ ) -H ( p ′ )) y t ∥ 2 ˆ Σ( p ′ ) -1 ≤ ( ∥ ( ˆ Σ( p t ) -H ( p t )) y t ∥ 2 ˆ Σ( p t ) -1 -∥ ( ˆ Σ( p ′ ) -H ( p ′ )) y t ∥ 2 ˆ Σ( p ′ ) -1 ) + O ( d log( d | P ′ | /δ ) t -1 ) (by Lemma 14 with an union bound over P ′ ) = θ ⊤ t ˆ Σ( p t ) -1 θ t -θ ′⊤ ˆ Σ( p ′ ) -1 θ ′ + O ( d log( d | P ′ | /δ ) t -1 ) (define θ t = ( ˆ Σ( p t ) -H ( p t )) y t and θ ′ = ( ˆ Σ( p ′ ) -H ( p ′ )) y t ) = ( θ t -θ ′ ) ⊤ ˆ Σ( p t ) -1 θ t + θ ′⊤ ( ˆ Σ( p t ) -1 -ˆ Σ( p ′ ) -1 ) θ t + θ ′⊤ ˆ Σ( p ′ ) -1 ( θ t -θ ′ ) + O ( d log( d | P ′ | /δ ) t -1 ) ≤ ( θ t -θ ′ ) ⊤ ( ˆ Σ( p t ) -1 θ t + ˆ Σ( p ′ ) -1 θ ′ ) + θ ′⊤ ˆ Σ( p ′ ) -1 ( ˆ Σ( p ′ ) -ˆ Σ( p t ) ) ˆ Σ( p t ) -1 θ t + O ( d log( d | P ′ | /δ ) t -1 )

The first two terms above can be bounded by the order of O ( ϵt 2 ) by Eq. (38). Plugging in the choice of ϵ = 1 T 3 and recalling that log | P ′ | = O ( d 2 log( d/ϵ )) finishes the proof.

D Regret Analysis

Consider the regret decomposition in Section 3.5.

where A 0 is drawn from D and is independent from the interaction between the learning and the environment. Recall that our algorithm is FTRL:

The FTRL-Reg term can be handled by the standard FTRL analysis (Lemma 5). In order to deal with the issue that F can be unbounded on the boundary of H A 0 , we apply Lemma 5 with the regret comparator U A 0 defined as

where H A 0 ∗ ≜ argmin H ∈H A 0 F ( H ) . Thus,

FTRL-Reg

In the rest of this section, we bound the following terms individually: Bias , Bonus , Penalty , Stability-1 , Stability-2 , Error .

For any t = 2 , · · · , T , let E t -1 be the event that the high-probability event in Lemma 17, Lemma 18, and Lemma 19 happens for all 1 , · · · , t -1 and E t -1 be the opposite event of E t -1 (i.e. any of these three lemmas fails for any 1 , · · · , t -1 ). We have P [ E t -1 ] = 1 -O ( δ ) and P [ E t -1 ] = O ( δ ) . Let E [ · | E t -1 ] be the conditional expectation that event E t -1 happens and let E E t = E [ · | F t -1 , E t -1 ]

D.1 Bounding the Bias term

Lemma 20.

∥ ˆ y t ∥ 2 = ∥ Σ t ( a t -ˆ x t ) a t y t ∥ 2 ≤ ∥ Σ t ( a t -ˆ x t ) ∥ 2 ≤ O ( T )

Thus, we have trivial bound

Therefore, we have

D.2 Bounding the Bonus term

We first prove the following useful technique lemma to bound the inner product of lifted matrices.

Proof. From Theorem 2.1 of Lu and Shiou [2002], for any block matrix R = [ A B C D ] if A is invertible and its Schur complement S A = D -CA -1 B is invertible, then

Using above equation, for the first equation, Since ( H + hh ⊤ ) -1 = H -1 -H -1 hh ⊤ H -1 1+ h ⊤ H -1 h . The inverse Schur complement of H + hh ⊤ is 1 + h ⊤ H -1 h . Thus

and

For the second equation, observe that

Applying the first equality, we have

Below, we continue to lower bound this term. By the same formula above, we have

Thus

Using Lemma 21, we are able to show Corollary 22 which bound part of the second term.

Proof. From Lemma 21, we have

Since ˆ Σ t ⪰ β t I , ˆ Σ -1 t ⪯ 1 β t I . Since ∥ ˆ x t ∥ 2 ≤ 1 , we have ∥ ˆ x t ∥ 2 ˆ Σ -1 t ≤ 1 β t . Then

Lemma 23.

Proof. For any t , we have

On the other hand, since ˆ Σ ˆ Σ ˆ Σ t ⪰ 1 t I ⪰ 1 T I , we have trivial bound

Therefore, we have

D.3 Bounding the Penalty term

Lemma 24. U A 0 , we have

Proof. Since U A 0 = ( 1 -1 T 2 ) U A 0 + 1 T 2 H A 0 ∗ , we have U A 0 ⪰ 1 T 2 H A 0 ∗ . Then

D.4 Bounding the Stability-1 term

Zimmert and Lattimore [2022] gave a useful identity to bound the Bregman divergence. We restate it in Lemma 25 for completeness.

Proof.

Lemma 26. For any H ∈ H A 0 , we have

Proof. Recall that H A 0 t = ̂ Cov( p A 0 t ) and ̂ Cov( p ) = [ Cov( p ) + µ ( p ) µ ( p ) ⊤ µ ( p ) µ ( p ) ⊤ 1 ] , we have

Since E A∼D E a ∼ p A [ ( a -ˆ x t )( a -ˆ x t ) ⊤ ] = H t , taking expectations over A t , a t and A 0 conditioned on E t -1 , we have

Notice that given E t -1 ,

Hence we continue to upper bound the last expression by

On the other hand, since t ⪰ t I ⪰ T I , we have trivial bound

Combining everything, we get

D.5 Bounding the Stability-2 term

Note that Lemma 8 does not require matrix A,B to be positive semi-definite. We will use it to prove the following lemma based on Lemma 34 in Dann et al. [2023b].

Lemma 27. If η t α t ≤ 1 64 t , then

Proof. We first show that max H ∈H A 0 〈 H A 0 t -H, -α t ˆ Σ ˆ Σ ˆ Σ -1 t 〉 -D ( H,H A 0 t ) 2 η t ≤ α t 2 ∥ ˆ Σ ˆ Σ ˆ Σ -1 t ∥ ∇ -2 F ( H A 0 t ) .

Define

and λ = ∥ α t ˆ Σ ˆ Σ ˆ Σ -1 t ∥ ∇ -2 F ( H A 0 t ) . Since ˆ Σ ˆ Σ ˆ Σ t ⪰ 1 t I , H A 0 t ⪯ 2 I , η t α t ≤ 1 64 t , we have

Let H ′ be the maximizer of G . Since G ( H A 0 t ) = 0 , we have G ( H ′ ) ≥ 0 . It suffices to show ∥ H ′ -H A 0 t ∥ ∇ 2 F ( H A 0 t ) ≤ 16 η t λ because from Lemma 8, it leads to

To show ∥ H ′ -H A 0 t ∥ ∇ 2 F ( H A 0 t ) ≤ 16 η t λ , it suffices to show that for all U such that ∥ U -H A 0 t ∥ ∇ 2 F ( H A 0 t ) = 16 η t λ , G ( U ) ≤ 0 . This is because given this condition, if ∥ H ′ -H A 0 t ∥ ∇ 2 F ( H A 0 t ) > 16 η t λ , then there is a U in the line segment between H A 0 t and H ′ such that ∥ U -H A 0 t ∥ ∇ 2 F ( H A 0 t ) = 16 η t λ . From the condition, G ( U ) ≤ 0 ≤ min { G ( H A 0 t ) , G ( H ′ ) } which contradicts to the strictly concave of G .

Now consider any U such that ∥ U -H A 0 t ∥ ∇ 2 F ( H A 0 t ) = 16 η t λ . By Taylor expansion, there exists U ′ in the line segment between U and H A 0 t such that

We have ∥ U ′ -H A 0 t ∥ ∇ 2 F ( H A 0 t ) ≤ ∥ U -H A 0 t ∥ ∇ 2 F ( H A 0 t ) = 16 η t λ ≤ 1 2 . From the Equation 2.2 in page 23 of Nemirovski [2004] (also appear in Eq.(5) of Abernethy et al. [2009]) and log det is a self-concordant function, we have ∥ U -H A 0 t ∥ 2 ∇ 2 F ( U ′ ) ≥ 1 4 ∥ U -H A 0 t ∥ 2 ∇ 2 F ( H A 0 t ) . Thus, we have

We have ∥ ˆ Σ ˆ Σ ˆ Σ -1 t ∥ ∇ -2 F ( H A 0 t ) = √ Tr( H A 0 t ˆ Σ ˆ Σ ˆ Σ -1 t H A 0 t ˆ Σ ˆ Σ ˆ Σ -1 t ) = √ Tr(( H A 0 t ˆ Σ ˆ Σ ˆ Σ -1 t ) 2 ) . Observe the following two facts: 1) all eigenvalues of H A 0 t ˆ Σ ˆ Σ ˆ Σ -1 t are non-negative since H A 0 t and ˆ Σ ˆ Σ ˆ Σ -1 t are both positive semi-definite, 2) for a square matrix A with all non-negative eigenvalues, Tr( A 2 ) ≤ Tr( A ) 2 because Tr( A 2 ) = ∑ i λ i ( A 2 ) = ∑ i λ i ( A ) 2 ≤ ( ∑ i λ i ( A )) 2 . We have

This allows us to conclude

where we use that ˆ Σ ˆ Σ ˆ Σ t ⪰ 1 2 E A 0 ∼ D [ H A 0 t ] given E t -1 .

On the other hand, since ˆ Σ ˆ Σ ˆ Σ t ⪰ 1 t I ⪰ 1 T I , for any t = 1 , · · · , T , we have trivial bound

Overall,

D.6 Bounding the Error term

Lemma 28.

Proof. Since U A 0 = ( 1 -1 T 2 ) U A 0 + 1 T 2 H A 0 ∗ , and ˆ Σ t ⪰ 1 T I, ˆ Σ ˆ Σ ˆ Σ t ⪰ 1 T I we have

D.7 Finishing up

Recall the regret decomposition at the beginning of Appendix D. From Lemma 24, Lemma 26, Lemma 27, and Lemma 28, we have

FTRL-Reg = Penalty + Stability-1 + Stability-2 + Error

From Lemma 20 and Lemma 23, we can cancel out the additional regret induced by bias through the well-designed bonus term. Namely,

Thus, we have

Recall that we have an additional condition in Lemma 27 such that for any t , η t α t ≤ 1 64 t . Picking α t = d √ t , η t = 1 64 d √ t and δ = 1 T 2 , we get

where we assume d 2 ≤ √ T without loss of generality (otherwise the bound is vacuous).

E Handling Misspecification

In this section, we discuss how to handle misspecification as defined in Section 3.6. In Appendix E.1, we study the case where the amount of misspecification ε is known by the learner. In Appendix E.2, we use a blackbox approach to turn it into an algorithm that achieves almost the same regret bound (up to log T factors) without knowning ε .

E.1 Known misspecification

As discussed in Section 3.6, when the amount of misspecification ε is known, we still use Algorithm 1, but with different α t and η t . Throughout this subsection, we let α t = d √ t + ε √ d and η t = 1 64 ( d √ t + ε √ d t ) , and point out the modifications of the analysis from Appendix D.

We start with the regret decomposition similar to that in Appendix D, but here we define

The regret decomposition goes as follows:

Now ˆ y t = ˆ Σ -1 t ( a t -ˆ x t ) ℓ t with E [ ℓ t ] = a ⊤ t y t + c t ( a t ) .

For the Bias term, the proof is almost the same as Lemma 20. The only difference is that from the fourth line, we have

for some c t ( a t ) such that | c t ( a t ) | ≤ ε t . This leads to an additional term of

Plugging it into the proof of Lemma 20, we have

Other parts of the proof follow those in Lemma 20. Finally, we get

The Bonus term will not be affected, according to Lemma 23, we have

The Penalty term will not be affected, according to Lemma 24, we have

Stability-1 term is also unchanged, as we assume that ℓ t still lies in [ -1 , 1] even under misspecification. We still have

The Stability-2 term will not be affected as long as η t α t ≤ 1 64 t . According to Lemma 27, we have

The Error term is also unaffected. We still have Error = O (1) .

Adding these terms together, the regret caused by bias and the negative term induced by bonus cancel out. We have

Recall that we pick α t = d √ t + ε √ d . η t = 1 64 d √ t +64 ε √ d t and δ = 1 T 2 . This gives where we assume d 2 ≤ √ T without loss of generality.

E.2 Unknown misspecification

In this subsection, we use a model selection technique to convert the algorithm in Appendix E.1 which requires knowledge on ε into an algorithm that achieves a similar regret bound without knowing ε . Such a procedure to handle unknown misspecification/corruption has appeared in several previous works [Foster et al., 2020, Wei et al., 2022], though we adopt the technique in Jin et al. [2023] to handle the adversarial case.

The idea here is a black-box reduction which turns an algorithm that only deals with known ε to one that handles unknown ε . More specifically, the reduction has two layers. The bottom layer takes as input an arbitrary misspecification-robust algorithm that operates under known ε (e.g., Algorithm 1), and outputs a stable misspecification-robust algorithm (formally defined later) that still operates under known ε . The top layer follows the standard Corral idea and takes as input a stable algorithm that operates under known ε , and outputs an algorithm that operates under unknown ε . Below, we explain these two layers of reduction in details.

Bottom Layer (from an Arbitrary Algorithm to a Stable Algorithm) The input of the bottom layer is an arbitrary misspecification-robust algorithm, formally defined as:

Definition 29. An algorithm is misspecification-robust if it takes θ as input, and achieves the following regret for any random stopping time t ′ ≤ T and any policy u :

for problem-dependent and log( T ) factors c 1 , c 2 ≥ 1 and ε 1: t ′ ≜ √ t ′ ∑ t ′ τ =1 ε 2 τ .

Algorithm 3 ST able A lgorithm B y I ndependent L earners and I nstance SE lection (STABILISE)

Input : ε and a base algorithm satisfying Definition 29. Initialize : ⌈ log 2 T ⌉ instances of the base algorithm ALG 1 , . . . , ALG ⌈ log 2 T ⌉ , where ALG j is configured with the parameter

In our case, c 1 = Θ( d 2 log T ) and c 2 = Θ( √ d ) . While the regret bound in Definition 29 might look cumbersome, it is in fact fairly reasonable: if the guess θ is not smaller than the true amount of ε 1: t ′ , the regret should be of order d 2 √ t ′ + √ dθ ; otherwise, the regret bound is vacuous since T is its largest possible value. The only extra requirement is that the algorithm needs to be anytime (i.e., the regret bound holds for any stopping time t ′ ), but even this is known to be easily achievable by using a doubling trick over a fixed-time algorithm. It is then clear that Algorithm 1 (together with a doubling trick) indeed satisfies Definition 29.

As mentioned, the output of the bottom layer is a stable robust algorithm. To characterize stability, we follow Agarwal et al. [2017] and define a new learning protocol that abstracts the interaction between the output algorithm of the bottom layer and the master algorithm from the top layer:

Protocol 1. In every round t , before the learner makes a decision, a probability w t ∈ [0 , 1] is revealed to the learner. After making a decision, the learner sees the desired feedback from the environment with probability w t , and sees nothing with probability 1 -w t .

One can convert any misspecification-robust algorithm (defined in Definition 29) into a stable misspecification-robust algorithm (characterized in Theorem 30).

This conversion is achieved by a procedure that called STABILISE (see Algorithm 3 for details). The high-level idea of STABILISE is as follows. Noticing that the challenge when learning in Protocol 1 is that w t varies over time, we discretize the value of w t and instantiate one instance of the input algorithm to deal with one possible discretized value, so that it is learning in Protocol 1 but with a fixed w t , making it straightforward to bound its regret based on what it promises in Definition 29.

More concretely, STABILISE instantiates O (log 2 T ) instances { ALG j } ⌈ log 2 T ⌉ j =0 of the input algorithm that satisfies Definition 29, each with a different parameter θ j . Upon receiving w t from the environment, it dispatches round t to the j -th instance where j is such that w t ∈ (2 -j -1 , 2 -j ] , and uses the policy generated by ALG j to interact with the environment (if w t ≤ 1 T , simply ignore this round). Based on Protocol 1, the feedback for this round is received with probability w t . To equalize the probability of ALG j receiving feedback as mentioned in the high-level idea, when the feedback is actually obtained, STABILISE sends it to ALG j only with probability 2 -j -1 w t (and discards it otherwise). This way, every time ALG j is assigned to a round, it always receives the desired feedback with probability w t · 2 -j -1 w t = 2 -j -1 . This equalization step allows us to use the original guarantee of the base algorithm (Definition 29) and run it as it is, without requiring it to perform extra importance weighting steps as in Agarwal et al. [2017].

The choice of θ j is crucial in making sure that STABILISE only has εT regret overhead instead of εT min t ∈ [ T ] w t . Since ALG j only receives feedback with probability 2 -j -1 , the expected total misspecification it experiences is on the order of 2 -j -1 εT . Therefore, its input parameter θ j only needs to be of this order instead of the total amount of misspecification εT .

The formal guarantee of the conversion is stated in the following Theorem 30.

Theorem 30. If an algorithm is misspecification robust according to Definition 29 for some constants ( c 1 , c 2 ) , then Algorithm 3 ensures

under Protocol 1, where ρ T = 1 min t ∈ [ T ] w t , with c ′ 1 = Θ(( c 1 + c 2 ) √ log T ) .

Proof of Theorem 30. Define indicators

Now we consider the regret of ALG j . Notice that ALG j makes an update only when g t,j h t,j = 1 . By the guarantee of the base algorithm (Definition 29), we have

We first bound the last term: Notice that E [ h t,j | g t,j ] = 2 -j -1 g t,j by Algorithm 3. Therefore,

By Freedman's inequality, with probability at least 1 -1 T 2 ,

which gives

with probability at least 1 -1 T 2 using Eq. (41). Similarly,

with probability at least 1 -1 T 2 . Therefore, with probability at least 1 -2 T 2 ,

Therefore, the last term in Eq. (40) is bounded by 2 T 2 T ≤ 2 T .

Next, we deal with other terms in Eq. (40). Again, by E [ h t,j | g t,j ] = 2 -j -1 g t,j , Eq. (40) implies

which implies after rearranging:

(using that when g t,j = 1 , 1 2 -j -1 ≤ 2 w t , and the definition of θ j )

Now, summing this inequality over all j ∈ { 0 , 1 , . . . , ⌈ log 2 T ⌉} , we get

where N ≤ O (log T ) is the number of ALG j 's that has been executed at least once. On the other hand,

Combining the two parts and using the assumption c 2 ≥ 1 finishes the proof.

Top Layer (from Known ε to Unknown ε ) In this subsection, we use the algorithm that we construct in Theorem 30 as a base algorithm, and further construct an algorithm with √ T + ε regret under unknown ε . The idea is to run multiple base algorithms, each with a different hypothesis on ε ; on top of them, run another multi-armed bandit algorithm to adaptively choose among them. The goal is to let the top-level bandit algorithm perform almost as well as the best base algorithm. This is the Corral idea outlined in Agarwal et al. [2017], Foster et al. [2020], Luo et al. [2022], and the algorithm is presented in Algorithm 4.

Theorem 31. Using an algorithm constructed in Theorem 30 as a base algorithm, Algorithm 4 ensures Reg = O ( c ′ 1 √ T log 3 T + c ′ 2 εT ) without knowing ε .

Algorithm 4 (A Variant of) Corral

Initialize : a log-barrier algorithm with each arm being an instance of an algorithm satisfying the guarantee in Theorem 30. The hypothesis on εT is set to 2 i for arm i ( i = 1 , 2 , . . . , M ≜ ⌈ log 2 T ⌉ ). Initialize : ρ 0 ,i = M, ∀ i .

where η = 1 4 c ′ 1 √ T .

For all i , send w t,i to instance i .

Draw i t ∼ w t .

Execute the a t output by instance i t

Receive the loss z t,i t for action a t (whose expectation is f t ( a t ) ) and send it to instance i t . Define for all i :

The top-level bandit algorithm is an FTRL with log-barrier regularizer. We first state the standard regret bound of FTRL under log-barrier regularizer, whose proof can be found in, e.g., Theorem 7 of Wei and Luo [2018].

Lemma 32. The FTRL algorithm over a convex subset Ω of the ( M -1) -dimensional simplex ∆( M ) :

as long as ηw t,i | ℓ t,i | ≤ 1 2 for all t, i .

Proof of Theorem 31. The Corral algorithm is essential an FTRL with log-barrier regularizer. To apply Lemma 32, we first verify the condition ηw t,i | ℓ t,i | ≤ 1 2 where ℓ t,i = ˆ z t,i -r t,i . By our choice of η ,

ensures for all u ∈ Ω ,

The right-hand side of the last equality is non-zero only when ρ t,i > ρ t -1 ,i , implying that ρ t,i = 1 w t,i . Therefore, we further bound it by

which can be combined to get the desired property ηw t,i | ˆ z t,i -r t,i | ≤ 1 2 . Hence, by the regret guarantee of log-barrier FTRL (Lemma 32), we have

where i ⋆ is the smallest i such that 2 i upper bounds the true total misspecification amount εT Bounding term 1 :

.

where

and

Bounding term 2 :

Combining the two terms and using η = Θ ( 1 c ′ 1 √ T + c ′ 2 ) , M = Θ(log T ) , we get

On the other hand, by the guarantee of the base algorithm (Theorem 30) and that εT ∈ [2 i ⋆ -1 , 2 i ⋆ ] , we have

Combining Eq. (44) and Eq. (45), we get

which finishes the proof.

Proof of Theorem 3. As shown in Appendix E.1, our Algorithm 1 can be adapted to satisfy Definition 29 with c 1 = Θ( d 2 log T ) and c 2 = Θ( √ d ) . By a concatenation of Theorem 30 and Theorem 31, we conclude that there is an algorithm that achieves

regret under unknown ε .

F Analysis for Linear EXP4

Proof of Theorem 4. We first show that

The magnitude of the loss is bounded by

If γ ≥ 2 dη , then we have | ˆ ℓ t,π | ≤ 1 2 and we can use the standard regret bound of exponential weights:

Let H t = E a ∼ p t [ aa ⊤ ] . Then we have ˜ H -1 t ⪯ 1 1 -γ H -1 t , and thus

Combining all proves Eq. (46).

Next, we show that there exists θ ∈ Θ such that

Let ˆ θ be the closest element in Θ to ∑ T t =1 y t . By the definition of Θ and the assumption that ∥ y t ∥ ≤ 1 , we have ∥ ∥ ∥ ˆ θ -∑ T t =1 y t ∥ ∥ ∥ ≤ ϵ . Thus, for any A ,

where the last inequality is by the fact that π ˆ θ ( A ) = argmin a ∈A a ⊤ ˆ θ . Taking expectation over A gives Eq. (47).

Finally, combining Eq. (46) and Eq. (47), choosing ϵ = 1 and γ = 2 dη = 2 d √ log T T , we get

finishing the proof.

G Comparison with Dai et al. [2023] and Sherman et al. [2023]

We state the exponential weight algorithm adopted by Luo et al. [2021], Dai et al. [2023], Sherman et al. [2023] in Algorithm 5, which is an algorithm that we know to achieve the prior-art regret bound in our setting (though they studied a more general MDP setting).

Their algorithm proceeds in epochs (indexed by k ), where every epoch consists of W rounds. The policy on action set A in the k -th epoch is defined as

where ˆ y k is the loss estimator for epoch k , and b k ( a ) is a (non-linear) bonus. In all W rounds in epoch k , the same policy is executed. The samples obtained in these W rounds are randomly divided into two halfs. One half is used to estimate the covariance matrix ˆ Σ k , and the other half is used to construct the loss estimator ˆ y k (see Line 5 of Algorithm 5).

Algorithm 5 Exponential weights with magnitude-reduced loss estimators

2

For all

A

, define

3 Randomly partition { ( k -1) W +1 , . . . , kW } into two equal parts T k , T ′ k .

4 for t = ( k -1) W +1 , . . . , kW do receive A t , sample a t ∼ p A t k , and receive ℓ t . 5 Define

G.1 Regret Analysis Sketch

The regret analysis starts with a standard decomposition that is similar to ours. We abuse the notation by defining y k = 1 W ∑ kW t =( k -1) W y t . Then

Bounding the regret term follows the standard analysis of exponential weight:

where H k = E A∼ D E a ∼ p A k [ aa ⊤ ] . Then they use the following fact to bound the stability term: as long as W ≥ d β 2 , it holds with high probability that ˆ Σ -1 k H k ˆ Σ -1 k ⪯ 2 ˆ Σ -1 k . Thus EW-Reg can be

further bounded by

By the definition of the bonus function b t , it holds that

Finally, the bias term can be bounded as follows:

The bias here has a similar form as in our case. They use the following fact to bound the bias: as long as W ≥ d β 2 , it holds that ∥ ( ˆ Σ k -H k ) y k ∥ ˆ Σ -1 k ≤ √ βd . Therefore, the bias can further be upper bounded by

Combining the three parts, we get that the overall regret is of order

Choosing α ≈ √ βd , we further bound it by

Recall the constraint W ≥ d β 2 . Choosing W = d β 2 gives

which gives d (ln |A 0 | ) 1 6 T 5 6 with the optimally chosen η and β .

Remark Due to the restrictions on the magnitude of the loss estimator required by the exponential weight algorithm, there is actually another constraint η β ≤ 1 , which makes Eq. (48) be d (ln |A 0 | ) 1 7 T 6 7 at best. This is exactly the bound obtained by Sherman et al. [2023]. A more sophisticated way to construct ˆ y k developed by Dai et al. [2023] removes this additional requirement and allows a bound of d (ln |A 0 | ) 1 6 T 5 6 . The sub-optimal bound T 8 9 reported in Dai et al. [2023] is due to issues related to MDPs, which are not presented in the contextual bandit case here.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of adversarial linear contextual bandits, where loss vectors are selected adversarially and the context for each round is drawn from a fixed distribution; traditional approaches require access to a simulator for generating free i.i.d. contexts or achieve a sub-optimal regret no better than $\\widetilde O(T^{5/6})$.\n\nThe authors significantly improve on these methods by achieving a regret of $\\widetilde O(\\sqrt{T})$ without the need for a simulator, while maintaining computational efficiency when the action set per round is small. The results affirmatively answer an open question about the existence of a polynomial-time algorithm with $\\mathrm{poly}(d)\\sqrt{T}$ regret. The approach allows for the case where the loss is linear up to an additive misspecification error, showing a near-optimal dependence on the magnitude of this error. \n\nThe paper also presents a computationally inefficient algorithm that provides an improved $\\widetilde O(d\\sqrt{T})$ regret without a simulator, expanding on the EXP4 algorithm.\n\nSTRENGTHS:\nThe paper presents an algorithm that innovatively instantiates an individual Follow-The-Regularized-Leader (FTRL) algorithm on each action set. This development has a wide range of implications for fields. It promises more efficient and effective optimization algorithms in the face of adversarial environments.\n\nOne of the major strengths of the paper lies in the construction of loss estimators and feature covariance matrix estimators, especially in challenging situations where the learner has no knowledge about the context distribution and there is no simulator available.\n\nThe paper manages to achieve an improved bound on the bias. The techniques used surpass those used in previous studies [DLWZ23, SKM23], leading to a significant improvement in the algorithm's performance and making it more effective in practice.\n\nThe introduction of feature centralization is also a notable strength. By centralizing the features by $\\hat x_t$, an estimation of the mean features under the current policy, the bias $y_t - \\hat y_t$ appears in a nice form that can be compensated by a bonus term. This approach provides an elegant solution to handling bias in the estimator.\n\nThe paper successfully handles the issue of strong dependence between the policy and the empirical context distribution, Dˆt, which typically rules out canonical concentration bounds.\n\nThe paper addresses the need for high efficiency in reusing the context samples, which is a challenging problem often not adequately handled by existing methods.\n\nWEAKNESSES:\nA notable weakness of the paper is the absence of empirical or numerical experiments to validate the theoretical results. Without practical implementation and testing of the proposed estimators on different datasets, it becomes difficult to assess their efficacy and performance.\n\nQUESTIONS:\nHow might these estimators handle high-dimensional data? Are there potential ways to improve their performance in such scenarios in future work?\n\nCould the presented estimators be generalized to other types of bandit problems, such as non-linear contextual bandit problems?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations as I could tell.\n\n\n\n==================================================\n\nPost-Rebuttal:\n\nI appreciate the response from the authors, and it addresses most of my concerns. Though I still believe the empirical experiments and the comparisons to other existing methods should be an important part of the work, this is still a strong theoretical paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies an adversarial linear bandit problem. At each round, the adversary selects a hidden loss vector $y_t$, the environment 'stochastically generates' an action set $A_t$, and the learner chooses an action in the action set. The learner is able to observe noisy loss $\\ip{y_t}{a_t}$.\n\nSTRENGTHS:\nUpdate after rebuttal. This paper proposed an optimal algorithm for the adversarial contextual linear bandit problem. The algorithm consists of OMD with logdet barrier as well as the lifting technique.\n\nWEAKNESSES:\nUpdate after rebuttal. \nCan dependence on d be improved?\n\nQUESTIONS:\nFirst, I did not fully understand the problem setting studied in this paper and would need some clarifications. \n\n1. In this work context vectors = actions, I have never seen this formulation in previous literature. The action set $A_t$ is generated from the distribution $D$. What exactly is distribution $D$? Is it defined over the power set of $B_2^d$? What assumption do you make on the size of $A_t$? \n\n2. The formulation does not align with previous work (e.g. NO20). In NO20, they assume the action is chosen by an adversary for each arm in $[K]$ (action for arm $a$ is $\\theta_{t,a}$), and the context $x_t$ is drawn from (known) distribution and revealed to the learner. The learner chooses one of the arms and observes the noisy loss centered at ${x_t^T}\\cdot{\\theta_{t, A_t}}$. Regret is compared to best policy, which is mapping from context to action. \nThis current work assumes the loss vector is chosen adversarially and the action set is drawn 'stochastically' (which I don't completely understand, see point above. ) Regret is compared to best policy, which is mapping from action set to a distribution over action set. \nGiven the different formulations, I do not believe the claims in this work, stating they improve over prior work, are accurate. \n\nBased on my understanding of this work, if the action set $A_t$ were constant each round, then this is the bandit linear optimization problem, as in (competing in the dark, abernethy et al. ). Hence this paper really seems to be solving a bandit linear optimization problem where the action set is changing each round, and not the 'adversarial linear contextual bandit' problem as introduced in the intro section.\n\nLIMITATIONS:\nThe author did not address limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary: \nThis paper presents a near-optimal computationally efficient simulator-free algorithm in contextual bandits setting with i.i.d contexts and adversarial reward functions. Most of the past work on this setting either requires a simulator that allows them to draw a large number of contexts from a distribution, or is computationally inefficient. Existing algorithms that do not require a simulator and are computationally efficient only achieve a regret of O(T^{5/6}). This work greatly improves the dependence on T, achieving a regret of O(d^2 \\sqrt{T}), while still computationally efficient without a simulator. Their algorithm is based on FTRL with a log-determinant barrier. They also present an algorithm that improves the dependence on d, achieving a regret of O(d \\sqrt{T}) but is only computationally efficient for a finite policy class.\n\nSTRENGTHS:\nI think this paper makes a solid theoretical contribution to the field. The improvement from O(T^{5/6}) to O(\\sqrt{T}) is substantial and simulator-free computationally efficient algorithms are important. The authors exploit the algorithmic ideas of ZL22, the ghost sample trick of NO20, along with novel analysis to provide an improved regret bound. I am slightly concerned about the originality of the algorithmic approach - but this is made up for with the improved analysis and improved regret bound \n\nThe paper was clearly well written and the contributions clearly stated. I enjoyed how the authors decomposed the regret and the analysis of the estimator in 3.2-3.4.\n\nWEAKNESSES:\nThe main weakness is that the paper was challenging to read inplaces, especially for readers not already familiar with the ideas of ZL22. There were other places where a bit of extra detail could have helped the exposition greatly. I have tried to remark on these in the list below.\n\nRemarks/Comments\n1. On bottom of page 5, you mention the loss in the “original space”, can you explain precisely what that loss is? For readers not familiar with ZL22, Line 2 in the algorithm may be opaque - in particular the sequence H_s^{-1} is chosen to control the bias. Moving a bit of the discussion from 3.4/3.5 up - or even simply remarking that it will be explained later may help.\n2. The discussion on line 188-189 is similar to 193-194. You can drop one.\n3. In equation 2, you use a \\hat{Sigma} - however the point of this equations seems to be that Sigma is known! Maybe remove the empirical hat?\n4. Is it possible to elaborate a bit more in the paper how your methods “surpass those in [DLWZ23,SKM23]” as described in line 246?\n5. \\gamma_t should be defined in algorithm for readability\n\nQUESTIONS:\n1. To analyze the first term |\\hat{Sigma}_t - H_ty_t|_hat{Sigma}_t^{-1} in line 242 did you need to use any kind of matrix concentration? If so, did you need control of the minimal eigenvalue of H_t? If not, how were you able to avoid it?\n2. The linear EXP4 algorithm looks very much similar to the original EXP4 algorithm. Could you provide more explanation of why we cannot directly use other computationally inefficient benchmark algorithms to get the same regret, i.e. why it’s necessary to present and prove linear EXP4? \n3. The computationally efficient algorithm, Algorithm 1, achieves a O(d^2\\sqrt{T}) regret while having a squared dependence on d; while the algorithm that achieves O(d\\sqrt{T}) regret is optimal but computationally inefficient. Do you believe algorithm 1 could achieve a O(d\\sqrt(T)) regret?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper studied the contextual bandits with i.i.d. contexts and adversarial linear loss functions without simulators and proposed a follow-the-regularized-leader with log-determinant  barrier (Logdet-FTRL) algorithm with a carefully designed covariance matrix estimator that is computationally efficient given small contexts set and achieves order-optimal (in T) regret \tilde{O}(d^2 \sqrt{T}). Previous algorithms achieving \sqrt{T} regret either require the knolwedge of contexts distribution or a simulator from which the learner can learn the contexts distribution by drawing free i.i.d. contexts. The proposed algorithm estimates the key covariance matrix via previously observes contexts with centralization and ridge regularizer. 

The setting contains stochastic sleeping bandits as special case and the proposed algorithm  answers the open problem in stochastic sleeping bandits affirmtively. A computationally inefficient algorithm that achieves regret with improved d factor is also proposed.

STRENGTHS:
The analysis is novel and the contribution is significant to linear adversarial bandits with i.i.d. contexts without simulator. 
Multiple related scenarios and extentions are discussed. 
I went over the proofs of the main regret analysis in App. D and several technical lemmas. The proofs are sound to me.

WEAKNESSES:
I do not see perticular weaknesses of the current version.

QUESTIONS:
Minor typos:
Line 164, has been taken
The second equality between line 796 and line 797, U^{\Ac_0} instead of \bar{U}^{\Ac_0}.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07