SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haolin Liu*
srs8rh@virginia.edu
Chen-Yu Wei*†
chenyu.wei@virginia.edu
Julian Zimmert*
zimmert@google.com
We consider the adversarial linear contextual bandit problem, where the loss vectors are selected fully adversarially and the per-round action set (i.e. the context) is drawn from a fixed distribution. Existing methods for this problem either require access to a simulator to generate free i.i.d. contexts, achieve a sub-optimal regret no better than  $\widetilde{\mathcal{O}}(T^{5/6})$ , or are computationally inefficient. We greatly improve these results by achieving a regret of  $\widetilde{\mathcal{O}}(\sqrt{T})$  without a simulator, while maintaining computational efficiency when the action set in each round is small. In the special case of sleeping bandits with adversarial loss and stochastic arm availability, our result answers affirmatively the open question by Saha et al. [2020] on whether there exists a polynomial-time algorithm with  $\mathrm{poly}(d)\sqrt{T}$  regret. Our approach naturally handles the case where the loss is linear up to an additive misspecification error, and our regret shows near-optimal dependence on the magnitude of the error.
Contextual bandit is a widely used model for sequential decision making. The interaction between the learner and the environment proceeds in rounds: in each round, the environment provides a context; based on it, the learner chooses an action and receive a reward. The goal is to maximize the total reward across multiple rounds. This model has found extensive applications in fields such as medical treatment [Tewari and Murphy, 2017], personalized recommendations [Beygelzimer et al., 2011], and online advertising [Chu et al., 2011].
Algorithms for contextual bandits with provable guarantees have been developed under various assumptions. In the linear regime, the most extensively studied model is the stochastic linear contextual bandit, in which the context can be arbitrarily distributed in each round, while the reward is determined by a fixed linear function of the context-action pair. Near-optimal algorithms for this setting have been established in, e.g., [Chu et al., 2011, Abbasi-Yadkori et al., 2011, Li et al., 2019, Foster et al., 2020]. Another model, which is the focus of this paper, is the adversarial linear contextual bandit, in which the context is drawn from a fixed distribution, while the reward is determined by a time-varying linear function of the context-action pair. A computationally efficient algorithm for this setting is first proposed by Neu and Olkhovskaya [2020]. However, existing research for this setting still faces challenges in achieving near-optimal regret and sample complexity when the context distribution is unknown.
*The authors are listed in alphabetical order.
This work was done when Chen-Yu Wei was at MIT Institute for Data, Systems, and Society.
<sup>3</sup>Apparently, the stochastic and adversarial linear contextual bandits defined here are incomparable, and their names do not fully capture their underlying assumptions. However, these are the terms commonly used in the literature (e.g., [Abbasi-Yadkori et al., 2011, Neu and Olkhovskaya, 2020]).
The algorithm by Neu and Olkhovskaya [2020] requires the learner to have full knowledge on the context distribution, and access to an exploratory policy that induces a feature covariance matrix with a smallest eigenvalue at least  $\lambda$ . Under these assumptions, their algorithm provides a regret guarantee of  $\widetilde{\mathcal{O}} (\sqrt{d\log(|\mathcal{A}|)T / \lambda})^4$ , where  $d$  is the feature dimension,  $|\mathcal{A}|$  is the maximum size of the action set, and  $T$  is the number of rounds. These assumptions are relaxed in the work of Luo et al. [2021], who studied a more general linear MDP setting. When specialized to linear contextual bandits, Luo et al. [2021] only requires access to a simulator from which the learner can draw free i.i.d. contexts. Their algorithm achieves a  $\widetilde{\mathcal{O}} ((d\log (|\mathcal{A}|)T^2)^{1 / 3}))$  regret. The regret is further improved to the near-optimal one  $\widetilde{\mathcal{O}} (\sqrt{d\log(|\mathcal{A}|)T})$  by Dai et al. [2023] through refined loss estimator construction.
All results that attain  $\tilde{\mathcal{O}}(T^{2/3})$  or  $\tilde{\mathcal{O}}(\sqrt{T})$  regret bound discussed above rely on access to the simulator. In their algorithms, the number of calls to the simulator significantly exceeds the number of interactions between the environment and the learner, but this is concealed from the regret bound. Therefore, their regret bounds do not accurately reflect the sample complexity of their algorithms. Another set of results for linear MDPs [Luo et al., 2021, Dai et al., 2023, Sherman et al., 2023, Kong et al., 2023] also consider the simulator-free scenario, essentially using interactions with the environment to fulfill the original purpose of the simulator. When applying their techniques to linear contextual bandits, their algorithms only achieve a regret bound of  $\tilde{\mathcal{O}}(T^{5/6})$  at best (see detailed analysis and comparison in Appendix G).
Our result significantly improves the previous ones: without simulators, we develop an algorithm that ensures a regret bound of order  $\widetilde{\mathcal{O}}(d^2\sqrt{T})$ , and it is computationally efficient as long as the size of the action set is small in each round (similar to all previous work). Unlike previous algorithms which always collect new contexts (through simulators or interactions with the environment) to estimate the feature covariance matrix, we leverage the context samples the learner received in the past to do this. Although natural, establishing a near-tight regret requires highly efficient use of context samples, necessitating a novel way to construct the estimator of feature covariance matrix and a tighter concentration bound for it. Additionally, to address the potentially large magnitude and the bias of the loss estimator, we turn to the use of log-determinant (logdet) barrier in the follow-the-regularized-leader (FTRL) framework. Logdet accommodates larger loss estimators and induces a larger bonus term to cancel the bias of the loss estimator, both of which are crucial for our result.
Our setting subsumes sleeping bandits with stochastic arm availability [Kanade et al., 2009, Saha et al., 2020] and combinatorial semi-bandits with stochastic action sets [Neu and Valko, 2014]. Our result answers affirmatively the main open question left by Saha et al. [2020] on whether there exists a polynomial-time algorithm with  $\mathrm{poly}(d)\sqrt{T}$  regret for sleeping bandits with adversarial loss and stochastic availability.
As a side result, we give a computationally inefficient algorithm that achieves an improved  $\widetilde{\mathcal{O}}(d\sqrt{T})$  regret without a simulator. While this is a direct extension from the EXP4 algorithm [Auer et al., 2002], such a result has not been established to our knowledge, so we include it for completeness.
We review the literature of various contextual bandit problems, classifying them based on the nature of the context and the reward function, specifically whether they are stochastic/fixed or adversarial.
Contextual bandits with i.i.d. contexts and fixed reward functions (S-S) Significant progress has been made in contextual bandits with i.i.d. contexts and fixed reward functions, under general reward function classes or policy classes [Langford and Zhang, 2007, Dudik et al., 2011, Agarwal et al., 2012, 2014, Simchi-Levi and Xu, 2022, Xu and Zeevi, 2020]. In the work by Dudik et al. [2011], Agarwal et al. [2012, 2014], the algorithms also use previously collected contexts to estimate the inverse probability of selecting actions under the current policy. However, these results only obtain regret bounds that polynomially depend on the number of actions. Furthermore, these results rely on having a fixed reward function, making their techniques not directly applicable to our case.
<sup>4</sup>The linear contextual bandit problem formulation in Neu and Olkhovskaya [2020] is different from ours. However, it can be reduced to our setting with dimension  $d|\mathcal{A}|$ , where  $|\mathcal{A}|$  is the maximum number of actions in each round. The  $\tilde{\mathcal{O}} (\sqrt{d\log(|\mathcal{A}|)T / \lambda})$  bound reported here is obtained by adopting their techniques to our setting.
Table 1: Related works in the "S-A" category. CB stands for contextual bandits and SB stands for semi-bandits. The relations among settings are as follows: Sleeping Bandit  $\subset$  Contextual SB  $\subset$  Linear CB, Linear CB  $\subset$  Linear MDP, and Linear CB  $\subset$  General CB. The table compares our results with the Pareto frontier of the literature. For algorithms dealing more general settings, we have carefully translated their techniques to Linear CB and reported the resulting bounds.  $\Sigma_{\pi}$  denotes the feature covariance matrix induced by policy  $\pi$ .  $|\mathcal{A}|$  and  $|\Pi|$  are sizes of the action set and the policy set.
Target Setting	Algorithm	Regret	Simulator	Computation	Assumption
General CB	Syrgkanis et al. [2016b]	(log |Π|)1/3(|A|T)2/3	✓	poly(|A|, log |Π|, T)	ERM Oracle
Linear MDP	Dai et al. [2023]	√dTlog |A|	✓	poly(|A|, d, T)
Dai et al. [2023]	d(log |A|)1/6T5/6	poly(|A|, d, T)
Sherman et al. [2023]
Kong et al. [2023]	(d7T4)1/5 + poly (1/λ)	Td	∃π, Σπ ≥ λI
Linear CB	Algorithm 1	d2√T	poly(|A|, d, T)
Algorithm 2	d√T	Td
Contextual SB	Neu and Valko [2014]	(dT)2/3	poly(d, T)
Sleeping Bandit	Saha et al. [2020]	√2dT	poly(d, T) (|A| ≤ d)
Contextual bandits with adversarial contexts and fixed reward functions (A-S) In this category, the most well-known results are in the linear setting [Chu et al., 2011, Abbasi-Yadkori et al., 2011, Zhao et al., 2023]. Besides the linear case, previous work has investigated specific reward function classes [Russo and Van Roy, 2013, Li et al., 2022, Foster et al., 2018]. Recently, Foster and Rakhlin [2020] introduced a general approach to deal with general function classes with a finite number of actions, which has since been improved or extended by Foster and Krishnamurthy [2021], Foster et al. [2021], Zhang [2022]. This category of problems is not directly comparable to the setting studied in this paper, but both capture a certain degree of non-stationarity of the environment.
Contextual bandits with i.i.d. contexts and adversarial reward functions (S-A) This is the category which our work falls into. Several oracle efficient algorithms that require simulators have been proposed for general policy classes [Rakhlin and Sridharan, 2016, Syrgkanis et al., 2016b]. The oracle they use (i.e., the empirical risk minimization, or ERM oracle), however, is not generally implementable in an efficient manner. For the linear case, the first computationally efficient algorithm is by Neu and Olkhovskaya [2020], under the assumption that the context distribution is known. This is followed by Olkhovskaya et al. [2023] to obtain refined data-dependent bounds. A series of works [Neu and Olkhovskaya, 2021, Luo et al., 2021, Dai et al., 2023, Sherman et al., 2023] apply similar techniques to linear MDPs, but when specialized to linear contextual bandits, they all assume known context distribution, or access to a simulator, or only achieves a regret no better than  $\widetilde{\mathcal{O}}(T^{5/6})$ . The work of Kong et al. [2023] also studies linear MDPs; when specialized to contextual bandits, they obtain a regret bound of  $\widetilde{\mathcal{O}}(T^{4/5} + \mathrm{poly}\left(\frac{1}{\lambda}\right))$  without a simulator but with a computationally inefficient algorithm and an undesired inverse dependence on the smallest eigenvalue of the covariance matrix. Related but simpler settings have also been studied. The sleeping bandit problem with stochastic arm availability and adversarial reward [Kleinberg et al., 2010, Kanade et al., 2009, Saha et al., 2020] is a special case of our problem where the context is always a subset of standard unit vectors. Another special case is the combinatorial semi-bandit problem with stochastic action sets and adversarial reward [Neu and Valko, 2014]. While these are special cases, the regret bounds in these works are all worse than  $\widetilde{\mathcal{O}}(\mathrm{poly}(d)\sqrt{T})$ . Therefore, our result also improves upon theirs.
Contextual bandits with adversarial contexts and adversarial reward functions (A-A) When both contexts and reward functions are adversarial, there are computational [Kanade and Steinke, 2014] and oracle-call [Hazan and Koren, 2016] lower bounds showing that no sublinear regret is achievable unless the computational cost scales polynomially with the size of the policy set. Even for the linear case, Neu and Olkhovskaya [2020] argued that the problem is at least as hard as online learning a one-dimensional threshold function, for which sublinear regret is impossible. For this
For combinatorial semi-bandit problems, our algorithm is not as computationally efficient as Neu and Valko [2014], which can handle exponentially large action sets.
challenging category, besides using the inefficient EXP4 algorithm, previous work makes stronger assumptions on the contexts [Syrgkanis et al., 2016a] or resorts to alternative benchmarks such as dynamic regret [Luo et al., 2018, Chen et al., 2019] and approximate regret [Emamjomeh-Zadeh et al., 2021].
Lifting and exploration bonus for high-probability adversarial linear bandits Our technique is related to those obtaining high-probability bounds for linear bandits. Early development in this line of research only achieves computational efficiency when the action set size is small [Bartlett et al., 2008] or only applies to special action sets such as two-norm balls [Abernethy and Rakhlin, 2009]. Recently, near-optimal high-probability bounds for general convex action sets have been obtained by lifting the problem to a higher dimensional one, which allows for a computationally efficient way to impose bonuses [Lee et al., 2020, Zimmert and Lattimore, 2022]. The lifting and the bonus ideas we use are inspired by them, though for different purposes. However, due to the extra difficulty arising in the contextual case, currently we only obtain a computationally efficient algorithm when the action set size is small.
1.2 Computational Complexity
Our main algorithm is based on log-determinant barrier optimization similar to Foster et al. [2020], Zimmert and Lattimore [2022]. Computing its action distribution is closely related to computing the D-optimal experimental design [Khachiyan and Todd, 1990]. Per step, this is shown to require  $\tilde{\mathcal{O}}(|\mathcal{A}_t| \mathrm{poly}(d))$  computational and  $\tilde{\mathcal{O}}(\log(|\mathcal{A}_t|) \mathrm{poly}(d))$  memory complexity [Foster et al., 2020, Proposition 1], where  $|\mathcal{A}_t|$  is the action set size at round  $t$ . The computational bottleneck comes from (approximately) maximizing a quadratic function over the action set. It is an open question whether linear optimization oracles or other type of oracles can lead to efficient implementation of our algorithm for continuous action sets.
In the literature, there are few linear context bandit algorithms that provably avoid  $|\mathcal{A}|$  computation per round. The LinUCB algorithm [Chu et al., 2011, Abbasi-Yadkori et al., 2011] suffers from the same quadratic function maximization issue, and therefore is computationally comparable to our algorithm. The SquareCB.Lin algorithm by Foster et al. [2020] is based on the same log-determinant barrier optimization. Another recent algorithm by Zhang [2022] only admits an efficient implementation for continuous action sets in the Bayesian setting but not in the frequentist setting (though they provided an efficient heuristic implementation in their experiments). The Thompson sampling algorithm by Agrawal and Goyal [2013], which has efficient implementation, also relies on well-specified Gaussian prior. The only work that we know can avoids  $|\mathcal{A}|$  computation in the frequentist setting is Zhu et al. [2022], but their technique is only known to handle the A-S setting.
We study the adversarial linear contextual bandit problem where the loss vectors are selected fully adversarially and the per-round action set (i.e. the context) is drawn from a fixed distribution. The learner and the environment interact in the following way. Let  $\mathbb{B}_2^d$  be the L2-norm unit ball in  $\mathbb{R}^d$ .
For  $t = 1,\dots ,T$
1. The environment decides an adversarial loss vector  $y_{t} \in \mathbb{B}_{2}^{d}$ , and generates a random action set (i.e., context)  $\mathcal{A}_t \subset \mathbb{B}_2^d$  from a fixed distribution  $D$  independent from anything else.
2. The learner observes  $\mathcal{A}_t$ , and (randomly) chooses an action  $a_t \in \mathcal{A}_t$ .
3. The learner receives the loss  $\ell_t \in [-1, 1]$  with  $\mathbb{E}[\ell_t] = \langle a_t, y_t \rangle$ .
A policy  $\pi$  is a mapping which, given any action set  $\mathcal{A} \subset \mathbb{R}^d$ , maps it to an element in the convex hull of  $\mathcal{A}$ . We use  $\pi(\mathcal{A})$  to refer to the element that it maps  $\mathcal{A}$  to. The learner's regret with respect to policy  $\pi$  is defined as the expected performance difference between the learner and policy  $\pi$ :
$$
\operatorname {R e g} (\pi) = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle a _ {t}, y _ {t} \rangle - \sum_ {t = 1} ^ {T} \langle \pi (\mathcal {A} _ {t}), y _ {t} \rangle \right]
$$
where the expectation is taken over all randomness from the environment  $(y_{t}$  and  $\mathcal{A}_t)$  and from the learner  $(a_{t})$ . The pseudo-regret (or just regret) is defined as  $\mathrm{Reg} = \max_{\pi}\mathrm{Reg}(\pi)$ , where the maximization is taken over all possible policies.
Notations For any matrix  $A$ , we use  $\lambda_{\max}(A)$  and  $\lambda_{\min}(A)$  to denote the maximum and minimum eigenvalues of  $A$ , respectively. We use  $\operatorname{Tr}(A)$  to denote the trace of matrix  $A$ . For any action set  $\mathcal{A}$ , let  $\Delta(\mathcal{A})$  be the space of probability measures on  $\mathcal{A}$ . Let  $\mathcal{F}_t = \sigma(\mathcal{A}_s, a_s, \forall s \leq t)$  be the  $\sigma$ -algebra at round  $t$ . Define  $\mathbb{E}_t[\cdot] = \mathbb{E}[\cdot|\mathcal{F}_{t-1}]$ . Given a differentiable convex function  $F: \mathbb{R}^d \to \mathbb{R} \cup \{\infty\}$ , the Bregman divergence with respect to  $F$  is defined as  $D_F(x,y) = F(x) - F(y) - \langle \nabla F(y), x - y \rangle$ . Given a positive semi-definite (PSD) matrix  $A$ , for any vector  $x$ , define the norm generated by  $A$  as  $\|x\|_A = \sqrt{x^\top A x}$ . For any context  $\mathcal{A} \subset \mathbb{R}^d$  and  $p \in \Delta(\mathcal{A})$ , define  $\mu(p) = \mathbb{E}_{a \sim p}[a]$  and  $\operatorname{Cov}(p) = \mathbb{E}_{a \sim p}[(a - \mu(p))(a - \mu(p))^\top]$ . For any  $a$ , define the lifted action  $\boldsymbol{a} = (a,1)^\top$  and the lifted covariance matrix  $\widehat{\operatorname{Cov}}(p) = \mathbb{E}_{a \sim p}[\boldsymbol{a} \boldsymbol{a}^\top] = \mathbb{E}_{a \sim p}\left[ \begin{array}{cc} aa^\top & a \\ a^\top & 1 \end{array} \right] = \left[ \begin{array}{ccc} \operatorname{Cov}(p) + \mu(p)\mu(p)^\top & \mu(p) & 1 \\ \mu(p)^\top & 1 & 1 \end{array} \right]$ .
We use bold matrices to denote matrices in the lifted space (e.g., in Algorithm 1 and Definition 1).
3 Follow-the-Regularized-Leader with the Log-Determinant Barrier
In this section, we present our main algorithm, Algorithm 1. This algorithm can be viewed as instantiating an individual Follow-The-Regularized-Leader (FTRL) algorithm on each action set (Line 2), with all FTRLs sharing the same loss vectors. This perspective has been taken by previous works Neu and Olkhovskaya [2020], Olkhovskaya et al. [2023] and simplifies the understanding of the problem. The rationale comes from the following calculation due to Neu and Olkhovskaya [2020]: for any policy  $\pi$  that may depend on  $\mathcal{F}_{t-1}$ ,
$$
\mathbb {E} _ {t} \left[ \left\langle \pi (\mathcal {A} _ {t}), y _ {t} \right\rangle \right] = \mathbb {E} _ {\mathcal {A} _ {t}} \left[ \mathbb {E} _ {y _ {t}} \left[ \left\langle \pi (\mathcal {A} _ {t}), y _ {t} \right\rangle \mid \mathcal {F} _ {t - 1} \right] \right] = \mathbb {E} _ {\mathcal {A} _ {0}} \left[ \mathbb {E} _ {y _ {t}} \left[ \left\langle \pi (\mathcal {A} _ {0}), y _ {t} \right\rangle \mid \mathcal {F} _ {t - 1} \right] \right] = \mathbb {E} _ {t} \left[ \left\langle \pi (\mathcal {A} _ {0}), y _ {t} \right\rangle \right]
$$
where  $\mathcal{A}_0$  is a sample drawn from  $D$  independent of all interaction history. This allows us to calculate the regret as
$$
\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \pi_ {t} \left(\mathcal {A} _ {t}\right) - \pi \left(\mathcal {A} _ {t}\right), y _ {t} \right\rangle \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \pi_ {t} \left(\mathcal {A} _ {0}\right) - \pi \left(\mathcal {A} _ {0}\right), y _ {t} \right\rangle \right] \tag {1}
$$
where  $\pi_t$  is the policy used by the learner at time  $t$ . Note that this view does not require the learner to simultaneously "run" an algorithm on every action set since the learner only needs to calculate the policy on  $\mathcal{A}$  whenever  $\mathcal{A}_t = \mathcal{A}$ . In the regret analysis, in view of Eq. (1), it suffices to consider a single fixed action set  $\mathcal{A}_0$  drawn from  $D$  and bound the regret on it, even though the learner may never execute the policy on it. This  $\mathcal{A}_0$  is called a "ghost sample" in Neu and Olkhovskaya [2020].
3.1 The lifting idea and the execution of Algorithm 1
Our algorithm is built on the logdet-FTRL algorithm developed by Zimmert and Lattimore [2022] for high-probability adversarial linear bandits, which lifts the original  $d$ -dimensional problem over the feature space to a  $(d + 1) \times (d + 1)$  one over the covariance matrix space, with the regularizer being the negative log-determinant function. In our case, we instantiate an individual logdet-FTRL on each action set. The motivation behind Zimmert and Lattimore [2022] to lift the problem to the space of covariance matrix is that it casts the problem to one in the positive orthant, which allows for an easier way to construct the bonus term that is crucial to compensate the variance of the losses, enabling a high-probability bound in their case. In our case, we use the same technique to introduce the bonus term, but the goal is to compensate the bias resulting from the estimation error in the covariance matrix (see Section 3.4). This bias only appears in our contextual case but not in the linear bandit problem originally considered in Zimmert and Lattimore [2022].
As argued previously, we can focus on the learning problem over a fixed action set  $\mathcal{A}$ , and our algorithm operates in the lifted space of covariance matrices  $\mathcal{H}^{\mathcal{A}} = \{\widehat{\mathrm{Cov}}(p) : p \in \Delta(\mathcal{A})\} \subset \mathbb{R}^{(d+1) \times (d+1)}$ . For this space, we define the lifted loss  $\gamma_{t} = \begin{bmatrix} 0 & \frac{1}{2}y_{t} \\ \frac{1}{2}y_{t}^{\top} & 0 \end{bmatrix} \in \mathbb{R}^{(d+1) \times (d+1)}$  so
that  $\langle \widehat{\mathrm{Cov}}(p), \gamma_t \rangle = \mathbb{E}_{a \sim p}[a^\top y_t] = \langle \mu(p), y_t \rangle$  and thus the loss value in the lifted space (i.e.,  $\langle \widehat{\mathrm{Cov}}(p), \gamma_t \rangle$ ) is the same as that in the original space (i.e.,  $\langle \mu(p), y_t \rangle$ ).