SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Algorithm 1 Logdet-FTRL for linear contextual bandits
Definitions:  $F(\pmb {H}) = -\log \operatorname *{det}(\pmb {H}),\eta_t = \frac{1}{64d\sqrt{t}},\alpha_t = \frac{d}{\sqrt{t}},\beta_t = \frac{100(d + 1)^3\log(3T)}{t - 1}.$    
for  $t = 1,2,\ldots$  do For all  $\mathcal{A}$  define  $\begin{array}{r}\pmb {H}_t^{\mathcal{A}} = \underset {\pmb {H}\in \mathcal{H}^{\mathcal{A}}}{\mathrm{argmin}}\sum_{s = 1}^{t - 1}\langle \pmb {H},\hat{\gamma}_s - \alpha_s\hat{\pmb{\Sigma}}_s^{-1}\rangle +\frac{\boldsymbol{F}(\boldsymbol{H})}{\eta_t}. \end{array}$    
For all  $\mathcal{A}$  define  $p_t^A\in \Delta (\mathcal{A})$  such that  $\pmb {H}_t^{\mathcal{A}} = \widehat{\mathrm{Cov}} (p_t^{\mathcal{A}})$    
Receive  $\mathcal{A}_t$  and sample  $a_{t}\sim p_{t}^{\mathcal{A}_{t}}$    
Observe  $\ell_t\in [-1,1]$  with  $\mathbb{E}[\ell_t] = a_t^\top y_t$  and construct  $\hat{y}_t = \hat{\Sigma}_t^{-1}(a_t - \hat{x}_t)\ell_t$  , where  $\hat{x}_t = \frac{1}{t - 1}\sum_{\tau = 1}^{t - 1}\mathbb{E}_{a\sim p_t^{A_\tau}}[a],\hat{H}_t = \frac{1}{t - 1}\sum_{\tau = 1}^{t - 1}\mathbb{E}_{a\sim p_t^{A_\tau}}[(a - \hat{x}_t)(a - \hat{x}_t)^\top ],\hat{\Sigma}_t = \hat{H}_t + \beta_tI.$  Define  $\hat{\pmb{H}}_t = \frac{1}{t - 1}\sum_{\tau = 1}^{t - 1}\pmb {H}_t^{A_\tau}$  and  $\hat{\pmb{\Sigma}}_t = \hat{\pmb{H}}_t + \beta_t\pmb {I}$  and  $\hat{\gamma}_t = \left[ \begin{array}{cc}0 & \frac{1}{2}\hat{y}_t\\ \frac{1}{2}\hat{y}_t^\top & 0 \end{array} \right].$  (If  $t = 1$  , define  $\hat{\Sigma}_t^{-1}$  and  $\hat{\Sigma}_t^{-1}$  as zeros).
In each round  $t$ , the FTRL on  $\mathcal{A}$  outputs a lifted covariance matrix  $\pmb{H}_t^A \in \mathcal{H}^A$  that corresponds to a probability distribution  $p_t^A \in \Delta(\mathcal{A})$  such that  $\widehat{\mathrm{Cov}}(p_t^A) = H_t^A$  (Line 2 and Line 3). Upon receiving  $\mathcal{A}_t$ , the learner samples an action from  $p_t^{A_t}$  and the agent constructs the loss estimator  $\hat{y}_t$  (Line 5). Similarly to the construction of  $\gamma_t$ , we define the lifted loss estimator  $\hat{\gamma}_t = \begin{bmatrix} 0 & \frac{1}{2}\hat{y}_t \\ \frac{1}{2}\hat{y}_t^\top & 0 \end{bmatrix}$  which makes  $\langle \widehat{\mathrm{Cov}}(p), \hat{\gamma}_t \rangle = \mathbb{E}_{a \sim p}[a^\top\hat{y}_t] = \langle \mu(p), \hat{y}_t \rangle$ . The lifted loss estimator, along with the bonus term  $-\alpha_t\hat{\Sigma}_t^{-1}$ , is then fed to the FTRL on all  $\mathcal{A}$ 's. The purpose of the bonus term will be clear in Section 3.4.
In the rest of this section, we use the following notation in addition to those defined in Algorithm 1. Definition 1. Define  $x_{t}^{\mathcal{A}} = \mathbb{E}_{a\sim p_{t}^{\mathcal{A}}}[a]$ ,  $x_{t} = \mathbb{E}_{\mathcal{A}\sim D}[x_{t}^{\mathcal{A}}]$ ,  $H_{t}^{\mathcal{A}} = \mathbb{E}_{a\sim p_{t}^{\mathcal{A}}}[(a - \hat{x}_{t})(a - \hat{x}_{t})^{\top}]$ ,  $H_{t} = \mathbb{E}_{\mathcal{A}\sim D}[H_{t}^{\mathcal{A}}]$ ,  $\pmb{H}_{t} = \mathbb{E}_{\mathcal{A}\sim D}[\pmb{H}_{t}^{\mathcal{A}}]$ . Let  $p_{\star}^{\mathcal{A}} \in \Delta(\mathcal{A})$  be the action distribution used by the benchmark policy on  $\mathcal{A}$ , and define  $u^{\mathcal{A}} = \mathbb{E}_{a\sim p_{\star}^{\mathcal{A}}}[a]$ ,  $u = \mathbb{E}_{\mathcal{A}\sim D}[u^{\mathcal{A}}]$ ,  $\pmb{U}^{\mathcal{A}} = \mathbb{E}_{a\sim p_{\star}^{\mathcal{A}}}[\pmb{a}\pmb{a}^{\top}]$ ,  $\pmb{U} = \mathbb{E}_{\mathcal{A}\sim D}[\pmb{U}^{\mathcal{A}}]$ . Notice that the  $x_{t}^{\mathcal{A}}$  and  $u^{\mathcal{A}}$  defined here is equivalent to the  $\pi_t(\mathcal{A})$  and  $\pi (\mathcal{A})$  in Eq. (1), respectively.
3.2 The construction of loss estimators and feature covariance matrix estimators
Our goal is to make  $\hat{y}_t$  in Line 5 an estimator of  $y_{t}$  with controllable bias and variance. If the context distribution is known (as in Neu and Olkhovskaya [2020]), then a standard unbiased estimator of  $y_{t}$  is
$$
\hat {y} _ {t} = \Sigma_ {t} ^ {- 1} a _ {t} \ell_ {t}, \quad \text {w h e r e} \quad \Sigma_ {t} = \mathbb {E} _ {\mathcal {A} \sim D} \mathbb {E} _ {a \sim p _ {t} ^ {\mathcal {A}}} \left[ a a ^ {\top} \right]. \tag {2}
$$
To see its unbiasedness, notice that  $\mathbb{E}[a_t\ell_t] = \mathbb{E}_{A\sim D}\mathbb{E}_{a\sim p_t^A}[aa^\top y_t]$  and thus  $\mathbb{E}[\hat{y}_t] = y_t$ . This  $\hat{y}_t$ , however, can have a variance that is inversely related to the smallest eigenvalue of the covariance matrix  $\hat{\Sigma}_t$ , which can be unbounded in the worst case. This is the main reason why Neu and Ollkhovskaya [2020] does not achieve the optimal bound, and requires the bias-variance-tradeoff techniques in Dai et al. [2023] to close the gap. When the context distribution is unknown but the learner has access to a simulator [Luo et al., 2021, Dai et al., 2023, Sherman et al., 2023, Kong et al., 2023], the learner can draw free contexts to estimate the covariance matrix  $\hat{\Sigma}_t$  up to a very high accuracy without interacting with the environment, making the problem close to the case of known context distribution.
Challenges arise when the learner has no knowledge about the context distribution and there is no simulator. In this case, there are two natural ways to estimate the covariance matrix under the current policy. One is to draw new samples from the environment, treating the environment like a simulator. This approach is essentially taken by all previous work studying linear models in the "S-A" category. However, this is very expensive, and it causes the simulator-equipped bound  $\sqrt{T}$  in Dai et al. [2023]
to deteriorate to the simulator-free bound  $T^{5/6}$  at best (see Appendix G for details). The other is to use the contexts received in time 1 to  $t$  to estimate the covariance matrix under the policy at time  $t$ . This demands a very high efficiency in reusing the contexts samples, and existing ways of constructing the covariance matrix and the accompanied analysis by Dai et al. [2023], Sherman et al. [2023] are insufficient to achieve the near-optimal bound even with context reuse. This necessitates our tighter construction of the covariance matrix estimator and tighter concentration bounds for it.
Our construction of the loss estimator (Line 5) is
$$
\hat {y} _ {t} = \hat {\Sigma} _ {t} ^ {- 1} \left(a _ {t} - \hat {x} _ {t}\right) \ell_ {t} \quad \text {w h e r e} \quad \hat {\Sigma} _ {t} = \mathbb {E} _ {\mathcal {A} \sim \hat {D} _ {t}} \mathbb {E} _ {a \sim p _ {t} ^ {\mathcal {A}}} \left[ (a - \hat {x} _ {t}) (a - \hat {x} _ {t}) ^ {\top} \right] + \beta_ {t} I \tag {3}
$$
where  $\hat{D}_t = \mathrm{Uniform}\{\mathcal{A}_1,\mathcal{A}_2,\dots ,\mathcal{A}_{t - 1}\} ,\hat{x}_t = \mathbb{E}_{\mathcal{A}\sim \hat{D}_t},\mathbb{E}_{a\sim p_t^A}[a]$ , and  $\beta_{t} = \widetilde{\mathcal{O}} (d^{3} / t)$ . Comparing Eq. (3) with Eq. (2), we see that besides using the empirical context distribution  $\hat{D}_t$  in place of the ground truth  $D$  and adding a small term  $\beta_{t}I$  to control the smallest eigenvalue of the covariance matrix, we also centralize the features by  $\hat{x}_t$ , an estimation of the mean features under the current policy. The centralization is important in making the bias  $y_{t} - \hat{y}_{t}$  appear in a nice form that can be compensated by a bonus term. The estimator might seem problematic on first sight, because  $p_t^A$  is strongly dependent on  $\hat{D}_t$ , which rules out canonical concentration bounds. We circumvent this issue by leveraging the special structure of  $p_t$  in Algorithm 1, which allows for a union bound over a sufficient covering of all potential policies (Appendix C.3). The analysis on the bias of this loss estimator is also non-standard, which is the key to achieve the near-optimal bound. In the next two subsections, we explain how to bound the bias of this loss estimator (Section 3.3), and how the bonus term can be used to compensate the bias (Section 3.4).
3.3 The bias of the loss estimator
Since the true loss vector is  $y_{t}$  and we use the loss estimator  $\hat{y}_t$  in the update, there is a bias term emerging in the regret bound at time  $t$ :
$$
\mathbb {E} _ {t} \left[ \langle x _ {t} ^ {\mathcal {A} _ {0}} - u ^ {\mathcal {A} _ {0}}, y _ {t} - \hat {y} _ {t} \rangle \right] = \mathbb {E} _ {t} \left[ \langle x _ {t} - u, y _ {t} - \hat {y} _ {t} \rangle \right] = \mathbb {E} _ {t} \left[ (x _ {t} - u) ^ {\top} \left(I - \hat {\Sigma} _ {t} ^ {- 1} (a _ {t} - \hat {x} _ {t}) a _ {t} ^ {\top}\right) y _ {t} \right]
$$
where definitions of  $x_{t}^{\mathcal{A}}, u^{\mathcal{A}}, x_{t}, u$  can be found in Definition 1, and we use the definition of  $\hat{y}_t$  in Eq. (3) in the last equality. Now taking expectation over  $\mathcal{A}_t$  and  $a_{t}$  conditioned on  $\mathcal{F}_{t - 1}$ , we can further bound the expectation in the last expression by
$$
\begin{array}{l} \left(x _ {t} - u\right) ^ {\top} \left(I - \hat {\Sigma} _ {t} ^ {- 1} H _ {t}\right) y _ {t} - \left(x _ {t} - u\right) ^ {\top} \hat {\Sigma} _ {t} ^ {- 1} \left(x _ {t} - \hat {x} _ {t}\right) \hat {x} _ {t} ^ {\top} y _ {t} \\ \leq \left\| x _ {t} - u \right\| _ {\hat {\Sigma} _ {t} ^ {- 1}} \left\| \left(\hat {\Sigma} _ {t} - H _ {t}\right) y _ {t} \right\| _ {\hat {\Sigma} _ {t} ^ {- 1}} + \left\| x _ {t} - u \right\| _ {\hat {\Sigma} _ {t} ^ {- 1}} \left\| x _ {t} - \hat {x} _ {t} \right\| _ {\hat {\Sigma} _ {t} ^ {- 1}} \tag {4} \\ \end{array}
$$
(see Definition 1 for the definition of  $H_{t}$ ). The two terms  $\| (\hat{\Sigma}_{t} - H_{t})y_{t}\|_{\hat{\Sigma}_{t}^{-1}}$  and  $\| x_{t} - \hat{x}_{t}\|_{\hat{\Sigma}_{t}^{-1}}$  in Eq. (4) are related to the error between the empirical context distribution  $\hat{D}_t = \mathrm{Uniform}\{\mathcal{A}_1,\dots ,\mathcal{A}_{t - 1}\}$  and the true distribution  $D$ . We handle them through novel analysis and bound both of them by  $\widetilde{\mathcal{O}}\big(\sqrt{d^3 / t}\big)$ . See Lemma 13, Lemma 14, Lemma 18, and Lemma 19 for details. The techniques we use in these lemmas surpass those in Dai et al. [2023], Sherman et al. [2023]. As a comparison, a similar term as  $\| (\hat{\Sigma}_{t} - H_{t})y_{t}\|_{\hat{\Sigma}_{t}^{-1}}$  is also presented in Eq. (16) of Dai et al. [2023] and Lemma B.5 of Sherman et al. [2023] when bounding the bias. While their analysis uses off-the-shelf matrix concentration inequalities, our analysis expands this expression by its definition, and applies concentration inequalities for scalars on individual entries. Overall, our analysis is more tailored for this specific expression. Previous works ensure that this term can be bounded by  $\mathcal{O}(\sqrt{\beta})$  after collecting  $\mathcal{O}(\beta^{-2})$  new samples (Lemma 5.1 of Dai et al. [2023] and Lemma B.1 of Sherman et al. [2023]), we are able to bound it by  $\mathcal{O}(1 / \sqrt{t})$  only using  $t$  samples that the learner received up to time  $t$ . This essentially improves their  $\mathcal{O}(\beta^{-2})$  sample complexity bound to  $\mathcal{O}(\beta^{-1})$ . See Appendix G for detailed comparison with Dai et al. [2023] and Sherman et al. [2023].
Now we have bounded the regret due to bias of  $\hat{y}_t$  by the order of  $\sqrt{d^3/t}\|x_t - u\|_{\hat{\Sigma}_t^{-1}}$ . The next problem is how to mitigate this term. This is also a problem in previous work [Luo et al., 2021, Dai et al., 2023, Sherman et al., 2023], and it has become clear that this can be handled by incorporating bonus in the algorithm.
3.4 The bonus term
To handle a bias term in the form of  $\| x_{t} - u\|_{\hat{\Sigma}_{t}^{-1}}$ , we resort to the idea of *bonus*. To illustrate this, suppose that instead of feeding  $\hat{y}_t$  to the FTRLs, we feed  $\hat{y}_t - b_t$  for some  $b_{t}$ . Then this would give us a regret bound of the following form:
$$
\begin{array}{l} \operatorname {R e g} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle x _ {t} - u, \hat {y} _ {t} - b _ {t} \rangle \right] + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle x _ {t} - u, y _ {t} - \hat {y} _ {t} \rangle \right] + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle x _ {t} - u, b _ {t} \rangle \right] \\ \lesssim \widetilde {\mathcal {O}} \left(d ^ {2} \sqrt {T}\right) + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \sqrt {\frac {d ^ {3}}{t}} \| x _ {t} - u \| _ {\hat {\Sigma} _ {t} ^ {- 1}} \right] + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle x _ {t} - u, b _ {t} \rangle \right] \tag {5} \\ \end{array}
$$
where we assume that FTRL can give us  $\widetilde{\mathcal{O}}(d^2\sqrt{T})$  bound for the loss sequence  $\hat{y}_t - b_t$ . Our hope here is to design a  $b_t$  such that  $\langle x_t - u, b_t \rangle$  provides a negative term that can be used to cancel the bias term  $\sqrt{d^3/t} \|x_t - u\|_{\hat{\Sigma}_t^{-1}}$  in the following manner:
$$
\text {b i a s} + \text {b o u n s} = \sum_ {t = 1} ^ {T} \left(\sqrt {\frac {d ^ {3}}{t}} \| x _ {t} - u \| _ {\hat {\Sigma} _ {t} ^ {- 1}} + \langle x _ {t} - u, b _ {t} \rangle\right) \lesssim \widetilde {\mathcal {O}} \left(d ^ {2} \sqrt {T}\right). \tag {6}
$$
which gives us a  $\widetilde{\mathcal{O}}(d^2\sqrt{T})$  overall regret by Eq. (5). This approach relies on two conditions to be satisfied. First, we have to find a  $b_t$  that makes Eq. (6) hold. Second, we have to ensure that the FTRL algorithm achieves a  $\widetilde{\mathcal{O}}(d^2\sqrt{T})$  bound under the loss sequence  $\hat{y}_t - b_t$ .
To meet the first condition, we take inspiration from Zimmert and Lattimore [2022] and lift the problem to the space of covariance matrix in  $\mathbb{R}^{(d + 1)\times (d + 1)}$ . Considering the bonus term  $\alpha_{t}\hat{\Sigma}_{t}^{-1}$  in the lifted space, we have
$$
\left\langle \boldsymbol {H} _ {t} - \boldsymbol {U}, \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \right\rangle = \alpha_ {t} \operatorname {T r} \left(\boldsymbol {H} _ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1}\right) - \alpha_ {t} \operatorname {T r} \left(\boldsymbol {U} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1}\right) \tag {7}
$$
Using Lemma 17 and Corollary 22, we can upper bound Eq. (7) by  $\mathcal{O}\left(d\alpha_{t}\right)-\frac{\alpha_{t}}{4}\|u-\hat{x}_{t}\|_{\hat{\Sigma}_{t}^{-1}}^{2}$ . This gives
$$
\begin{array}{l} \text {b i a s} + \text {b o n u s} \leq \sum_ {t = 1} ^ {T} \left(\sqrt {\frac {d ^ {3}}{t}} \| x _ {t} - u \| _ {\hat {\Sigma} _ {t} ^ {- 1}} + d \alpha_ {t} - \frac {\alpha_ {t}}{4} \| \hat {x} _ {t} - u \| _ {\hat {\Sigma} _ {t} ^ {- 1}} ^ {2}\right) \\ \leq \widetilde {\mathcal {O}} (d ^ {2} \sqrt {T}) + \sum_ {t = 1} ^ {T} \sqrt {\frac {d ^ {3}}{t}} \| x _ {t} - \hat {x} _ {t} \| _ {\hat {\Sigma} _ {t} ^ {- 1}} + \sum_ {t = 1} ^ {T} \left(\sqrt {\frac {d ^ {3}}{t}} \| \hat {x} _ {t} - u \| _ {\hat {\Sigma} _ {t} ^ {- 1}} - \frac {\alpha_ {t}}{4} \| \hat {x} _ {t} - u \| _ {\hat {\Sigma} _ {t} ^ {- 1}} ^ {2}\right). \\ \end{array}
$$
Using Lemma 18 to bound the second term above by  $\widetilde{\mathcal{O}} (\sum_{t}d^{3} / t) = \widetilde{\mathcal{O}} (d^{3})$ , and AM-GM to bound the third term by  $\widetilde{\mathcal{O}} (\sum_{t}d^{3} / (t\alpha_{t})) = \widetilde{\mathcal{O}} (d^{2}\sqrt{T})$ , we get Eq. (6), through the help of lifting.
To meet the second condition, we have to analyze the regret of FTRL under the loss  $\hat{y}_t - b_t$ . The key is to show that the bonus  $\alpha_t \hat{\Sigma}_t^{-1}$  introduces small stability term overhead. Thanks to the use of the logdet regularizer and its self-concordance property, the extra stability term introduced by the bonus can indeed be controlled by the order  $\sqrt{T}$ . The key analysis is in Lemma 27.
Previous works rely on exponential weights [Luo et al., 2021, Dai et al., 2023, Sherman et al., 2023] rather than logdet-FTRL, which comes with the following drawbacks. 1) In Luo et al. [2021], Sherman et al. [2023] where exponential weights is combined with standard loss estimators, the bonus introduces large stability term overhead. Therefore, their bound can only be  $T^{2/3}$  at best even with simulators. 2) In Dai et al. [2023] where exponential weights is combined with magnitude-reduced loss estimators, the loss estimator for action  $a$  can no longer be represented as a simple linear function  $a^\top \hat{y}_t$ . Instead, it becomes a complex non-linear function. This restricts the algorithm's potential to leverage linear optimization oracle over the action set and achieve computational efficiency.
3.5 Overall regret analysis
With all the algorithmic elements discussed above, now we give a formal statement for our regret guarantee and perform a complete regret analysis. Our main theorem is the following.
Proof sketch. Let  $\mathcal{A}_0$  be drawn from  $D$  independently from all the interaction history between the learner and the environment. Recalling the definitions in Definition 1, we have
$$
\begin{array}{l} \operatorname {R e g} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle a _ {t} - u ^ {\mathcal {A} _ {t}}, y _ {t} \rangle \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {t}} - \boldsymbol {U} ^ {\mathcal {A} _ {t}}, \gamma_ {t} \rangle \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {U} ^ {\mathcal {A} _ {0}}, \gamma_ {t} \rangle \right] \\ \leq \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {U} ^ {\mathcal {A} _ {0}} , \gamma_ {t} - \hat {\gamma} _ {t} \rangle \right]} _ {\text {B i a s}} + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {U} ^ {\mathcal {A} _ {0}} , \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \rangle \right]} _ {\text {B o u n s}} + \underbrace {\mathbb {E} \left[ \sum_ {t = 1} ^ {T} \langle \boldsymbol {H} _ {t} ^ {\mathcal {A} _ {0}} - \boldsymbol {U} ^ {\mathcal {A} _ {0}} , \hat {\gamma} _ {t} - \alpha_ {t} \hat {\boldsymbol {\Sigma}} _ {t} ^ {- 1} \rangle \right]} _ {\text {F T R L - R e g}} \\ \end{array}
$$
Each term can be bounded as follows:
- Bias  $\leq \mathcal{O}(d^2\sqrt{T}\log T) + \frac{1}{4}\sum_{t = 1}^{T}\alpha_t\| u - x_t\|_{\hat{\Sigma}_t^{-1}}^2$  (discussed in Section 3.3).
- **Bonus**  $\leq \mathcal{O}(d^2\sqrt{T}\log T) - \frac{1}{4}\sum_{t = 1}^{T}\alpha_t\| u - x_t\|_{\hat{\Sigma}_t^{-1}}^2$  (discussed in Section 3.4).
- FTRL-Reg  $\leq \mathcal{O}(d^2\sqrt{T}\log T)$
Combining all terms gives the desired bound. The complete proof is provided in Appendix D.
3.6 Handling Misspecification
In this subsection, we show how our approach naturally handles the case when the expectation of the loss cannot be exactly realized by a linear function but with a misspecification error. In this case, we assume that the expectation of the loss is given by  $\mathbb{E}[\ell_t|a_t = a] = f_t(a)$  for some  $f_{t}:\mathbb{R}^{d}\to [-1,1]$ , and the realized loss  $\ell_t$  still lies in  $[-1,1]$ . We define the following notion of misspecification (slightly more refined than that in Neu and Olkhovskaya [2020]):
Assumption 1 (misspecification).  $\sqrt{\frac{1}{T}\sum_{t=1}^{T}\inf_{y\in\mathbb{B}_2^d}\sup_{\mathcal{A}\in\mathrm{supp}(D)}\sup_{a\in\mathcal{A}}(f_t(a)-\langle a,y\rangle)^2} \leq \varepsilon.$
Based on previous discussions, the design idea of Algorithm 1 is to 1) identify the bias of the loss estimator, and 2) add necessary bonus to compensate the bias. When there is misspecification, this design idea still applies. The difference is that now the loss estimator  $\hat{y}_t$  potentially has more bias due to misspecification. Therefore, the bias becomes larger by an amount related to  $\varepsilon$ . Consequently, we need to enlarge bonus (raising  $\alpha_t$ ) to compensate it. Due to the larger bonus, we further need to tune down the learning rate  $\eta_t$  to make the algorithm stable. Overall, to handle misspecification, when  $\varepsilon$  is known, it boils down to using the same algorithm (Algorithm 1) with adjusted  $\alpha_t$  and  $\eta_t$ . The case of unknown  $\varepsilon$  can be handled by the standard meta-learning technique Corral [Agarwal et al., 2017, Foster et al., 2020, Luo et al., 2022]. We defer all details to Appendix E and only state the final bound here.
Theorem 3. Under misspecification, there is an algorithm ensuring  $\mathrm{Reg} \leq \tilde{\mathcal{O}}(d^2\sqrt{T} + \sqrt{d}\varepsilon T)$  without knowing  $\varepsilon$  in advance.
4 Linear EXP4
To tighten the  $d$ -dependence in the regret bound, we can use the computationally inefficient algorithm EXP4 [Auer et al., 2002]. The original regret bound for EXP4 has a polynomial dependence on the number of actions, but here we take the advantage of the linear structure to show a bound that only depends on the feature dimension  $d$ . The algorithm is presented in Algorithm 2.
Theorem 2. Algorithm 1 ensures  $\operatorname{Reg} \leq \mathcal{O}(d^2\sqrt{T}\log T)$ .
Algorithm 2 Linear EXP4
input:  $\Pi, \eta, \gamma$ .
for  $t = 1,2,\ldots$  do
Receive  $\mathcal{A}_t\subset \mathbb{R}^d$
Construct  $\nu_{t} \in \Delta(\mathcal{A}_{t})$  such that  $\max_{a \in \mathcal{A}_{t}} \|a\|_{G_{t}^{-1}}^{2} \leq d$ , where  $G_{t} = \mathbb{E}_{a \sim \nu_{t}}[aa^{\top}]$ . Set
$$
P _ {t, \pi} = \frac {\exp \left(- \eta \sum_ {s = 1} ^ {t - 1} \hat {\ell} _ {s , \pi}\right)}{\sum_ {\pi^ {\prime} \in \Pi} \exp \left(- \eta \sum_ {s = 1} ^ {t - 1} \hat {\ell} _ {s , \pi^ {\prime}}\right)}
$$
and define  $p_{t,a} = \sum_{\pi \in \Pi}P_{t,\pi}\mathbb{I}\{\pi (\mathcal{A}_t) = a\}$ .
Sample  $a_{t}\sim \tilde{p}_{t} = (1 - \gamma)p_{t} + \gamma \nu_{t}$  and receive  $\ell_t\in [-1,1]$  with  $\mathbb{E}[\ell_t] = \langle a_t,y_t\rangle$
Construct  $\forall \pi \in \Pi$ :  $\hat{\ell}_{t,\pi} = \langle \pi (\mathcal{A}_t),\tilde{H}_t^{-1}a_t\ell_t\rangle$ , where  $\tilde{H}_t = \mathbb{E}_{a\sim \tilde{p}_t}[aa^\top]$ .
To run Algorithm 2, we restrict ourselves to a finite policy class. The policy class we use in the algorithm is the set of linear policies defined as
$$
\Pi = \left\{\pi_ {\theta}: \theta \in \Theta , \pi_ {\theta} (\mathcal {A}) = \underset {a \in \mathcal {A}} {\operatorname {a r g m i n}} a ^ {\top} \theta \right\} \tag {8}
$$
where  $\Theta$  is an 1-net of  $[-T, T]^d$ . The next theorem shows that this suffices to give us near-optimal bounds for our problem. The proof is given in Appendix F.
Theorem 4. With  $\gamma = 2d\sqrt{(\log T) / T}$  and  $\eta = \sqrt{(\log T) / T}$ , Algorithm 2 with the policy class defined in Eq. (8) guarantees  $\operatorname{Reg} = \mathcal{O}\left(d\sqrt{T\log T}\right)$ .
Note that this result technically also holds in the "A-A" category with respect to the policy class defined in Eq. (8). However, this policy class is not necessarily a sufficient cover of all policies of interest when the contexts and losses are adversarial.
We derived the first algorithm that obtains  $\sqrt{T}$  regret in contextual linear bandits with stochastic action sets in the absence of a simulator or prior knowledge on the distribution. As a side result, we obtained the first computationally efficient  $\mathrm{poly}(d)\sqrt{T}$  algorithm for adversarial sleeping bandits with general stochastic arm availabilities. We believe the techniques in this paper will be useful for improving results for simulator-free linear MDPs as well.