SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This result was proven in Lemma 1 in Ito et al. (2020), and also follows from Lemma 5.7 in Lovász and Vempala (2007).
First, we need to introduce some notation which will be useful for the reduction to the linear bandit setting and for the accompanying proofs. We denote  $z_{a}(q,x) = q_{a}x$  and  $z(q,x) = (z_{1}(q,x),\ldots ,z_{K}(q,x))^{\intercal}$ . We also define  $\Sigma_t = \mathrm{diag}_{a\in [K]}(\Sigma_{a,t})$  as a block diagonal arrangement of the covariance matrices per arm. Using this notation, the distribution of the sampling algorithm (10) may be rewritten as
$$
\tilde {p} _ {t} (q | x) = \frac {p _ {t} (q | x) \mathbb {1} \left\{\| z (q , x) \| _ {\Sigma_ {t} ^ {- 1}} ^ {2} \leq d K \gamma^ {2} \right\}}{\mathbb {P} _ {t} \left[ \| z (q , x) \| _ {\Sigma_ {t} ^ {- 1}} ^ {2} \leq d K \gamma^ {2} \right]}. \tag {17}
$$
Let  $\widetilde{Q}_t(x) \sim \tilde{p}_t(q|x)$ ,  $Q_t(x) \sim p_t(q|x)$  and  $\widetilde{Z}_t(x) = z(\widetilde{Q}_t(x), x)$ ,  $Z_t(x) = z(Q_t(x), x)$ ,  $Z^*(x) = z(\pi^*(x), x)$ . And we denote the aggregated loss parameter  $\theta_t = (\theta_1, \ldots, \theta_K)^\top$  and its estimate  $\widehat{\theta}_t = (\widehat{\theta}_1, \ldots, \widehat{\theta}_K)^\top$ . Then we can express the regret as follows:
$$
R _ {T} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \ell_ {t} \left(X _ {t}, A _ {t}\right) - \ell_ {t} \left(X _ {t}, \pi^ {*} \left(X _ {t}\right)\right) \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \tilde {Z} _ {t} \left(X _ {t}\right) - Z ^ {*} \left(X _ {t}\right), \theta_ {t} \right\rangle \right]. \tag {18}
$$
The crucial observation is that the log-concavity of the distribution of  $Z_{t}(X_{t})$  follows from that of the distribution of  $X_{t}$ :
Lemma 4.2. Suppose  $z(q,x) = \sum_{a} q_{a} \varphi(x,a)$  for  $\varphi(x,a) = (\bar{0}^{\intercal}, \dots, x^{\intercal}, \dots)$  such that  $x$  is on the da'th co-ordinate and  $Q(x) \sim p_t(\cdot|x)$  for  $p_t(\cdot|x)$  defined in (8). If  $X \sim p_X(\cdot)$  and  $p_X(\cdot)$  is log-concave and  $Z(x) = z(Q_t(x), x)$ , then  $Z(X)$  also follows a log-concave distribution.
The proof of this result is a rather straightforward computation of the density of  $Z_{t}(X_{t})$  and can be found in Appendix C. To proceed, we write regret as a sum of two terms
$$
R _ {t} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle \widetilde {Z} _ {t} \left(X _ {t}\right) - Z _ {t} \left(X _ {t}\right), \theta_ {t} \right\rangle \right] + \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle Z _ {t} \left(X _ {t}\right) - Z ^ {*} \left(X _ {t}\right), \theta_ {t} \right\rangle \right]. \tag {19}
$$
Having shown that  $Z_{t}(X_{t})$  is log-concave, and since the log-concavity is preserved under linear transformations, for  $y = \Sigma_{t}^{-1 / 2^{\mathrm{T}}}Z_{t}(X_{t})$  we can see that  $\mathbb{E}\left[yy^{\mathrm{T}}\right] = I$ , and thus by Lemma 4.1 it immediately follows that the probability that (6) is not satisfied is small for a proposed choice of  $\gamma = 4\log (10dKT)$ :
$$
\mathbb {P} _ {t} \left[ \| Z _ {t} (X _ {t}) \| _ {\Sigma_ {t} ^ {- 1}} ^ {2} > d K \gamma^ {2} \right] \leq d K \exp (1 - \gamma) \leq 3 d K \exp (- \gamma) \leq \frac {1}{6 T ^ {2}}.
$$
Using this observation, we show that the first term of (19) is just  $\mathcal{O}(1)$ , which is formally proved in Lemma C.2 in the appendix.
To control the second term of the regret decomposition (19), consider the reduction of the contextual bandit problem to a combination of auxiliary online learning problems that are defined separately for each context, as proposed in Neu and Olkhovskaya (2020), Lemma 3. More details and a full proof can be found in Appendix C.
Lemma 4.3. Let  $\pi^{*}$  be any fixed stochastic policy and let  $X_0\sim \mathcal{D}$  be a sample from the context distribution independent from  $\mathcal{F}_T$ . Suppose that  $p_t\in \mathcal{F}_{t - 1}$ , such that  $p_t(\cdot |x)$  is a probability density with respect to Lebesgue measure with support  $\Delta^K$  and let  $Q_{t}(x)\sim p_{t}(\cdot |x)$ . Then,
$$
\mathbb {E} _ {t} \left[ \left\langle Z _ {t} \left(X _ {t}\right) - Z ^ {*} \left(X _ {t}\right), \theta_ {t} \right\rangle \right] = \mathbb {E} _ {t} \left[ \left\langle Z _ {t} \left(X _ {0}\right) - Z ^ {*} \left(X _ {0}\right), \widehat {\theta} _ {t} \right\rangle \right]. \tag {20}
$$
To see why this would be useful further in the proof, we interpret the right-hand side of (20) as follows. Consider the online learning problem for a fixed  $x$  with the decision set to be  $\Delta^K$  and losses  $\ell_t(x,q) = \langle z(q,x),\widehat{\theta}_t\rangle$  and consider running a version of a contextual bandit problem with a fixed context  $x$ , such that the probability of an action  $q$  defined as in Equation 8, so  $p_t(q|x)\propto \exp \left(-\eta_t\sum_{a = 1}^K q_a\left\langle x,\sum_{s = 1}^{t - 1}\widehat{\theta}_{s,a}\right\rangle\right)$ . Then, the regret for the fixed  $x$  against  $\pi^* (x)$  can be written as:
$$
\widehat {R} _ {T} (x) = \sum_ {t = 1} ^ {T} \mathbb {E} _ {Q _ {t} (x) \sim p _ {t} (\cdot | x)} \left[ \langle z (Q _ {t} (x), x) - z (\pi^ {*} (x), x), \widehat {\theta} _ {t} \rangle \right].
$$
Then it is easy to see that the right-hand side of (20) is equal to  $\mathbb{E}\left[\widehat{R}_T(X_0)\right]$ . Thus, we first show a bound on  $\widehat{R}_T(x)$  that holds almost surely for any  $x$  and then take an expectation with respect to  $X_0$ . We control the regret  $\widehat{R}_T(x)$  by following the general schema of the optimistic mirror descent analysis developed in (Rakhlin and Sridharan, 2013; Ito et al., 2020). With this analysis, we get the following bound for any  $x \in \mathcal{X}$ :
Lemma 4.4. Assume that  $\eta_{t + 1} \leq \eta_t$  for all  $t$ , let  $q_0$  be a uniform distribution over  $[K]$  and  $\psi(y) = \exp(y) - y - 1$ . Then, the regret  $\widehat{R}_T(x)$  of CONTEXTEW almost surely satisfies
$$
\begin{array}{l} \widehat {R} _ {T} (x) \leq \frac {1}{T} \sum_ {t = 1} ^ {T} \left\langle z \left(q _ {0} - \pi^ {*} (x), x\right), \widehat {\theta} _ {t} \right\rangle + \frac {K \log T}{\eta_ {T}} \\ \left. + \sum_ {t = 1} ^ {T} \frac {1}{\eta_ {t}} \mathbb {E} _ {Q _ {t} (x) \sim p _ {t} (\cdot | x)} \left[ \psi \left(- \eta_ {t} \left\langle z \left(Q _ {t} (x), x\right), \widehat {\theta} _ {t} - m _ {t} \right\rangle\right) \right], \right. \tag {21} \\ \end{array}
$$
$$
f o r \psi (y) = \exp (y) - y - 1.
$$
We place the derivation of the this bound in the appendix. The crucial ingredient is to show that the square of the estimated loss can be bounded by the square of the true loss. Using the definition of  $\theta_t$ , denoting  $Var_t = \operatorname{tr}\left(\widetilde{\Sigma}_t^{-1}Z_t(X_0)Z_t(X_0)^{\top}\widetilde{\Sigma}_t^{-1}Z_t(X_t)Z_t(X_t)^{\top}\right)$ , we get
$$
\mathbb {E} _ {t} \left[ \left(- \eta_ {t} \left\langle Z _ {t} \left(X _ {0}\right), \widehat {\theta} _ {t} - m _ {t} \right\rangle\right) ^ {2} \right] = \mathbb {E} _ {t} \left[ \eta_ {t} ^ {2} \left(\ell_ {t} \left(A _ {t}, X _ {t}\right) - X _ {t} ^ {\mathrm {T}} m _ {t, A _ {t}}\right) ^ {2} V a r _ {t} \right], \tag {22}
$$
As additional corollary of the concentration result for log-concave random variables, we can show the following relation between matrices  $\Sigma_t$  and  $\tilde{\Sigma}_t$ :
$$
\frac {3}{4} \Sigma_ {t} \preceq \widetilde {\Sigma} _ {t} \preceq \frac {4}{3} \Sigma_ {t}, \tag {23}
$$
which we prove in Lemma C.2 in the appendix. Then we can show that, almost surely:
$$
\begin{array}{l} \mathbb {E} _ {X _ {0}} \left[ V a r _ {t} \right] = \mathbb {E} _ {X _ {0}} \left[ \operatorname {t r} \left(\widetilde {\Sigma} _ {t} ^ {- 1} Z _ {t} \left(X _ {0}\right) Z _ {t} \left(X _ {0}\right) ^ {\intercal} \widetilde {\Sigma} _ {t} ^ {- 1} Z _ {t} \left(X _ {t}\right) Z _ {t} \left(X _ {t}\right) ^ {\intercal}\right) \right] \\ = \operatorname {t r} \left(\widetilde {\Sigma} _ {t} ^ {- 1} \Sigma_ {t} \widetilde {\Sigma} _ {t} ^ {- 1} Z _ {t} (X _ {t}) Z _ {t} (X _ {t}) ^ {\intercal}\right) \leq \frac {4}{3} \operatorname {t r} \left(\widetilde {\Sigma} _ {t} ^ {- 1} \widetilde {\Sigma} _ {t} \widetilde {\Sigma} _ {t} ^ {- 1} Z _ {t} (X _ {t}) Z _ {t} (X _ {t}) ^ {\intercal}\right) \\ = \frac {4}{3} Z _ {t} \left(X _ {t}\right) ^ {\top} \tilde {\Sigma} _ {t} ^ {- 1} Z _ {t} \left(X _ {t}\right) \leq Z _ {t} \left(X _ {t}\right) ^ {\top} \Sigma_ {t} ^ {- 1} Z _ {t} \left(X _ {t}\right) \leq d K \gamma^ {2}. \tag {24} \\ \end{array}
$$
where the first inequality follows from (23) and the second inequality is immediate from (23) and the fact that for symmetric positive definite matrices  $A \succeq B$  follows from  $B^{-1} \succeq A^{-1}$ . The last inequality follows from (6) in the CONTEXT. So, from (22) and (35), we get
$$
\mathbb {E} _ {t} \left[ \left(- \eta_ {t} \left\langle Z _ {t} \left(X _ {0}\right), \widehat {\theta} _ {t} - m _ {t} \right\rangle\right) ^ {2} \right] \leq d K \gamma^ {2} \mathbb {E} _ {t} \left[ \eta_ {t} ^ {2} \left(\ell_ {t} \left(A _ {t}, X _ {t}\right) - X _ {t} ^ {\mathrm {T}} m _ {t, A _ {t}}\right) ^ {2} \right],
$$
which, as we stated above, is the key step to prove Theorem 3.1.
First-order regret bound To prove result of Theorem 3.2, we show that the bound in the Theorem 3.1 can instantiated to obtain a first-order regret bound with a different choice of the learning rate  $\eta_t$ . Going along the same lines with regard to the concentration of  $\widehat{L}_t$  as for  $\widehat{V}_t$ , by setting  $m_t = \bar{0}$  and noticing that then  $V_T \leq L_T$  we get
$$
R _ {T} \leq 2 d K \gamma^ {2} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \eta_ {t} \ell_ {t} \left(A _ {t}, X _ {t}\right) ^ {2} \right] + \widetilde {\mathcal {O}} \left(K \sqrt {d V _ {T}}\right) \leq 4 \sqrt {d} K \gamma^ {2} \sqrt {L _ {T}} + \widetilde {\mathcal {O}} \left(K \sqrt {d L _ {T}}\right).
$$
Since  $R_{T} = L_{t} - L_{T}^{*}$ , by solving the quadratic inequality with respect to  $L_{T}^{*}$ , we get that  $L_{T} \leq L_{T}^{*} + \widetilde{\mathcal{O}}(K\sqrt{d})$ , yielding the final bound.
In conclusion, by applying the approach of (Ito et al., 2020) we have constructed the first scheme achieving  $\tilde{O}\left(K\sqrt{dL_T^*}\right)$  regret with a runtime of  $\mathcal{O}\left((K^5 + \log T) \cdot g_{\Sigma}\right)$ , where  $g_{\Sigma}$  is the time taken to construct the covariance matrix per round - a potentially large polynomial improvement over the  $\mathcal{O}\left(T^{Kd}\right)$  runtime of MYGA. The application of linear bandit algorithms to the contextual bandit problem constitutes, to the best of our knowledge, a novel approach. In doing so we've found a number of positive aspects, including efficiency, but also the direct applicability of other properties enjoyed by the algorithm such as second order bounds (Ito et al., 2020).
Our approach is based on reducing the linear contextual bandit problem to a linear bandit problem, as opposed to a multi-armed bandit problem as in (Neu and Olkhovskaya, 2020). While the specifics of this reduction heavily relied on the joint log-concavity of the context distributions and the exponential-weights posterior over the simplex of actions, we wonder if such approaches can be successfully applied to achieve other types of improvements for linear contextual bandits. In particular, it is curious to what extent other recent advances in the linear bandit problem can be translated to the linear contextual bandit setting. Note that, while the truncation step in Algorithm 1 has an insignificant computational cost as the condition is satisfied with probability  $\mathcal{O}(1 - 1 / T)$ , it can be removed by paying a  $\log(1 / \lambda_{min}(\Sigma))$  multiplicative term in the regret by implementing additional exploration with probability  $1 / T$ . It is natural to ask whether or not approaches based on other instantiations of online mirror descent would also yield first-order bounds, and possibly improve the dependence on  $K$ . The answer is not obvious: for an example of how a naive application of an instantiation of FTRL fails to achieve a first-order bound, see Appendix D.
A relevant question pertains to whether or not such an application of algorithms for linear bandits is necessary at all, but standard approaches such as direct adaptation of Exp3, and first-order adaptations thereof such as GREEN Allenberg et al. (2006) do not seem to give the desired result. In addition, thresholding the worst performing arms inevitably biases the loss estimator due to undersampling of those arms for which the threshold has been applied, and the resulting additional bias term picked up in the regret scales with  $1 / \lambda_{\min}(\Sigma_{t,a})$ , which may be arbitrarily large. Another standard approach of finding an optimistic estimator yielded no fruit during the course of this study due to the lack of the existence of such an estimator without saving all previous losses explicitly.
Our algorithm achieves the regret bound  $\mathcal{O}(K\sqrt{dV_T})$ , while the worst case guarantee of LINEXP3 of Neu and Olkhovskaya (2020) is  $\mathcal{O}(\sqrt{dKT})$ . This discrepancy is not surprising as the Algorithm 1 of Ito et al. (2020) scales as  $\mathcal{O}(n\sqrt{T})$  ( $n$  being the dimension of the action space for the linear bandit), which arises from the deployment of continuous exponential weights. MYGA achieves the same  $\mathcal{O}(K\sqrt{dL_T^*})$  bound due to the number of experts needed to cover the joint set of additive loss parameters. It is worth here emphasising that no known algorithm achieves a better dependence on  $K$  than  $\mathcal{O}(K\sqrt{dL_T^*})$  for the linear adversarial contextual bandit problem. Meanwhile, if the linear bandit is played on the  $n$ -simplex, an improvement to  $\sqrt{nT}$  is possible. For further discussion of this point, see Section 28.5 of Lattimore and Szepesvári (2020). It is thus still unclear whether or not the extra factor of  $\sqrt{K}$  is necessary if one aims for a first-order bound.
An additional point is that while the MYGA algorithm Allen-Zhu et al. (2018) allows for adversarially chosen contexts, the analysis of MYGA for our setting relies heavily on the assumption that contexts are drawn i.i.d. at each iteration. A natural question is then whether or not a similar result is achievable in the adversarial context case. It is known that achieving sub-linear regret is not possible even for full-information online learning of one-dimensional threshold classifiers when both contexts and losses are adversarial (Ben-David et al., 2009; Syrgkanis et al., 2016), which renders sub-linear regret similarly impossible to guarantee for the even harder setting that we consider in this paper. However, we do conjecture that we could overcome the assumption that the distribution is known or that we can sample from it by employing a more elaborate algorithm to estimate the distribution from the data. Indeed, it is not obvious if the distributional assumption of a lower bound to the covariance matrix eigenvalues is entirely necessary, since the regret does not depend on this.
Lastly, it would be an interesting challenge to see if a high-probability regret bound could be obtained in the form stated in the COLT 2017 open problem Agarwal et al. (2017) for this setting, but since a high-probability  $O(\sqrt{T})$  has not yet been proved for the problem here considered, the latter may be more worthy of focus in the short term.
Tim van Erven and Jack Mayo were supported by the Netherlands Organization for Scientific Research (NWO) under grant number VI.Vidi.192.095. Gergely Neu was supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (Grant agreement No. 950180). Chen-Yu Wei would like to acknowledge the support from Simons-Berkeley Research Fellowship.
Foster, D. J. and Krishnamurthy, A. (2021). Efficient first-order contextual bandits: Prediction, allocation, and triangular discrimination. volume 23, page 18907 - 18919.
Freedman, D. A. (1975). On Tail Probabilities for Martingales. The Annals of Probability, 3(1):100 - 118.
Ito, S., Hirahara, S., Soma, T., and Yoshida, Y. (2020). Tight first- and second-order regret bounds for adversarial linear bandits. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 2028-2038. Curran Associates, Inc.
Lattimore, T. and Szepesvári, C. (2020). Bandit Algorithms. Cambridge University Press.
Lee, C.-W., Luo, H., Wei, C.-Y., and Zhang, M. (2020). Bias no more: high-probability data-dependent regret bounds for adversarial bandits and mdps. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 15522-15533. Curran Associates, Inc.
Li, L., Chu, W., Langford, J., and Schapire, R. E. (2010). A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pages 661-670. ACM.
Lovász, L. and Vempala, S. (2006). Hit-and-run from a corner. SIAM Journal on Computing, 35(4):985-1005.
Lovasz, L. and Vempala, S. (2007). The geometry of logconcave functions and sampling algorithms. *Random Structures & Algorithms*, 30(3):307-358.
Narayanan, H. and Rakhlin, A. (2017). Efficient sampling from time-varying log-concave distributions. Journal of Machine Learning Research, 18(112):1-29.
Neu, G. (2015). First-order regret bounds for combinatorial semi-bandits. In Conference on Learning Theory, pages 1360-1375. PMLR.
Neu, G. and Olkhovskaya, J. (2020). Efficient and robust algorithms for adversarial linear contextual bandits. In Proceedings of the 33rd Annual Conference on Learning Theory (COLT 2020), pages 3049-3068.
Rakhlin, A. and Sridharan, K. (2013). Optimization, learning, and games with predictable sequences. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS'13, page 3066-3074, Red Hook, NY, USA. Curran Associates Inc.
Stoltz, G. (2005). Incomplete information and internal regret in prediction of individual sequences. PhD thesis, Université Paris Sud-Paris XI.
Syrgkanis, V., Krishnamurthy, A., and Schapire, R. (2016). Efficient algorithms for adversarial contextual learning. In Balcan, M. F. and Weinberger, K. Q., editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2159-2168, New York, New York, USA. PMLR.
Tewari, A. and Murphy, S. A. (2017). From ads to interventions: Contextual bandits in mobile health. In Mobile Health - Sensors, Analytic Methods, and Applications.
van der Hoeven, D., van Erven, T., and Kotlowski, W. (2018). The many faces of exponential weights in online learning. In Conference On Learning Theory, pages 2067-2092.
A First-order Bound by Reduction to MYGA
Proof. The MYGA algorithm of Allen-Zhu et al. (2018) competes with a class of experts  $E$ , where each expert  $e \in E$  provides a stochastic prediction  $\xi_t^e \in \Delta_K$  in each round  $t$ . It provides the following expected regret bound with respect to the best expert:
$$
R _ {T} = O \left(\sqrt {K \log (| E | + T) L _ {T} ^ {*}} + K \log (| E | + T)\right). \tag {25}
$$
Losses for the arms can be adversarial, and are assumed to take values in  $[0,1]$ .
We will instantiate the experts to cover the parameter space  $\{\beta \in \mathbb{R}^{K\times d}:\max_a\| \beta_a\| \leq RT\}$  of potentially optimal parameters for deterministic policies of the form (3), which we know must contain the optimal policy  $\pi_T^*$  with corresponding parameters  $\beta^{*} = \mathbb{E}[\sum_{t = 1}^{T}\theta_{t}]$ . The covering number for a ball of radius  $RT$  at precision  $\epsilon >0$  is between  $\left(\frac{RT}{\epsilon}\right)^d$  and  $\left(\frac{3RT}{\epsilon}\right)^d$ , so by taking the Cartesian product of this covering with itself  $K$  times we can cover all  $\beta$  with  $\left(\frac{RT}{\epsilon}\right)^{Kd}\leq |E|\leq \left(\frac{3RT}{\epsilon}\right)^{Kd}$  points  $\beta^1,\ldots ,\beta^{|E|}$ . Let  $\ddot{\beta}\in \{\beta^1,\dots ,\beta^{|E|}\}$  be the closest point in the covering to the optimal parameters  $\beta^{*}$ . Then its expected approximation error can be upper bounded as follows:
$$
\begin{array}{l} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle X _ {t}, \theta_ {t, \pi_ {\ddot {\beta}} (X _ {t})} \right\rangle - \left\langle X _ {t}, \theta_ {t, \pi_ {\beta^ {*}} (X _ {t})} \right\rangle \right] = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle X _ {0}, \theta_ {t, \pi_ {\ddot {\beta}} (X _ {0})} \right\rangle - \left\langle X _ {0}, \theta_ {t, \pi_ {\beta^ {*}} (X _ {0})} \right\rangle \right] \\ = \mathbb {E} \left[ \left\langle X _ {0}, \beta_ {\pi_ {\ddot {\beta}} (X _ {0})} ^ {*} \right\rangle - \left\langle X _ {0}, \beta_ {\pi_ {\beta^ {*}} (X _ {0})} ^ {*} \right\rangle \right] \\ \leq \mathbb {E} \left[ \left\langle X _ {0}, \ddot {\beta} _ {\pi_ {\hat {\beta}} (X _ {0})} \right\rangle - \left\langle X _ {0}, \beta_ {\pi_ {\beta^ {*}} (X _ {0})} ^ {*} \right\rangle \right] + \sigma \epsilon \\ = \mathbb {E} \left[ \min  _ {a} \left\langle X _ {0}, \ddot {\beta} _ {a} \right\rangle - \left\langle X _ {0}, \beta_ {\pi_ {\beta^ {*}} (X _ {0})} ^ {*} \right\rangle \right] + \sigma \epsilon \\ \leq \mathbb {E} \left[ \left\langle X _ {0}, \ddot {\beta} _ {\pi_ {\beta^ {*}} (x _ {0})} \right\rangle - \left\langle X _ {0}, \beta_ {\pi_ {\beta^ {*}} (x _ {0})} ^ {*} \right\rangle \right] + \sigma \epsilon \\ \leq \mathbb {E} \left[ \left\langle X _ {0}, \beta_ {\pi_ {\beta^ {*}} (x _ {0})} ^ {*} \right\rangle - \left\langle X _ {0}, \beta_ {\pi_ {\beta^ {*}} (x _ {0})} ^ {*} \right\rangle \right] + 2 \sigma \epsilon \\ = 2 \sigma \epsilon . \\ \end{array}
$$
Adding this to (25), instantiated with  $|E| \leq \left(\frac{3RT}{\epsilon}\right)^{Kd}$ , and choosing  $\epsilon = \frac{dK^2}{2}$  completes the proof.
B Auxiliary lemmas
To ensure that step 2 in CONTEXTEW is defined correctly, we show that the matrix  $\Sigma_t$  is full rank: Lemma B.1. Let the distribution of  $X_{t}$  be such that  $\lambda_{\mathrm{min}}(\mathbb{E}[X_tX_t^\top]) > 0$ . Then, we can show
$$
\lambda_ {\min } \left(\Sigma_ {t, a}\right) > 0 \tag {26}
$$
for any  $a \in [K]$ , and consequently
$$
\lambda_ {\min } \left(\Sigma_ {t}\right) > 0 \tag {27}
$$
Proof. To show that  $\Sigma_{t,a}$  is full rank, it suffices to show that there is no  $v\in \mathbb{R}^d$  such that  $v^{\intercal}\Sigma_{t,a}v = 0$ . Suppose, to the contrary, that such a  $v$  does exist. Then  $0 = v^{\intercal}\mathbb{E}_t\left[Q_{t,a}^2 (X_t)X_tX_t^{\intercal}\right]v = \mathbb{E}_t\left[Q_{t,a}^2 (v^{\intercal}X_t)^2\right]$ , which implies that  $Q_{t,a}v^{\intercal}X_{t} = 0$  almost surely. Since  $Q_{t,a} > 0$  almost surely, it follows that in fact  $v^{\intercal}X_{t} = 0$  almost surely and therefore  $0 = \mathbb{E}_t\left[(v^{\intercal}X_t)^2\right] = v^{\intercal}\mathbb{E}_t\left[X_tX_t^{\intercal}\right]v$ . But this contradicts our assumption that  $\lambda_{\min}(\mathbb{E}_t\left[X_tX_t^{\intercal}\right]) > 0$ .
We will use a simple corollary of Freedman's inequality Freedman (1975) that was introduced in Lemma 2 in Bartlett et al. (2008):
Lemma B.2. Let  $Y_{1}, \ldots, Y_{t}$  be a martingale difference sequence with respect to a filtration  $\mathcal{F}_{1} \subset \dots \subset \mathcal{F}_{t}$  such that  $\mathbb{E}[Y_s|\mathcal{F}_s] = 0$ . Suppose that  $Y_{s} \leq b$  holds almost surely. Then with probability at least  $1 - \delta$  we have  $\sum_{s=1}^{t} Y_{s} \leq 2\max\{2\sqrt{\sum_{s=1}^{t}\mathbb{E}[Y_{s}^{2}|\mathcal{F}_{s}]}, b\sqrt{\ln(1/\delta)}\} \sqrt{\ln(1/\delta)}$ .
In our analysis, we will use some results from Ito et al. (2020). We will make use of the following concentration property of log-concave distributions, Lemma 1 from Ito et al. (2020):
Lemma B.3. If  $y$  follows a log-concave distribution  $p$  over  $\mathbb{R}^d$  and  $\mathbb{E}_{y\sim p}[yy^{\intercal}] \preccurlyeq I$ , we have
$$
\mathbb {P} \left[ \| y \| _ {2} ^ {2} \geq d \alpha^ {2} \right] \leq d \exp (1 - \alpha),
$$
for arbitrary  $\alpha \geq 0$