SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Julia Olkhovskaya<sup>1</sup> Jack Mayo<sup>2</sup> Tim van Erven<sup>2</sup> Gergely Neu<sup>3</sup> Chen-Yu Wei<sup>4</sup>
$^{1}$ Department of Intelligent Systems, Delft University of Technology, Delft, The Netherlands*
$^{2}$ Korteweg-de Vries Institute for Mathematics, University of Amsterdam, Amsterdam, The Netherlands
$^{3}$ AI group, DTIC, Universitat Pompeu Fabra, Barcelona, Spain
<sup>4</sup>MIT Institute for Data, Systems, and Society, Massachusetts Institute of Technology, Cambridge, MA, USA
We consider the adversarial linear contextual bandit setting, which allows for the loss functions associated with each of  $K$  arms to change over time without restriction. Assuming the  $d$ -dimensional contexts are drawn from a fixed known distribution, the worst-case expected regret over the course of  $T$  rounds is known to scale as  $\tilde{O}(\sqrt{KdT})$ . Under the additional assumption that the density of the contexts is log-concave, we obtain a second-order bound of order  $\tilde{O}(K\sqrt{dV_T})$  in terms of the cumulative second moment of the learner's losses  $V_T$ , and a closely related first-order bound of order  $\tilde{O}(K\sqrt{dL_T^*})$  in terms of the cumulative loss of the best policy  $L_T^*$ . Since  $V_T$  or  $L_T^*$  may be significantly smaller than  $T$ , these improve over the worst-case regret whenever the environment is relatively benign. Our results are obtained using a truncated version of the continuous exponential weights algorithm over the probability simplex, which we analyse by exploiting a novel connection to the linear bandit setting without contexts.
The contextual bandit problem is a generalization of the multi-armed bandit setting in which a learner observes relevant contextual information before choosing an arm. The goal of the learner is to minimize the excess cumulative loss of the chosen arms compared to the best fixed policy for mapping contexts to arms. This framework addresses a broad range of important real-world problems like sequential treatment allocation (Tewari and Murphy, 2017), online recommendation (Beygelzimer et al., 2011) or online advertising (Li et al., 2010), and is actively used in practice (Agarwal et al., 2016). Numerous variants of the setting have been studied, which differ in the assumptions they make about the losses and the contexts. In this paper, we focus on the recently introduced setting of Neu and Olkhovskaya (2020) where the contexts are finite-dimensional i.i.d. random vectors, and the losses are time-varying linear functions of the context that may potentially be generated by an adversary. In this setting, the worst-case rate for the expected regret is known to be  $\hat{O} (\sqrt{T})$  for time horizon  $T$  (Neu and Olkhovskaya, 2020).
Our main contribution is to replace the worst-case rate by adaptive bounds. Specifically, we obtain a bound of  $\tilde{O} (\sqrt{V_T})$  in terms of a quadratic measure of variance  $V_{T}$  for the losses of the algorithm, and a bound of  $\tilde{O} (\sqrt{L_T^*})$ , where  $L_{T}^{*}$  is the cumulative loss incurred by the optimal policy. Such bounds in terms of  $L_{T}^{*}$  or  $V_{T}$  are generally referred to as first-order and second-order bounds, respectively,
*Work was done when the author was affiliated with Vrije Universiteit Amsterdam.
and have been extensively studied in the bandit literature. They can lead to much stronger guarantees in the often realistic case when  $T$  is large, but the losses vary little or when there exists a policy with very low cumulative loss.
Worst-case guarantees in terms of  $T$  have first been proved for the contextual bandit problem with finite policy classes by Auer et al. (2002b), with further improvements by Beygelzimer et al. (2011). These methods can deal with adversarial losses and contexts, but only work for finite policy classes and have run-time scaling linearly with the size of the class—which is generally unacceptable in practice. This latter challenge has been addressed by a line of work culminating in Agarwal et al. (2014), which only requires access to an optimization oracle over the policy class. Their results, however, remain restricted to i.i.d. contexts and losses. An alternative line of work has been initiated by Auer (2002); Chu et al. (2011); Abbasi-Yadkori et al. (2011), who studied the special case of i.i.d. linear loss functions with changing decision sets. The case of i.i.d. contexts and adversarial linear losses has first been studied by Neu and Olkhovskaya (2020).
Improvements of worst-case guarantees of order  $\sqrt{T}$  to first-order bounds scaling with  $\sqrt{L_T^*}$  have been known for a variety of bandit settings since the works of Stoltz (2005); Allenberg et al. (2006), and Neu (2015). Regarding contextual bandits, the COLT 2017 open problem of Agarwal et al. (2017) asks for efficient algorithms that achieve first-order bounds for large, but finite, policy classes, either when both contexts and losses are i.i.d. or when both are fully adversarial. First to answer the open problem were Allen-Zhu et al. (2018), who obtained an optimal first-order regret guarantee for adversarial losses and contexts, but with an algorithm that is inefficient for large policy classes. Foster and Krishnamurthy (2021) provide the first efficient algorithm for the non-adversarial setting where the loss function is fixed over time and one has access to an oracle that can solve various optimization tasks over the policy class. We improve on these works in terms of the computational efficiency of our algorithm and by allowing the loss function to vary adversially over time, although we do rely on the extra assumption that the loss functions are linear.
Another relevant framework is the adversarial linear bandit setting (without contexts), where there also exist adaptive results (Bubeck et al., 2019; Lee et al., 2020; Ito et al., 2020). While conceptually related, an important distinction is that the linear bandit setting assumes a fixed decision set, whereas reducing the linear contextual bandit problem to a linear bandit problem requires the use of decision sets that change as a function of the contexts.
Main Contributions. We consider a  $K$ -armed linear contextual bandit problem with  $d$ -dimensional contexts over  $T$  rounds. The contexts are assumed to be drawn i.i.d., but the linear loss functions mapping contexts to losses for the arms are chosen by an adaptive adversary. The aim of the learner is to minimize their regret, which is the gap between the expected cumulative loss of the learner and the expected cumulative loss of the best fixed policy  $\pi_T^*$  chosen in full knowledge of the sequence of losses. In this setting,  $\pi_T^*$  is known to be a linear classifier, i.e. it chooses the arm with smallest predicted loss, where the predictions are fixed linear functions of the context (see Section 2). The goal is therefore to compete with all linear classifiers. We first obtain the following second-order bound on the expected regret
$$
R _ {T} = \tilde {O} \left(K \sqrt {d V _ {T}}\right), \tag {1}
$$
where  $V_{T}$  is defined in (5) as a measure of the cumulative second moments of the losses for the arms played by the algorithm. Following Ito et al. (2020), we allow these moments to be centered around optimistic estimates that can further improve the bound when available or can simply be set to zero when they are not. We further obtain a first order bound of the form
$$
R _ {T} \left(\pi_ {T} ^ {*}\right) = \tilde {O} \left(K \sqrt {d L _ {T} ^ {*}}\right). \tag {2}
$$
The second-order bound is obtained using a truncated version of the continuous exponential weights algorithm over the probability simplex, similar to the algorithm for linear non-contextual bandits of Ito et al. (2020), and the first-order bound may be obtained as a corollary. As discussed in Section 3.3, the computational complexity of this method is dominated by two steps that together require  $\tilde{O}(K^5) + (d/\epsilon)^{O(1)}$  per round for approximation up to precision  $\epsilon > 0$ , which is computationally feasible for moderate  $K$  and  $\epsilon$ . Both results are not strict improvements on the worst-case rate of  $\tilde{O}(\sqrt{KdT})$  by Neu and Olkhovskaya (2020): first, they have a slightly worse dependence on  $K$ . We consider this a price worth paying for the first adaptive bounds in this setting. Second, they require the extra
assumption that the distribution of the contexts is log-concave. Although log-concavity is weaker than assuming the contexts follow e.g. (truncated) Gaussian distributions, we conjecture that it may not be necessary to obtain a computationally efficient algorithm. This conjecture is based on the observation that there exists in fact an easy way to obtain at least the first-order bound (2) without the log-concavity assumption, but with an algorithm that has no hope of being efficiently implemented. As described in Section 2.2, this is possible by running the MYGA algorithm (Allen-Zhu et al., 2018) on  $O\left(\frac{T}{dK^2}\right)^{Kd}$  experts that cover the set of linear classifiers to sufficient precision. The run-time of this approach is prohibitive, because it scales linearly with the number of experts, which is a large polynomial in  $T$ .
Techniques. The LinExp3 method of Neu and Olkhovskaya (2020) is based on an adaptation of the classic Exp3 algorithm for regular multi-armed bandits (Auer et al., 2002a). A natural approach would therefore be to replace the Exp3 component in LinExp3 by a method with first-order guarantees for the multi-armed bandit setting, but, as discussed in Section D, this leads to difficulties controlling the variance. Instead of building on Exp3, we therefore follow the perhaps surprising approach of building our algorithm on continuous exponential weights over the probability simplex (van der Hoeven et al., 2018). In particular, our approach is based on a combination of the recently proposed techniques of Ito et al. (2020) for linear bandits with tools designed by Neu and Olkhovskaya (2020) to deal with the contextual case.
Outline. The rest of the paper is organized as follows. After describing the setting in the next section, we state a formal version of the simple first-order bound that can be obtained using the MYGA algorithm (Theorem 2.1). This is followed by Section 3, which states our main results corresponding to the regret bounds in Equations 1 and 2. Section 4 then gives a high-level overview of the proofs, with pointers provided to the details in the appendix. Finally, Section 5 concludes with discussion.
Notation Let  $\Delta^K = \{w\in \mathbb{R}^K |w_1\geq 0,\ldots ,w_K\geq 0,\sum_{a = 1}^K w_a = 1\}$  denote the  $(K - 1)$ -dimensional probability simplex. For any positive semi-definite matrix  $M\in \mathbb{R}^{d\times d}$ ,  $\| v\| _M = \sqrt{v^\top Mv}$  denotes the corresponding Mahalanobis norm, and for any positive integer  $n$ , we abbreviate  $[n] = \{1,\dots ,n\}$ .
2.1 Setting
We consider the setting of (Neu and Olkhovskaya, 2020), in which there is an interaction between a learner and an unknown environment. This interaction proceeds in rounds indexed by  $t \in [T]$ , such that for each  $t$ :
1. The environment commits to  $[K]$  parameter vectors  $\theta_{t,1},\ldots ,\theta_{t,K}\in \mathbb{R}^d$  without revealing any to the learner.
2. A context vector  $X_{t}\in \mathbb{R}^{d}$  is drawn i.i.d. from some fixed distribution  $\mathcal{D}$  according to  $X_{t}\sim \mathcal{D}$ , and revealed to the learner.
3. The learner commits to an action  $A_{t} \in [K]$ , and incurs the loss  $\ell_t(X_t, A_t)$ , where  $\ell_t(X, a) = \langle X, \theta_{t,a} \rangle$ .
The environment is allowed to randomize its choices of  $\theta_{t,a}$ . These must be independent from the context  $X_{t}$  in round  $t$ , but they may depend on previous contexts  $X_{s}$  and actions  $A_{s}$  for  $s < t$ .
We write  $\pi_t(a|X_t)$  for the policy of the learner in round  $t$  conditional on observing context  $X_t$ , so that  $A_t \sim \pi_t(X_t)$ , and we use the following notation for the expected cumulative losses of the algorithm and policy  $\pi$ , respectively:
$$
L _ {T} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \ell_ {t} \left(X _ {t}, A _ {t}\right) \right], L _ {T} ^ {\pi} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \ell_ {t} \left(X _ {t}, \pi \left(X _ {t}\right)\right) \right].
$$
Let  $\Pi$  be the set of all all stationary deterministic policies  $\pi : \mathbb{R}^d \to [K]$ , we define the optimal policy  $\pi^*$  as  $\pi^* = \arg \min_{\pi \in \Pi} L_T^\pi$ . Then the learner's goal is to compete with policy  $\pi^*$ , as measured by the expected regret:
$$
R _ {T} = L _ {T} - L _ {T} ^ {\pi^ {*}} = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} \left\langle X _ {t}, \theta_ {t, A _ {t}} - \theta_ {t, \pi^ {*} \left(X _ {t}\right)} \right\rangle \right],
$$
where the expectation is taken over each  $X_{t}\sim \mathcal{D}$ , and any randomness applied by the learner or environment in their respective choices. Using the linearity of the loss functions it can be shown that the optimal policy is always a linear classifier (Neu and Olkhovskaya, 2020):
$$
\pi_ {T} ^ {*} (x) = \underset {a} {\arg \min } \left\langle x, \sum_ {t = 1} ^ {T} \mathbb {E} [ \theta_ {t, a} ] \right\rangle .
$$
We may therefore restrict attention to competing with policies of the form
$$
\pi_ {\beta} (x) = \underset {a} {\arg \min } \left\langle x, \beta_ {a} \right\rangle \quad (\beta \in \mathbb {R} ^ {K \times d}). \tag {3}
$$
For deriving our technical results, it will be useful to define the filtration  $\mathcal{F}_t = \sigma(\{X_s, A_s : s \leq t\})$  and the notations  $\mathbb{E}_t[\cdot] = \mathbb{E}[\cdot | \mathcal{F}_{t-1}]$  and  $\mathbb{P}_t[\cdot] = \mathbb{P}[\cdot | \mathcal{F}_{t-1}]$ .
Assumptions Following Neu and Olkhovskaya (2020), we assume that  $\| X_t\| \leq \sigma$ ,  $\|\theta_{t,a}\| \leq R$  and  $\ell_t(x,a) \in [-1,1]$  almost surely. In addition, the covariance matrix  $\Sigma = \mathbb{E}[XX^{\top}]$  of the context distribution is assumed to be positive definite, with smallest eigenvalue  $\lambda_{\min}(\Sigma) > 0$ .
2.2 An Inefficient Algorithm
A first order bound for our problem can be obtained by instantiating the MYGA algorithm of Allen-Zhu et al. (2018) for a set of  $\Theta\left(\frac{T}{K^2d}\right)^{Kd}$  experts that cover the parameter space of policies of the form (3), which is guaranteed to contain the optimal policy  $\pi_T^*$ :
Theorem 2.1. Suppose that  $0 \leq \ell_t(a, X_t) \leq 1$  almost surely for all  $a \in [K]$ . Then, by instantiating MYGA with  $\Theta\left(\frac{T}{K^2d}\right)^{Kd}$  experts, it obtains the following first-order bound for the adversarial linear contextual bandit problem:
$$
R _ {T} = O \left(K \sqrt {d L _ {T} ^ {*} \log T} + K ^ {2} d \log T\right). \tag {4}
$$
Although this provides a quick way to see that first-order bounds are possible, the resulting algorithm is completely impractical, because its run-time is proportional to the number of experts, which grows as a large polynomial in  $T$ . The proof, including a more detailed description of the experts, can be found in Appendix A.
3 First- and Second-Order Bounds
In this section we present an algorithm using a novel adaptation of a method developed for the adversarial linear bandit to be suitable for use in the adversarial linear contextual bandit setting. The method proposed is based on a form of continuous exponential weights that has been shown to lead to a first-order bound in the former (Ito et al., 2020). The algorithm allows for optimistic estimates  $m_{t,a} \in \mathbb{R}^d$  for the environment's choices  $\theta_{t,a}$ , which can always be set to 0 when they are not available. We show two types of guarantees. First, in Theorem 3.1, we obtain a second-order regret bound in terms of the cumulative squared error of the estimates  $m_{t,a}$ :
$$
V _ {T} = \mathbb {E} \left[ \sum_ {s = 1} ^ {T} \left\langle X _ {s}, \theta_ {s, A _ {s}} - m _ {s, A _ {s}} \right\rangle^ {2} \right]. \tag {5}
$$
Taking  $m_{t,a} = 0$ , this provides a second-order regret bound in terms of the squared losses. Alternatively,  $m_{t,a}$  may be estimated using an online regression algorithm, as described by Ito et al. (2020). As our second result, we show in Theorem 3.2 that a first-order bound can be derived for the same algorithm with a different choice of hyperparameters and the assumption that the losses are non-negative.
Algorithm 1 CONTEXTEW
Parameters:  $\gamma > 0, \eta_1 \geq \ldots \geq \eta_T > 0, m_1, \ldots, m_T$
1. Observe  $X_{t}$
2. Repeat:
Pick  $Q_{t}$  from the distribution  $p_t$  defined in (8), until
$$
\sum_ {a = 1} ^ {K} \left\| Q _ {t, a} X _ {t} \right\| _ {\Sigma_ {t, a} ^ {- 1}} ^ {2} \leq d K \gamma^ {2}, \tag {6}
$$
where  $\Sigma_{t,a}$  is defined in (9).
3. Set  $\tilde{Q}_t = Q_t$  equal to the last sample of  $Q_{t}$ , which caused the loop to exit, and choose an arm according to  $A_{t} \sim \tilde{Q}_{t}$ .
4. Observe the loss  $\ell_t(X_t, A_t)$  and estimate  $\widehat{\theta}_{t,a}$  for all  $a$  according to (12).
Our full algorithm is shown in Algorithm 1. As it is an adaptation of continuous exponential weights for the contextual bandits setting, we refer to it as CONTEXTEW. It runs a two-stage sampling procedure: after observing context  $X_{t}$ , the first stage of the algorithm samples a random policy  $\widetilde{Q}_t \in \Delta^K$ , and then the second stage consists of drawing an arm  $A_{t}$  randomly from  $\widetilde{Q}_t$ . The distribution of  $\widetilde{Q}_t$  is constructed as follows: first we sample a different policy  $Q_{t}$  from the exponential weights distribution over the probability simplex with density proportional to
$$
w _ {t} (q | X _ {t}) = \exp \left(- \eta_ {t} \sum_ {a = 1} ^ {K} q _ {a} \langle X _ {t}, \sum_ {s = 1} ^ {t - 1} \hat {\theta} _ {s, a} + m _ {t, a} \rangle\right), \tag {7}
$$
where  $m_{s,a}$  is a function that is measurable with respect to  $\mathcal{F}_{s-1}$ . The sum  $\sum_{a=1}^{K} q_a \left\langle X_t, \sum_{s=1}^{t-1} \widehat{\theta}_{s,a} \right\rangle$  estimates the cumulative loss that the policy  $q$  would have incurred if it had been played in all previous rounds. It relies on estimates  $\widehat{\theta}_{s,a}$  of the loss vectors  $\theta_{s,a}$ , which will be defined below, and a time-varying learning rate  $\eta_t > 0$ , which is hyperparameter of the algorithm. The normalized density function corresponding to the weights in (7) is:
$$
p _ {t} (q | X _ {t}) = \frac {w _ {t} (q | X _ {t})}{\int_ {\Delta^ {K}} w _ {t} (q | X _ {t}) d q}. \tag {8}
$$
Following Ito et al. (2020), we then introduce a rejection sampling step (6) to reduce the variance, which is based on the following covariance matrices  $\Sigma_{t,a}$  corresponding to  $Q_{t}$ :
$$
\Sigma_ {t, a} = \mathbb {E} _ {t} \left[ Q _ {t, a} ^ {2} X _ {t} X _ {t} ^ {\intercal} \right], \tag {9}
$$
so that  $\widetilde{Q}_t$  ends up being sampled according to the following truncated exponential weights density:
$$
\tilde {p} _ {t} (q | X _ {t}) = \frac {p _ {t} (q | X _ {t}) \mathbb {1} \left\{\sum_ {a = 1} ^ {K} \| q _ {a} X _ {t} \| _ {\Sigma_ {t , a} ^ {- 1}} ^ {2} \leq d K \gamma^ {2} \right\}}{\mathbb {P} _ {t} \left[ \sum_ {a = 1} ^ {K} \| q _ {a} X _ {t} \| _ {\Sigma_ {t , a} ^ {- 1}} ^ {2} \leq d K \gamma^ {2} | X _ {t} \right]}, \tag {10}
$$
with truncation level hyperparameter  $\gamma > 0$ . We will show that all  $\Sigma_{t,a}$  are invertible, as are their analogues in which  $Q_{t}$  is replaced by  $\widetilde{Q}_{t}$ :
$$
\widetilde {\Sigma} _ {t, a} = \mathbb {E} _ {t} \left[ \widetilde {Q} _ {t, a} ^ {2} X _ {t} X _ {t} ^ {\top} \right]. \tag {11}
$$
It remains to specify our estimators for  $\theta_{t,a}$ , which are defined as follows:
$$
\widehat {\theta} _ {t, a} = m _ {t, a} + \widetilde {Q} _ {t, a} \widetilde {\Sigma} _ {t, a} ^ {- 1} X _ {t} \left(\left\langle X _ {t}, \theta_ {t, a} \right\rangle - \left\langle X _ {t}, m _ {t, a} \right\rangle\right) \mathbb {1} \left\{A _ {t} = a \right\}. \tag {12}
$$
These estimates can be shown to be unbiased:
$$
\begin{array}{l} \mathbb {E} _ {t} \left[ \widehat {\theta} _ {t, a} \right] = m _ {t, a} + \widetilde {\Sigma} _ {t, a} ^ {- 1} \mathbb {E} _ {t} \left[ \widetilde {Q} _ {t, a} X _ {t} X _ {t} ^ {\top} \mathbb {1} \left\{A _ {t} = a \right\} \right] (\theta_ {t, a} - m _ {t, a}) \\ = m _ {t, a} + \widetilde {\Sigma} _ {t, a} ^ {- 1} \mathbb {E} _ {t} \left[ \widetilde {Q} _ {t, a} ^ {2} X _ {t} X _ {t} ^ {\intercal} \right] (\theta_ {t, a} - m _ {t, a}) = \theta_ {t, a}. \\ \end{array}
$$
We instantiate CONTEXTEW with adaptive learning rates  $\eta_t$ . For our second-order result, these are defined in terms of the empirical counterpart to  $V_t$ :  $\widehat{V}_t = \sum_{s=1}^t \langle X_s, \theta_{s,A_s} - m_{s,A_s} \rangle^2$ , and we abbreviate  $G_t = 8\sqrt{\widehat{V}_{t-1}\ln(2T^2) + 144\ln^2T} + 176\ln T$ . Then we set
$$
\eta_ {t} = \left(1 0 0 d K \gamma^ {2} + d \left(\widehat {V} _ {t - 1} + 1 + G _ {t - 1}\right)\right) ^ {- 1 / 2}. \tag {13}
$$
This leads to the following second-order bound:
Theorem 3.1 (Second-Order). Suppose  $\mathcal{D}$  has a log-concave density. Then, for  $\gamma = 4\log(10dKT)$ ,  $\eta_t$  as in (13) and any  $\mathcal{F}_{t-1}$ -measurable estimates  $m_t$ , the expected regret of CONTEXTEW is at most  $R_T = \widetilde{O}(K\sqrt{dV_T})$ .
To tune  $\eta_t$  adaptively for our first-order bound, we define it using the algorithm's empirical cumulative loss  $\widehat{L}_t = \sum_{s=1}^t \ell_t(X_s, A_s)$ , which acts as a self-confident empirical estimate of  $L_T^*$ . We further abbreviate
$$
H _ {t} = 8 \sqrt {2 \widehat {L} _ {t} \ln T + 4 0 \ln^ {2} T} + 7 2 \ln T, \tag {14}
$$
and then set
$$
\eta_ {t} = \left(1 0 0 d \gamma^ {2} + d K \left(\widehat {L} _ {t - 1} + 1 + H _ {t - 1}\right)\right) ^ {- 1 / 2}. \tag {15}
$$
This leads to the following first-order bound:
Theorem 3.2 (First-Order). Suppose that  $\mathcal{D}$  has a log-concave density and that  $0 \leq \ell_t(a, X_t) \leq 1$  almost surely for all  $a \in [K]$ . Then, for  $\gamma = 4\log(10dKT)$ ,  $\eta_t$  as in (15) and  $m_t = 0$ , the expected regret of CONTEXTEW is at most  $R_T = \widetilde{O}(K\sqrt{dL_T^*})$ .
3.3 Computational Efficiency
The two computational bottlenecks in the algorithm are the cost of sampling from the output distribution  $p_t(q|X_t)$  and computation of the covariance matrices  $\Sigma_{t,a}$  in each round.
Due to the log-linearity of our method, there exists several practical methods of sampling. As mentioned in Ito et al. (2020), one can employ the methods of Lovász and Vempala (2007), which was shown in Lovász and Vempala (2006) to enjoy a bound of  $O(K^4 \log(1/\epsilon))$  (where  $\epsilon$  is a bound on the total variation distance between the output distribution and the target), but this still requires knowledge of a density dominating the target distribution on all but a set with total starting mass  $\leq \epsilon/2$ . In Narayanan and Rakhlin (2017), a method is developed for general log-concave distributions which, specialized to log-linear distributions (and without additional assumptions on the initial distribution) yields an  $O(K^3 \nu^2 + \log(1/\epsilon))$  method when the geometry admits a  $\nu$ -self concordant barrier. Since there always exists a  $K$ -self-concordant barrier for a  $K$ -dimensional convex body, and thus the running time of this method for our problem is  $O(K^5 + \log T)$  up to a precision  $\epsilon \sim \frac{1}{T^\beta}$  for some  $\beta > 0$ . As referred to in Ito et al. (2020), the covariance matrix  $\Sigma_{t,a}$  is computable in  $\mathcal{O}((d/\epsilon)^{O(1)})$  sampling steps drawing upon the results of Lovász and Vempala (2007).
In this section we provide the analysis of CONTEXTEW from which Theorems 3.1 and 3.2 follow. Throughout the analysis, we will be extensively using the following property of log-concave distributions:
Lemma 4.1. If  $x$  follows a log-concave distribution  $p$  over  $\mathbb{R}^d$  and  $\mathbb{E}[xx^\tau] \preccurlyeq I$ , we have, for any  $\alpha \geq 0$ :
$$
\mathbb {P} \left[ \| x \| _ {2} ^ {2} \geq d \alpha^ {2} \right] \leq d \exp (1 - \alpha). \tag {16}
$$