SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ritu Xu
New Haven, CT 06511
ruitu.xu@yale.edu
Yifei Min
yifei.min@yale.edu
Tianhao Wang
tianhao.wang@yale.edu
Linear contextual bandits represent a fundamental class of models with numerous real-world applications, and it is critical to developing algorithms that can effectively manage noise with unknown variance, ensuring provable guarantees for both worst-case constant-variance noise and deterministic reward scenarios. In this paper, we study linear contextual bandits with heteroscedastic noise and propose the first noise-adaptive Thompson sampling-style algorithm that achieves a variance-dependent regret upper bound of  $\widetilde{\mathcal{O}}\left(d^{3/2} + d^{3/2}\sqrt{\sum_{t=1}^{T}\sigma_t^2}\right)$ , where  $d$  is the dimension of the context vectors and  $\sigma_t^2$  is the variance of the reward in round  $t$ . This recovers the existing  $\widetilde{\mathcal{O}}(d^{3/2}\sqrt{T})$  regret guarantee in the constant-variance regime and further improves to  $\widetilde{\mathcal{O}}(d^{3/2})$  in the deterministic regime, thus achieving a smooth interpolation in between. Our approach utilizes a stratified sampling procedure to overcome the too-conservative optimism in the linear Thompson sampling algorithm for linear contextual bandits.
Linear contextual bandits represent a natural extension of multi-armed bandit problems (Auer et al., 2002a; Robbins, 1952; Lai et al., 1985), where the reward of each arm is assumed to be a linear function of the contextual information associated with the arm. Such problems manifest in numerous real-world applications, encompassing online advertising (Wu et al., 2016), recommendation systems (Deshpande and Montanari, 2012), and personalized medicine (Varatharajah and Berry, 2022; Lu et al., 2021). A multitude of algorithms have been proposed, tailored to diverse settings within the domain of linear contextual bandits (Auer et al., 2002b; Abe et al., 2003). Notably, two main streams of approaches to address the exploration-exploitation dilemma in linear contextual bandits have emerged: Upper Confidence Bound (UCB) (Chu et al., 2011; Abbasi-Yadkori et al., 2011) and Thompson sampling (TS) (Agrawal and Goyal, 2013; Abeille and Lazaric, 2017). In practical scenarios, the varying and non-transparent noise variance inherent in each reward is a common phenomenon (Towse et al., 2015; Omari et al., 2018; Somu et al., 2018; Cheng and Kleijnen, 1999). Such noise, which may exhibit heteroscedasticity and correlation with the context, can significantly impact algorithm performance, particularly when the variance of the noise is unknown a priori (Kirschner
and Krause, 2018; Zhao et al., 2022). Thus, it is imperative to develop adaptive algorithms capable of handling noise with unknown heteroscedastic variance, ensuring provable guarantees when operating with both constant-variance noise and deterministic reward scenarios. Recently, a few variance-aware UCB algorithms have been proposed for linear bandits. In particular, Zhou et al. (2021) examined the scenario where the variance is known and observed after each arm pull, while Zhang et al. (2021); Kim et al. (2022b); Zhao et al. (2023) explored the unknown variance case. Nevertheless, there is a scarcity of results concerning TS algorithms in this context.
Thompson Sampling (Thompson, 1933), a notable Bayesian approach, provides a computationally efficient means of addressing the exploration-exploitation trade-off (Rusmevichientong and Tsitsiklis, 2010; Chapelle and Li, 2011). This method applies posterior sampling to generate a parameter estimator for arm selection, thereby naturally balancing exploration and exploitation by selecting arms with high expected rewards and those with substantial uncertainty (Russo et al., 2018; Riquelme et al., 2018). In the existing literature, the regret upper bound for the linear TS algorithm for linear contextual bandits is of order  $\widetilde{\mathcal{O}}(d^{3/2}\sqrt{T})$  (Agrawal and Goyal, 2013; Abeille and Lazaric, 2017), and the existing algorithm cannot adapt to higher-order structure of the noise.
As an initial endeavor to incorporate variance information into TS-style algorithmic design for linear bandits, we adopt a weighted ridge regression approach to construct a TS algorithm, LinVDTS, detailed in Algorithm 2. This algorithm achieves a variance-dependent performance guarantee on the expected regret, as demonstrated in Theorem 4.1. Nonetheless, a technical barrier intrinsic to TS-style algorithms, yet absent in UCB, emerges during this naive integration of weighted regression with TS. Specifically, the issue stems from inadequate control over the variance-adjusted norm of unselected context vectors at each step, necessitating an anti-concentration argument that only allows for a variance-dependent upper bound on the expected regret and precludes the acquisition of a high-probability guarantee that is simultaneously variance-dependent. More discussion on this will be given in Section 4.2.
In a bid to navigate this intricate issue and exert explicit uncertainty control, we further propose a novel noise-adaptive TS variant in Algorithm 3. Our algorithmic design features (1) an uncertainty stratification scheme applied to contextual vectors, providing explicit regulation of the uncertainty quantification, (2) a cascading construction process for the feasible set, which, in tandem with the stratification scheme, operates to filter out unlikely arms, thereby eliminating the requirement for the anti-concentration argument in the analysis, and (3) the use of noise-adaptive confidence radii to direct the sampling procedure, thereby ensuring a more constricted variance-dependent regret. The theoretical analysis yields a high-probability regret bound that interpolates smoothly between the constant-variance and deterministic reward regimes.
Main contributions. The main contributions of our paper are summarized in three key aspects.
- We introduce a simple and efficient TS-style algorithm, named LinVDTS (Algorithm 2), that can utilize variance information for linear contextual bandits. We prove that the expected regret of LinVDTS is bounded by  $\widetilde{\mathcal{O}}\left(d^{3/2}\left(1 + \sqrt{\sum_{t=1}^{T}\sigma_t^2}\right)\right)$ , where  $d$  is the ambient dimension and  $\sigma_t^2$  is the variance of the noise in round  $t$ . We also prove that, with high probability, the regret of LinVDTS is bounded by  $\widetilde{\mathcal{O}}\left(d^{3/2}\left(1 + \sqrt{\sum_{t=1}^{T}\sigma_t^2}\right) + \sqrt{T}\right)$ . In the analysis of LinVDTS, we identify the inherent difficulty of controlling the uncertainty induced by posterior sampling, which precludes a high-probability guarantee that is aware of the noise variance.
- We devise a stratified sampling procedure that fulfills more efficient exploration for linear contextual bandits by exerting explicit uncertainty control over the context vectors. Based on this framework, we propose LinNATS (Algorithm 3), a noise-adaptive variant of linear TS. To the best of our knowledge, this is the first noise-adaptive TS algorithm for linear contextual bandits with heteroscedastic noise.
- We prove that LinNATS enjoys a high-probability regret bound of order  $\widetilde{\mathcal{O}}\left(d^{3/2}\left(1 + \sqrt{\sum_{t=1}^{T}\sigma_t^2}\right)\right)$ , under standard assumptions for linear contextual bandits with heteroscedastic noise. This improves over the existing  $\widetilde{\mathcal{O}}(d^{3/2}\sqrt{T})$  regret bound for linear Thompson sampling (Agrawal and Goyal, 2013; Abeille and Lazaric, 2017), especially when the noise variance diminishes. Our analysis bypasses the standard anti-concentration argument thanks to the stratification framework.
Notation. For any  $n \in \mathbb{N}^+$ , we denote  $[n] := \{1, 2, \ldots, n\}$ , and  $x_{1:n}$  is a shorthand for the set  $\{x_1, \ldots, x_n\}$ . We write scalars in normal font while representing vectors and matrices in bold font. For any vector  $\mathbf{x}$  and positive semi-definite matrix  $\boldsymbol{\Sigma}$ , we define  $\| \mathbf{x} \|_{\boldsymbol{\Sigma}} = \| \boldsymbol{\Sigma}^{1/2} \mathbf{x} \|_2$ . For two functions  $f$  and  $g$  defined on  $\mathbb{N}^+$ , we use  $f(n) \lesssim g(n)$  to indicate that there exists a universal constant  $C > 0$  such that  $f(n) \leq C \cdot g(n)$  for all  $n \in \mathbb{X}$ ;  $f(n) \gtrsim g(n)$  is defined analogously. We use  $f(n) \asymp g(n)$  to imply that both  $f(n) \lesssim g(n)$  and  $f(n) \gtrsim g(n)$  hold true. We use  $\mathcal{O}(\cdot)$  to hide constant factors and  $\widetilde{\mathcal{O}}(\cdot)$  to further hide poly-logarithmic terms.
Heteroscedastic linear bandits. The challenge of heteroscedastic noise in linear bandits has been studied through a multitude of perspectives: In particular, Kirschner and Krause (2018) studied linear bandits with heteroscedastic noise via the approach of information-directed sampling. Dai et al. (2022) examined heteroscedastic sparse linear bandits and demonstrated a versatile framework capable of transforming any heteroscedastic linear bandit algorithm into an algorithm tailored for heteroscedastic sparse linear bandits. Building on the UCB approach, Zhou et al. (2021) developed an adaptive algorithm that handles known-variance noise, providing performance guarantees of  $\widetilde{\mathcal{O}}\Big(\sqrt{dT} +d\sqrt{\sum_{t = 1}^{T}\sigma_{t}^{2}}\Big)$ . Zhang et al. (2021) introduced a variance-aware confidence set using elimination with peeling and proved a regret bound of  $\widetilde{\mathcal{O}}\Big(\mathrm{poly}(d)\sqrt{1 + \sum_{t = 1}^{T}\sigma_{t}^{2}}\Big)$ . Kim et al. (2022b) further refined this upper bound to  $\widetilde{\mathcal{O}}\Big(d^2 +d^{3 / 2}\sqrt{\sum_{t = 1}^{T}\sigma_{t}^{2}}\Big)$  for linear bandits. Zhao et al. (2022) proposed a multi-layered design for UCB algorithm that achieves a  $\widetilde{\mathcal{O}}\Big((R + 1)\sqrt{dT} + d\sqrt{\sum_{t = 1}^{T}\sigma_{t}^{2}}\Big)$  regret. More recently, Zhao et al. (2023) proposed a UCB-style algorithm with a unified regret upper bound of  $\widetilde{\mathcal{O}}\Big(d + d\sqrt{\sum_{t = 1}^{T}\sigma_{t}^{2}}\Big)$ , where the guarantee is facilitated by the introduction of a novel Freedman-type concentration inequality for self-normalized martingales.
Thompson sampling. TS as a method to address the exploration-exploitation trade-off has gained significant attention in recent years due to its simplicity, adaptability, and robust empirical performance (Chapelle and Li, 2011; Gopalan et al., 2014; Kandasamy et al., 2018). Its effectiveness has been demonstrated across various scenarios (Russo et al., 2018), and in particular, a multitude of theoretical results have been solidified within multi-armed bandit settings (May et al., 2012; Kaufmann et al., 2012; Korda et al., 2013; Russo and Van Roy, 2016). For linear contextual bandits, Agrawal and Goyal (2013) provided the first proof for the  $\widetilde{\mathcal{O}}(d^{3/2}\sqrt{T})$  regret of linear TS, and later Abeille and Lazaric (2017) delivered an alternate proof and further extended the analysis to more general linear problems. Notably, Hamidi and Bayati (2020) showed that the  $\widetilde{\mathcal{O}}(d^{3/2}\sqrt{T})$  rate cannot be improved in worse-case scenarios. While under additional assumptions such as regularity of the contexts, better rates can be achieved (Hamidi and Bayati, 2020; Kim et al., 2021, 2022a).
Moreover, Zhang (2022) introduced a modified version called Feel-Good TS, which is more aggressive in exploring new actions and achieves an  $\widetilde{\mathcal{O}}(d\sqrt{T})$  regret. Recently, Luo and Bayati (2023) also proposed a geometry-aware approach, enabling the establishment of a minimax optimal regret of order  $\widetilde{\mathcal{O}}(d\sqrt{T})$  for the TS algorithm. Xu et al. (2022) proposed a Langevin Monte Carlo TS algorithm that samples from the posterior distribution beyond Laplacian approximation. Investigations have also been undertaken into TS for kernelized bandits (Chowdhury and Gopalan, 2017) as well as integration of TS with neural networks (Wang and Zhou, 2020; Zhang et al., 2020). Recently, Saha and Kveton (2023) proposed a variance-aware TS algorithm on the topic of Bayesian bandits with a variance-dependent upper bound on its Bayesian regret. As far as our awareness extends, no existing results address noise-adaptive TS algorithms in the context of linear bandits.
In this section, we introduce the fundamental framework for contextual linear bandits. We also provide a concise overview of the linear TS algorithm, which functions as our benchmark methodology.
Algorithm 1 Linear Thompson Sampling
1: for  $t = 1, \dots, T$  do
2: Sample  $\pmb{\theta}_{t}^{\mathrm{TS}}$  from  $\mathcal{N}(\hat{\pmb{\theta}}_t,\beta^2\mathbf{V}_t^{-1})$
3: Select arm  $\mathbf{x}_t\gets \arg \max_{\mathbf{x}\in \mathcal{X}_t}\langle \mathbf{x},\pmb {\theta}_t^{\mathrm{TS}}\rangle$  and observe reward  $r_t$
4: end for
3.1 Linear contextual bandits
We investigate a linear contextual bandit problem with heteroscedastic noise. Let  $T$  be the total number of bandit selection rounds and let  $d$  be the ambient dimension. In each round  $t \in [T]$ , the environment generates an arbitrary set of context vectors  $\mathcal{X}_t \subseteq \mathbb{R}^d$ , potentially even in an adversarial manner, where each  $\mathbf{x} \in \mathcal{X}_t$  denotes the context vector of a feasible action. Upon observing the decision set  $\mathcal{X}_t$ , the agent selects a context vector  $\mathbf{x}_t \in \mathcal{X}_t$ , and then receives a reward  $r_t$  from the environment. In particular, here the reward is a linear function of the selected context further corrupted by noise, i.e.,
$$
r _ {t} = \mathbf {x} _ {t} ^ {\top} \boldsymbol {\theta} ^ {*} + \varepsilon_ {t},
$$
where  $\pmb{\theta}^{*}\in \mathbb{R}^{d}$  represents the true model parameter and  $\varepsilon_{t}$  is the stochastic noise. In this paper, we impose standard assumptions on the linear contextual bandit model, which are common in literature.
Assumption 3.1. The ground truth  $\theta^{*}$  satisfies  $\| \theta^{*}\|_{2}\leq 1$ . For all  $t\in [T]$ , the decision set  $\mathcal{X}_t$  is contained in the unit ball, i.e.,  $\| \mathbf{x}\| _2\leq 1$  for all  $\mathbf{x}\in \mathcal{X}_t$ . There exists a constant  $R > 0$  such that  $|\varepsilon_t|\le R$  for all  $t\in [T]$ . For every  $t\in [T]$ ,  $\mathbb{E}[\varepsilon_t\mid \mathbf{x}_{1:t},\varepsilon_{1:t - 1}] = 0$  and  $\mathbb{E}[\varepsilon_t^2\mid \mathbf{x}_{1:t},\varepsilon_{1:t - 1}] = \sigma_t^2$ .
In each round  $t$ , an algorithm selects an arm  $\mathbf{x}_t \in \mathcal{X}_t$ , and we denote the optimal arm as  $\mathbf{x}_t^*$ , i.e.,  $\mathbf{x}_t^* = \arg \max_{\mathbf{x} \in \mathcal{X}_t} \mathbf{x}^\top \boldsymbol{\theta}^*$ . As a result, the suboptimality of the selected arm at time  $t$  can be expressed as  $\Delta_t = \mathbf{x}_t^{*\top} \boldsymbol{\theta}^* - \mathbf{x}_t^\top \boldsymbol{\theta}^*$ . The objective of the agent is to minimize the cumulative regret incurred over the time horizon  $T$ , which is defined as
$$
\mathcal {R} (T) = \sum_ {t = 1} ^ {T} \Delta_ {t} = \sum_ {t = 1} ^ {T} \left\langle \mathbf {x} _ {t} ^ {*} - \mathbf {x} _ {t}, \boldsymbol {\theta} ^ {*} \right\rangle .
$$
3.2 Thompson sampling for linear contextual bandits
A standard TS algorithm for linear contextual bandits (i.e., Algorithm 1) was first proposed in Agrawal and Goyal (2013), employing Gaussian priors and likelihood functions in the design. At each round  $t \in [T]$ , a sample  $\pmb{\theta}_t^{\mathrm{TS}}$  is drawn from the Gaussian posterior distribution, centered at the estimate  $\widehat{\pmb{\theta}}_t = \mathbf{V}_t^{-1} \sum_{s=1}^{t-1} r_s \mathbf{x}_s$  with covariance matrix  $\beta^2 \mathbf{V}_t^{-1}$ , where  $\mathbf{V}_t = \mathbf{I}_d + \sum_{s=1}^{t-1} \mathbf{x}_s \mathbf{x}_s^{\intercal}$  and the confidence radius  $\beta = 3R \sqrt{d \log(T / \delta)}$ . The arm  $\mathbf{x}_t$  is then selected against the sample  $\pmb{\theta}_t^{\mathrm{TS}}$ .
The main idea here is to achieve exploitation by updating the posterior using collected information, and the exploration among the arms is performed naturally in posterior sampling. Agrawal and Goyal (2013) demonstrated that the algorithm presented in Algorithm 1 achieves a regret bound of order  $\widetilde{O}(d^{3/2}\sqrt{T})$ . This bound is worse than the minimax lower bound by an unavoidable factor of  $\sqrt{d}$  (Hamidi and Bayati, 2020). Despite this discrepancy, the regret bound effectively showcases the algorithm's ability to balance exploration and exploitation in the linear contextual bandit setting.
4 Warm up: a simple and variance-dependent Thompson sampling algorithm
In this section, we introduce a preliminary algorithm, LinVDTS, delineated in Algorithm 2, which combines TS with a weighted ridge regression estimator in a straightforward manner. For the sake of clear exposition, we focus on a less complex linear bandit problem in which the agent is privy to both the reward  $r_t$  and its variance  $\sigma_t^2$  subsequent to the selection of an arm.
4.1 Algorithmic design
In order to integrate variance information into the estimation process, the use of weighted regression has emerged as a prevalent approach in the field of variance-aware online learning (Zhou et al., 2021;
Algorithm 2 Linear Variance-Dependent Thompson Sampling (LinVDTS)
Require: Total number of iterations  $T$ , number of total arms  $n$
1: Initialize  $\mathbf{V}_1\gets \lambda \mathbf{I}_d$ $\nu_{1}\leftarrow 0,\widehat{\pmb{\theta}}_{1}\leftarrow 0,\widehat{\beta}_{1}\leftarrow \sqrt{\lambda},$  and  $\Sigma_1\gets \widehat{\beta}_1^2\mathbf{V}_1^{-1}$
2: for  $t = 1,\dots ,T$  do
3: Observe context vectors  $\mathcal{X}_t$
4: Draw  $\pmb{\theta}_t^{\mathrm{TS}}\sim \mathcal{N}(\widehat{\pmb{\theta}}_t,\pmb {\Sigma}_t)$
5:  $\mathbf{x}_t\gets \arg \max_{\mathbf{x}\in \mathcal{X}_t}\langle \mathbf{x},\pmb{\theta}_t^{\mathrm{TS}}\rangle$
6: Observe reward  $r_t$  and variance  $\sigma_t^2$
7: Update  $\mathbf{V}_{t + 1}\gets \mathbf{V}_t + \mathbf{x}_t\mathbf{x}_t^\top /\overline{\sigma}_t^2$
8: Update  $\pmb{\nu}_{t + 1}\gets \pmb {\nu}_t + r_t\mathbf{x}_t / \overline{\sigma}_t^2$
9: Update  $\hat{\pmb{\theta}}_{t + 1}\gets \mathbf{V}_{t + 1}^{-1}\pmb{\nu}_{t + 1}$
10: Compute covariance matrix  $\pmb{\Sigma}_{t + 1}\gets \widehat{\beta}_{t + 1}^2\mathbf{V}_{t + 1}^{-1}$
Min et al., 2021b, 2022a; Zhou and Gu, 2022; Yin et al., 2022; Zhao et al., 2023). This methodology is intimately associated with the concept of the Best Linear Unbiased Estimator (BLUE), a well-established statistical technique for achieving optimal linear estimation (Henderson, 1975).
Let us first explain the details of the proposed algorithm LinVDTS. In every round  $t$ , the algorithm computes an estimate  $\widehat{\theta}_t$  of the ground truth  $\theta^*$  by solving a weighted ridge regression problem:
$$
\widehat {\boldsymbol {\theta}} _ {t} = \underset {\boldsymbol {\theta}} {\arg \min } \sum_ {s = 1} ^ {t} \frac {1}{\overline {{\sigma}} _ {s} ^ {2}} (r _ {s} - \langle \mathbf {x} _ {s}, \boldsymbol {\theta} \rangle) ^ {2} + \lambda \| \boldsymbol {\theta} \| _ {2} ^ {2}.
$$
Here  $\overline{\sigma}_t$  represents a weight parameter and  $\lambda$  denotes a regularization constant. For each approximation  $\widehat{\pmb{\theta}}_t$ , we stochastically draw  $\pmb{\theta}_t^{\mathrm{TS}}$  from a Gaussian distribution with mean  $\widehat{\pmb{\theta}}_t$  and covariance  $\pmb{\Sigma}_t = \widehat{\beta}_t\mathbf{V}_t^{-1}$ , where  $\mathbf{V}_t = \lambda \mathbf{I}_d + \sum_{s=1}^{t-1}\mathbf{x}_s\mathbf{x}_s^{\intercal}/\overline{\sigma}_s^2$  is computed from previous context and reward pairs, and the confidence radius,  $\widehat{\beta}_t$ , is chosen as
$$
\widehat {\beta} _ {t} \asymp \sqrt {d \log \left(\frac {t}{d \lambda \alpha^ {2}}\right) \log \left(\frac {t ^ {2}}{\delta} \log \frac {\gamma^ {2}}{\alpha}\right)} + \frac {R}{\gamma^ {2}} \log \left(\frac {t ^ {2}}{\delta} \log \frac {\gamma^ {2}}{\alpha}\right) + \sqrt {\lambda}.
$$
Refer to (A.2) for a detailed delineation of  $\widehat{\beta}_t$ . For each context vector  $\mathbf{x} \in \mathcal{X}_t$ , we compute the estimated reward as  $\langle \mathbf{x}, \boldsymbol{\theta}_t^{\mathrm{TS}} \rangle$ , and the algorithm outlined in Algorithm 2 then selects the arm  $\mathbf{x}_t$  that optimizes this estimated reward. Following the approach of Zhou and Gu (2022), we construct the weight parameter  $\overline{\sigma}_t = \max \{\sigma_t, \alpha, \gamma \| \mathbf{x}_t \|_{\mathbf{V}_t^{-1}}^{1/2}\}$ , which is designed as the maximum among the variance, a constant  $\alpha$ , and the uncertainty associated with the selected arm  $\mathbf{x}_t$ . It ensures the formulation of a tight confidence radius  $\widehat{\beta}_t$ , guided by a Bernstein-type concentration inequality, cf. Theorem A.1.
4.2 Regret guarantee and technical challenges
The theorem below establishes a variance-dependent upper bound on the expected regret of LinVDTS. See Appendix A.3 for detailed proof.
Theorem 4.1. Set parameters  $\alpha = 1 / \sqrt{T}$ ,  $\gamma = R^{\frac{1}{2}} / d^{\frac{1}{4}}$ ,  $\lambda = d$ . For any  $\delta \in (0,1)$ , the expected regret of Algorithm 2 is upper bounded as follows:
$$
\mathbb {E} [ \mathcal {R} (T) ] = \mathcal {O} \left(d ^ {3 / 2} \left(1 + \sqrt {\sum_ {t = 1} ^ {T} \sigma_ {t} ^ {2}}\right) \log T + \delta T\right).
$$
Further, it holds with probability  $1 - \delta$  that
$$
\mathcal {R} (T) = \mathcal {O} \left(d ^ {3 / 2} \left(1 + \sqrt {\sum_ {t = 1} ^ {T} \sigma_ {t} ^ {2}}\right) \log T + \sqrt {T \log \frac {1}{\delta}}\right).
$$
Remark 4.2. Choosing  $\delta = 1 / T$ , then we see that the expected regret of Algorithm 2 is bounded by  $\widetilde{\mathcal{O}}\Big(d^{3 / 2}\Big(1 + \sqrt{\sum_{t = 1}^{T}\sigma_t^2}\Big)\Big)$ . However, the high-probability regret bound of Algorithm 2 is of order  $\widetilde{\mathcal{O}}\Big(d^{3 / 2}\Big(1 + \sqrt{\sum_{t = 1}^{T}\sigma_t^2}\Big) + \sqrt{T}\Big)$ , where the additional  $\sqrt{T}$  term fails to be variance-dependent. Nonetheless, this regret bound already improves over the existing  $\widetilde{\mathcal{O}}(d^{3 / 2}\sqrt{T})$  bound for linear Thompson sampling in the literature (Agrawal and Goyal, 2013; Abeille and Lazaric, 2017).
The limitation on the high-probability bound arises from a set of technical challenges that are implicit and inherent in the posterior sampling used in Algorithm 2. To illustrate this, we now provide an outline of the proof for Theorem 4.1, and then discuss the technical intricacies therein.
Proof sketch of Theorem 4.1. Let us denote  $\Delta_t(\mathbf{x}) = \langle \mathbf{x}_t^* - \mathbf{x}, \boldsymbol{\theta}^* \rangle$  as the suboptimality of  $\mathbf{x} \in \mathcal{X}_t$ . We introduce the saturated set  $\mathcal{S}(t)$  for round  $t$  as the ensemble of arms whose confidence radius is dwarfed by its suboptimality, i.e.,  $\mathcal{S}(t) = \left\{\mathbf{x} \in \mathcal{X}_t : \Delta_t(\mathbf{x}) > (2\sqrt{d\log t} + 1)\widehat{\beta_t}\| \mathbf{x}\|_{\mathbf{V}_t^{-1}}\right\}$  (Agrawal and Goyal, 2013). Then the suboptimality  $\Delta_t$  of the chosen context  $\mathbf{x}_t$  can be decomposed as follows:
$$
\Delta_ {t} = \langle \mathbf {x} _ {t} ^ {*} - \mathbf {x} _ {t}, \boldsymbol {\theta} ^ {*} \rangle = \Delta_ {t} (\mathbf {x} _ {t} ^ {\dagger}) + \mathbf {x} _ {t} ^ {\dagger \mathsf {T}} \boldsymbol {\theta} ^ {*} - \mathbf {x} _ {t} ^ {\mathsf {T}} \boldsymbol {\theta} ^ {*},
$$
where the term  $\mathbf{x}_t^\dagger = \arg \min_{\mathbf{x} \notin S(t)} \| \mathbf{x} \|_{\mathbf{V}_t^{-1}}$  refers to the unsaturated arm that possesses the smallest  $\mathbf{V}_t^{-1}$  norm. Leveraging the concentration results delineated in Lemmata A.2 and A.3 pertaining to  $\widehat{\pmb{\theta}}_t$  and  $\pmb{\theta}_t^{\mathrm{TS}}$ , we have
$$
\Delta_ {t} \lesssim \sqrt {d \log t} \left(\| \mathbf {x} _ {t} ^ {\dagger} \| _ {\mathbf {V} _ {t} ^ {- 1}} + \| \mathbf {x} _ {t} \| _ {\mathbf {V} _ {t} ^ {- 1}}\right) \widehat {\beta} _ {t}. \tag {4.1}
$$