SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For upper bound on the expected regret, we use an anti-concentration argument (Lemma A.6) to relate  $\| \mathbf{x}_t^\dagger \|_{\mathbf{V}_t^{-1}}$  back to  $\| \mathbf{x}_t\|_{\mathbf{V}_t^{-1}}$ . This yields
$$
\mathbb {E} \left[ \Delta_ {t} \mid \mathscr {H} _ {t} \right] \lesssim \sqrt {d \log t} \mathbb {E} \left[ \| \mathbf {x} _ {t} \| _ {\mathbf {V} _ {t} ^ {- 1}} \right] \hat {\beta} _ {t} + \frac {2}{t ^ {2}}, \tag {4.2}
$$
where  $\{\mathcal{H}_t\}_{t\geq 1}$  represents a filtration of the information available up to the observation of the set of context vectors at each round. Then collecting the above inequality for all  $t\in [T]$ , together with an elliptical potential lemma, we get the desired upper bound for  $\mathbb{E}[\mathcal{R}(T)]$ .
Next, for the high-probability regret bound, we decompose
$$
\mathcal {R} (T) = \sum_ {t = 1} ^ {T} \mathbb {E} \left[ \Delta_ {t} \mid \mathcal {H} _ {t} \right] + \sum_ {t = 1} ^ {T} \left(\Delta_ {t} - \mathbb {E} \left[ \Delta_ {t} \mid \mathcal {H} _ {t} \right]\right) \tag {4.3}
$$
where the first term corresponds to the expected regret bound. We control the second term using martingale concentration, which results in an additional  $\sqrt{T}$  term in the final regret bound.
Why cannot simultaneously achieve variance awareness and high-probability bound?
First, the optimism induced by the posterior sampling is too conservative. A primary challenge in the analysis is to effectively control each suboptimality term  $\Delta_t$ , cf.(4.1). It is important to note that one key algorithmic design in UCB-type algorithms is the exact optimism from the UCB bonus  $\beta \| \mathbf{x}_t\|_{\mathbf{V}_t^{-1}}$ , which ensures  $\langle x_t^*,\widehat{\pmb{\theta}}_t\rangle +\beta \| \mathbf{x}_t^*\|_{\mathbf{V}_t^{-1}}\leq \langle x_t,\widehat{\pmb{\theta}}_t\rangle +\beta \| \mathbf{x}_t\|_{\mathbf{V}_t^{-1}}$ . However, for TS-style algorithms, the absence of this exact optimism hinders our control over the term  $\| \mathbf{x}_t^*\|_{\mathbf{V}_t^{-1}}$  throughout all rounds  $t\in [T]$ , i.e., a standard decomposition of  $\Delta_t = \langle \mathbf{x}_t^* -\mathbf{x}_t,\pmb {\theta}^*\rangle$  yields  $\Delta_t\leq (2\sqrt{d\log t} + 1)\widehat{\beta}_t(2\| \mathbf{x}_t^*\|_{\mathbf{V}_t^{-1}} + \| \mathbf{x}_t\|_{\mathbf{V}_t^{-1}})$  in this case. As a result, standard analyses turn to methodologies that employ an anti-concentration inequality on the sampling distribution (Agrawal and Goyal, 2013). This leads to the inequality  $\| \mathbf{x}_t^\dagger \|_{\mathbf{V}_t^{-1}}\leq \| \mathbf{x}_t\|_{\mathbf{V}_t^{-1}}$  with a constant probability, thereby providing an upper bound on the expected regret.
Second, in the conversion of the expected regret bound to a high-probability bound, we have no help from the noise variance information. One may wonder if it is possible to incorporate the
This is also identified in Zhang (2022) as the source of suboptimal dependence on dimension in the regret bound for TS.
variance of  $\Delta_t - \mathbb{E}[\Delta_t \mid \mathcal{H}_t]$  to get a Bernstein-type concentration result. However, the variance of  $\Delta_t - \mathbb{E}[\Delta_t \mid \mathcal{H}_t]$  does not necessarily conform to the noise variance, especially when the decision set  $\mathcal{X}_t$  is revealed adversarially. To see this, consider the following situation: Given  $\widehat{\theta}_t$ , the environment reveals a decision set  $\mathcal{X}_t = \{\mathbf{x}_{\perp}, -\mathbf{x}_{\perp}\}$  where  $\mathbf{x}_{\perp}$  is orthogonal to  $\widehat{\theta}_t$ . Then a small perturbation in  $\boldsymbol{\theta}_t^{\mathrm{TS}}$  along the direction of  $\mathbf{x}_{\perp}$  can cause the change of arm selection. This implies that a small uncertainty of the posterior distribution along the direction of  $\mathbf{x}_{\perp}$  will be amplified to a constant variation of the selected arm  $\mathbf{x}_t$ . Therefore, in general, we do not have any delicate control of the variance of  $\Delta_t - \mathbb{E}[\Delta_t \mid \mathcal{H}_t]$ . Consequently, we can only apply Azuma-Hoeffding inequality to get a non-noise-adaptive term of  $\widetilde{\mathcal{O}}(\sqrt{T})$ . We discuss in the next section how to conquer this via more advanced algorithmic design.
5 General case: a noise-adaptive Thompson sampling algorithm
In this section, we propose another TS algorithm LinNATS for linear contextual bandits, one that is provably adaptive to unknown heteroscedastic noise. For greater generality, we adopt the identical problem setting delineated in Section 3, and we refrain from assuming access to the variance information  $\{\sigma_t^2\}_{t=1}^T$  associated with the rewards  $\{r_t\}_{t=1}^T$ .
The proposed algorithm LinNATS is displayed in Algorithm 3. Below we go through the details of LinNATS and explain the algorithmic design along the way.
Stratification over contextual uncertainty. For more efficient uncertainty control, we adopt a stratification strategy akin to that previously used for UCB-type algorithms (Chu et al., 2011; Li et al., 2023; Zhao et al., 2023). Our algorithm segments the context vectors at each round  $t$  into  $L$  distinct layers, enabling the precise control of both  $\| \mathbf{x}_t\|_{\mathbf{V}_t^{-1}}$  and  $\| \mathbf{x}_t^*\|_{\mathbf{V}_t^{-1}}$ .
Specifically, the  $\ell$ -th layer establishes a threshold of  $2^{-\ell}$  and retains a separate estimate  $\widehat{\pmb{\theta}}_{t,\ell}$  and sampled variable  $\pmb{\theta}_{t,\ell}^{\mathrm{TS}}$  for each  $\ell \in [L]$  and  $t \in [T]$  (Line 3). Commencing from  $\ell = 1$ , a sequence of decision sets  $\mathcal{X}_t = \mathcal{X}_{t,1} \supseteq \mathcal{X}_{t,2} \supseteq \ldots$  of diminishing sizes is derived by excluding the arms  $\mathbf{x} \in \mathcal{X}_{t,\ell}$  that are less likely to maximize  $\langle \mathbf{x}, \pmb{\theta}^* \rangle$  at each layer  $\ell$ . More specifically, on Line 11 we let
$$
\mathcal {X} _ {t, \ell + 1} = \left\{\mathbf {x} \in \mathcal {X} _ {t, \ell}: \langle \mathbf {x}, \boldsymbol {\theta} _ {t, \ell} ^ {\mathrm {T S}} \rangle \geq \max  _ {\mathbf {x} ^ {\prime} \in \mathcal {X} _ {t, \ell}} \left\langle \mathbf {x} ^ {\prime}, \boldsymbol {\theta} _ {t, \ell} ^ {\mathrm {T S}} \right\rangle - 2 ^ {- \ell + 1} \left(\sqrt {2 d \log \left(4 t ^ {2} L / \delta\right)} + 1\right) \widehat {\beta} _ {t, \ell} \right\}, \tag {5.1}
$$
which ensures that the optimal arm  $\mathbf{x}_t^*$  falls into all decision sets with high probability. The context vector  $\mathbf{x}_t = \arg \max_{\mathbf{x} \in \mathcal{X}_{t,\ell}} \langle \mathbf{x}, \boldsymbol{\theta}_{t,\ell}^{\mathrm{TS}} \rangle$  is chosen if the uncertainty term  $\| \mathbf{x} \|_{\mathbf{V}_{t,\ell}^{-1}} \leq \alpha$  for all  $\mathbf{x} \in \mathcal{X}_{t,\ell}$  (Line 8). Otherwise,  $\mathbf{x}_t$  is selected only when the uncertainty surpasses the threshold, i.e.,  $\| \mathbf{x}_t \|_{\mathbf{V}_{t,\ell}^{-1}} \geq 2^{-\ell}$  for some  $\ell$ , and the elimination process in (5.1) terminates (Line 13).
The round index  $t$  is incorporated into a growing index set  $\Psi_{t+1,\ell}$  (Line 9 & 15), and the observation pair  $(\mathbf{x}_t, r_t)$  participates the later estimation of  $\widehat{\pmb{\theta}}_{s,\ell}$  for all  $s > t$  only if the chosen arm exhibits large uncertainty within the current layer, i.e.,  $\| \mathbf{x}_t \|_{\mathbf{V}_{t,\ell}^{-1}} \geq 2^{-\ell}$  (Line 21). This process guarantees that every context vector  $\mathbf{x} \in \mathcal{X}_{t,\ell}$ , cascading through the first  $(\ell-1)$ -th layer, satisfies  $\| \mathbf{x} \|_{\mathbf{V}_{t,\ell-1}^{-1}} \leq 2^{-\ell+1}$ .
Parameter estimation with weighted ridge regression. To estimate  $\theta^{*}$ , we again utilize weighted ridge regression, but this time within each uncertainty level. For each layer  $\ell \in [L]$  and round  $t \in [T]$ , the associated estimator  $\widehat{\theta}_{t,\ell}$  is given by
$$
\widehat {\pmb {\theta}} _ {t, \ell} = \underset {\pmb {\theta} \in \mathbb {R} ^ {d}} {\arg \min} \sum_ {s \in \Psi_ {t, \ell}} w _ {s} ^ {2} (r _ {s} - \langle \mathbf {x} _ {s}, \pmb {\theta} \rangle) ^ {2} + 2 ^ {- 2 \ell} \| \pmb {\theta} \| _ {2} ^ {2},
$$
where the weight parameter  $w_{t} > 0$  is selected to fulfill the condition  $\| w_{t}\mathbf{x}_{t}\|_{\mathbf{V}_{t,\ell}^{-1}} = 2^{-\ell}$  (Line 14). Owing to the specific formulation of the weighting parameter, it follows that  $\sup_{s\in \Psi_{t,\ell}}\| w_s\mathbf{x}_s\|_{\mathbf{V}_{s,\ell}^{-1}} = 2^{-\ell}$  holds universally for all  $t\in [T]$  and  $\ell \in [L]$ . This allows the application of a Freedman-type
Algorithm 3 Linear Noise-Adaptive Thompson Sampling (LinNATS)
Require: Time horizon  $T$ , number of arms  $n$ , and threshold  $\alpha = T^{-3/2}/R$   
1: Initialize  $L \leftarrow [\log_2(1/\alpha)]$ ,  $\mathbf{V}_{1,\ell} \leftarrow 2^{-2\ell}\mathbf{I}_d$ ,  $\nu_{1,\ell} \leftarrow 0$ ,  $\widehat{\theta}_{1,\ell} \leftarrow 0$ ,  $\Psi_{1,\ell} \leftarrow \emptyset$  and  $\widehat{\beta}_{1,\ell} \leftarrow 2^{-\ell+1}$  for all  $\ell \in [L]$   
2: for  $t = 1, \ldots, T$  do  
3: Draw  $\theta_{t,\ell}^{\mathrm{TS}} \sim \mathcal{N}(\widehat{\theta}_{t,\ell}, \Sigma_{t,\ell})$  for all  $\ell \in [L]$   
4: Observe context vectors  $\mathcal{X}_t = (\mathbf{x}_{t,1}, \ldots, \mathbf{x}_{t,n})$   
5: Let  $\ell \leftarrow 1$ ,  $\mathcal{X}_{t,\ell} \leftarrow \mathcal{X}_t$   
6: while  $\mathbf{x}_t$  is not selected do  
7: if  $\|\mathbf{x}\|_{\mathbf{V}_{t,\ell}^{-1}} \leq \alpha$  for all  $\mathbf{x} \in \mathcal{X}_{t,\ell}$  then  
8: Choose  $\mathbf{x}_t \leftarrow \arg \max_{\mathbf{x} \in \mathcal{X}_{t,\ell}}\langle \mathbf{x}, \theta_{t,\ell}^{\mathrm{TS}} \rangle$  and observe reward  $r_t$   
9: Update  $\Psi_{t+1,\ell'} \leftarrow \Psi_{t,\ell'}$  for all  $\ell' \in [L]$   
10: else if  $\|\mathbf{x}\|_{\mathbf{V}_{t,\ell}^{-1}} \leq 2^{-\ell}$  for all  $\mathbf{x} \in \mathcal{X}_{t,\ell}$  then  
11: Update  $\mathcal{X}_{t,\ell+1}$  following (5.1)  
12: else  
13: Choose any  $\mathbf{x}_t \in \mathcal{X}_{t,\ell}$  such that  $\|\mathbf{x}_t\|_{\mathbf{V}_{t,\ell}^{-1}} > 2^{-\ell}$  and observe reward  $r_t$   
14: Compute  $w_t \leftarrow 2^{-\ell}/\|\mathbf{x}_t\|_{\mathbf{V}_{t,\ell}^{-1}}$   
15: Update  $\Psi_{t+1,\ell} \leftarrow \Psi_{t,\ell} \cup \{t\}$  and  $\Psi_{t+1,\ell'} \leftarrow \Psi_{t,\ell'}$  for all  $\ell' \in [L] \setminus \{\ell\}$   
16: end if  
17: Update  $\ell \leftarrow \ell+1$   
18: end while  
19: for  $\ell \in [L]$  do  
20: if  $\Psi_{t+1,\ell} \neq \Psi_{t,\ell}$  then  
21: Update  $\mathbf{V}_{t+1,\ell} \leftarrow \mathbf{V}_{t,\ell} + w_t^2\mathbf{x}_t\mathbf{x}_t^\top$ ,  $\nu_{t+1,\ell} \leftarrow \nu_{t,\ell} + w_t^2r_t\mathbf{x}_t$   
22: Update  $\widehat{\theta}_{t+1,\ell} \leftarrow \mathbf{V}_{t+1,\ell}^{-1}\nu_{t+1,\ell}$   
23: Compute confidence radius  $\widehat{\beta}_{t+1,\ell}$  following Eq. (5.3)  
24: Compute covariance matrix  $\Sigma_{t+1,\ell} \leftarrow \widehat{\beta}_{t+1,\ell}^2\mathbf{V}_{t+1,\ell}^{-1}$   
25: else  
26:  $\mathbf{V}_{t+1,\ell} \leftarrow \mathbf{V}_{t,\ell}, \nu_{t+1,\ell} \leftarrow \nu_{t,\ell}, \widehat{\theta}_{t+1,\ell} \leftarrow \widehat{\theta}_{t,\ell}, \widehat{\beta}_{t+1,\ell} \leftarrow \widehat{\beta}_{t,\ell}, \Sigma_{t+1,\ell} \leftarrow \Sigma_{t,\ell}$   
27: end if  
28: end for  
29: end for
concentration inequality to guarantee with high probability for all layer  $\ell \in [L]$  that
$$
\left\| \widehat {\boldsymbol {\theta}} _ {t, \ell} - \boldsymbol {\theta} ^ {*} \right\| _ {\mathbf {V} _ {t, \ell}} = \widetilde {\mathcal {O}} \left(\frac {R}{2 ^ {\ell}} + \frac {1}{2 ^ {\ell}} \sqrt {\sum_ {s \in \Psi_ {t , \ell}} w _ {s} ^ {2} \sigma_ {s} ^ {2}}\right). \tag {5.2}
$$
Variance-dependent confidence radius. The variance-dependent error bound (5.2) is sufficient for the formulation of a TS confidence radius, provided that the variance  $\sigma_t^2$  is known. However, under scenarios where the variance is unknown, it is necessary to further estimate them adaptively and on the fly. Specifically, we estimate the summation of past weighted variances, represented as  $\sum_{s\in \Psi_{t,\ell}}w_s^2\sigma_s^2$ , each using an empirical estimator  $(r_t - \langle \mathbf{x}_t,\widehat{\pmb{\theta}}_{t,\ell}\rangle)^2$  as a substitute for  $\sigma_t^2$ . The weighted summation of these estimators  $\zeta_{t,\ell} = \sum_{s\in \Psi_{t,\ell}}w_s^2 (r_s - \langle \mathbf{x}_s,\widehat{\pmb{\theta}}_{s,\ell}\rangle)^2$  effectively acts as a precise estimator of  $\sum_{s\in \Psi_{t,\ell}}w_s^2\sigma_s^2$  (Zhao et al., 2023). Utilizing this estimator, we adjust the posterior distribution according to the following confidence radius
$$
\widehat {\beta} _ {t, \ell} = \frac {1}{2 ^ {\ell - 4}} \sqrt {\left(8 \zeta_ {t , \ell} + 6 R ^ {2} \log \frac {8 t ^ {2} L}{\delta} + 2 ^ {- 2 \ell + 4}\right) \log \frac {8 t ^ {2} L}{\delta}} + \frac {3 R}{2 ^ {\ell - 1}} \log \frac {8 t ^ {2} L}{\delta} + \frac {1}{2 ^ {\ell - 1}}, \tag {5.3}
$$
where the variance estimator takes a slight variant for technical considerations:
$$
\zeta_ {t, \ell} = \left\{ \begin{array}{l l} \sum_ {s \in \Psi_ {t, \ell}} w _ {s} ^ {2} \left(r _ {s} - \langle \mathbf {x} _ {s}, \widehat {\boldsymbol {\theta}} _ {t, \ell} \rangle\right) ^ {2}, & \text {i f} 2 ^ {\ell} \geq 6 4 \sqrt {\log \left(8 t ^ {2} L / \delta\right)} \\ R ^ {2} | \Psi_ {t, \ell} |, & \text {o t h e r w i s e .} \end{array} \right. \tag {5.4}
$$
5.2 Regret analysis
We provide the regret guarantee of Algorithm 3 in the following theorem. We also present a proof sketch of Theorem 5.1 to emphasize the main idea. A complete proof is given in Appendix B.3.
Theorem 5.1. Set parameters  $\alpha = T^{-3/2} / R$  and  $L = \lceil \log_2(1 / \alpha) \rceil$ . For any  $\delta \in (0,1)$ , choose the confidence radii  $\{\widehat{\beta}_{t,\ell}\}_{t \in [T], \ell \in [L]}$  and the variance estimators  $\{\zeta_{t,\ell}\}_{t \in [T], \ell \in [L]}$  according to (5.3) and (5.4) respectively. Then with probability at least  $1 - 2\delta$ , Algorithm 3 has regret guarantee
$$
\mathcal {R} (T) = \mathcal {O} \bigg (d ^ {3 / 2} \bigg (1 + R + \sqrt {\sum_ {t = 1} ^ {T} \sigma_ {t} ^ {2}} \bigg) \log T \bigg).
$$
Remark 5.2. Theorem 5.1 shows that our proposed algorithm, LinNATS, delivers a variance-dependent regret guarantee free of any prior knowledge of the noise variance. The regret upper bound is also stipulated to be of the order  $\widetilde{\mathcal{O}}\Big(d^{3/2}\Big(1 + \sqrt{\sum_{t=1}^{T}\sigma_t^2}\Big)\Big)$ , considering  $R$  as a constant. Notably, in scenarios where the variance is constant, our LinNATS algorithm recovers the regret guarantee of  $\widetilde{\mathcal{O}}(d^{3/2}\sqrt{T})$  for linear TS (Agrawal and Goyal, 2013; Abeille and Lazaric, 2017), and in situations where the reward is deterministic, it attains a regret guarantee of  $\widetilde{\mathcal{O}}(d^{3/2})$ . This highlights the versatility and adaptive performance of our proposed algorithm across various environmental settings.
Next, we present the main ingredients of our analysis for Algorithm 3.
Proof sketch of Theorem 5.1. The choice of the weight parameters in LinNATS allows us to first apply a standard Freedman-type concentration argument (Theorem B.1), and deduce in Lemma B.2 that the estimator  $\widehat{\pmb{\theta}}_{t,\ell}$  is accurate up to a variance-dependent quantity  $\widetilde{\beta}_{t,\ell}$ :
$$
\| \widehat {\boldsymbol {\theta}} _ {t, \ell} - \boldsymbol {\theta} ^ {*} \| _ {\mathbf {V} _ {t, \ell}} \leq \widetilde {\beta} _ {t, \ell} \asymp \frac {1}{2 ^ {\ell}} \sqrt {\sum_ {s \in \Psi_ {t , \ell}} w _ {s} ^ {2} \sigma_ {s} ^ {2} \log \frac {t ^ {2} L}{\delta}} + \frac {R}{2 ^ {\ell}} \log \frac {t ^ {2} L}{\delta} + \frac {1}{2 ^ {\ell - 1}}.
$$
However,  $\widetilde{\beta}_{t,\ell}$  comprises the variance information  $\{\sigma_s^2\}_{s\in \Psi_{t,\ell}}$ , which is unknown to the algorithm and thus cannot be employed in the construction of the TS process. Thanks to Lemma B.4, the sum of past weighted variances  $\sum_{s\in \Psi_{t,\ell}}w_s^2\sigma_s^2$  can be upper-bounded by the empirical estimator  $\zeta_{t,\ell}$  defined in (5.4). Therefore, we can use  $\widehat{\beta}_{t,\ell}$  as defined in (5.3) for the confidence radius, and show that
$$
\left\| \boldsymbol {\theta} _ {t, \ell} ^ {\mathrm {T S}} - \boldsymbol {\theta} ^ {*} \right\| \mathbf {v} _ {t, \ell} \leq (\sqrt {2 d \log (4 t ^ {2} L / \delta)} + 1) \widehat {\beta} _ {t, \ell}.
$$
Then, by leveraging the stratified structure in conjunction with the cascading arm selection procedure across  $\ell \in [L]$ , we are equipped to exert uniform control over  $\| \mathbf{x}\|_{\mathbf{V}_{t,\ell -1}^{-1}}$  for every  $\mathbf{x}\in \mathcal{X}_{t,\ell}$  in each instance of  $t\in \Psi_{T + 1,\ell}$ , which resolves the technical challenge laid out in Section 4.2. Note that for any context vector  $\mathbf{x}\in \mathcal{X}_{t,\ell}$ , we have  $\| \mathbf{x}\|_{\mathbf{V}_{t,\ell -1}^{-1}}\leq 2^{-\ell +1}$ , and thus by Cauchy-Schwarz inequality
$$
\left| \langle \mathbf {x}, \boldsymbol {\theta} _ {t, \ell} ^ {\mathrm {T S}} - \boldsymbol {\theta} ^ {*} \rangle \right| \leq \left\| \mathbf {x} \right\| _ {\mathbf {V} _ {t, \ell - 1} ^ {- 1}} \left\| \boldsymbol {\theta} _ {t, \ell} ^ {\mathrm {T S}} - \boldsymbol {\theta} ^ {*} \right\| _ {\mathbf {V} _ {t, \ell - 1}} \leq 2 ^ {- \ell + 1} \left(\sqrt {2 d \log \left(4 t ^ {2} L / \delta\right)} + 1\right) \widehat {\beta} _ {t, \ell}. \tag {5.5}
$$
Indeed, we ascertain in Lemma B.5 via an induction argument that for all  $t \geq 1$  and  $\ell \in [L]$ , where  $\mathcal{X}_{t,\ell}$  exists,  $\mathbf{x}_t^* \in \mathcal{X}_{t,\ell}$ . Hence (5.5) holds for both  $\mathbf{x}_t^*$  and  $\mathbf{x}_t$  as a consequence of the stratified structure. This allows us to execute the following decomposition of suboptimality for all  $t \in \Psi_{T+1,\ell}$ :
$$
\begin{array}{l} \Delta_ {t} \leq \underbrace {\left\langle \mathbf {x} _ {t} ^ {*} , \boldsymbol {\theta} _ {t , \ell - 1} ^ {\mathrm {T S}} \right\rangle - \left\langle \mathbf {x} _ {t} , \boldsymbol {\theta} _ {t , \ell - 1} ^ {\mathrm {T S}} \right\rangle} _ {\leq 0 \text {b y t h e d e f i n i t i o n o f} \mathbf {x} _ {t}} + \underbrace {| \left\langle \mathbf {x} _ {t} ^ {*} , \boldsymbol {\theta} _ {t , \ell - 1} ^ {\mathrm {T S}} - \boldsymbol {\theta} ^ {*} \right\rangle | + | \left\langle \mathbf {x} _ {t} , \boldsymbol {\theta} _ {t , \ell - 1} ^ {\mathrm {T S}} - \boldsymbol {\theta} ^ {*} \right\rangle |} _ {\text {c o n t r o l l e d v i a (5 . 5)}} \\ \leq 8 \cdot 2 ^ {- \ell} (\sqrt {2 d \log (4 t ^ {2} L / \delta)} + 1) \widehat {\beta} _ {t, \ell - 1}. \\ \end{array}
$$
The main result is then obtained by collating the preceding inequalities for all  $\ell \in [L]$  followed by an elliptical potential inequality, taking into account that  $L = \mathcal{O}(\log T)$ .
In this paper, we aimed to address the task of noise-adaptive learning on linear contextual bandits with indeterminate heteroscedastic variance. As part of our endeavor, we put forth a straightforward algorithm, LinVDTS, which assures a variance-dependent guarantee on the expected regret. Our analytical exploration reveals that a simplistic implementation of Thompson Sampling culminates in a sub-optimal regret bound. To mitigate this issue, we put forth an innovative Thompson Sampling algorithm, LinNATS. With the incorporation of a stratification scheme, the algorithm successfully navigates the technical challenges of uncertainty control, securing a variance-dependent regret under unknown variance. Consequently, it effectively bridges the performance chasm between the worst-case scenarios involving constant variance and those concerning deterministic rewards.
Looking forward, it would be intriguing to conceive a noise-adaptive variant of the recently devised Feel-Good TS as introduced by Zhang (2022), which enhances the dependency on the dimension  $d$ . Furthermore, extending the noise-adaptive methodology to TS algorithms designed for more general settings, e.g., generalized linear bandits and Reinforcement Learning with linear function approximation, constitutes an appealing direction for future research.
ABBASI-YADKORI, Y., PAL, D. and SZEPESVÁRÍ, C. (2011). Improved algorithms for linear stochastic bandits. Advances in neural information processing systems 24.
ABE, N., BIERMANN, A. W. and LONG, P. M. (2003). Reinforcement learning with immediate rewards and linear hypotheses. Algorithmica 37 263-293.
ABEILLE, M. and LAZARIC, A. (2017). Linear thompson sampling revisited. In Artificial Intelligence and Statistics. PMLR.
ABRAMOWITZ, M., STEGUN, I. A. ET AL. (1964). Handbook of mathematical functions, vol. 55. Dover New York.
AGRAWAL, S. and GOYAL, N. (2013). Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning. PMLR.
AUER, P., CESA-BIANCHI, N. and FISCHER, P. (2002a). Finite-time analysis of the multiarmed bandit problem. Machine learning 47 235-256.
AUER, P., CESA-BIANCHI, N., FREUND, Y. and SCHAPIRE, R. E. (2002b). The nonstochastic multiarmed bandit problem. SIAM journal on computing 32 48-77.
CHAPELLE, O. and LI, L. (2011). An empirical evaluation of thompson sampling. Advances in neural information processing systems 24.
CHEN, L., MIN, Y., BELKIN, M. and KARBASI, A. (2021). Multiple descent: Design your own generalization curve. Advances in Neural Information Processing Systems 34 8898-8912.
CHEN, L., MIN, Y., ZHANG, M. and KARBASI, A. (2020). More data can expand the generalization gap between adversarially robust and standard models. In International Conference on Machine Learning. PMLR.
CHENG, R. C. and KLEIJEN, J. P. (1999). Improved design of queueing simulation experiments with highly heteroscedastic responses. Operations research 47 762-777.
CHOWDHURY, S. R. and GOPALAN, A. (2017). On kernelized multi-armed bandits. In International Conference on Machine Learning. PMLR.
CHU, W., LI, L., REYZIN, L. and SCHAPIRE, R. (2011). Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings.
DAI, Y., WANG, R. and DU, S. S. (2022). Variance-aware sparse linear bandits. arXiv preprint arXiv:2205.13450.
DESHPANDE, Y. and MONTANARI, A. (2012). Linear bandits in high dimension and recommendation systems. In 2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton). IEEE.
FEI, Y. and Xu, R. (2022a). Cascaded gaps: Towards gap-dependent regret for risk-sensitive reinforcement learning. arXiv preprint arXiv:2203.03110.
FEI, Y. and Xu, R. (2022b). Cascaded gaps: Towards logarithmic regret for risk-sensitive reinforcement learning. In International Conference on Machine Learning. PMLR.
GOPALAN, A., MANNOR, S. and MANSOUR, Y. (2014). Thompson sampling for complex online problems. In International conference on machine learning. PMLR.
HAMIDI, N. and BAYATI, M. (2020). On worst-case regret of linear thompson sampling. arXiv preprint arXiv:2006.06790.
HE, J., WANG, T., MIN, Y. and GU, Q. (2022). A simple and provably efficient algorithm for asynchronous federated contextual linear bandits. Advances in neural information processing systems 35 4762-4775.