Title: Noise-Adaptive Thompson Sampling for Linear Contextual Bandits
Venue: NeurIPS 2023
Accepted: yes
Keywords: Linear Contextual Bandit

Thompson Sampling

Noise-Adaptive
Abstract: Linear contextual bandits represent a fundamental class of models with numerous real-world applications, and it is critical to develop algorithms that can effectively manage noise with unknown variance, ensuring provable guarantees for both worst-case constant-variance noise and deterministic reward scenarios. In this paper, we study linear contextual bandits with heteroscedastic noise and propose the first noise-adaptive Thompson sampling-style algorithm that achieves a variance-dependent regret upper bound of $\widetilde O\Big(d^{3/2} + d^{3/2} \sqrt{\sum_{t=1}^T \sigma_t^2}\Big)$, where $d$ is the dimension of the context vectors and $\sigma_t^2$ is the variance of the reward in round $t$. This recovers the existing $\widetilde O(d^{3/2}\sqrt{T})$ regret guarantee in the constant-variance regime and further improves to $\widetilde O(d^{3/2})$ in the deterministic regime, thus achieving a smooth interpolation in between. Our approach utilizes a stratified sampling procedure to overcome the too-conservative optimism in the linear Thompson sampling algorithm for linear contextual bandits.

Noise-Adaptive Thompson Sampling for Linear Contextual Bandits

Abstract

Linear contextual bandits represent a fundamental class of models with numerous real-world applications, and it is critical to developing algorithms that can effectively manage noise with unknown variance, ensuring provable guarantees for both worst-case constant-variance noise and deterministic reward scenarios. In this paper, we study linear contextual bandits with heteroscedastic noise and propose the first noise-adaptive Thompson sampling-style algorithm that achieves a variance-dependent regret upper bound of ˜ O ( d 3 / 2 + d 3 / 2 √ ∑ T t =1 σ 2 t ) , where d is the dimension of the context vectors and σ 2 t is the variance of the reward in round t . This recovers the existing ˜ O ( d 3 / 2 √ T ) regret guarantee in the constant-variance regime and further improves to ˜ O ( d 3 / 2 ) in the deterministic regime, thus achieving a smooth interpolation in between. Our approach utilizes a stratified sampling procedure to overcome the too-conservative optimism in the linear Thompson sampling algorithm for linear contextual bandits.

1 Introduction

Linear contextual bandits represent a natural extension of multi-armed bandit problems (Auer et al., 2002a; Robbins, 1952; Lai et al., 1985), where the reward of each arm is assumed to be a linear function of the contextual information associated with the arm. Such problems manifest in numerous real-world applications, encompassing online advertising (Wu et al., 2016), recommendation systems (Deshpande and Montanari, 2012), and personalized medicine (Varatharajah and Berry, 2022; Lu et al., 2021). A multitude of algorithms have been proposed, tailored to diverse settings within the domain of linear contextual bandits (Auer et al., 2002b; Abe et al., 2003). Notably, two main streams of approaches to address the exploration-exploitation dilemma in linear contextual bandits have emerged: Upper Confidence Bound (UCB) (Chu et al., 2011; Abbasi-Yadkori et al., 2011) and Thompson sampling (TS) (Agrawal and Goyal, 2013; Abeille and Lazaric, 2017). In practical scenarios, the varying and non-transparent noise variance inherent in each reward is a common phenomenon (Towse et al., 2015; Omari et al., 2018; Somu et al., 2018; Cheng and Kleijnen, 1999). Such noise, which may exhibit heteroscedasticity and correlation with the context, can significantly impact algorithm performance, particularly when the variance of the noise is unknown a priori (Kirschner

and Krause, 2018; Zhao et al., 2022). Thus, it is imperative to develop adaptive algorithms capable of handling noise with unknown heteroscedastic variance, ensuring provable guarantees when operating with both constant-variance noise and deterministic reward scenarios. Recently, a few variance-aware UCB algorithms have been proposed for linear bandits. In particular, Zhou et al. (2021) examined the scenario where the variance is known and observed after each arm pull, while Zhang et al. (2021); Kim et al. (2022b); Zhao et al. (2023) explored the unknown variance case. Nevertheless, there is a scarcity of results concerning TS algorithms in this context.

Thompson Sampling (Thompson, 1933), a notable Bayesian approach, provides a computationally efficient means of addressing the exploration-exploitation trade-off (Rusmevichientong and Tsitsiklis, 2010; Chapelle and Li, 2011). This method applies posterior sampling to generate a parameter estimator for arm selection, thereby naturally balancing exploration and exploitation by selecting arms with high expected rewards and those with substantial uncertainty (Russo et al., 2018; Riquelme et al., 2018). In the existing literature, the regret upper bound for the linear TS algorithm for linear contextual bandits is of order ˜ O ( d 3 / 2 √ T ) (Agrawal and Goyal, 2013; Abeille and Lazaric, 2017), and the existing algorithm cannot adapt to higher-order structure of the noise.

As an initial endeavor to incorporate variance information into TS-style algorithmic design for linear bandits, we adopt a weighted ridge regression approach to construct a TS algorithm, LinVDTS , detailed in Algorithm 2. This algorithm achieves a variance-dependent performance guarantee on the expected regret, as demonstrated in Theorem 4.1. Nonetheless, a technical barrier intrinsic to TS-style algorithms, yet absent in UCB, emerges during this naive integration of weighted regression with TS. Specifically, the issue stems from inadequate control over the variance-adjusted norm of unselected context vectors at each step, necessitating an anti-concentration argument that only allows for a variance-dependent upper bound on the expected regret and precludes the acquisition of a high-probability guarantee that is simultaneously variance-dependent. More discussion on this will be given in Section 4.2.

In a bid to navigate this intricate issue and exert explicit uncertainty control, we further propose a novel noise-adaptive TS variant in Algorithm 3. Our algorithmic design features (1) an uncertainty stratification scheme applied to contextual vectors, providing explicit regulation of the uncertainty quantification, (2) a cascading construction process for the feasible set, which, in tandem with the stratification scheme, operates to filter out unlikely arms, thereby eliminating the requirement for the anti-concentration argument in the analysis, and (3) the use of noise-adaptive confidence radii to direct the sampling procedure, thereby ensuring a more constricted variance-dependent regret. The theoretical analysis yields a high-probability regret bound that interpolates smoothly between the constant-variance and deterministic reward regimes.

Main contributions. The main contributions of our paper are summarized in three key aspects.

We introduce a simple and efficient TS-style algorithm, named LinVDTS (Algorithm 2), that can utilize variance information for linear contextual bandits. We prove that the expected regret of LinVDTS is bounded by ˜ O ( d 3 / 2 ( 1 + √ ∑ T t =1 σ 2 t )) , where d is the ambient dimension and σ 2 t is the variance of the noise in round t . We also prove that, with high probability , the regret of LinVDTS is bounded by ˜ O ( d 3 / 2 ( 1 + √ ∑ T t =1 σ 2 t ) + √ T ) . In the analysis of LinVDTS , we identify the inherent difficulty of controlling the uncertainty induced by posterior sampling, which precludes a high-probability guarantee that is aware of the noise variance.

We devise a stratified sampling procedure that fulfills more efficient exploration for linear contextual bandits by exerting explicit uncertainty control over the context vectors. Based on this framework, we propose LinNATS (Algorithm 3), a noise-adaptive variant of linear TS. To the best of our knowledge, this is the first noise-adaptive TS algorithm for linear contextual bandits with heteroscedastic noise.

Weprove that LinNATS enjoys a high-probability regret bound of order ˜ O ( d 3 / 2 ( 1+ √ ∑ T t =1 σ 2 t )) , under standard assumptions for linear contextual bandits with heteroscedastic noise. This improves over the existing ˜ O ( d 3 / 2 √ T ) regret bound for linear Thompson sampling (Agrawal and Goyal, 2013; Abeille and Lazaric, 2017), especially when the noise variance diminishes. Our analysis bypasses the standard anti-concentration argument thanks to the stratification framework.

Notation. For any n ∈ N + , we denote [ n ] := { 1 , 2 , . . . , n } , and x 1: n is a shorthand for the set { x 1 , . . . , x n } . We write scalars in normal font while representing vectors and matrices in bold font. For any vector x and positive semi-definite matrix Σ , we define ∥ x ∥ Σ = ∥ Σ 1 / 2 x ∥ 2 . For two functions f and g defined on N + , we use f ( n ) ≲ g ( n ) to indicate that there exists a universal constant C > 0 such that f ( n ) ≤ C · g ( n ) for all n ∈ X ; f ( n ) ≳ g ( n ) is defined analogously. We use f ( n ) ≍ g ( n ) to imply that both f ( n ) ≲ g ( n ) and f ( n ) ≳ g ( n ) hold true. We use O ( · ) to hide constant factors and ˜ O ( · ) to further hide poly-logarithmic terms.

2 Related work

Heteroscedastic linear bandits. The challenge of heteroscedastic noise in linear bandits has been studied through a multitude of perspectives: In particular, Kirschner and Krause (2018) studied linear bandits with heteroscedastic noise via the approach of information-directed sampling. Dai et al. (2022) examined heteroscedastic sparse linear bandits and demonstrated a versatile framework capable of transforming any heteroscedastic linear bandit algorithm into an algorithm tailored for heteroscedastic sparse linear bandits. Building on the UCB approach, Zhou et al. (2021) developed an adaptive algorithm that handles known-variance noise, providing performance guarantees of ˜ O ( √ dT + d √ ∑ T t =1 σ 2 t ) . Zhang et al. (2021) introduced a variance-aware confidence set using elimination with peeling and proved a regret bound of ˜ O ( poly( d ) √ 1 + ∑ T t =1 σ 2 t ) . Kim et al. (2022b) further refined this upper bound to ˜ O ( d 2 + d 3 / 2 √ ∑ T t =1 σ 2 t ) for linear bandits. Zhao et al. (2022) proposed a multi-layered design for UCB algorithm that achieves a ˜ O ( ( R +1) √ dT + d √ ∑ T t =1 σ 2 t ) regret. More recently, Zhao et al. (2023) proposed a UCB-style algorithm with a unified regret upper bound of ˜ O ( d + d √ ∑ T t =1 σ 2 t ) , where the guarantee is facilitated by the introduction of a novel Freedman-type concentration inequality for self-normalized martingales.

Thompson sampling. TS as a method to address the exploration-exploitation trade-off has gained significant attention in recent years due to its simplicity, adaptability, and robust empirical performance (Chapelle and Li, 2011; Gopalan et al., 2014; Kandasamy et al., 2018). Its effectiveness has been demonstrated across various scenarios (Russo et al., 2018), and in particular, a multitude of theoretical results have been solidified within multi-armed bandit settings (May et al., 2012; Kaufmann et al., 2012; Korda et al., 2013; Russo and Van Roy, 2016). For linear contextual bandits, Agrawal and Goyal (2013) provided the first proof for the ˜ O ( d 3 / 2 √ T ) regret of linear TS, and later Abeille and Lazaric (2017) delivered an alternate proof and further extended the analysis to more general linear problems. Notably, Hamidi and Bayati (2020) showed that the ˜ O ( d 3 / 2 √ T ) rate cannot be improved in worse-case scenarios. While under additional assumptions such as regularity of the contexts, better rates can be achieved (Hamidi and Bayati, 2020; Kim et al., 2021, 2022a).

Moreover, Zhang (2022) introduced a modified version called Feel-Good TS, which is more aggressive in exploring new actions and achieves an ˜ O ( d √ T ) regret. Recently, Luo and Bayati (2023) also proposed a geometry-aware approach, enabling the establishment of a minimax optimal regret of order ˜ O ( d √ T ) for the TS algorithm. Xu et al. (2022) proposed a Langevin Monte Carlo TS algorithm that samples from the posterior distribution beyond Laplacian approximation. Investigations have also been undertaken into TS for kernelized bandits (Chowdhury and Gopalan, 2017) as well as integration of TS with neural networks (Wang and Zhou, 2020; Zhang et al., 2020). Recently, Saha and Kveton (2023) proposed a variance-aware TS algorithm on the topic of Bayesian bandits with a variance-dependent upper bound on its Bayesian regret. As far as our awareness extends, no existing results address noise-adaptive TS algorithms in the context of linear bandits.

3 Preliminaries

In this section, we introduce the fundamental framework for contextual linear bandits. We also provide a concise overview of the linear TS algorithm, which functions as our benchmark methodology.

Algorithm 1 Linear Thompson Sampling

1: for t = 1 , . . . , T do 2: Sample θ TS t from N ( ̂ θ t , β 2 V -1 t ) 3: Select arm x t ← arg max x ∈ X t ⟨ x , θ TS t ⟩ and observe reward r t

4: end for

3.1 Linear contextual bandits

We investigate a linear contextual bandit problem with heteroscedastic noise. Let T be the total number of bandit selection rounds and let d be the ambient dimension. In each round t ∈ [ T ] , the environment generates an arbitrary set of context vectors X t ⊆ R d , potentially even in an adversarial manner, where each x ∈ X t denotes the context vector of a feasible action. Upon observing the decision set X t , the agent selects a context vector x t ∈ X t , and then receives a reward r t from the environment. In particular, here the reward is a linear function of the selected context further corrupted by noise, i.e. ,

where θ ∗ ∈ R d represents the true model parameter and ε t is the stochastic noise. In this paper, we impose standard assumptions on the linear contextual bandit model, which are common in literature.

Assumption 3.1. The ground truth θ ∗ satisfies ∥ θ ∗ ∥ 2 ≤ 1 . For all t ∈ [ T ] , the decision set X t is contained in the unit ball, i.e. , ∥ x ∥ 2 ≤ 1 for all x ∈ X t . There exists a constant R > 0 such that | ε t | ≤ R for all t ∈ [ T ] . For every t ∈ [ T ] , E [ ε t | x 1: t , ε 1: t -1 ] = 0 and E [ ε 2 t | x 1: t , ε 1: t -1 ] = σ 2 t .

In each round t , an algorithm selects an arm x t ∈ X t , and we denote the optimal arm as x ∗ t , i.e. , x ∗ t = arg max x ∈ X t x ⊺ θ ∗ . As a result, the suboptimality of the selected arm at time t can be expressed as ∆ t = x ∗ t ⊺ θ ∗ -x ⊺ t θ ∗ . The objective of the agent is to minimize the cumulative regret incurred over the time horizon T , which is defined as

3.2 Thompson sampling for linear contextual bandits

A standard TS algorithm for linear contextual bandits ( i.e. , Algorithm 1) was first proposed in Agrawal and Goyal (2013), employing Gaussian priors and likelihood functions in the design. At each round t ∈ [ T ] , a sample θ TS t is drawn from the Gaussian posterior distribution, centered at the estimate ̂ θ t = V -1 t ∑ t -1 s =1 r s x s with covariance matrix β 2 V -1 t , where V t = I d + ∑ t -1 s =1 x s x ⊺ s and the confidence radius β = 3 R √ d log( T/δ ) . The arm x t is then selected against the sample θ TS t .

The main idea here is to achieve exploitation by updating the posterior using collected information, and the exploration among the arms is performed naturally in posterior sampling. Agrawal and Goyal (2013) demonstrated that the algorithm presented in Algorithm 1 achieves a regret bound of order ˜ O ( d 3 / 2 √ T ) . This bound is worse than the minimax lower bound by an unavoidable factor of √ d (Hamidi and Bayati, 2020). Despite this discrepancy, the regret bound effectively showcases the algorithm's ability to balance exploration and exploitation in the linear contextual bandit setting.

4 Warm up: a simple and variance-dependent Thompson sampling algorithm

In this section, we introduce a preliminary algorithm, LinVDTS , delineated in Algorithm 2, which combines TS with a weighted ridge regression estimator in a straightforward manner. For the sake of clear exposition, we focus on a less complex linear bandit problem in which the agent is privy to both the reward r t and its variance σ 2 t subsequent to the selection of an arm.

4.1 Algorithmic design

In order to integrate variance information into the estimation process, the use of weighted regression has emerged as a prevalent approach in the field of variance-aware online learning (Zhou et al., 2021;

Algorithm 2 Lin ear V arianceD ependent T hompson S ampling ( LinVDTS )

Require: Total number of iterations T , number of total arms

1: Initialize V 1 ← λ I d , ν 1 ← 0 , ̂ θ 1 ← 0 , ̂ β 1 ← √ λ , and Σ 1 ← ̂ β 2 1 V -1 1

2: for t = 1 , . . . , T do

3: Observe context vectors X t

4: Draw θ TS t ∼ N ( ̂ θ t , Σ t )

6: Observe reward r t and variance σ 2 t

5: x t ← arg max x ∈ X t ⟨ x , θ TS t ⟩

7: Update V t +1 ← V t + x t x ⊺ t /σ 2 t

8: Update ν t +1 ← ν t + r t x t /σ 2 t

9: Update ̂ θ t +1 ← V -1 t +1 ν t +1

10: Compute covariance matrix Σ t +1 ← ̂ β 2 t +1 V -1 t +1

11: end for

Min et al., 2021b, 2022a; Zhou and Gu, 2022; Yin et al., 2022; Zhao et al., 2023). This methodology is intimately associated with the concept of the Best Linear Unbiased Estimator (BLUE), a wellestablished statistical technique for achieving optimal linear estimation (Henderson, 1975).

Let us first explain the details of the proposed algorithm LinVDTS . In every round t , the algorithm computes an estimate ̂ θ t of the ground truth θ ∗ by solving a weighted ridge regression problem:

Here σ t represents a weight parameter and λ denotes a regularization constant. For each approximation ̂ θ t , we stochastically draw θ TS t from a Gaussian distribution with mean ̂ θ t and covariance Σ t = ̂ β t V -1 t , where V t = λ I d + ∑ t -1 s =1 x s x ⊺ s /σ 2 s is computed from previous context and reward pairs, and the confidence radius, ̂ β t , is chosen as

Refer to (A.2) for a detailed delineation of ̂ β t . For each context vector x ∈ X t , we compute the estimated reward as ⟨ x , θ TS t ⟩ , and the algorithm outlined in Algorithm 2 then selects the arm x t that optimizes this estimated reward. Following the approach of Zhou and Gu (2022), we construct the weight parameter σ t = max { σ t , α, γ ∥ x t ∥ 1 / 2 V -1 t } , which is designed as the maximum among the variance, a constant α , and the uncertainty associated with the selected arm x t . It ensures the formulation of a tight confidence radius ̂ β t , guided by a Bernstein-type concentration inequality, cf. Theorem A.1.

4.2 Regret guarantee and technical challenges

The theorem below establishes a variance-dependent upper bound on the expected regret of LinVDTS . See Appendix A.3 for detailed proof.

Theorem 4.1. Set parameters α = 1 / √ T , γ = R 1 2 /d 1 4 , λ = d . For any δ ∈ (0 , 1) , the expected regret of Algorithm 2 is upper bounded as follows:

Further, it holds with probability 1 -δ that

n

Remark 4.2 . Choosing δ = 1 /T , then we see that the expected regret of Algorithm 2 is bounded by ˜ O ( d 3 / 2 ( 1 + √ ∑ T t =1 σ 2 t )) . However, the high-probability regret bound of Algorithm 2 is of order ˜ O ( d 3 / 2 ( 1 + √ ∑ T t =1 σ 2 t ) + √ T ) , where the additional √ T term fails to be variance-dependent. Nonetheless, this regret bound already improves over the existing ˜ O ( d 3 / 2 √ T ) bound for linear Thompson sampling in the literature (Agrawal and Goyal, 2013; Abeille and Lazaric, 2017).

The limitation on the high-probability bound arises from a set of technical challenges that are implicit and inherent in the posterior sampling used in Algorithm 2. To illustrate this, we now provide an outline of the proof for Theorem 4.1, and then discuss the technical intricacies therein.

Proof sketch of Theorem 4.1. Let us denote ∆ t ( x ) = ⟨ x ∗ t -x , θ ∗ ⟩ as the suboptimality of x ∈ X t . We introduce the saturated set S ( t ) for round t as the ensemble of arms whose confidence radius is dwarfed by its suboptimality, i.e. , S ( t ) = { x ∈ X t : ∆ t ( x ) > (2 √ d log t +1) ̂ β t ∥ x ∥ V -1 t } (Agrawal and Goyal, 2013). Then the suboptimality ∆ t of the chosen context x t can be decomposed as follows:

where the term x † t = arg min x / ∈ S ( t ) ∥ x ∥ V -1 t refers to the unsaturated arm that possesses the smallest V -1 t norm. Leveraging the concentration results delineated in Lemmata A.2 and A.3 pertaining to ̂ θ t and θ TS t , we have

For upper bound on the expected regret, we use an anti-concentration argument (Lemma A.6) to relate ∥ x † t ∥ V -1 t back to ∥ x t ∥ V -1 t . This yields

where { H t } t ≥ 1 represents a filtration of the information available up to the observation of the set of context vectors at each round. Then collecting the above inequality for all t ∈ [ T ] , together with an elliptical potential lemma, we get the desired upper bound for E [ R ( T )] .

Next, for the high-probability regret bound, we decompose

where the first term corresponds to the expected regret bound. We control the second term using martingale concentration, which results in an additional √ T term in the final regret bound.

Why cannot simultaneously achieve variance awareness and high-probability bound?

First, the optimism induced by the posterior sampling is too conservative. 1 A primary challenge in the analysis is to effectively control each suboptimality term ∆ t , cf. (4.1). It is important to note that one key algorithmic design in UCB-type algorithms is the exact optimism from the UCB bonus β ∥ x t ∥ V -1 t , which ensures ⟨ x ∗ t , ̂ θ t ⟩ + β ∥ x ∗ t ∥ V -1 t ≤ ⟨ x t , ̂ θ t ⟩ + β ∥ x t ∥ V -1 t . However, for TS-style algorithms, the absence of this exact optimism hinders our control over the term ∥ x ∗ t ∥ V -1 t throughout all rounds t ∈ [ T ] , i.e. , a standard decomposition of ∆ t = ⟨ x ∗ t -x t , θ ∗ ⟩ yields ∆ t ≤ (2 √ d log t + 1) ̂ β t (2 ∥ x ∗ t ∥ V -1 t + ∥ x t ∥ V -1 t ) in this case. As a result, standard analyses turn to methodologies that employ an anti-concentration inequality on the sampling distribution (Agrawal and Goyal, 2013). This leads to the inequality ∥ x † t ∥ V -1 t ≤ ∥ x t ∥ V -1 t with a constant probability, thereby providing an upper bound on the expected regret.

Second, in the conversion of the expected regret bound to a high-probability bound, we have no help from the noise variance information. One may wonder if it is possible to incorporate the

variance of ∆ t -E [∆ t | H t ] to get a Bernstein-type concentration result. However, the variance of ∆ t -E [∆ t | H t ] does not necessarily conform to the noise variance, especially when the decision set X t is revealed adversarially. To see this, consider the following situation: Given ̂ θ t , the environment reveals a decision set X t = { x ⊥ , -x ⊥ } where x ⊥ is orthogonal to ̂ θ t . Then a small perturbation in θ TS t along the direction of x ⊥ can cause the change of arm selection. This implies that a small uncertainty of the posterior distribution along the direction of x ⊥ will be amplified to a constant variation of the selected arm x t . Therefore, in general, we do not have any delicate control of the variance of ∆ t -E [∆ t | H t ] . Consequently, we can only apply Azuma-Hoeffding inequality to get a non-noise-adaptive term of ˜ O ( √ T ) . We discuss in the next section how to conquer this via more advanced algorithmic design.

5 General case: a noise-adaptive Thompson sampling algorithm

In this section, we propose another TS algorithm LinNATS for linear contextual bandits, one that is provably adaptive to unknown heteroscedastic noise. For greater generality, we adopt the identical problem setting delineated in Section 3, and we refrain from assuming access to the variance information { σ 2 t } T t =1 associated with the rewards { r t } T t =1 .

5.1 Algorithm

The proposed algorithm LinNATS is displayed in Algorithm 3. Below we go through the details of LinNATS and explain the algorithmic design along the way.

Stratification over contextual uncertainty. For more efficient uncertainty control, we adopt a stratification strategy akin to that previously used for UCB-type algorithms (Chu et al., 2011; Li et al., 2023; Zhao et al., 2023). Our algorithm segments the context vectors at each round t into L distinct layers, enabling the precise control of both ∥ x t ∥ V -1 t and ∥ x ∗ t ∥ V -1 t .

Specifically, the ℓ -th layer establishes a threshold of 2 -ℓ and retains a separate estimate ̂ θ t,ℓ and sampled variable θ TS t,ℓ for each ℓ ∈ [ L ] and t ∈ [ T ] (Line 3). Commencing from ℓ = 1 , a sequence of decision sets X t = X t, 1 ⊇ X t, 2 ⊇ . . . of diminishing sizes is derived by excluding the arms x ∈ X t,ℓ that are less likely to maximize ⟨ x , θ ∗ ⟩ at each layer ℓ . More specifically, on Line 11 we let

which ensures that the optimal arm x ∗ t falls into all decision sets with high probability. The context vector x t = arg max x ∈ X t,ℓ ⟨ x , θ TS t,ℓ ⟩ is chosen if the uncertainty term ∥ x ∥ V -1 t,ℓ ≤ α for all x ∈ X t,ℓ (Line 8). Otherwise, x t is selected only when the uncertainty surpasses the threshold, i.e. , ∥ x t ∥ V -1 t,ℓ ≥ 2 -ℓ for some ℓ , and the elimination process in (5.1) terminates (Line 13).

The round index t is incorporated into a growing index set Ψ t +1 ,ℓ (Line 9 & 15), and the observation pair ( x t , r t ) participates the later estimation of ̂ θ s,ℓ for all s > t only if the chosen arm exhibits large uncertainty within the current layer, i.e. , ∥ x t ∥ V -1 t,ℓ ≥ 2 -ℓ (Line 21). This process guarantees that every context vector x ∈ X t,ℓ , cascading through the first ( ℓ -1) -th layer, satisfies ∥ x ∥ V -1 t,ℓ -1 ≤ 2 -ℓ +1 .

Parameter estimation with weighted ridge regression. To estimate θ ∗ , we again utilize weighted ridge regression, but this time within each uncertainty level. For each layer ℓ ∈ [ L ] and round t ∈ [ T ] , the associated estimator ̂ θ t,ℓ is given by

where the weight parameter w t > 0 is selected to fulfill the condition ∥ w t x t ∥ V -1 t,ℓ = 2 -ℓ (Line 14). Owing to the specific formulation of the weighting parameter, it follows that sup s ∈ Ψ t,ℓ ∥ w s x s ∥ V -1 s,ℓ = 2 -ℓ holds universally for all t ∈ [ T ] and ℓ ∈ [ L ] . This allows the application of a Freedman-type

Algorithm 3 Lin ear N oiseA daptive T hompson S ampling ( LinNATS )

̸

concentration inequality to guarantee with high probability for all layer ℓ ∈ [ L ] that

Variance-dependent confidence radius. The variance-dependent error bound (5.2) is sufficient for the formulation of a TS confidence radius, provided that the variance σ 2 t is known. However, under scenarios where the variance is unknown, it is necessary to further estimate them adaptively and on the fly. Specifically, we estimate the summation of past weighted variances, represented as ∑ s ∈ Ψ t,ℓ w 2 s σ 2 s , each using an empirical estimator ( r t -⟨ x t , ̂ θ t,ℓ ⟩ ) 2 as a substitute for σ 2 t . The weighted summation of these estimators ζ t,ℓ = ∑ s ∈ Ψ t,ℓ w 2 s ( r s - ⟨ x s , ̂ θ s,ℓ ⟩ ) 2 effectively acts as a precise estimator of ∑ s ∈ Ψ t,ℓ w 2 s σ 2 s (Zhao et al., 2023). Utilizing this estimator, we adjust the posterior distribution according to the following confidence radius

where the variance estimator takes a slight variant for technical considerations:

5.2 Regret analysis

We provide the regret guarantee of Algorithm 3 in the following theorem. We also present a proof sketch of Theorem 5.1 to emphasize the main idea. A complete proof is given in Appendix B.3.

Theorem 5.1. Set parameters α = T -3 / 2 /R and L = ⌈ log 2 (1 /α ) ⌉ . For any δ ∈ (0 , 1) , choose the confidence radii { ̂ β t,ℓ } t ∈ [ T ] ,ℓ ∈ [ L ] and the variance estimators { ζ t,ℓ } t ∈ [ T ] ,ℓ ∈ [ L ] according to (5.3) and (5.4) respectively. Then with probability at least 1 -2 δ , Algorithm 3 has regret guarantee

Remark 5.2 . Theorem 5.1 shows that our proposed algorithm, LinNATS , delivers a variance-dependent regret guarantee free of any prior knowledge of the noise variance. The regret upper bound is also stipulated to be of the order ˜ O ( d 3 / 2 ( 1 + √ ∑ T t =1 σ 2 t )) , considering R as a constant. Notably, in scenarios where the variance is constant, our LinNATS algorithm recovers the regret guarantee of ˜ O ( d 3 / 2 √ T ) for linear TS (Agrawal and Goyal, 2013; Abeille and Lazaric, 2017), and in situations where the reward is deterministic, it attains a regret guarantee of ˜ O ( d 3 / 2 ) . This highlights the versatility and adaptive performance of our proposed algorithm across various environmental settings.

Next, we present the main ingredients of our analysis for Algorithm 3.

Proof sketch of Theorem 5.1. The choice of the weight parameters in LinNATS allows us to first apply a standard Freedman-type concentration argument (Theorem B.1), and deduce in Lemma B.2 that the estimator ̂ θ t,ℓ is accurate up to a variance-dependent quantity ˜ β t,ℓ :

However, ˜ β t,ℓ comprises the variance information { σ 2 s } s ∈ Ψ t,ℓ , which is unknown to the algorithm and thus cannot be employed in the construction of the TS process. Thanks to Lemma B.4, the sum of past weighted variances ∑ s ∈ Ψ t,ℓ w 2 s σ 2 s can be upper-bounded by the empirical estimator ζ t,ℓ defined in (5.4). Therefore, we can use ̂ β t,ℓ as defined in (5.3) for the confidence radius, and show that

Then, by leveraging the stratified structure in conjunction with the cascading arm selection procedure across ℓ ∈ [ L ] , we are equipped to exert uniform control over ∥ x ∥ V -1 t,ℓ -1 for every x ∈ X t,ℓ in each instance of t ∈ Ψ T +1 ,ℓ , which resolves the technical challenge laid out in Section 4.2. Note that for any context vector x ∈ X t,ℓ , we have ∥ x ∥ V -1 t,ℓ -1 ≤ 2 -ℓ +1 , and thus by Cauchy-Schwarz inequality

Indeed, we ascertain in Lemma B.5 via an induction argument that for all t ≥ 1 and ℓ ∈ [ L ] , where X t,ℓ exists, x ∗ t ∈ X t,ℓ . Hence (5.5) holds for both x ∗ t and x t as a consequence of the stratified structure. This allows us to execute the following decomposition of suboptimality for all t ∈ Ψ T +1 ,ℓ :

The main result is then obtained by collating the preceding inequalities for all ℓ ∈ [ L ] followed by an elliptical potential inequality, taking into account that L = O (log T ) .

6 Conclusion and future work

In this paper, we aimed to address the task of noise-adaptive learning on linear contextual bandits with indeterminate heteroscedastic variance. As part of our endeavor, we put forth a straightforward algorithm, LinVDTS , which assures a variance-dependent guarantee on the expected regret. Our analytical exploration reveals that a simplistic implementation of Thompson Sampling culminates in a sub-optimal regret bound. To mitigate this issue, we put forth an innovative Thompson Sampling algorithm, LinNATS . With the incorporation of a stratification scheme, the algorithm successfully navigates the technical challenges of uncertainty control, securing a variance-dependent regret under unknown variance. Consequently, it effectively bridges the performance chasm between the worst-case scenarios involving constant variance and those concerning deterministic rewards.

Looking forward, it would be intriguing to conceive a noise-adaptive variant of the recently devised Feel-Good TS as introduced by Zhang (2022), which enhances the dependency on the dimension d . Furthermore, extending the noise-adaptive methodology to TS algorithms designed for more general settings, e.g. , generalized linear bandits and Reinforcement Learning with linear function approximation, constitutes an appealing direction for future research.

References

ABBASI-YADKORI, Y., P ´ AL, D. and SZEPESV ´ ARI, C. (2011). Improved algorithms for linear stochastic bandits. Advances in neural information processing systems 24 .

ABE, N., BIERMANN, A. W. and LONG, P. M. (2003). Reinforcement learning with immediate rewards and linear hypotheses. Algorithmica 37 263-293.

ABEILLE, M. and LAZARIC, A. (2017). Linear thompson sampling revisited. In Artificial Intelligence and Statistics . PMLR.

ABRAMOWITZ, M., STEGUN, I. A. ET AL. (1964). Handbook of mathematical functions , vol. 55. Dover New York.

AGRAWAL, S. and GOYAL, N. (2013). Thompson sampling for contextual bandits with linear payoffs. In International conference on machine learning . PMLR.

AUER, P., CESA-BIANCHI, N. and FISCHER, P. (2002a). Finite-time analysis of the multiarmed bandit problem. Machine learning 47 235-256.

AUER, P., CESA-BIANCHI, N., FREUND, Y. and SCHAPIRE, R. E. (2002b). The nonstochastic multiarmed bandit problem. SIAM journal on computing 32 48-77.

CHAPELLE, O. and LI, L. (2011). An empirical evaluation of thompson sampling. Advances in neural information processing systems 24 .

CHEN, L., MIN, Y., BELKIN, M. and KARBASI, A. (2021). Multiple descent: Design your own generalization curve. Advances in Neural Information Processing Systems 34 8898-8912.

CHEN, L., MIN, Y., ZHANG, M. and KARBASI, A. (2020). More data can expand the generalization gap between adversarially robust and standard models. In International Conference on Machine Learning . PMLR.

CHENG, R. C. and KLEIJNEN, J. P. (1999). Improved design of queueing simulation experiments with highly heteroscedastic responses. Operations research 47 762-777.

CHOWDHURY, S. R. and GOPALAN, A. (2017). On kernelized multi-armed bandits. In International Conference on Machine Learning . PMLR.

CHU, W., LI, L., REYZIN, L. and SCHAPIRE, R. (2011). Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics . JMLR Workshop and Conference Proceedings.

DAI, Y., WANG, R. and DU, S. S. (2022). Variance-aware sparse linear bandits. arXiv preprint arXiv:2205.13450 .

DESHPANDE, Y. and MONTANARI, A. (2012). Linear bandits in high dimension and recommendation systems. In 2012 50th Annual Allerton Conference on Communication, Control, and Computing (Allerton) . IEEE.

FEI, Y. and XU, R. (2022a). Cascaded gaps: Towards gap-dependent regret for risk-sensitive reinforcement learning. arXiv preprint arXiv:2203.03110 .

FEI, Y. and XU, R. (2022b). Cascaded gaps: Towards logarithmic regret for risk-sensitive reinforcement learning. In International Conference on Machine Learning . PMLR.

GOPALAN, A., MANNOR, S. and MANSOUR, Y. (2014). Thompson sampling for complex online problems. In International conference on machine learning . PMLR.

HAMIDI, N. and BAYATI, M. (2020). On worst-case regret of linear thompson sampling. arXiv preprint arXiv:2006.06790 .

HE, J., WANG, T., MIN, Y. and GU, Q. (2022). A simple and provably efficient algorithm for asynchronous federated contextual linear bandits. Advances in neural information processing systems 35 4762-4775.

HENDERSON, C. R. (1975). Best linear unbiased estimation and prediction under a selection model. Biometrics 423-447.

JACOT, A., GABRIEL, F. and HONGLER, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems 31 .

KANDASAMY, K., KRISHNAMURTHY, A., SCHNEIDER, J. and P ´ OCZOS, B. (2018). Parallelised bayesian optimisation via thompson sampling. In International Conference on Artificial Intelligence and Statistics . PMLR.

KAUFMANN, E., KORDA, N. and MUNOS, R. (2012). Thompson sampling: An asymptotically optimal finite-time analysis. In Algorithmic Learning Theory: 23rd International Conference, ALT 2012, Lyon, France, October 29-31, 2012. Proceedings 23 . Springer.

KIM, W., KIM, G.-S. and PAIK, M. C. (2021). Doubly robust thompson sampling with linear payoffs. Advances in Neural Information Processing Systems 34 15830-15840.

KIM, W., LEE, K. and PAIK, M. C. (2022a). Double doubly robust thompson sampling for generalized linear contextual bandits. arXiv preprint arXiv:2209.06983 .

KIM, Y., YANG, I. and JUN, K.-S. (2022b). Improved regret analysis for variance-adaptive linear bandits and horizon-free linear mixture mdps. Advances in Neural Information Processing Systems 35 1060-1072.

KIRSCHNER, J. and KRAUSE, A. (2018). Information directed sampling and bandits with heteroscedastic noise. In Conference On Learning Theory . PMLR.

KORDA, N., KAUFMANN, E. and MUNOS, R. (2013). Thompson sampling for 1-dimensional exponential family bandits. Advances in neural information processing systems 26 .

LAI, T. L., ROBBINS, H. ET AL. (1985). Asymptotically efficient adaptive allocation rules. Advances in applied mathematics 6 4-22.

LI, Y., WANG, Y. and ZHOU, Y. (2023). Nearly minimax-optimal regret for linearly parameterized bandits. IEEE Transactions on Information Theory .

LILLICRAP, T. P., HUNT, J. J., PRITZEL, A., HEESS, N., EREZ, T., TASSA, Y., SILVER, D. and WIERSTRA, D. (2015). Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 .

LING, S., XU, R. and BANDEIRA, A. S. (2019). On the landscape of synchronization networks: A perspective from nonconvex optimization. SIAM Journal on Optimization 29 1879-1907.

LU, M., MIN, Y., WANG, Z. and YANG, Z. (2022). Pessimism in the face of confounders: Provably efficient offline reinforcement learning in partially observable markov decision processes. In The Eleventh International Conference on Learning Representations .

LU, Y., XU, Z. and TEWARI, A. (2021). Bandit algorithms for precision medicine. arXiv preprint arXiv:2108.04782 .

LUO, Y. and BAYATI, M. (2023). Geometry-aware approaches for balancing performance and theoretical guarantees in linear bandits. arXiv preprint arXiv:2306.14872 .

MAY, B. C., KORDA, N., LEE, A. and LESLIE, D. S. (2012). Optimistic bayesian sampling in contextual-bandit problems. Journal of Machine Learning Research 13 2069-2106.

MIN, Y., CHEN, L. and KARBASI, A. (2021a). The curious case of adversarially robust models: More data can help, double descend, or hurt generalization. In Uncertainty in Artificial Intelligence . PMLR.

MIN, Y., HE, J., WANG, T. and GU, Q. (2022a). Learning stochastic shortest path with linear function approximation. In International Conference on Machine Learning . PMLR.

MIN, Y., HE, J., WANG, T. and GU, Q. (2023). Cooperative multi-agent reinforcement learning: Asynchronous communication and linear function approximation. International Conference on Machine Learning .

MIN, Y., WANG, T., XU, R., WANG, Z., JORDAN, M. and YANG, Z. (2022b). Learn to match with no regret: Reinforcement learning in markov matching markets. Advances in Neural Information Processing Systems 35 19956-19970.

MIN, Y., WANG, T., ZHOU, D. and GU, Q. (2021b). Variance-aware off-policy evaluation with linear function approximation. Advances in neural information processing systems 34 7598-7610.

OMARI, C. O., MWITA, P. N. and GICHUHI, A. W. (2018). Currency portfolio risk measurement with generalized autoregressive conditional heteroscedastic-extreme value theory-copula model .

RIQUELME, C., TUCKER, G. and SNOEK, J. (2018). Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. arXiv preprint arXiv:1802.09127 .

ROBBINS, H. (1952). Some aspects of the sequential design of experiments .

RUSMEVICHIENTONG, P. and TSITSIKLIS, J. N. (2010). Linearly parameterized bandits. Mathematics of Operations Research 35 395-411.

RUSSO, D. and VAN ROY, B. (2016). An information-theoretic analysis of thompson sampling. The Journal of Machine Learning Research 17 2442-2471.

RUSSO, D. J., VAN ROY, B., KAZEROUNI, A., OSBAND, I., WEN, Z. ET AL. (2018). A tutorial on thompson sampling. Foundations and Trends® in Machine Learning 11 1-96.

SAHA, A. and KVETON, B. (2023). Only pay for what is uncertain: Variance-adaptive thompson sampling. arXiv preprint arXiv:2303.09033 .

SOMU, N., MR, G. R., KALPANA, V., KIRTHIVASAN, K. and VS, S. S. (2018). An improved robust heteroscedastic probabilistic neural network based trust prediction approach for cloud service selection. Neural Networks 108 339-354.

SONG, G., XU, R. and LAFFERTY, J. (2021). Convergence and alignment of gradient descent with random backpropagation weights. Advances in Neural Information Processing Systems 34 19888-19898.

THOMPSON, W. R. (1933). On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika 25 285-294.

TOWSE, A., JONSSON, B., MCGRATH, C., MASON, A., PUIG-PEIRO, R., MESTRE-FERRANDIZ, J., PISTOLLATO, M. and DEVLIN, N. (2015). Understanding variations in relative effectiveness: A health production approach. International journal of technology assessment in health care 31 363-370.

VARATHARAJAH, Y. and BERRY, B. (2022). A contextual-bandit-based approach for informed decision-making in clinical trials. Life 12 1277.

WANG, Z. and ZHOU, M. (2020). Thompson sampling via local uncertainty. In International Conference on Machine Learning . PMLR.

WU, Q., WANG, H., GU, Q. and WANG, H. (2016). Contextual bandits in a collaborative environment. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval .

XU, P., ZHENG, H., MAZUMDAR, E. V., AZIZZADENESHELI, K. and ANANDKUMAR, A. (2022). Langevin monte carlo for contextual bandits. In International Conference on Machine Learning . PMLR.

XU, R., CHEN, L. and KARBASI, A. (2021). Meta learning in the continuous time limit. In International Conference on Artificial Intelligence and Statistics . PMLR.

XU, R., MIN, Y., WANG, T., JORDAN, M. I., WANG, Z. and YANG, Z. (2023). Finding regularized competitive equilibria of heterogeneous agent macroeconomic models via reinforcement learning. In International Conference on Artificial Intelligence and Statistics . PMLR.

YE, C., XIONG, W., GU, Q. and ZHANG, T. (2023). Corruption-robust algorithms with uncertainty weighting for nonlinear contextual bandits and markov decision processes. In International Conference on Machine Learning . PMLR.

YIN, M., DUAN, Y., WANG, M. and WANG, Y.-X. (2022). Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism. arXiv preprint arXiv:2203.05804 .

ZHANG, T. (2022). Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science 4 834-857.

ZHANG, W., ZHOU, D., LI, L. and GU, Q. (2020). Neural thompson sampling. arXiv preprint arXiv:2010.00827 .

ZHANG, Z., YANG, J., JI, X. and DU, S. S. (2021). Improved variance-aware confidence sets for linear bandits and linear mixture mdp. Advances in Neural Information Processing Systems 34 4342-4355.

ZHAO, H., HE, J., ZHOU, D., ZHANG, T. and GU, Q. (2023). Variance-dependent regret bounds for linear bandits and reinforcement learning: Adaptivity and computational efficiency. arXiv preprint arXiv:2302.10371 .

ZHAO, H., ZHOU, D., HE, J. and GU, Q. (2022). Bandit learning with general function classes: Heteroscedastic noise and variance-dependent regret bounds. arXiv preprint arXiv:2202.13603 .

ZHOU, D. and GU, Q. (2022). Computationally efficient horizon-free reinforcement learning for linear mixture mdps. arXiv preprint arXiv:2205.11507 .

ZHOU, D., GU, Q. and SZEPESVARI, C. (2021). Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory . PMLR.

A Regret analysis of Algorithm 2

A.1 Clarification of notation

Before elaborating on proofs, we delineate several shorthands to facilitate our discourse. For each t ∈ [ T ] , we establish the events E θ t and E TS t . These events are denoted as the inclusion of ̂ θ t and θ TS t in the pertinent confidence ellipse with some confidence radius ̂ β t , respectively, i.e. ,

and

We proceed to define the set S ( t ) representing the saturated arms at round t as follows:

where we denote ∆ t ( x ) = ⟨ x ∗ t -x , θ ∗ ⟩ as the suboptimality of x ∈ X t . Moreover, we present a filtration, denoted by { H t } t ≥ 1 , which is defined as

This filtration encapsulates the information that has been gleaned up until the observation of the set of context vectors X t at each particular round.

Concentration inequalities are widely used in literature (Fei and Xu, 2022b; Min et al., 2022b; He et al., 2022; Lu et al., 2022; Xu et al., 2023), and here we introduce a Bernstein-type inequality applicable to vector-valued martingales. This theorem, when coalesced with the approach of weighted ridge regression, enables us to construct stringent bounds that are dependent on the variance { σ 2 t } t ≥ 1 .

Theorem A.1 (Zhou and Gu (2022), Theorem 4.3) . Let { H t } ∞ t =1 be a filtration, { y t , ϱ t } t ≥ 1 a stochastic process so that for each t ≥ 1 , y t ∈ R d is H t -measurable and ϱ t ∈ R is H t +1 -measurable. For each t ≥ 1 , define r t := ⟨ y t , θ ∗ ⟩ + ϱ t and suppose that ϱ t , y t satisfy

for some constants L, R, σ, ϵ > 0 and θ ∗ ∈ R d . Then, for any 0 < δ < 1 , with probability at least 1 -δ we have for all t > 0 that

where U t = λ I d + ∑ t s =1 y t y ⊺ t , ν t = ∑ t s =1 r s y t , θ t = U -1 t ν t , and

A.2 Proof of supporting lemmata

The forthcoming Lemmas A.2 and A.3 show that the events E θ t and E TS t hold with high probability for all t ∈ [ T ] . The anti-concentration argument can then be developed in a manner analogous to the approach employed by Agrawal and Goyal (2013), which is often not necessary in UCB-type of arguments (Fei and Xu, 2022a; Chu et al., 2011; Min et al., 2023; Zhao et al., 2023).

Lemma A.2. Under Assumption 3.1, for any δ ∈ (0 , 1) , if we define for all t ≥ 1 that

with the coefficients α, γ, λ introduced in Algorithm 2, then the estimate ̂ θ t given by the associated weighted regressions satisfy

Proof. Observe that for any t ≥ 1 and x ∈ X t , by Cauchy-Schwarz inequality we have

so it suffices to bound each ∥ x ∥ V -1 t . By Assumption 3.1, the following conditions on ε t and x t hold:

Moreover, we can obtain following σ t = max { σ t , α, γ ∥ x t ∥ 1 / 2 V -1 t } that

Therefore, by applying Theorem A.1 with the stochastic process { x t /σ t , ε t /σ t } t ≥ 1 and ϵ = R/γ 2 , we deduce that with probability at least 1 -δ/ 2 for all t ≥ 1

Since ∥ θ ∗ ∥ 2 ≤ 1 by Assumption 3.1, it follows from the definition of ̂ β t that ∥ ̂ θ t -θ ∗ ∥ V t ≤ ̂ β t . Combining this with (A.3), we conclude that with probability at least 1 -δ/ 2 ,

for all t ≥ 1 and x ∈ X t .

The following lemma establishes the control over | x ⊺ θ TS t -x ⊺ ̂ θ t | .

Lemma A.3. Under Assumption 3.1, for any δ ∈ (0 , 1) , let ̂ β t be as defined in Lemma A.2 for all t ≥ 1 . Then the sampled variable θ TS t in Algorithm 2 satisfies

Proof. By definition, θ TS t ∼ N ( ̂ θ t , Σ t ) where Σ t = ̂ β 2 t V -1 t , thus ∥ θ TS t -̂ θ t ∥ Σ -1 t ∼ N (0 , 1) . Then using the concentration inequality in Lemma C.1 for standard normal distribution, for every t ∈ [ T ] , we have with probability at least 1 -δ 4 t 2 that

where the first inequality follows from Cauchy-Schwarz inequality. Applying a union bound over t ∈ [ T ] and utilizing the fact that ∑ ∞ t =1 1 t 2 ≤ 2 , we obtain the final result.

In the following, we delve into the anti-concentration argument presented in Lemmas A.4 to A.6, which constitute the upper bound on expected regret.

Lemma A.4. For any t ≥ 1 , if H t is a filtration such that E θ t holds, we have a constant probability that the reward estimate ⟨ x ∗ t , θ TS t ⟩ of the optimal arm constitutes an upper confidence bound, i.e. ,

Proof. Note that we can bound the quantity J t = ( x ∗ t ⊺ θ ∗ -x ∗ t ⊺ ̂ θ t ) / ( ̂ β t ∥ x ∗ t ∥ V -1 t ) as follows:

where the inequality is due to Lemma A.2. Recalling that θ TS t has a mean of ̂ θ t , we can establish that the probability of the reward estimate x ∗ t ⊺ θ TS t serving as an upper bound for the true reward is lower-bounded by

The last inequality follows from the anti-concentration inequality in Lemma C.1 for standard normal distribution and the fact that

Consequently, combining (A.4) and (A.5), it follows that the desired probability is lower bounded by a constant, i.e. , P ( x ∗ t ⊺ θ TS t > x ∗ t ⊺ θ ∗ | H t ) ≥ 1 4 e √ π .

Lemma A.5. For any t ≥ 1 , if H t is a filtration such that E θ t holds, we have with a constant probability that the chosen arm x t is not a saturated arm, i.e. ,

Proof. Recall the definition of the set of saturated arms S ( t ) in (A.1) and that the selected arm x t = arg max x ∈ X t x ⊺ θ TS t aims to maximize the estimated reward. Consequently, x t / ∈ S ( t ) if x ∗ t ⊺ θ TS t ≥ x ⊺ θ TS t for all saturated arms x ∈ S ( t ) , implying that the estimated reward for the optimal arm surpasses the estimated rewards of all saturated arms. It follows that:

Further note that by the definition of saturated arms, we have

when both E θ t and E TS t hold. Therefore,

The last inequality is a consequence of Lemmas A.3 and A.4.

Lemma A.6. For any t ≥ 1 , if H t is a filtration such that both E θ t and E TS t hold, we have

Proof. Let x † t represent the unsaturated arm with the smallest ∥ · ∥ V -1 t norm, i.e. ,

The existence of such an unsaturated arm x † t is guaranteed since x ∗ t / ∈ S ( t ) . When both E θ t and E TS t hold, we can express the suboptimality as follows:

Then since x t is the optimal arm under θ TS t , we further have

where the second inequality results from x † t / ∈ S ( t ) . Note that ∥ x t ∥ V -1 t ≥ ∥ x † t ∥ V -1 t with constant probability, i.e. ,

The last inequality is due to Lemma A.5 and the definition of x † t as the unsaturated arm with the smallest ∥ · ∥ V -1 t . Consequently,

where the first inequality is due to ∆ t ≤ 2 for all t , the second inequality follows from applying (A.6), Lemmas A.3 and A.4.

A.3 Proof of Theorem 4.1

Let us define a shorthand E θ = ∩ T t =1 E θ t . The expected regret can be decomposed as follows:

where we write C = max t ≥ 1 2 | 1 4 e √ π -1 t 2 | +1 . The first inequality is due to Lemma A.2, the second inequality is due to Lemma A.6, and the last inequality follows from ∆ t ≤ 2 for all t ∈ [ T ] . To alleviate the burden on notation, we write g t = min { 2 , (2 √ d log t +1) C ̂ β t ∥ x t ∥ V -1 t } and an index set T = { t ∈ [ T ] : ∥ x t /σ t ∥ V -1 t ≥ 1 } . It follows that

where the first inequality follows from the definition of g t , the last equality is due to ∥ x t /σ t ∥ V -1 t < 1 for all t / ∈ T , and the second inequality is due to

The last inequality is due to Lemma C.2 and ∥ x t /σ t ∥ ≤ 1 /α . Further, we decompose T = [ T ] \ T = T 1 ∪ T 2 where

The second term in (A.7) can be controlled through a decomposition

where the first inequality is due to Cauchy-Schwarz inequality, and the second inequality is due to Lemma C.2 and ∥ x t /σ t ∥ ≤ 1 /α . For t ∈ T 2 , we have σ t = γ 2 ∥ x t /σ t ∥ V -1 t , and then

Hence, combining the inequalities above, we have

and an upper bound on expected regret follows noting that ∑ T t =1 min { 2 , (2 √ d log t + 1) C ̂ β t ∥ x t ∥ V -1 t } enjoys a deterministic upper bound:

E [ R ( T )] ≤ T ∑ t =1 g t + δT +4 ≤ 4 d log(1 + T/ ( dλα 2 )) + 3 C · 2 √ √ √ √ d T ∑ t =1 ̂ β 2 t ( σ 2 t + α 2 ) log T √ d log(1 + T/ ( dλα 2 )) +3 C · 2 γ 2 d √ d log T log(1 + T/ ( dλα 2 ))(max t ∈ [ T ] ̂ β t ) + δT +4 Take λ = d , α = 1 / √ T , and γ = √ R/d 1 / 4 , it follows that log(1 + T/ ( dλα 2 )) ≤ log( T 2 /d 2 +1) and ̂ β t = 12 √ d log(1 + t/ ( dλα 2 )) log(32(1 + log( γ 2 /α )) t 2 /δ ) +30 R log(32(1 + log( γ 2 /α )) t 2 /δ ) /γ 2 + √ λ ≤ 12 √ d log(1 + tT/d 2 ) log(32(1 + log( R √ T/ √ d )) t 2 /δ ) +30 √ d log(32(1 + log( R √ T/ √ d )) t 2 /δ ) + √ d.

Combining Lemma A.2 and the inequalities above with δ = 1 T , we conclude that

Recall that we have

For notational convenience, we further define S t = ∑ t s =1 D t where D t = ∆ t -g t -2 t 2 , and

It follows that E [ D t | H t ] ≤ 0 for all t ∈ [ T ] , and { S t } t ≥ 1 therefore forms a super-martingale process with respect to the filtrations { H t } t . Note that | D t | ≤ 2 g t + 2 ≤ 4 , and with AzumaHoeffding inequality we have with probability at least 1 -δ/ 2 that

B Regret Analysis of Algorithm 3

B.1 Clarification of notation

Analogous to Appendix A, we first introduce two concentration events E θ t,ℓ and E TS t,ℓ for each t ∈ [ T ] and ℓ ∈ [ L ] , i.e. ,

and

These events are also characterized as the inclusion of ̂ θ t,ℓ and θ TS t,ℓ in the corresponding confidence ellipse, respectively. It is noteworthy that the event E θ t,ℓ is defined using

rather than the confidence radius ̂ β t,ℓ .

Subsequently, we introduce a Freedman-type concentration inequality pertinent to vector-valued martingales. This theorem, distinct from Theorem A.1, necessitates a nuanced control on the uncertainty term ∥ y t ∥ U -1 t -1 for all t ≥ 1 . Consequently, it furnishes an even more stringent bound in comparison to Theorem A.1.

Theorem B.1 (Zhao et al. (2023), Theorem 2.1) . Let { H t } ∞ t =1 be a filtration, { y t , ϱ t } t ≥ 1 a stochastic process so that y t ∈ R d is H t -measurable and ϱ t ∈ R is H t +1 -measurable. For t ≥ 1 let r t = ⟨ y t , θ ∗ ⟩ + ϱ t and suppose that ϱ t , y t satisfy

for some constants R, { v t } t > 0 and θ ∗ ∈ R d . Then, for any 0 < δ < 1 , with probability at least 1 -δ we have for all t > 0 that

where U t = λ I d + ∑ t s =1 y s y ⊺ s , ν t = ∑ t s =1 r s y s , θ t = U -1 t ν t , and

β t = 16 ρ √ v t log(4 t 2 /δ ) + 6 ρR log(4 t 2 /δ ) , where ρ ≥ sup t ≥ 1 ∥ y t ∥ U -1 t -1 .

B.2 Proof of supporting lemmata

Lemmas B.2 and B.3 show that the concentration events E θ t,ℓ and E TS t,ℓ hold with high probability for all t ∈ [ T ] and ℓ ∈ [ L ] .

Lemma B.2. For any δ ∈ (0 , 1) , if we define ˜ β t,ℓ for all t ≥ 1 and ℓ ∈ [ L ] as in (B.1) , then the estimate ̂ θ t,ℓ given by the associated weighted regressions in Algorithm 3 satisfy

Proof. Notice that for any fixed ℓ ∈ [ L ] , t ∈ Ψ t +1 ,ℓ , and x ∈ X t that

Following the assumption on ε t , it holds that

where w t = 2 -ℓ / ∥ x t ∥ V -1 t,ℓ ≤ 1 . Apply Theorem B.1 with stochastic process { w t x t , w t ε t } to get that with probability at least 1 -δ 2 L for all t ∈ Ψ T +1 ,ℓ

∥

θ

t,ℓ

̂

-

θ

∗

∥

V

β

t,ℓ

˜

,

where

It follows from a union bound that with probability at least 1 -δ/ 2 that for all ℓ ∈ [ L ] , t ∈ Ψ t +1 ,ℓ , and x ∈ X t

Lemma B.3. For any δ ∈ (0 , 1) and ̂ β t,ℓ defined in (5.3) for all t ≥ 1 and ℓ ∈ [ L ] , the sampled variables θ TS t,ℓ in Algorithm 3 satisfy

Proof. Note that θ TS t,ℓ ∼ N ( ̂ θ t,ℓ , Σ t,ℓ ) where Σ t = ̂ β 2 t,ℓ V -1 t,ℓ , we have with probability at least 1 -δ 4 t 2 L

t,ℓ

where the last inequality follows the concentration inequality in Lemma C.1 for standard normal distribution and that

A union bound over all ℓ ∈ [ L ] and t ≥ 1 yields the desired result, where the probability bound follows ∑ ∞ t =1 1 /t 2 < 2 .

Next, we show that the empirical estimator ζ t,ℓ provides an accurate estimate of the summation of weighted variance up to a logarithmic factor. The proof is analogous to Zhao et al. (2023), and we provide it here for the sake of maintaining a self-contained discourse.

Lemma B.4. For all t ≥ 1 and ℓ ∈ [ L ] such that 2 ℓ ≥ 64 √ log(4( t +1) 2 L/δ ) , if E θ t,ℓ is satisfied and ζ t,ℓ = ∑ s ∈ Ψ t,ℓ w 2 s ( r s -⟨ x s , ̂ θ s,ℓ ⟩ ) 2 , then the following inequalities hold:

Proof. For any ℓ ∈ [ L ] and t ∈ [ T ] , we have ∑ s ∈ Ψ t +1 ,ℓ w 2 i ε 2 s being an unbiased estimator of ∑ s ∈ Ψ t +1 ,ℓ w 2 i σ 2 s conditioned on past observations. More specifically, we have E [ ε 2 t | x 1: t , r 1: t -1 ] -E [ σ 2 t | x 1: t , r 1: t -1 ] = 0 for all t ≥ 1 and

where the first inequality follows from V ar( w s ε 2 s | x 1: t , r 1: t -1 ) ≤ E [ w 2 s ε 4 s | x 1: t , r 1: t -1 ] . Following Freedman inequality, it holds that with probability at least 1 -2 δ/L that for all t ≥ 1

where the second inequality follows from Young's inequality.

It then follows that

where the second inequality follows from ( x + y ) 2 ≤ 2 x 2 +2 y 2 . Notice that the last term is bounded by

where the first inequality follows from w 2 s x s x ⊺ s ⪯ V t +1 ,ℓ and the second inequality is due to Lemma B.2. If we have 2 ℓ ≥ 64 √ log(8 t 2 L/δ ) , then

and

Move 1 2 ∑ s ∈ Ψ t +1 ,ℓ w 2 s σ 2 s to the other side of the inequality, we get the desired result.

Moreover, we can establish the inequality in the other direction:

where the first inequality follows from ̂ θ t,ℓ being the minimizer of the weighted ridge regression problem. Combining the inequality with (B.2) yields the desired result.

Lemma B.5. Suppose that E θ t,ℓ and E TS t,ℓ hold for all ℓ ∈ [ L ] and t ∈ Ψ t +1 ,ℓ . If all the elements in series { ˜ β t,ℓ } t,ℓ and { ̂ β t,ℓ } t,ℓ , as defined in (B.1) and (5.3) respectively, satisfy

then for all t ≥ 1 and ℓ ∈ [ L ] such that X t,ℓ exists, it holds x ∗ t ∈ X t,ℓ .

Proof. We prove the statement by induction. If ℓ = 1 , then x ∗ t ∈ X t,ℓ = X t trivially holds. Suppose x ∗ t ∈ X t,ℓ for some ℓ ∈ Z + and X t,ℓ +1 exists, we have for all x ∈ X t,ℓ that

where the second inequality is due to the assumption ˜ β t,ℓ ≤ ̂ β t,ℓ , Lemmas B.2 and B.3, and the last inequality is due to ∥ x ∥ V -1 t,ℓ ≤ 2 -ℓ for all x ∈ X t,ℓ . If we define ˜ x t = arg max x ∈ X t,ℓ ⟨ x , θ TS t,ℓ ⟩ , the it holds that

where the second inequality follows from ⟨ ˜ x t -x ∗ t , θ ∗ ⟩ ≤ 0 . Hence, the optimal action x ∗ t ∈ X t,ℓ +1 and the statement follows from induction.

Lemma B.6. Suppose that ˜ β t,ℓ ≤ ̂ β t,ℓ , E θ t,ℓ , and E TS t,ℓ hold for all ℓ ∈ [ L ] and t ∈ Ψ t +1 ,ℓ . For any ℓ ∈ [ L ] \{ 1 } , we have

Proof. For all t ∈ Ψ T +1 ,ℓ , we have x t , x ∗ t ∈ X t,ℓ following Lemma B.5 and the assumptions on ˜ β t,ℓ ≤ ̂ β t,ℓ , E θ t,ℓ , and E TS t,ℓ . Due to Line 11 in Algorithm 3, it holds

In addition, for any t ∈ Ψ T +1 ,ℓ , we have both ∥ x t ∥ V -1 t,ℓ -1 ≤ 2 -ℓ +1 and ∥ x ∗ t ∥ V -1 t,ℓ -1 ≤ 2 -ℓ +1 . It follows that

where the second inequality is due to (B.3), Lemmas B.2 and B.3. Summing over t ∈ Ψ T +1 ,ℓ , we have

where the second inequality is due to ∥ w t x t ∥ V -1 t,ℓ = 2 -ℓ for all t ∈ Ψ T +1 ,ℓ , and the last inequality is due to Lemma C.2.

B.3 Proof of Theorem 5.1

With probability at least 1 -2 δ , all the statements of Lemmas B.2 and B.3 hold, and the rest of our proof will build on such an event. Let ℓ 0 = ⌈ 1 2 log 2 log(8( T + 1) 2 L/δ ) ⌉ + 8 , and we have ˜ β t,ℓ ≤ ̂ β t,ℓ for all t ∈ [ T ] and ℓ ∈ [ L ] \ [ ℓ 0 ] following Lemma B.4. It then follows Lemma B.6 that for all ℓ ∈ [ L ] \ [ ℓ 0 ]

where the first inequality is due to Lemma B.6, the second inequality is due to the definition of ̂ β T,ℓ -1 , and the last inequality is due to Lemma B.4. For any layer ℓ ∈ [ ℓ 0 ] and t ∈ Ψ T +1 ,ℓ , we have

where the first inequality follows from ⟨ x , θ ∗ ⟩ ≤ 1 for all x ∈ X t and t ∈ [ T ] , the equality is due to ∥ w t x t ∥ V -1 t,ℓ = 2 -ℓ for all t ∈ Ψ T +1 ,ℓ , and the last inequality is due to Lemma C.2 and 2 ℓ 0 ≲ √ log( TL/δ ) .

For any t ∈ [ T ] \ ( ∪ ℓ ∈ [ L ] Ψ T +1 ,ℓ ) , Algorithm 3 selects an arm at some ℓ t -th layer, and it follows that

where the first inequality is due to ˜ β t,ℓ ≤ ̂ β t,ℓ , Lemmas B.2, B.3 and B.5, the second inequality is due to the arm selection rule of Algorithm 3, the third inequality is due to Cauchy-Schwarz inequality, and the fourth inequality is due to ˜ β t,ℓ ≤ ̂ β t,ℓ , Lemmas B.2 and B.3, and the fifth inequality follows from the definition of α and ̂ β t,ℓ t ≤ ˜ O ( R √ T ) .

Combining the above inequalities together, we conclude that

where the inequality follows from L ≲ log T and ℓ 0 = ⌈ 1 2 log 2 log(8( T +1) 2 L/δ ) ⌉ +8 .

C Auxiliary Lemmata

Lemma C.1 (Abramowitz et al. (1964)) . For a Gaussian distributed random variable X with mean m and variance σ 2 , it holds that for any x ≥ 1 that

LemmaC.2 (Abbasi-Yadkori et al. (2011), Lemma 11) . Let { X t } ∞ t =1 be a sequence in R d , V ∈ R d × d positive definite matrix, and V t = V + ∑ t s =1 X s X ⊺ s . Then we have that

Further, if ∥ X t ∥ 2 ≤ L for all t , then

and finally, if λ min ( V ) ≥ max(1 , L 2 ) then

D Numerical Results

In our simulation, we test our algorithm on a linear contextual bandit problem with different sets of reward perturbations. Specifically, the simulation is carried out over T = 2000 rounds with an ambient dimension of d = 25 . For each selection round t ∈ [ T ] , the environment yields a set of contexts X t , which contains K = 50 context vectors. The context vectors are randomly drawn and structured as truncated Gaussian vectors, ensuring their norms remain bounded by 1 . Within each trial, a random ground truth vector θ ∗ is generated as another Gaussian vector. The reward associated with every context vector is then computed based on the expected reward, perturbed by Gaussian noise. We employ our proposed algorithms, LinVDTS and LinNATS , to scrutinize their performance within this configuration.

Cumulative Regret Over Time

600

Baseline on Constant Variance

VDTSonConstantVariance

500

Baseline onScheduledVariance

VDTS on Scheduled Variance

NATS on Scheduled Variance

400

Cumulative Regret

300

200

100

0

0

250

500

750

1000

1250

1500

1750

2000

Round

Figure Fig. 1 presents the cumulative regrets of the LinVDTS and LinNATS algorithms, using the vanilla TS as a benchmark. The algorithms are evaluated in two distinct bandit environments: one characterized by constant variance and the other by a quadratically decaying variance. Performance under the constant variance setup is captured by dashed lines, whereas the decaying variance scenario is depicted by solid lines, with the observations being grounded on 50 randomized trials. It is noteworthy that while LinVDTS is privy to the ground truth variance, LinNATS operates without this knowledge in our experimental framework. This distinction underscores LinNATS 's commendable adaptability in environments where the variance is elusive. Given its access to variance data, the superior performance of LinVDTS vis-a-vis LinNATS in fluctuating variance situations is anticipated. The presented outcomes showcase the variance-awareness of our proposed algorithms and the inherent flexibility of these algorithms in modulating their regret according to the prevailing variance dynamics.

Future directions and broader impact. The recent progress in understanding the training dynamics of deep neural networks (Jacot et al., 2018; Song et al., 2021) has paved the way for investigations at the intersection of deep learning and reinforcement learning (Lillicrap et al., 2015; Chen et al., 2021; Xu et al., 2021). This integrative approach promises to yield transformative insights and substantially enhance the performance of existing learning architectures. Further, the incorporation of non-linear function approximation into our frameworks emerges as a pivotal research direction,

with the potential to significantly improve their robustness (Ling et al., 2019; Chen et al., 2020; Min et al., 2021a; Ye et al., 2023). We believe that engaging in these research endeavors is important for the refinement of our methodologies.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers linear bandit problems with adaptive noise sequences. They propose LinVDTS that takes the noise level as input, applies a weighted RLS to estimate the parameter, and adopts a TS manner to choose the actions. LinVDTS achieves a noise-dependent regret that matches the best $O(d^{\\frac{3}{2}} \\sqrt{d})$ frequentist order. They also propose LinNATS, which drops the requirement of the knowledge of heteroscedastic noise using a weighted RLS estimator with multi-layer uncertainty structures. LinNATS achieves similar regret bounds.\n\nSTRENGTHS:\nThe paper is well-organized and easy to follow. The proofs seem to be technically sound. In general, I think this paper does a great and complete job in providing theoretical guarantees for the noise-adaptive linear bandit problems.\n\nWEAKNESSES:\nMy main concern is the significance and contributions of the job. The key contributions are claimed to be LinVDTS, LinNATS and their regret bounds. However, the RLS estimator and its proof technique are standard in the literature of Linear Thomson Sampling (LinTS). Compared with LinTS, the only new component of LinVDTS is the weighted RLS estimator (which is also standard in its literature) to handle adaptive noise. The remaining parts, including the associated regret proof, are essentially the same. The multi-layer design of LinNATS is also common in papers dealing with unknown parameters/uncertainty/non-stationarity. It's not a surprise the authors could get the presented results\n\nI don't agree with the third contribution the author claimed: \"This improves over the existing $\\mathcal{O}(d^{\\frac{3}{2}}\\sqrt{T})$ regret bound for LinTS\". When all variances are fixed at constant, the bound is still $\\mathcal{O}(d^{\\frac{3}{2}}\\sqrt{T})$. The mentioned ``diminishing variances'' are too simplistic and shall not be over-emphasized. \n\nThird, the presented Bayesian regret is of order $\\mathcal{O}(d^{\\frac{3}{2}}\\sqrt{T}+\\delta T)$, which is worse than the known best achievable $\\mathcal{O}(d\\sqrt{T})$ [1]. \n\nRefs:\n1. Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling. Mathematics of Operations Research, 39(4):1221–1243, 2014. doi: 10.1287/moor.2014.0650. 2, 3\n\nQUESTIONS:\nAs discussed above in the Weakness part, I would appreciate it if the authors could explain more regarding the significance and contributions of this work. Also, can the authors comment on why they fail to achieve optimal Bayesian bound?\n\nLIMITATIONS:\nI don't see any limitations or potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work delves into the field of the linear contextual bandit, a widely used model for addressing sequential decision-making problems with broad applications in real-world scenarios. In this framework, the player selects an arm from a predefined set and receives a random reward from an unknown distribution upon pulling it. The objective of this learning problem is to minimize the cumulative regret, which is the sum of the differences between the rewards obtained and the rewards of the optimal arm (which is unknown to the player) across all rounds. In particular, Thompson sampling (TS) emerges as a prominent family of algorithms for solving the linear contextual bandit problem, with an extensive body of literature dedicated to analyzing their theoretical properties. TS has garnered considerable attention in research, making it an active area of exploration. This work specifically focuses on the linear Thompson Sampling (TS) algorithms, where existing studies have established an upper bound of $O(d^{3/2} \\sqrt{T})$ for the regret in Linear TS algorithms. However, this bound lacks noise adaptivity, meaning it does not consider the variance in the reward distribution. Intuitively, the bandit learning problem becomes easier when the reward distribution has lower variance, and vice versa.\n\nThis work addresses the limitations of previous methods by providing the first noise adaptive linear TS algorithms and theoretically proves that it achieves a variance-aware regret upper bound. Specifically, two different algorithms are presented. The first algorithm, VDTS, is shown to achieve a variance-aware regret bound in expectation. This work briefly explains the technical difficulty in establishing a high-probability variance-aware bound by identifying two factors: (1) the optimism in VDTS is too conservative; (2) a purely technical challenge for controlling a difference term in the regret analysis. The work also presents another high-probability bound for VDTS, but it is not variance-aware. \n\nThis study then introduces a new algorithm, NATS, which successfully achieves a variance-aware regret bound in high probability. This is the first TS-based algorithm for linear contextual bandit that is noise adaptive. The key design of NATS is a layering technique, which is motivated by some UCB-based algorithms. On a high level, this technique divides the uncertainty quantities into different layers depending on their magnitudes. By doing so, a variance-dependent confidence radius for model parameters can be built, which eventually leads to a variance-aware regret bound. This differs from the anti-concentration argument adopted by existing work in analyzing TS-based algorithms.\n\nSTRENGTHS:\n1. This work provides a solid theoretical analysis of two proposed TS-based linear bandit algorithms with well-grounded motivation. In particular, the study on how to achieve a high-probability noise-adaptive regret bound is comprehensive. The following are my detailed comments.\n2. The focus on noise-awareness is well-motivated. I think it is particularly interesting to have a result for TS that bridges the gap between deterministic reward and constant-variance reward. Furthermore, since such types of results have been developed for UCB-based algorithms, it is a natural question to ask whether this is achievable in a TS setting. \n3. This work is the first work to establish a high probability regret bound with noise-adaptivity for linear TS-based algorithms. Although such kind of results has been achieved by UCB-based algorithms in quite a few existing works, there is a scarcity of results concerning TS-based algorithms. Since TS is believed to have higher computational efficiency than UCB, variance-awareness will make TS more favorable, especially in scenarios characterized by noise heteroscedasticity. \n4. Naively adopting the algorithmic design based on weighted linear regression from UCB will not grant a high probability of regret bound in the TS setting, which is due to the technical barrier in the analysis as mentioned on page 6. Therefore, the work takes a different route by borrowing the idea of stratification over the uncertainty quantity, as introduced in Section 5.1, which results in a variance-dependent concentration inequality (eq 5.2). \n5. As a theoretical work, this current manuscript is well-written. Most parts are relatively easy to follow. The paragraphs are eloquently composed, giving a clear explanation of the complicated algorithmic design of VDTS and NATS. For example, after main theorems 4.1 and 5.1, remarks and discussions are added to provide implications of the results. The discussion on page 6 regarding why VDTS cannot achieve a high-probability variance-aware bound is especially appreciated.\n\nWEAKNESSES:\n1. For theoretical papers on bandit algorithms, numerical simulations are usually quite necessary for demonstrating the empirical performance of the proposed algorithms. For this work, this is especially the case as its main theoretical contribution is a variance-aware regret bound which often involves subtlety. I would not doubt that NATS/VDTS will produce a sublinear ($O(\\sqrt{T})$) cumulative regret. The key implication of this work is whether the algorithm can truly adapt to the noise level, but it is not immediately clear. For more specific questions, please refer to the Question section below.\n2. This is more of a question than a weakness, but the work mentioned that the current regret bound’s dependence on dimension d is $O(d^{3/2}$), in comparison with $O(d)$ in the Feel-good TS algorithm. This makes me very curious about whether the current method can be improved to reach optimality in d. For example, if it is not possible, what technical barrier prevents the method from doing so? It would be helpful to have a bit more discussion on this point.\n\nQUESTIONS:\n1. I would be curious to see how the algorithm performs. The following questions are worth investigating: \n\n​\t (a) How does the regret of NATS respond to the change in the noise level? For example, if one runs a numerical simulation on two synthetic bandit instances with different $\\sigma^2$, will the regret curves manifest obvious differences? \n \t\n(b) It seems that the absence of a high-probability variance-aware bound for VDTS is more of a technical issue. So, I am interested to know whether VDTS indeed has a worse performance than NATS.\n\n2. This manuscript uses both terminologies “variance aware” and “noise adaptive” without an explicit definition (though it's not a big issue). In my understanding, they refer to the same property. So, I am just a bit curious why using two terms instead of one. Or perhaps an explicit definition/explanation can be added to prevent any confusion for future readers.\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the contextual linear bandit model, a widely utilized framework for online decision-making problems. Specifically, it focuses on the heteroscedastic noise setting, where each reward is associated with a different variance. The authors propose a novel noise-adaptive Thompson sampling algorithm that establishes a regret upper bound dependent on the variance. This theoretical contribution not only encompasses the findings from the constant-variance and deterministic-reward regimes but also provides an interpolation between these two settings.\n\nThe paper presents two algorithms in total. The first algorithm, VDTS, achieves a variance-dependent regret bound in expectation. However, the authors acknowledge the inherent challenge of controlling uncertainty induced by posterior sampling, preventing them from obtaining a high-probability regret upper bound.\n\nTo address this limitation, the authors introduce the NATS algorithm, which employs stratification to partition uncertainty terms into distinct layers. By doing so, it bypasses the conventional anti-concentration argument typically employed in Thompson sampling analysis. The proposed NATS algorithm is proven to achieve a high-probability regret upper bound, surpassing existing bounds for linear Thompson sampling.\n\nSTRENGTHS:\n**Novelty**\n\nThis paper makes a significant contribution by introducing the first variance-aware Thompson sampling algorithm. Previous theoretical work solely focused on regret bounds that were independent of noise variance. The authors employ novel techniques, drawing inspiration from the stratification framework initially proposed for UCB-based algorithms.\n\n**Theoretical Contribution**\n\nThe theoretical contributions of this paper are commendable. The introduction of a variance-aware bound represents a substantial improvement over existing bounds for linear Thompson sampling. Traditional bounds may prove suboptimal when the true noise variance is relatively small. Additionally, the authors' analysis incorporates stratification, enabling them to circumvent the commonly adopted anti-concentration argument found in classical Thompson sampling analyses.\n\n**Clarity**\n\nThis paper is presented in a clear and coherent manner. The mathematical definitions are precise and well-defined. The VDTS and NATS algorithms are thoroughly explained and accompanied by demonstrations. The main theorems (Theorem 4.1 and 5.1) are followed by discussions regarding their implications.\n\nWEAKNESSES:\nThis paper has no empirical results at all. For such kinds of theoretical bandit algorithms, numerical experiments are usually required to demonstrate the correctness of the proposed methods and the validity of the proved theorems.\n\nQUESTIONS:\n1. Theoretically, NATS is shown to achieve a high probability regret upper bound, while VDTS only in-expectation regret upper bound. I am interested in whether NATS and VDTS will perform differently (or similarly) in empirical results. A numerical simulation would be interesting to check.\n2. In this paper, variance-awareness of the proposed algorithm is only proven theoretically. But I doubt whether this can really be realized empirically. For example, if we apply NATS to two bandit instances with different noise levels, would the observed regret really be smaller if the variance is smaller?\n\nLIMITATIONS:\nPlease refer to the above Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a Thompson-Sampling style algorithm for linear contextual bandits under a heteroscedastic noise setting. When the heteroskedastic noise is known, they present a simple algorithm, LinVDTS, that achieves a variance-dependent regret upper bound, by utilizing a weighted ridge regression estimator. When the noise is unknown, they present a more general algorithm, LinNATS, that adapts to the unknown noise by using the idea of optimism together with elimination to stratify the uncertainty level.\n\nSTRENGTHS:\nThe paper is well-written. The author provides a clear structure of the two settings and intuitions for their algorithm design. They also provide a good transition between their two algorithms from an easier to a harder problem setting. Their theoretical bound is impressive in the sense that it is not obvious if one could hit the same regret bound \\tilde{O}(d^{3/2}\\sqrt{T}) as the constant variance case, but their result recovers this bound. Also, the discussion of lines 207-239 is very insightful.\n\nWEAKNESSES:\nThe explanation of certain details of the algorithm could be improved. For example, the stratification process in the paper is a bit complicated. A picture to explain it would help. \nAlso, the author could add more annotations to the description of their algorithms so that it is easier to read. As another note, the paper is missing experiments, although the theoretical contribution is solid and it could be considered as a theory paper. \n\nMinor Comments:\n1. Line 137: the dependence of the noise on x_t should be more emphasized. At first glance, the fact that the noise at time t depends on the arm x_t was hidden. \n2. Check the sentence structure/grammar. Eg in Line 207\n\nQUESTIONS:\n1. When the noise is known, can you provide a more elaborate explanation of why you choose \\bar{\\sigma}_s instead of \\sigma_s? Intuitively, if the variance of the noise is known, I would tend to just normalize the variance so that the problem reduces to a homoscedastic noise setting and use classical algorithms to solve it, and I’d like to know more about why that’s not enough. \n2. Can the authors comment on a matching lower bound that incorporates the correct dependence on the changing variance?\n3. Is there an off-by-1 error in line 166? Namely, the definition of \\theta_t uses x_t, whereas \\theta_t in the algorithms seems to only depend on the history up to time t-1. \n4. I think the algorithm is computationally inefficient for large or infinite arm sets, so it is more of a theoretical interest but the practical side of it is less impactful. Do the authors think the algorithm can be made practical?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}