Title: Variance-Aware Feel-Good Thompson Sampling for Contextual Bandits
Venue: NeurIPS 2025
Accepted: yes
Keywords: Thompson sampling

variance aware

contextual bandits
Abstract: Variance-dependent regret bounds have received increasing attention in recent studies on contextual bandits. However, most of these studies are focused on upper confidence bound (UCB)-based bandit algorithms, while sampling based bandit algorithms such as Thompson sampling are still understudied. The only exception is the `LinVDTS` algorithm (Xu et al., 2023), which is limited to linear reward function and its regret bound is not optimal with respect to the model dimension. In this paper, we present `FGTSVA`, a variance-aware Thompson Sampling algorithm for contextual bandits with general reward function with optimal regret bound. At the core of our analysis is an extension of the decoupling coefficient, a technique commonly used in the analysis of Feel-good Thompson sampling (FGTS) that reflects the complexity of the model space. With the new decoupling coefficient denoted by $\mathrm{dc}$, `FGTS-VA` achieves the regret of $\tilde{\mathcal{O}}(\sqrt{\mathrm{dc}\cdot\log|\mathcal{F}|\sum_{t=1}^T\sigma_t^2}+\mathrm{dc})$, where $|\mathcal{F}|$ is the size of the model space, $T$ is the total number of rounds, and $\sigma_t^2$ is the subgaussian norm of the noise (e.g., variance when the noise is  Gaussian) at round $t$. In the setting of contextual linear bandits, the regret bound of `FGTSVA` matches that of UCB-based algorithms using weighted linear regression (Zhou and Gu, 2022).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper brings the feel-good Thompson sampling (FGTS) algorithm into the second-order (variance-aware) contextual bandit. They show that the regret of the algorithm can upper bounded by the decoupling coefficient, specifically, if the decoupling coefficient of the function class is d, then the regret of variance-aware FGTS is bounded by $\\sqrt{d*\\Sigma*\\log F}+d$. This is the first paper to provide variance aware guarantees for feel good Thompson sampling algorithm.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. This paper is well-written. All assumptions required by the algorithms are made clear and easily understood in this paper.\n2. The study of variance-aware FGTS is of interest to the community, as FGTS is a kind of posterior sampling algorithm, which has better guarantee in Bayesian regret setting than other frequentist regret algorithm (like UCB or inverse-gap-weighting), and also the variance-aware study also provides tighter guarantees than the common regret bound.\n3. In this paper, they considered different parameter $\\eta$ across different time ste. This modification is novel and not appeared in previous feel-good TS papers.\n\nWeakness:\n1. This paper does not provide numerical experiments to demonstrate the influence of total variances and regret bound. Some numerical experiments will be much helpful to illustrate the theoretical findings.\n2. There is no lower bound provided. It will be better if there exists some lower bound which shows that the regret of the upper bound is tight up to some factors.\n3. The decoupling coefficient is a little bit difficult to understand than the normally used eluder dimension. It will be better if there is more explanation on how the decoupling coefficient can be bounded for some specific function classes.\n\nQUESTIONS:\nI have the following questions about this paper:\n1. Is there an computationally efficient way for updating the distribution and performing FGTS?\n2. When the variance $\\sigma_t$ is revealed to the learner after making action at round $t$, or even not revealed to the learner, does the algorithm still work?\n3. In reference [1], they also present an upper bound scaled with $\\sqrt{A*\\Sigma}$ for the inverse-gap-weighting algorithms in that paper, does a similar guarantee holds for FGTS?\n4. This paper only considers the prior to be uniform over all functions, what about other priors? Can the algorithm achieves similar guarantees, or under what assumptions over the prior can the algorithm has similar guarantees?\n5. Since the Thompson sampling algorithm was firstly introduced for Bayesian regret, is there tight characterization of the variance-aware FGTS in Bayesian regret setting?\n\n[1] Jia, Zeyu, et al. \"How does variance shape the regret in contextual bandits?.\" Advances in Neural Information Processing Systems 37 (2024): 83730-83785.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces **FGTS-VA**, a variance-aware variant of Feel-Good Thompson Sampling for contextual bandits.  The algorithm samples a reward model from a modified posterior that (i) weights squared-loss terms by the per-round noise variance and (ii) adds a “feel-good” exploration bonus.  A new **generalized decoupling coefficient** \\(dc_{\\lambda,\\varepsilon,T}\\) is defined to control the dependence between the sampled model and the action it optimizes; it upper-bounds to \\(O(d)\\) for linear models and to the generalized Eluder dimension for arbitrary function classes.  The authors prove an expected regret which matches the best known UCB-style bounds for linear bandits and collapses to \\(O(\\tilde d)\\) in deterministic settings.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths and Weaknesses\n**Quality** – The theoretical analysis is careful: bounds hold for finite function classes. Matching optimal variance-dependent regret for linear bandits closes a known gap between UCB and Thompson-sampling methods.  However, the work is entirely theoretical; there is no empirical demonstration.\n\n**Clarity** – Algorithm 1 and Table 1 are easy to follow, and key assumptions are explicit.  The proofs are long and notation-heavy; readers unfamiliar with prior FGTS analyses may struggle.\n\n**Significance** – Providing the first variance-aware, horizon-free FGTS algorithm is a meaningful advance for bandit theory.  Practical impact is limited because the algorithm assumes each round’s variance \\(\\sigma_t^2\\) is revealed. This is in contrast to other existing variance aware methods.\n\n**Originality** – Extending FGTS with variance weighting and introducing the generalized decoupling coefficient are novel ideas, though they build on established FGTS and UCB techniques.\n\nQUESTIONS:\n### Questions and Suggestions\n1. **Variance-revealing assumption** – Many real systems do not expose \\(\\sigma_t^2\\).  Could the method be adapted with online variance estimation, or maybe a doubling scheme?  Even a heuristic discussion would help.  \n2. **Computability of \\(dc_{\\lambda,\\varepsilon,T}\\)** – In practice, how would one estimate or bound the generalized decoupling coefficient for non-linear models?  Any thoughts or heuristics?\n3. **Finite vs. infinite \\(\\mathcal F\\)** – The main theorem requires a finite function class with a uniform prior. Remark 5.5 talks about the linear case with F being set to an \\epsilon net of the unit ball. What about general function classes? A complete analysis with an epsilon net argument is missing. Also in comparison to the IGW methods (https://arxiv.org/pdf/2107.02237, https://arxiv.org/pdf/2107.02237), if we assume finite arms K, would the bound only depend on K and not something like log |F| or Eluder dimension?\n4. **Empirical check** – A small simulation (e.g., linear bandit with heteroscedastic noise) comparing FGTS-VA to LinVDTS and weighted OFUL would demonstrate that the theory transfers to practice.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes FGTS-VA, a new variance-aware Thompson Sampling algorithm for contextual bandits that achieves optimal regret bounds. The key innovation is incorporating variance-dependent weights and a \"feel-good\" exploration term into the posterior distribution, allowing it to handle general reward functions while matching the performance of UCB-based methods. Theoretical analysis introduces a generalized decoupling coefficient to prove regret bounds for linear bandits, which is minimax optimal and improves upon prior TS approaches. Though requiring known variances, this work represents significant progress in making Thompson Sampling both theoretically sound and practically competitive for variance-sensitive bandit problems.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Theoretical guarantees of this paper are sound, when the variances are known, the upper bound of the algorithm matches the UCB-type algorithm.\n\n2. The writing of this paper is clear and complete, and they also mentioned the technical challenges in the proofs.\n\nWeaknesses:\n\n1. This algorithm requires knowing the variances, but for UCB-type algorithms in the field of Variance-Aware contextual bandits, there are already good algorithms like Save+, which do not need to know the variances for prior knowledge. This means that the algorithm proposed in this paper can not match the state-of-the-art in UCB-type algorithms.\n\n2. The algorithm design seems to be just a trivial combination of weighted linear regression and feel-good Thompson Sampling.\n\n3. There are no empirical studies in this paper.\n\nQUESTIONS:\n1. Is it possible to design a feel-good Thompson Sampling type algorithm without needing to know the variances?\n\n2. Could you explain in more intuitive terms how the generalized decoupling coefficient differs from standard decoupling coefficients?\n\n3. How does the feel-good exploration term in Type B posterior differ fundamentally from Type A in handling variance awareness? Why does Type A fail in this setting?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of regret minimization for contextual bandits. More specifically, they study the setting of \"weak adversary with variance revealing\" in which the noise (or an upper bound of the noise) is revealed to the learner at each time step. In that setting, previous work obtain a regret bound scaling with the cumulative variances $\\lambda$ and the dimension $d$ at a rate $d \\sqrt{\\Lambda} + d$ for UCB-based methods and at a rate $d^{1.5} \\sqrt{\\Lambda} + d^{1.5}$ for Thompson Sampling based methods. The authors bridge this gap and propose an adaptation of the Feel-Good Thompson Sampling algorithm that attains the correct scaling with respect to the cumulative variance and the dimension. To do that, they propose a new version of the decoupling coefficient, the generalized decoupling coefficient that controls the Bellman Error in a variance-aware way. Then, they propose a new analysis of the Feel-Good Thompson Sampling algorithm that doesn't rely on the usual telescoping analysis of exponential weights. This allows them to use time-dependent learning rates and control the regret of their algorithm in a variance-aware way.\n\nSTRENGTHS AND WEAKNESSES:\nThe main result of the paper shows that Thompson-Sampling based method can obtain the same variance-aware rate as UCB based methods. Moreover, as noted by the authors, their algorithm can accomodate time variying learning rates and doesn't require previous knowledge of the total variance. This results holds with a high degree of generality as it mostly requires bounded generalized eluder dimension. \nThe analysis is also insightful, they show the limitations of the usual telescoping analysis of the exponential weights algorithms when dealing with the concurrent choice of the learning rates. They show how to circumvent this limitation by avoiding the telescoping all together. The work of relating the instantaneous regret to the potential of the exponential weight is then handled by the generalized decoupling coefficient and some careful analysis. The arguments they use are (to my knowledge) new and could be used beyond the scope of this paper to analyze exponential weights algorithms.\n\nThe main limitation of the result is that the algorithms requires the knowledge of $\\sigma_t$ unlike the SAVE algorithm mentionned in previous work. Also, while the algorithm is able to scale with the cumulative variance $\\Lambda_t$, the algorithm is not anytime as only $\\nu_t $ and $\\lambda_t$ are scaling with time. The rest of the hyperparameters need to be set with knowledge of the horizon.\n\n\nThe analysis is somewhat difficult to follow in its current form. A significant portion of the main paper focuses on illustrating the failure modes of Type A FGTS, which results in many crucial elements of the new analysis being deferred to the appendix. As a consequence, the presentation feels scattered. It would greatly improve clarity to include a few introductory lines in the main text that summarize the gist of the analysis starting from the regret and Bellman Error Feel-Good decomposition, introducing the generalized decoupling coefficient, and then outlining the bounds on the LS and FG terms. For example, Line 307 is difficult to interpret without such context.\n\nAdditionally, there are some missing details, unclear references, and minor errors or typos that hinder comprehension:\n\n* In the main text, pointers to the appendix are not always clear. For instance, Line 301 states: “the second inequality holds due to Assumption 3.1 with an argument similar to (6.3). (6.7) is thus completely free of the expectation of exponential terms,” but does not clearly indicate where in the appendix the relevant derivation can be found.\n* Conversely, the appendix sometimes refers imprecisely to the main text. Line 461, for example, states: “we note that $Z \\leq \\log |\\mathcal{F}|$ using the argument in Section 6,” without a clear pointer to the specific argument.\n* In Line 121, there is a typo, the cumulative variance is not appearing in the regret bound.\n* Line 209: The decoupling coefficient should be defined such that the stated relationship holds for any $\\gamma$.\n* Line 270: The function to which Jensen’s inequality is applied is not explicitly stated. Since the function involved is the log-sum-exp, whose convexity is less immediate than that of more common functions, it would be helpful to mention this explicitly—or alternatively, use Hölder’s inequality.\n* Lemma 6.1 is cited as originating from Zhang (2023), but it is actually the well-known Donsker–Varadhan duality formula for the KL divergence.\n* Lines 192–193 mention that $\\sigma_t$ is revealed to the agent at the beginning of step $t$, and that this defines the “weak adversary” setting. This should ideally be clarified earlier in the paper, when the setting is first introduced.\n* Line 216 refers to an additional term in the generalized decoupling coefficient as arising from potentially large values of $\\beta_t$, with a promise of further explanation in Appendix A. However, the appendix contains little elaboration on this point.\n* The choice to set the hyperparameter $\\epsilon = T$ is surprising and counterintuitive, as one typically expects $\\epsilon$ to be a small quantity. This can be confusing at first (I initially assumed it was a typo). Moreover, this choice introduces an additional $\\log T$ factor in the generalized decoupling coefficient for the linear function class. When combined with the $\\log |\\mathcal{F}|$ term in the regret, it appears that the logarithmic term lies outside the square root, unlike in UCB-based methods where it is typically inside.\n\nQUESTIONS:\n- The fact that this analysis doesn't use the telescoping argument is quite surprising to me, in particular, it's not required that the learning rates are increasing or decreasing which is often the case for this type of analysis. While I agree with the technical details I would like to have a higher level understanding of the analysis. Could you provide a very high level of this part of the analyis and in particular why the telescoping is not needed anymore ?\n- If it were possible to use time-varying learning rate(and still telescope Z_t - Z_t-1) with Type-A FGTS, would it be possible to adapt to the variance ?\n- Is there a way to tune the other hyperparameters of the algorithm (like c) to obtain an anytime algorithm ? If not, what are the main limitations ?\n- Could the techniques of this paper be adapted to the \"strong\" adversary case ?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-08"}