Title: Variance-Dependent Regret Lower Bounds for Contextual Bandits
Venue: NeurIPS 2025
Accepted: no
Keywords: Bandit

Reinforcement Learning
Abstract: Variance-dependent regret bounds for linear contextual bandits, which improve upon the classical $\tilde{O}(d\sqrt{K})$ regret bound to $\tilde{O}(d\sqrt{\sum_{k=1}^K\sigma_k^2})$, where $d$ is the context dimension, $K$ is the number of rounds, and $\sigma^2_k$ is the noise variance in round $k$, has been widely studied in recent years. However, most existing works focus on the regret upper bounds instead of lower bounds. To our knowledge, the only lower bound is from Jia et al. (2024), which proved that for any eluder dimension $d_{\textbf{elu}}$ and total variance budget $\Lambda$, there exists an instance with $\sum_{k=1}^K\sigma_k^2\leq \Lambda$ for which  any algorithm incurs a variance-dependent lower bound of $\Omega(\sqrt{d_{\textbf{elu}}\Lambda})$. However, this lower bound has a $\sqrt{d}$ gap with existing upper bounds. Moreover, it only considers a fixed total variance budget $\Lambda$ and does not apply to a general variance sequence $\{\sigma_1^2,\ldots,\sigma_K^2\}$.
In this paper, to overcome the limitations of Jia et al. (2024), we consider the general variance sequence under two settings. For a prefixed sequence, where the entire variance sequence is revealed to the learner at the beginning of the learning process, we establish a variance-dependent lower bound of $\Omega(d \sqrt{\sum_{k=1}^K\sigma_k^2 }/\log K)$ for linear contextual bandits. For an adaptive sequence, where an adversary can generate the variance $\sigma_k^2$ in each round $k$ based on historical observations, we show that when the adversary must generate $\sigma_k^2$ before observing the decision set, a similar lower bound of $\Omega(d\sqrt{ \sum_{k=1}^K\sigma_k^2} /\log^6(dK))$ holds. In both settings, our results match the upper bounds of the SAVE algorithm (Zhao et al. 2023) up to logarithmic factors. Furthermore, if the adversary can generate the variance $\sigma_k$ after observing the decision set $\mathcal{D}_k$, we construct a counter-example showing that it is impossible to construct a variance-dependent lower bound if the adversary properly selects variances in collaboration with the learner.
Our lower bound proofs use a novel peeling technique that groups rounds by variance magnitude. For each group, we construct separate instances and assign the learner distinct decision sets. We believe this proof technique may be of independent interest.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides new lower bounds on the linear contextual bandit. The results are two-fold. First, they provide a lower bound to the setting where the variances are all given to the learner at the beginning; second they provide a similar lower bound when the variances are adaptive. The lower bound matches the upper bound in previous literatures.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper is well-written, and the proofs appear to be correct.\n2. The lower bounds established in this work match the existing upper bounds, thereby providing a tight characterization of the variance-aware linear contextual bandit problem.\n\nWeaknesses:\n1. The explanation of whether the learner and the adversary are in competition or collaboration is somewhat unclear. It would be very helpful if the authors could illustrate this relationship using real-world applications to provide greater intuition.\n2. This paper focuses exclusively on establishing a lower bound. However, in presenting the lower bound, the authors only provide a specific construction. It would be greatly beneficial if the paper could also discuss the types of settings or conditions under which this hard instance is likely to occur. Such insights would offer valuable guidance for future algorithm design, particularly in identifying and addressing worst-case scenarios.\n\nQUESTIONS:\nI have the following questions regarding this paper:\n1. Is it possible to generalize the results into contextual bandit with general function approximation?\n2. This paper mentioned that the lower bound does not work for stochastic linear bandit setting, e.g. when the first $d$ rounds the variances are all zero, then the optimal policy can already be learned. Can you provide a lower bound for this setting as well?\n3. The 'high probability bound' in Theorem 5.2 appears to be in the form 'with probability at least $1 - 1/K$, ...', so this means the probability of the event is fixed when $K$ is fixed. This seems a little bit weird. Can you prove a high probability bound in the form 'with probability at least $1 - \\delta$,...'?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work investigates variance-dependent contextual bandits, where the variance can be adaptively selected by an adversary who collaborates with the learner over time. Two types of adversaries are considered: (i) a weak adversary, who selects the variance without knowledge of the learner’s decision set, and (ii) a strong adversary, who has access to the decision set prior to choosing the variance. The paper also establishes a lower regret bound for the setting where the variance sequence is fixed in advance and known. Furthermore, a lower regret bound is provided for the case involving a weak adversary.\n\nSTRENGTHS AND WEAKNESSES:\nThe proposed lower regret bounds for both the pre-fixed variance sequence and the weak adversary setting are tight and represent significant contributions to the study of variance-dependent contextual bandits. Notably, these bounds match known upper bounds up to logarithmic factors. While the theoretical contributions are strong, there are several critical concerns regarding presentation and assumptions.\n\n**1. Lack of Application Context:**\nAlthough variance-dependent bandits have been studied for some time, the paper does not adequately motivate the problem with practical applications. A clear discussion of potential use cases should be included in the introduction to better frame the relevance of the problem.\n\n**2. Clarity of Presentation:**\nThe exposition requires significant improvement. Specifically, Section 1.1 (\"Our Contributions\") is difficult to follow on a first read and only becomes clearer after reading the full paper. It should be rewritten for better readability and accessibility. Additionally, the proof of Theorem 4.1, particularly lines 206–226, should be revised for clarity and structure.\n\n**3. Problematic Assumptions:**\nMore serious issues lie in the assumptions:\n\n* The condition $\\sum \\sigma_k^2 \\geq 1 + 384d^2$ effectively requires the number of arms to exceed 1000, which excludes most practical applications. This assumption is overly restrictive and should be reconsidered.\n* The assumption $\\Delta = 1/\\sqrt{96K}$ appears ad hoc and insufficiently justified. Since the regret scales with $\\Delta$, a $\\Delta$ that depends on the number of rounds $K$ undermines the meaningfulness of the lower bound. This dependency should be either removed or properly justified, as it currently weakens the claimed lower bounds.\n\nQUESTIONS:\nThe lower regret bound presented in the paper (ones in other papers as well) are turned out to depend on the simple sum of variances across time. This raises two important questions:\n\n(i) If the lower regret bound depends only on the simple sum of variances, how fundamentally different are the problems under a fixed variance budget versus those with a general (possibly adaptive) variance sequence? From a regret perspective, are they essentially equivalent, or does adaptivity introduce additional complexity not captured by the current analysis?\n\n(ii) A variance incurred in the early stages of learning may have a more pronounced impact on estimation and decision-making than one incurred later. In this sense, it seems natural that early-stage variances should be weighted more heavily than those in later stages. How, then, does it make sense to treat all variances equally in the regret bound? Is there a theoretical reason why the regret depends only on the total sum, without accounting for the temporal ordering of variances?\n\nLIMITATIONS:\naddressed in weakness", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of linear contextual bandits in the case in which the variance of the rewards changes at every time-step. Specifically it focuses on designing hard instances to prove lower bound for the case in which the sequence of variances is either given and known in advance, or chosen by an adversary either before or after having observed the decision set. Specifically, the paper provides a lower bound of the order of $\\Omega(d\\sqrt{\\sum_{i=1}^K \\sigma_k^2}/ \\log K)$, with $K$, number of rounds, when the sequence is fixed and known, an high probability bound of the order of  $\\Omega(d\\sqrt{\\sum_{i=1}^K \\sigma_k^2}/ \\log K)$, if the adversary can choose the sequence adaptively without observing the decision set. Lastly, they prove that in the case the adversary can observe the decision set beforehand there is an instance in which the regret is bounded by $d$, the dimension of the context space.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper studies a specific problem part of the vast literature on multi armed bandits which is of general interest to the community. Specifically it focuses on a less studied version of the problem in which the noise distribution is allowed to change at every time step. Hence, the topic studied an the result provided are novel.\n\nOn the other side, the paper could really be improved in terms of clarity: the writing results oftentimes confused and hard to read. Specifically, using alternatively the notation $\\sigma^2$ and $\\sigma$ to indicate the variance appear confusing, but mostly the choice of using $K$ instead of $T$ as the number of rounds, as it is well-established in the field, is, in my opinion, very counter intuitive.\n\nFurthermore, I have some doubts about the soundness of the proofs that I defer to the Questions section.\n\nQUESTIONS:\n1. Could you further elaborate what is Theorem 5.4 showing? I understand the proof but to my understanding it is only showing a particularly favorable instance of the problem in which the regret obtained is very low, and I do not understand the claim maid at lines 307-308 that this is meant to break the lower bound.\n\n2. It is not clear to me what are the hard instances used to derive the bound of Theorem 4.1. \n\n3. At line 224, isn't the value $\\sigma^2(i)$ provided a lower bound on the variance of the group $i$, instead of an upper-bound as stated? Furthermore what is the value $\\sigma_k$  which bounds the variance in all the groups?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The authors investigate variance-dependent lower bounds for contextual bandits. In the stochastic setting, their results improve the dependence on the variances and the dimension $d$ compared with existing bounds. They also study adversarial settings and establish impossibility results in certain cases.

STRENGTHS AND WEAKNESSES:
The paper addresses variance-dependent analysis in multi-armed bandits, an important topic in the field. While the contribution is solid, several points remain unclear.
- In Equation (3.1) the authors assume bounded errors. Does this assumption exclude Gaussian noise and location-shift models? In addition, how is the variance of the worst-case mean rewards defined, given that the variance may also depend on the means?
- For best-arm identification, prior work such as Lu et al. (2021), Lalitha et al. (2023), and Kato et al. (2024) provides variance-dependent lower bounds. Can those results be applied to your setting, or can your results be transferred to theirs?
- Ito et al. (2022) develop variance-tight algorithms for linear bandits. Are their techniques applicable to your framework?

Related references:
- Lu, P., Tao, C., and Zhang, X. “Variance-Dependent Best Arm Identification,” UAI 2021.
- Lalitha, A., Kalantari, K., Ma, Y., Deoras, A., and Kveton, B. “Fixed-Budget Best-Arm Identification with Heterogeneous Variances,” UAI 2023.
- Kato, M., Okumura, K., Ishihara, T., and Kitagawa, T. “Adaptive Experimental Design for Policy Learning.”
- Ito, S., Tsuchiya, T., and Honda, J. “Adversarially Robust Multi-Armed Bandit Algorithm with Variance-Dependent Regret Bounds,” COLT 2022.

QUESTIONS:
See above.

LIMITATIONS:
N/A.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03