Title: Sharp Gap-Dependent Variance-Aware Regret Bounds for Tabular MDPs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement learning

gap-dependent

variance-dependent

instance-dependent
Abstract: We consider gap-dependent regret bounds for episodic MDPs.
We show that the Monotonic Value Propagation (MVP) algorithm (Zhang et al. [2024]) achieves a variance-aware gap-dependent regret bound of 
$$\tilde{O}\left(\left(\sum_{\Delta_h(s,a)>0} \frac{H^2 \log K \land \mathtt{Var}\_{\max}^{\textup{c}}}{\Delta_h(s,a)} +\sum_{\Delta_h(s,a)=0}\frac{ H^2 \land \mathtt{Var}\_{\max}^{\textup{c}}}{\Delta_{\mathrm{min}}} + SAH^4 (S \lor H) \right) \log K\right),$$ 
where $H$ is the planning horizon, $S$ is the number of states, $A$ is the number of actions, $K$ is the number of episodes, and $\tilde{O}$ hides $\mathsf{poly} \log (S, A, H, 1 / \Delta\_{\mathrm{min}}, 1 / \delta)$ terms. Here, $\Delta_h(s,a) =V_h^* (a) - Q_h^* (s, a)$ represents the suboptimality gap and $\Delta_{\mathrm{min}} := \min_{\Delta_h (s,a) > 0} \Delta_h(s,a)$.
The term $\mathtt{Var}\_{\max}^{\textup{c}}$ denotes the maximum conditional total variance, calculated as the maximum over all $(\pi, h, s)$ tuples of the expected total variance under policy $\pi$ conditioned on trajectories visiting state $s$ at step $h$. 
$\mathtt{Var}\_{\max}^{\textup{c}}$ characterizes the maximum randomness encountered when learning any $(h, s)$ pair.
Our result stems from a novel analysis of the weighted sum of the suboptimality gap and can be potentially adapted for other algorithms.
To complement the study, we establish a lower bound of 
$$\Omega \left( \sum_{\Delta_h(s,a)>0} \frac{H^2 \land \mathtt{Var}\_{\max}^{\textup{c}}}{\Delta_h(s,a)}\cdot \log K\right),$$ 
demonstrating the necessity of dependence on $\mathtt{Var}\_{\max}^{\textup{c}}$ even when the maximum unconditional total variance (without conditioning on $(h, s)$) approaches zero.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates gap-dependent regret bounds for tabular episodic Markov Decision Processes (MDPs).\n\nThe authors demonstrate that the previously proposed MVP algorithm can attain a variance-aware, gap-dependent regret bound.\n\nThey also establish a matching lower bound, highlighting the tightness and complementarity of both the upper and lower bounds.\n\nThe proofs use a novel analytical approach.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper effectively identifies an important gap in the RL literature: achieving the tightest problem-dependent regret bounds while accounting for both variance and gap.\n\nThe concept of maximum conditional total variance is novel to me.\n\nThe writing is clear and accessible, making the main contributions easy to understand. The related work section provides a well-structured comparison with prior approaches, helping readers contextualize the significance of the results.\n\nNew notations and concepts are introduced in a clear and organized manner. The overall structure of the paper (such as including a proof highlights section in the main text, as well as dedicated sections for notation and technical lemmas in the appendix) greatly enhances readability.\n\nOne limitation is that the MVP algorithm is restricted to tabular MDPs. It would be exciting to see future work that extends these ideas to more general settings beyond the tabular case.\n\nQUESTIONS:\nIn the abstract, should the definition of delta_min be restricted to (h,s,a) that delta_h(s,a) > 0 as is discussed in more detail later in the part of the paper?\n\nIn Definition 2, the summation starts from h'=1 and it is conditioned on a potentially later step s_h, which feels somewhat unconventional. Could you provide some intuition on why this formulation is important and how it plays a role in the regret bound?\n\nIn line 181, should the reference be to Definition 2?\n\nI’m particularly interested in the gap between the upper and lower bounds presented in the paper. Line 261 refers to Appendix H for further discussion, but I couldn’t locate it in the supplementary material. Could you provide clarification or additional insights on this point?\n\nLIMITATIONS:\nnot needed", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses regret bounds of tabular MDPs. They focus on providing gap-dependent regret bounds that are also variance-aware. They study the dependence of regret on the maximum conditional total variance (conditioned on the states) instead of the maximum unconditional total variance studied in existing literature. They show that the existing Monotonic Value Propagation (MVP) algorithm enjoys improved regret bounds. A lower bound on the gap-dependent regret is also provided.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper studies the dependence of regret on the maximum conditional total variance instead of the maximum unconditional total variance studied in existing literature, and shows how it allows a better regret bound for an existing algorithm (Monotonic Value Propagation). \n- The introduction of the conditional variance allows improvement in dependence on $H$.\n- The techniques used in this paper can be used to improve regret bounds in a variety of RL settings. \n\n\n**Weaknesses**\n- The clarity and structure of the paper can be improved. The paper only includes an informal version of theorems (upper bound and lower bound theorems. This is acceptable if the primary contribution of a paper is algorithmic or experimental. The primary contributions of this paper are regret bounds (upper / lower bounds), so it would be a better idea to include complete formal theorems in the paper. If space is a concern, authors can move the MVP algorithm to the appendix, as the algorithm is not a contribution of this work.\n\nQUESTIONS:\n- Why did the authors choose to present only the informal version of the regret bound in the main paper and leave the formal theorem to the appendix? The formal version of the lower bound is particularly different than the informal version presented in Theorem 3. I strongly recommend including the complete technical theorem in the main paper. \n\n- While the upper bound proof of the regret is well written, it was challenging to follow the presentation of the lower bound, due to the following reasons. \n\n1. A lot of details (including assumptions) are missing from the informal version of the lower bound theorem (Theorem 3).  \n\n2. Instead of giving a summarized, intuitive idea of the proof in the proof sketch, a simplified instance is discussed, which does not give much insight into how the instance can be generalized. \n\n3. Even though I am well aware of the multi-armed bandits lower bound proof (Lai & Robbins, 1985) utilized in appendix C, it was difficult to see how to it utilized to get a lower bound here, I request authors to expand and provide more detailed and easy to follow proof of the lower bound in the appendix.\n\n\n- The abstract includes the mathematical expression for regret bounds. Mathematical expressions are not displayed properly when the abstract is shown outside the PDFs (for e.g., openreview website). Hence, the authors might want to consider rewriting the abstract with fewer (or none) mathematical expressions to improve readability. \n\n- As evident in the questions, my primary concern with the paper is with the writing of Section 5 and the corresponding Appendix C; the score given currently is a reflection of that.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new theoretical analysis of the Monotonic Value Propagation (MVP) algorithm, establishing the first variance-aware, gap-dependent regret bound using a newly introduced metric: the maximum conditional total variance. The analysis achieves a state-of-the-art worst-case horizon dependence of H^2 in the regret bound, improving over prior work. The authors also prove a matching lower bound, showing that no algorithm can significantly outperform MVP in this setting. Key technical contributions include a novel reweighting analysis of suboptimal gaps and a refined clipping strategy for optimal actions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n**Quality**\n- Overall, the paper is very well written. \n- The arguments are supported by cleanly organized proofs.\n\n**Clarity**\n- The paper is written clearly, terms and concepts are clearly explained or defined. \n- Concepts such as suboptimality gaps, conditional variance, and the role of the MVP algorithm are well-motivated and easy to follow for readers with background in RL theory.\n\n**Significance**\n- The paper studies the important question of \"What is the tightest problem-dependent regret while considering both variance and gap?\". Obtaining such a bound can be helpful for the further adaptation of new and potentially better RL algorithms such as MVP. \n- The theoretical results are very interesting, providing a tight bound on MVP can be a significant result\n- The tight lower bound further strengthens the impact by showing near-optimality.\n- The new techniques discussed in the paper can be useful for other theoretical works. \n\n\n**Originality**\n- The theoretical results are new, the new techniques that are used to prove the results are also novel\n\nWeaknesses\n\n**Significance**\n- While the theoretical results are impressive, the paper does not include any empirical evaluation, whether on small toy problems or other more complex settings. It is unclear how these theoretical improvements translate to practical reinforcement learning problems. But I understand the main goal of this paper is to provide theoretical contributions.\n\nQUESTIONS:\n- The favorable bounds of the MVP algorithm shown in this work is very interesting, however, does this tight bound mean the MVP algorithm is very close to an optimal algorithm? Do you think we should still try to develop newer RL algorithms, or just focus on adapting existing ones into practical problems? \n- Is there any empirical evidence showing that MVP works much better than alternative RL algorithms? (perhaps even in some special MDPs)\n- How easily will the analysis made in the paper be extended to other more general MDP settings? \n- Does these theoretical results bring insight on how to better adapt MVP (or similar model-based algorithms)  to settings with function approximation or large state spaces?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The authors introduce the notion of conditional total variance into a new instance dependent (bounds depend on the gap of the MDPand total conditional variance quantity) performance bounds of the MVP algorithm of Zhang et al 2024. The paper also has a matching lower bound, which is the first in this style of performance bounds.

STRENGTHS AND WEAKNESSES:
S:
1)The paper is extremely well written and easy to follow. Clarity and organization is very good.
2)The lower bounds are a definitely an exciting addition to the literature.
3)The proof techniques seem novel to me.

W:
1)A detailed discussion about the significance of conditional variance is needed. e.g., what are some cases where conditional variance is significantly small? (better still , small experiments to depict this intuition)
2) Why is this a tighter or more appropriate notion than unconditional total variance (Def 1). 
3) How does the MVP algorithm behave with small \Delta_min.

QUESTIONS:
Can the authors comment if there are other lower instance dependent lower bounds which may not be "variance-aware" ? How does their lower bounds compare with those? 

Do the authors make any changes to MVP algorithm ? or one should consider this paper as proving properties for the MVP algorithm?

LIMITATIONS:
lack of empirical validation, potentially limited impact
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-03