Title: Faster Fixed-Point Methods for Multichain MDPs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Markov decision processes

value iteration

fixed-point methods

Halpern iteration

average-reward RL
Abstract: We study value-iteration (VI) algorithms for solving general (a.k.a. multichain) Markov decision processes (MDPs) under the average-reward criterion, a fundamental but theoretically challenging setting. Beyond the difficulties inherent to all average-reward problems posed by the lack of contractivity and non-uniqueness of solutions to the Bellman operator, in the multichain setting an optimal policy must solve the navigation subproblem of steering towards the best connected component, in addition to optimizing long-run performance within each component. We develop algorithms which better solve this navigational subproblem in order to achieve faster convergence for multichain MDPs, obtaining improved rates of convergence and sharper measures of complexity relative to prior work. Many key components of our results are of potential independent interest, including novel connections between average-reward and discounted problems, optimal fixed-point methods for discounted VI which extend to general Banach spaces, new sublinear convergence rates for the discounted value error, and refined suboptimality decompositions for multichain MDPs. Overall our results yield faster convergence rates for discounted and average-reward problems and expand the theoretical foundations of VI approaches.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents new value-iteration (VI) algorithms for solving multichain Markov Decision Processes (MDPs) under the average-reward criterion---a setting known for its analytical and computational difficulties. Unlike discounted MDPs, average-reward problems lack contractivity and may yield non-unique Bellman solutions. A central challenge in the multichain setting is the navigation subproblem, where an optimal policy must guide the agent toward the most rewarding recurrent class. The authors address this with novel algorithms that exhibit faster convergence by incorporating sharper complexity measures such as the gain-dropping time \\\\(T_{\\text{drop}}\\\\). Additional contributions include improved reductions between average-reward and discounted problems, optimal fixed-point methods applicable to Banach spaces, and refined suboptimality bounds. Collectively, these results strengthen the theoretical underpinnings of VI for general MDPs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Introduction of the gain-dropping time \\\\(T_{\\text{drop}}\\\\) allows more accurate characterization of the navigation difficulty in multichain MDPs.\n 2. Proposed methods achieve optimal or near-optimal non-asymptotic convergence rates, surpassing prior works by Lee \\& Ryu and Zurek \\& Chen.\n 3. Results extend to general Banach spaces and unify various aspects of fixed-point theory and MDP planning.\n  4. Intermediate results such as discounted-to-average-reward reductions and fixed-point error analyses can benefit a broad range of applications.\n   5. The paper presents well-structured algorithms (e.g., Shifted Halpern Iteration, Warm-start Halpern-then-Picard) with precise theoretical guarantees.\n\n\nWeaknesses\n\n1. The work is entirely theoretical; practical performance on real-world or benchmark problems is not demonstrated.\n 2. While asymptotically efficient, the proposed multi-phase algorithms may incur substantial runtime in large-scale problems.\n 3. Several results assume access to solutions of both modified and unmodified Bellman equations, which may not be readily obtainable.\n 4. The technical presentation is dense, potentially limiting readability for practitioners unfamiliar with fixed-point theory or advanced MDP concepts.\n\nQUESTIONS:\n1. How sensitive are the algorithms to inaccuracies in the estimated gain vector \\\\( \\rho^\\star \\\\)? Could a poor estimate significantly affect convergence or policy quality?\n2. Many guarantees require that the initial vector \\\\( h_0 \\\\) and the target \\\\( h \\\\) satisfy both the modified and unmodified Bellman equations. How practical is it to construct such an \\\\( h \\\\) in real-world settings?\n3. To what extent do the results depend on the specific commutativity conditions of the Bellman operator? Could similar results be generalized to operators lacking this structure?\n4. The warm-start procedures (e.g., using \\\\( T^{(t)}(0) \\\\)) are critical to performance. Are there other initialization schemes that could yield better empirical performance or simplify implementation?\n5. The value of \\\\( \\gamma = 1 - \\frac{1}{n} \\\\) plays a crucial role in performance. Could the algorithm adapt \\\\( \\gamma \\\\) dynamically during training based on observed convergence behavior?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents new iterative algorithms to compute the gain in general finite MDPs and the optimal value in discounted MDPs.\nThe algorithms combine Picard iterates and Halpern iterates to alleviate the non-contraction of the Bellman operator.\nIn both cases, the error bounds are within small factors to lower bounds.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper uses a well thought combination of classical (Picard) and Cezar-type (Halpern) iterates that seem to be innovative.\nMaybe a discussion on this coud be added.\n\n\nBoth algorithms  (discounted  and undiscounted) have  marginally better errors bounds than existing results as they introduce a new complexity parameter T_{drop} \nthat is smaller than existing ones, although it is of the same order in the worst cases.\n\nThe definition of T_{drop} is qualitative and not quantitative: why not take the average gain of the actual difference (max with 0) instead of the indicator that looks less sharp?\n\nQUESTIONS:\nMaybe a numerical section proving experiments on \"generic\" MDPs would show that the gap with previous approaches is not marginal?\n\n\nThe remark made in conclusion about the computability of Bellman operators is quite intriguing and deserves some explanations.\nThe reader does not understand what the authors have in mind.\nFor example, the case of compact action spaces is quite different because solutions of Bellman equations are not always optimal in that case.\nThis also rises the question of time and space complexity of the proposed algorithms.\n\nLIMITATIONS:\nThe main limitation is the incremental character of the paper. The discussion on previous work is precise and explains  the difference between both bounds. However, this difference can be marginal and can even disappear in some cases. \n\n\nAnother point of interest, that  is not discussed at all, is the use of alternative methods that do not use VI to compute the gain and value of the MDP.\n\n\nAn average or even smooth complexity analysis of fixed point methods in MDPs is also missing. That would have a more significant impact in practical uses.\n\nThe topic of this paper is borderline with respect to the main subject of NeurIPS (all aspects of Machine Learning).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-23"}

comments: SUMMARY:
They explicitly decompose the multichain average reward MDP problem into the “navigation/transient” subproblem (reaching the best recurrent class) and the “recurrent” subproblem (optimizing long-run reward within a class). Accelereated VI schemes are proposed and lower bounds are matched to show optimality of the theoretical convergence rate.

STRENGTHS AND WEAKNESSES:
Strengths

- Halpern‑then‑Picard achieves the lower bounds in Banach spaces; immediately drives faster VI for multichain MDPs.
- Lemma 4.1 ties average‑reward error to discounted residual without exploding terms, enabling a practical $\gamma$‑tuning recipe.

Weaknesses

- Convergence relies on a Blackwell‑optimal policy that simultaneously maximizes both Bellman equations and on knowing n≫1/Δ; these caveats are scattered in footnotes/appendix. 

- Notation is heavy - for example, nine new symbols appear in Section 2; key ideas (why Halpern warm‑start solves navigation) are buried on the other hand.

- The link to practical RL speed‑ups is weak. Bounds carry huge factors and require preset phase lengths based on unknown parameters, and no adaptive rule is proposed.

- Claims of “faster” remain theoretical; paper omits timing comparisons on even toy multichain MDPs.

QUESTIONS:
- It would greatly enhance the paper's impact by constructing a motivating example in the introduction why multichain average reward is really a pressing subject to study instead of discounting and weakly communicating MDPs in an RL setting.
- Numerical examples for the proposed algorithm would also greatly help.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a finite-time convergence analysis for value iteration in general (multichain) MDPs under the average-reward setting. Unlike traditional value iteration, which relies on the contraction property of the Bellman operator (typically present in discounted settings), the average-reward case lacks contraction, rendering standard analysis techniques inapplicable. While prior works have attempted to address this challenge using complexity parameters, their guarantees often degrade to worse than $\\mathcal{O}(1/n)$ in the general MDP setting. This paper introduces a novel complexity measure, gain-dropping time, which is both theoretically analyzed and shown to yield sharper convergence bounds. In addition, the authors develop new average-to-discounted reductions and establish optimality results for fixed-point methods in general Banach spaces.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths** \n* This paper introduces a novel complexity measure, gain-dropping time, to analyze the convergence rate of value iteration in multichain average-reward MDPs. Based on this measure, the authors derive convergence bounds that improve upon prior work.\n\n* The motivation is clearly stated, and the limitations of existing approaches are thoroughly discussed.\n\n* A new reduction technique is proposed that connects average-reward problems to discounted ones. This enables better bounds on the optimality gap under general MDP settings.\n\n\n**Weaknesses**\n* The paper is entirely theoretical. While the results are strong, empirical experiments demonstrating faster convergence compared to existing baselines would significantly strengthen the paper.\n\n* The convergence guarantees rely on the assumption that the bias vector $h$ satisfies both the modified and unmodified Bellman equations. Although Theorem D.4 discusses special cases (e.g., when $h$ is constant), a more general constant bound on $h$ would enhance the practical interpretability of the results.\n\nQUESTIONS:\n* A proof sketch for key results (e.g., Theorems 3.4 and 4.5) would improve readability and help clarify the paper's novel analytical techniques.\n* Lemma F.5 & F.6 shows that $T_{drop}$, which is asymptotically better than either bound in general. However, could $T_{drop}$ still be on the order of $in$ the worst case? Clarifying the tightness of this bound would be helpful.\n* The concept of gain-dropping time is quite interesting. Is it possible to extend this notion to analyze sample complexity or regret bounds in the offline or online reinforcement learning settings?\n\nLIMITATIONS:\nThe authors acknowledge that exact computation of Bellman operators may be infeasible in large-scale MDPs, and stochastic approximation may be necessary. Another limitation is that although the authors show that $T_{drop}$, no absolute constant upper bound is provided. As such, practical estimation or bounding of $T_{drop}$​ remains an open issue.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}