Title: Regret Analysis of Average-Reward Unichain MDPs via an Actor-Critic Approach
Venue: NeurIPS 2025
Accepted: yes
Keywords: Actor-Critic

Regret

Policy Gradient

Average-reward MDP

Unichain MDP
Abstract: Actor-Critic methods are widely used for their scalability, yet existing theoretical guarantees for infinite-horizon average-reward Markov Decision Processes (MDPs) often rely on restrictive ergodicity assumptions. We propose NAC-B, a Natural Actor-Critic with Batching, that achieves order-optimal regret of \$\tilde{O}(\sqrt{T})\$ in infinite-horizon average-reward MDPs under the unichain assumption, which permits both transient states and periodicity. This assumption is among the weakest under which the classic policy gradient theorem remains valid for average-reward settings. NAC-B employs function approximation for both the actor and the critic, enabling scalability to problems with large state and action spaces. The use of batching in our algorithm helps mitigate potential periodicity in the MDP and reduces stochasticity in gradient estimates, and our analysis formalizes these benefits through the introduction of the constants $C_{\text{hit}}$ and $C_{\text{tar}}$, which characterize the rate at which empirical averages over Markovian samples converge to the stationary distribution.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a natural actor critic algorithm with batching for average-reward Markov Decision Processes, and for the first time weakens the existing strong ergodicity assumptions to the unichain assumption and still preserves the order-optimal regret.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This work studies average-reward Markov Decision Processes, an important problem, and makes a fundamental breakthrough in the regret analysis by reducing the existing strong ergodicity assumptions to the unichain assumption while still preserving the order-optimal regret. The non-trivial challenges of transient states are solved by their novel proof techniques, such as using the convergence rate of transition kernel running average (Lemma 2) and conditioning on high probability event of bounded stopping time (Theorem 2). The general proof outline looks reasonable (I did not check details). The presentation is in general very clear. \n\nWeaknesses: A few points may need clarification or correction as shown in the questions below. This work could be stronger using experimental comparison with existing algorithms for average-reward MDPs.\n\nQUESTIONS:\n(1) Why are some actor-critic algorithms [26, 34, 14] and unichain settings [3, 20] excluded from Table 1? You could mention the properties of these works that do not fit the caption of Table 1, such as \"model-free\", \"infinite-horizon\" and \"average-reward\". \n\n(2) At the beginning of Section 4.1, is $b_v(\\theta, z)$ defined in (14)? (16) defines $b_u$ with 3 inputs. \n\n(3) Is the outer product defined as $a\\times b=ab^{\\top}$ or $ba^{\\top}$? You could define in the paper to be more clear. \n\n(4) In the proof of Lemma 2, should $T_{\\theta}$ be replaced with $T_{\\theta}-1$ since $s_{T _ {\\theta}}\\in\\mathcal{S} _ {\\mathcal{R}}^{\\theta}$? $\\mathbb{E} _ {\\theta}$ be conditioned on $\\mathcal{F} _ {T_{\\theta}}$ based on strong Markov property?\n\nLIMITATIONS:\nLinear function approximation of critic is assumed, which could be extended to more general function approximation. Also, the conclusion section mentions future directions to extending this analysis to other useful settings such as constrained MDPs, multi-agent RL etc. These indicate the limitation that the current analysis only focuses on very basic setting.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study infinite-horizon average-reward reinforcement learning under the assumption that the learner has access to a policy class and a feature mapping that enables linear approximation of the value functions for all policies in the class. A closely related prior work proposed an actor-critic algorithm for this setting but relied on a strong ergodicity assumption. This paper relaxes that requirement to a weaker unichain assumption by introducing a novel batch algorithm for gradient estimation and providing the associated analysis. The key to the analysis is handling the set of transient states.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper presents its contributions clearly. The proposed batching strategy for gradient estimation, along with the accompanying analysis under the unichain assumption, is novel and well-articulated.\n\nWeaknesses:\n- While the paper offers a novel contribution, its overall novelty and significance appear limited, as the algorithmic structure closely resembles that of MLMC-NAC ([14], ICML 2025).\n- The exposition could be improved for clarity. Understanding the proposed algorithm currently requires substantial prior knowledge of policy gradient methods in the average-reward setting. In particular, elaborating on the phrase \"follows standard constructions in Temporal Difference\" (Line 201) and incorporating relevant content from Appendix B into the main text would enhance accessibility.\n\nQUESTIONS:\nQ1) The proposed algorithm resembles MLMC-NAC [14], with a difference in how the inner loop size is determined. Specifically, MLMC-NAC samples the inner loop size from 2^Geom(1/2) while the proposed algorithm in this submission fixes the size to B. What is the intuition behind the difference? Is the difference necessary for the analysis?\n\nQ2) How challenging would it be to extend this work to settings where only the Bellman optimality equation is assumed? Since not all policies are exploratory under this weaker assumption, I would expect that some non-trivial algorithmic techniques are required to ensure sufficient exploration. Have there been any prior attempts to address this within the class of policy gradient algorithms?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates average-reward Markov decision processes under the unichain assumption. The authors introduce a model-free, policy-gradient algorithm called Natural Actor-Critic with Batching. A regret bound of $\\tilde{O}(\\sqrt{T})$ is proved with two parameters identified, $C_{hit}$ the worst-case expected time to enter the recurrent class, and $C_{tar}$ the expected time to reach a state drawn from stationary distribution.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. This paper proposes the algorithm NAC-B and provides the first order-optimal regret analysis for general PG in non-ergodic MDPs, which allows transient states and periodicity, for the average-reward RL problem.\n2. With the large-batch averaging, they novelly prove the total-variation bound for the time-averaged state distribution (Lemma 2).\n3. The authors propose the new techniques of building a “hit-the-recurrent-class” high-probability event and projecting updates onto the orthogonal complement of the shifting kernel to handle singular critics.\n\nWeakness:\n1. This paper comes with no experiment, especially considering the very large batch $B=O(\\sqrt{T})$ in their algorithm, which needs to be stored before performing the NPG update. Experiments are needed to verify the practical feasibility of this design.\n2. Two properties of the unichain MDP $C_{hit}$ and $C_{tar}$ might be very large. In large or poorly connected MDPs, these may scale with the size of the state space or the intermediate policies. A more detailed analysis and comparison with prior work would clarify their impact on sample complexity.\n\nQUESTIONS:\nPlease refer to the weakness\n\nLIMITATIONS:\nThis manuscript omits a societal impact discussion, but this is not a blocking issue since this paper’s theoretical focus and negligible immediate societal risk.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the natural actor-critic algorithm for infinite-horizon average-reward Markov Decision Processes (MDPs) under the unichain assumption, which allows for both periodicity and transient states. Unlike prior works that rely on ergodicity for theoretical analysis, this paper derives an $\\tilde{O}(\\sqrt{T})$ regret bound under substantially weaker assumptions. Key technical innovations include the use of batching to mitigate variance and periodicity, and the introduction of constants $C_{\\text{hit}}$ and $C_{\\text{tar}}$ to characterize convergence behavior. The algorithm employs general function approximation for both actor and critic components, enhancing scalability.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nNovelty: The paper makes a novel and substantial contribution by analyzing regret for actor-critic methods under the unichain assumption—this represents a significant step forward in understanding policy gradient methods in more general settings.\nAnalysis: The regret bound is carefully derived under general function approximation, with comprehensive proofs and clear articulation of assumptions. The techniques used to handle the challenges of periodicity and transient states are thoughtful and well-executed.\nClarity: Despite the technical depth, the writing is generally clear, and the theoretical insights are well-motivated.\n\nWeaknesses:\nWhile I do not consider the absence of experiments a major concern for a theoretical paper, it would still be valuable to evaluate the practical behavior of NAC-B—particularly its sensitivity to function approximation quality, step sizes, and batch sizes. For instance, although the use of large batches is theoretically well-justified, it could be costly in practice. A discussion or guidance on batch size selection in practical settings would help strengthen the paper.\nIn addition, although the unichain assumption is more general than the commonly studied ergodic setting, the paper would benefit from a more detailed discussion of its practical relevance. Specifically, it would be helpful to clarify in what types of real-world applications this assumption is likely to hold, as many practical environments may not satisfy the unichain condition.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nA theoretical work may not pose the societal impact as a high priority.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}