Title: Offline Actor-Critic for Average Reward MDPs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Offline

Actor-Critic
Abstract: We study offline policy optimization for infinite-horizon average-reward Markov decision processes (MDPs) with large or infinite state spaces. Specifically, we propose a pessimistic actor-critic algorithm that uses a computationally efficient linear function class for value function estimation. At the core of our method is a critic that computes a pessimistic estimate of the average reward under the current policy, as well as the corresponding policy gradient, by solving a fixed-point Bellman equation, rather than solving a successive sequence of regression problems as in finite horizon settings. This procedure reduces to solving a second-order cone program, which is computationally tractable. Our theoretical analysis is based on a weak partial data coverage assumption, which requires only that the offline data aligns well with the expected feature vector of a comparator policy. Under this condition, we show that our algorithm achieves the optimal sample complexity of  O(\varepsilon^{-2})  for learning a near-optimal policy, up to model misspecification errors.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a pessimistic actor-critic algorithm for offline reinforcement learning (RL) in infinite-horizon average reward Markov Decision Processes (AMDPs), using linear function approximation of the value function. The key technical contribution is a novel, computationally efficient critic step that avoids solving successive sequence of regression problems by directly solving a fixed-point Bellman equation using a linear optimization formulation with convex quadratic constraints. The algorithm is proven to be sample-efficient, with a near-optimal sample complexity of $\\mathcal{\\tilde{O}}(\\epsilon^{-2})$, under a weak notion of data coverage (the feature coverage ratio). This is the first such result in the offline average-reward setting that matches the best known rate and only requires for the MDP induced by all policies to be unichain.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The paper addresses the problem of offline RL infinite-horizon, average reward setting, an important gap in literature \n- The analysis requires mild assumption compared to prior literature - it requires only unichain (instead of uniformly ergodic) and a relatively weak notion of the feature coverage ratio\n- The algorithm cleverly constructs a pessimistic estimate of the average reward by solving a single fixed-point Bellman equation and address computational efficiency through the reformulation as a linear optimization with convex quadratic constraints\n- Significantly improves upon the sample complexity in comparison to prior offline RL literature and matches the best known in online RL domain\n- Well written paper with a clear presentation of the proof sketch and a through discussion of the assumptions used\n\n**Weaknesses:**\n- A discussion on the effects of scaling state and action space might be nice to include\n- Insight into how the current work may be extended to the general non-linear function approximation setting would make the paper comprehensive\n- While weaker than uniform ergodicity, the unichain assumption may not always hold. As mentioned in the section on future directions, it would be interesting to see how this affects algorithm design and analysis.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed an actor-critic offline reinforcement learning algorithm for average reward MDP, which first achieved O(\\eps^{-2}) sample complexity. As a paper focusing on the theoretical guarantee, the authors provided detailed proof and discussion on the relationship with the previous paper.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: 3\nThe main paper quality is good. The author provided an actor-critic framework to the offline RL problem. Decompose the sub-optimality into actor and critic parts. For the actor parts, the author adopts the Natural policy gradient method and achieves an optimization error convergence with order O(1/\\sqrt(k)). For the critic part, the authors applied Bellman equation regression with pessimism parameters. The proof sketch and details are solid. Moreover, the author discusses the assumption and previous work in depth, which is helpful.\n\nClarity: 2\nDespite the algorithm and proof being solid in this paper, some errors or typos may lead to misunderstanding.\n\n(1)\tIn the section 4.2 policy update, it is better to make it clear that the update follows Mirror Ascent with KL divergence or Natural policy gradient instead of general gradient ascent to make a better understanding\n(2)\tIn the proof of Lemma B.3, the notation \\bar{\\omega}_k and \\omega_k^* are mixed up? Make it hard to read the proof of Lemma B.3\n(3)\tIn the proof of Theorem 5.1, line 610, the optimization error term is missing.\n\nSignificance: 3\nThe authors discussed the convergence rate of the offline RL under the Average MDP setting, which is an important topic in RL theory and closes the gap.\n\nOriginality: 2\nAlthough the discussed problem is important to the community. However, it seems the novelty is somehow limited. The actor-critic framework is followed from [Zanette 2021]. Although I agree that [Zanette 2021] discussed the episodic setting and the induction technique is invalid in the average setting. However, it seems that the idea of proof procedure (Lemma 6.2 and 6.3) is similar to the FOPO algorithms' proof in [Wei et al, 2021]. The only difference is that [Wei et al, 2021] discusses online problems with Optimism, and this paper discusses offline problems with pessimism. \n\nIt would be great to highlight the contribution of the technical part of this paper if the authors could answer the following question.\n\nThe author mentioned that [Gabbianelli et al., 2024] is the only paper that also discusses offline RL under average MDP. However, they only achieve O(\\eps^{-4}). Could the author explain what the new idea or technique is adopted to improve the result to O(\\eps^{-2})?\n\nQUESTIONS:\nPlease refer to the clarity and Originality Parts for the questions. This paper is technically solid. I would increase the score if the author could confirm with me the question mentioned above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an actor-critic method for efficient offline RL in average reward MDPs with an intractably large or infinite number of states. They consider the linear function approximation framework, and make assumptions on the expressivity of their function approximator including approximate $q^{\\pi}$-realizability and Bellman completeness under any policy. Inspired by the actor-critic algorithm of Zanette et. al [2021] for the finite-horizon setting, for steps $k=1,...,K$ their algorithm obtains the critic’s weight $w_k$ as the solution of an optimization problem, then updates the actor parameter $\\theta_k$ such that the resulting actor policy in any state is equivalent to one step of mirror-ascent with KL regularization and critic evaluation $\\hat{q}_k(s,\\cdot)=\\langle \\phi(s,\\cdot),w_k\\rangle$ as the gradient. Similar to Zanette et al [2021], their optimization problem is constructed in each step using samples from the offline dataset, and constrains the critic weights to be equivalent to the perturbed solution of a ridge regression problem – with the sample estimate of $q^{\\pi_k}$ as the target. Furthermore, in the absence of the backward induction property enjoyed by finite-horizon MDPs, and the presence of the cumulative return $J^{\\pi}$ in the Bellman expression of $Q$-functions in the average-reward MDP setting, they propose two modifications to the optimization problem: 1) a new decision variable $J$ to estimate $J^{\\pi_k}$ and 2) a direct parametrization of an estimator for the state-value function $v^{\\pi_k}$ based on the critic’s weights. With these modifications, they prove $O(\\varepsilon^{-2})$ sample complexity guarantee under the weakest known data collection and coverage conditions. Precisely, they do not require that the dataset is generated i.i.d or by a single behaviour policy, and they only require the dataset to properly cover a single direction in the feature space. Their sample complexity guarantee also scales linearly (rather than quadratically) with the coverage parameter.\n\nSTRENGTHS AND WEAKNESSES:\nAs is evident in the long summary, I did enjoy reading the paper. The problem of offline RL in infinite-horizon MDPs with intractably many states, under weaker conditions like $q^{\\pi}$-realizability and Bellman completeness, is indeed a relevant open problem in the offline RL literature. The paper addresses this problem with an $O(\\varepsilon^{-2})$ sample complexity guarantee and their bound scales linearly with the single-direction coverage parameter, rather than quadratically – which is more common in related works.\n\nThere is no notable weakness. However, I would like to highlight that the algorithm is not fully computationally efficient. It is true that Algorithm 1 can be efficiently implemented with the help of an interior-point method to solve the optimization problem in step 5. While interior-point methods are highly precise, they are not exact. That said, the theoretical guarantees provided in the paper hold under the assumption that there exists an optimization solution which provides an exact solution to step 5. I strongly recommend that the authors add a comment to clarify this, especially in the abstract and other parts of the main text with explicit claims about computational efficiency.\n\nQUESTIONS:\nI have just one question:\n* Do the authors assume that the initial state is fixed?\n\nSome typos:\n1. L286: “However, their algorithm’s...” should be “However, their algorithm...”\n2. L299: “...while we study the more general function...” should be “...while we study the more general linear function...”\n3. After L335, the second inequality: $(s,a)\\sim\\mu^{\\pi}$ should be $s \\sim\\mu^{\\pi}$\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a pessimistic actor–critic algorithm for offline reinforcement learning in infinite-horizon average-reward MDPs (AMDPs) with linear function approximation.\n- The critic obtains a lower-bound (pessimistic) estimate of the average reward and its policy gradient by solving a single linear program with convex quadratic constraints that represents the fixed-point Bellman equation, avoiding the step-by-step regressions common in episodic settings .\n- The actor performs a conservative policy-improvement step on a soft-max policy class. Repeating the alternation $K$ times and outputting a mixture of intermediate policies yields an $\\varepsilon$-optimal policy (up to model-mis-specification error) with sample complexity $O(\\varepsilon^{-2})$—the first optimal-rate result for offline AMDPs .\n- The theory relies on (i) unichain dynamics with bounded Q-span, and (ii) approximate realizability + Bellman restricted closedness of the linear feature class .\n- Compared with the only prior provably efficient offline AMDP method, PDOR, the new algorithm improves the ε-dependence from $O(\\varepsilon^{-4})$ to $O(\\varepsilon^{-2})$ and removes the need for exact linear MDP structure .\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- Rigorous proofs culminating in Theorem 5.1 with explicit finite-sample bounds\n- Clear comparison against prior rates (PDOR, discounted-setting results).\n-  Tackles the average-reward offline regime, important for continuing tasks (inventory control, admission control) where resets are impossible\n- Optimal-rate guarantees were missing in this setting.\n\n# Weaknesses\n- Results hinge on linear approximation\n- Pessimism paradigm already popular in episodic/dis­counted RL\n\nQUESTIONS:\nThe conclusion suggests extending to “general function approximation.” What obstacles prevent directly replacing the linear critic with a neural network trained by convex-constrained regression?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}