Title: On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: policy gradient methods

reinforcement learning with general utilities
Abstract: Reinforcement learning with general utilities (RLGU) offers a unifying framework to capture several problems beyond standard expected returns, including imitation learning, pure exploration, and safe RL. Despite recent fundamental advances in the theoretical analysis of policy gradient (PG) methods for standard RL and recent efforts in RLGU, the understanding of these PG algorithms and their scope of application in RLGU still remain limited. In this work, we establish global optimality guarantees of PG methods for RLGU in which the objective is a general concave utility function of the state-action occupancy measure. In the tabular setting, we provide global optimality results using a new proof technique building on recent theoretical developments on the convergence of PG methods for standard RL using gradient domination. Our proof technique opens avenues for analyzing policy parameterizations beyond the direct policy parameterization for RLGU. In addition, we provide  global optimality results for large state-action space settings beyond prior work which has mostly focused on the tabular setting. In this large scale setting, we adapt PG methods by approximating occupancy measures within a function approximation class using maximum likelihood estimation. Our sample complexity only scales with the dimension induced by our approximation class instead of the size of the state-action space.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper establishes the global convergence of policy gradient in RL with concave utility functions. Further it provides sample complexity for the same, approximating the occupany measures.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: Theorem 1 is the most significant result of the paper. It establishes gradient domination for RLGU. Similar to linear RL, it upper bounds  the sub-optimality (a global quantity) with current  gradient (a local quantity).   Combined with sufficient increase lemma (smooth function), this result directly implies global convergece of policy gradient of RLGU with an iteration complexity of $O(\\frac{1}{\\epsilon})$ similar to linear RL. The proof for the theorem is also very short and sweet.  In my opinion, this result on its own is enough for the paper to be accepted.\n\n Weakness: (Minor) The paper misses to cite [1,2] that establishes convergence of policy gradient in finite steps (independent of state space).\n\n[1]@misc{liu2024elementaryanalysispolicygradient,\n      title={Elementary Analysis of Policy Gradient Methods}, \n      author={Jiacai Liu and Wenye Li and Ke Wei},\n      year={2024},\n      eprint={2404.03372},\n      archivePrefix={arXiv},\n      primaryClass={math.OC},\n      url={https://arxiv.org/abs/2404.03372}, \n}  \n\n[2] @misc{liu2024convergenceprojectedpolicygradient,\n      title={On the Convergence of Projected Policy Gradient for Any Constant Step Sizes}, \n      author={Jiacai Liu and Wenye Li and Dachao Lin and Ke Wei and Zhihua Zhang},\n      year={2024},\n      eprint={2311.01104},\n      archivePrefix={arXiv},\n      primaryClass={math.OC},\n      url={https://arxiv.org/abs/2311.01104}, \n}\n\nQUESTIONS:\n**Q1). Global convergence requires the mis-matchcoefficient to be fininte  which is typically satisfied by taking assuming initial state distribution $\\min_{s}\\mu(s)>0$ (Xiao 2022). Otherwise, it has been shown that policy gradient can (in linear RL) converges to some local soution [1]. The paper claims their result to be applicable to the continuous/infinite state space, it is not clear how this mismatch coefficient can be finite in these cases.**  \n\n\n[1] @misc{koren2025policygradienttreesearch,\n      title={Policy Gradient with Tree Search: Avoiding Local Optimas through Lookahead}, \n      author={Uri Koren and Navdeep Kumar and Uri Gadot and Giorgia Ramponi and Kfir Yehuda Levy and Shie Mannor},\n      year={2025},\n      eprint={2506.07054},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG},\n      url={https://arxiv.org/abs/2506.07054},}\n\n**Q2). Proposition 1 of the paper establishes sample complexity of $O(\\epsilon^{-2})$ for estimating occupation measure. I wonder, Lemma 1 of Kumar et al. 2022 that shows $\\gamma$-contracting operator for computing occupation measure (bootstraping just like Bellman operator), if this operator can help increase the sample complexity of the occupation measure?**\n\n**Q3 Could this work be extended to average reward case as studied in [4]? What could be possible challenges?**\n\n[4] @inproceedings{\nkumar2025global,\ntitle={Global Convergence of Policy Gradient in Average Reward {MDP}s},\nauthor={Navdeep Kumar and Yashaswini Murthy and Itai Shufaro and Kfir Yehuda Levy and R. Srikant and Shie Mannor},\nbooktitle={The Thirteenth International Conference on Learning Representations},\nyear={2025},\nurl={https://openreview.net/forum?id=2PRpcmJecX}\n}\n\n**Q4). (Bonus question) What do authors think about the sample complexity of single-time actor critic RELU as studied in [2,3]?**\n\n[2]@misc{chen2024finitetimeanalysissingletimescaleactorcritic,\n      title={Finite-time analysis of single-timescale actor-critic}, \n      author={Xuyang Chen and Lin Zhao},\n      year={2024},\n      eprint={2210.09921},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG},\n      url={https://arxiv.org/abs/2210.09921}, \n}\n\n[3]@misc{kumar2025convergencesingletimescaleactorcritic,\n      title={On the Convergence of Single-Timescale Actor-Critic}, \n      author={Navdeep Kumar and Priyank Agrawal and Giorgia Ramponi and Kfir Yehuda Levy and Shie Mannor},\n      year={2025},\n      eprint={2410.08868},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG},\n      url={https://arxiv.org/abs/2410.08868}, \n}\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses reinforcement learning with general utilities, in which the objective is a (potentially non-linear) function of the state-action occupancy measure. The paper provides a global optimality result for policy gradient when the utility is concave and the MDP is tabular through gradient domination technique. Next, the paper addresses the same problem beyond the tabular setting, where the state-action occupancy measure is estimated via maximum likelihood within an approximating class. In this more challenging settings, first-order and global optimality guarantees, together with corresponding sample complexities, are derived under some more restrictive assumptions.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths.**\n- RLGU is a framework that unifies a large body of settings of interest for RL, so any advancement on its understanding is potentially valuable;\n- The global optimality results for PG in tabular and larger scale RLGU settings are novel to the best of my knowledge;\n- The paper makes good use of recent advancements in the analysis policy gradient for RL to establish more general results beyond linear utilities;\n- The paper is well-written, with a mostly solid presentation.\n\n**Weaknesses.**\n- For both the tabular and larger scale settings, the significance of the results could be discussed in more details. It is not fully clear to which extent the reported results are a direct implication of previous results in RLGU and analysis of PG. While there might be important value in stating those results explicitly, still showing that the combination of what is previously known is non-trivial would be desirable;\n- The tabular setting could enjoy a more direct comparison with the analysis of the linear utility case;\n- The results with occupancy measure approximation is built upon a long string of assumptions. Some are more reasonable than others, but it would help to give more intuition/explanations on when they hold, why they are important, and how they might be overcome;\n- Some slippery details are briefly mentioned in the text and then buried in the appendix, most of all the need for i.i.d. state samples for MLE of the occupancy measure.\n\n**Evaluation.**\nOverall this looks like a solid and technically sound paper on a very interesting topic. While the readers who are familiar with previous results in RLGU may find some of the results not particularly surprising, seeing them stated explicitly and neatly in the theorem statements is great. However, as it is often the case in these kind of analysis, I think the assumptions are the core of the results, and I would like to see more discussion on them before judging the extent to which the paper is advancing the literature here. I am currently providing a borderline score, but I am open to increase if the authors can reassure me on the generality and strength of their results.\n\nQUESTIONS:\n**Tabular setting.**\n\n(1) I have some confusion about the strength of gradient domination result. From my understanding, Theorem 1 is saying that gradient domination for every pseudoreward gives a global optimality result. This makes a lot of sense, as Hazan et al. have established that RLGU can be cast as a sequence of (linear) RL problems, for which gradient domination leads to global optimality of PG as know from previous works. However, the question here is how much the distribution shift constant is bigger here w.r.t. standard gradient domination. From what I know, the constant is usually thought with $\\mu$ as the state distribution of the initial policy parametrization, which needs to cover an optimal policy state distribution. Here, we need a distribution $\\mu$ that covers all of the policies that are optimal for some pseudoreward (?) In general, this looks like the set of all the deterministic policies, which would make the constant very large and the result somewhat vacuous. Am I missing something? I would recommend the authors to make an explicit comparison between the gradient domination analysis in RL (e.g., Lemma 4.1 in Agarwal et al., 2021) and RLGU in the manuscript.\n\n**Large-scale setting.**\n\n(2) Assumptions. Understanding the assumptions here is crucial to assess the generality of the results and when do the guarantees hold. However, the discussion of the assumptions is somewhat lacking in my opinion. Realizability (Assumption 2) is ubiquitous in RL theory for value function estimation. I am wondering how stronger it can be for occupancy estimation. Assumption 4 also could be explained beyond saying \"This assumption captures most of the problems of interest in RLGU and beyond\". Why is that the case? Assumption 5 is quite hard to process, I don't think I understood the implications of any of these. Finally, the $\\epsilon_{MLE}$ assumption introduced in the statement of Theorem 3 warrant some discussion. Isn't this implying that the algorithm doesn't really converge ($\\epsilon_{MLE}$ is just a constant error that is not scaled by N)? Moreover, why $\\epsilon_{MLE}$ doesn't appear in Corollary 2? Shall not that be something like \"for any precision $\\epsilon > \\epsilon_{MLE}$...\"?\n\n(3) Occupancy measure estimation. Assuming i.i.d. sampling is somewhat problematic. It means $n$ is actually the number of trajectory, not state-action pairs. The authors may consider making this more explicit in the paper, adding proper factors to the sample complexity results (at least an $H$ factor shall be needed). Perhaps the results could look be refined by using more refined analysis for  sampling and estimation from Markov chains. For the undiscounted case \"Hoeffding's inequality for general Markov chains and its applications to statistical learning\" Fan et al. 2021. For the discounted case \"A tale of sampling and estimation in discounted reinforcement learning\" Metelli et al. 2023.\n\n(4) I am not very familiar with the analysis of Barakat et al. 2023. The paper here says that this previous work doesn't provide global optimality guarantees but just first-order. However, I am wondering whether their analysis would translate to global optimality when the same assumptions of this paper are considered. Aside from MSE and MLE step, what makes Barakat et al. analysis break for global optimality?\n\n**Other minor questions/comments.**\n\n(5) The results in the paper are provided for the discounted setting. In principle, can we use the same technical tools to study undiscounted settings, with finite or infinite horizon?\n\n(6) \"Assumption 2 is satisfied for instance for the class of generalized linear models\". Well, this is not supported when realizability is included in the assumption. Better to clarify in the text.\n\n(7) The discussion of the related work looks mostly complete. Perhaps \"Regret minimization for reinforcement learning with vectorial feedback and complex objectives\" Cheung 2019 and the pair \"Submodular reinforcement learning\" Prajapat et al. 2023, \"Global reinforcement learning: Beyond linear and convex rewards via submodular semi-gradient methods\" De Santi et al. 2024 could also be mentioned.\n\nLIMITATIONS:\nOn the limitations, see questions above. Potential negative societal impact are not applicable at this level of abstraction.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies reinforcement learning with general utilities (RLGU), where in a discounted Markov decision process (MDP), the objective is to maximize a utility function $F$ defined on the occupancy $\\lambda^\\pi$. Specifically, $\\lambda^\\pi (s, a)$ is the normalized discounted occupancy of $(s, a)$ under policy $\\pi$, and the utility is $F(\\lambda^\\pi)$. This work first establishes the gradient domination theorem for tabular MDPs and $F$ which is concave in $\\lambda$ (not $\\theta$, the parameter of $\\pi$). This theorem can further imply $\\epsilon^{-1}$ convergence for policy gradient (PG) methods. Then, when the occupancy falls inside a $m$-dimensional parametric class, the authors propose a two-stage PG algorithm, which iteratively does MLE for occupancy and policy update using the estimation. Under mild assumptions, the authors establish a $m \\epsilon^{-2}$ convergence to stationary points for general utility, and a $m \\epsilon^{-4}$ sub-optimality gap for concave utility.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- **Theories:** Establishes gradient domination properties for RLGU objectives, connecting them to recent advances in standard RL theory. First work to provide global optimality guarantees for RLGU beyond tabular settings. Results cover both concave and general utilities. Theorem 3 is a last-iterate global convergence guarantee which previous works fail to provide.\n- **Algorithm design:** PG-OMA is easily scalable to function classes beyond tabular.\n\n\n### Weaknesses\n- **Assumption on known utility:** This paper assumes that $F$ is known to calculate $\\nabla_\\lambda F(\\lambda)$, which is used in the \"policy gradient\" in Equations (4) and (5). While this assumption is without loss of generality for standard RL (known reward does not make problem easier), it might significantly reduce problem difficulty for more general (or even concave) utilities.\n- **Lacking empirical verification:** Did not provide empirical validations of the proposed algorithms, on either synthetic or practical environments. This weakens the importance of RLGU to real-world problems.\n\nQUESTIONS:\n1. Can you discuss about the cases of unknown $F$? For example, if the MDP is tabular and we only know that $F$ is concave, how much difficulty will removing the knowledge of $F$ impose on deriving a similar guarantee?\n\n2. Can you provide real-world examples aside from standard RL that RLGU plays an important role? Is there any existing environments or benchmarks for this problem?\n\nLIMITATIONS:\nOne limitation is the applicability of RLGU, which is from the problem side. This type of problem seems to mainly attract theoretical interests and lack practical significance, as I'm not aware of existing environments or benchmarks modeling RLGU. The other limitation is the assumption of known utility, which is from the algorithm side. There lacks a discussion on the necessity of this assumption.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

comments: SUMMARY:
Reinforcement learning with general utilities is an important topic in the RL community. This paper establishes global optimality guarantees of PG methods for RLGU in which the objective is a general concave utility function of the state-action occupancy measure. In the tabular setting, this paper provides global optimality results using a new proof technique building on recent theoretical developments on the convergence of PG methods for standard RL using gradient domination.

STRENGTHS AND WEAKNESSES:
Strengths:

This paper establishes tight bounds on the optimality gap using clear mathematical derivations and well-stated assumptions.

While prior works have analyzed convergence or approximation properties of RLGU in practice or for specific architectures, this work is among the first to offer a general analysis of global optimality, which is an important supplement in the RLGU community.

Weaknesses:

The analysis considers a single-step gradient update in the inner loop. While the authors briefly mention multi-step generalization, no formal analysis is provided. This limits applicability to practical implementations using multiple inner-loop steps.

Experiments are very limited. While they serve to confirm the theory, they offer little practical insight into real-world performance or limitations.

QUESTIONS:
Is the assumption of bounded gradients truly necessary, or could the results be generalized under gradient moment assumptions (e.g., bounded variance or sub-Gaussian tails?

Given that global optimality is approached as the inner-loop step size shrinks, how does this affect the sample efficiency and convergence speed in practice? Does the theory support any practical heuristics for tuning inner step sizes?

LIMITATIONS:
See weakness.
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-05