Title: Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents
Venue: NeurIPS 2025
Accepted: yes
Keywords: policy search

optimized certainty equivalents

risk aware learning
Abstract: Constrained optimization provides a common framework for dealing with conflicting objectives in reinforcement learning (RL). In most of these settings, the objectives (and constraints) are expressed though the expected accumulated reward. However, this formulation neglects risky or even possibly catastrophic events at the tails of the reward distribution, and is often insufficient for high-stakes applications in which the risk involved in outliers is critical. In this work, we propose a framework for risk-aware constrained RL, which exhibits per-stage robustness properties jointly in reward values and time using optimized certainty equivalents (OCEs). Our framework ensures an exact equivalent to the original constrained problem within a parameterized strong Lagrangian duality framework under appropriate constraint qualifications, and yields a simple algorithmic recipe which can be wrapped around standard RL solvers, such as PPO. Lastly, we establish the convergence of the proposed algorithm and verify the risk-aware properties of our approach through several numerical experiments.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a risk-aware constrained RL problem with both reward-based objective and constraints, for which this work established parameterized strong duality relation, online primal-dual algorithm and its convergence result, and verified the convergence by experimental results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: I can understand this work clearly. The risk-aware RL problem is popular and important for safe reinforcement learning. The algorithm and theoretical results are clear and reasonable based on my knowledge about minimax optimization and reinforcement learning. \n\nWeaknesses (See my questions below for details): The major concern is that problem novelty is not as strong as claimed. The convergence result does not directly imply the quality of policy that we are interested in. Some points need clarification.\n\nQUESTIONS:\n(1) **My major concern** is in the setting novelty. \"Our work is the first to handle reward-based constraints, covering a large class of risk measures (OCEs)\" in the introduction looks like an overclaim, since [12, a] use reward-based constraints. \n\nYour setting seems to be a special case of the risk measure-constrained RL (RCRL) [a], by restricting the function $g$ in Eq. (2) of [a] to $\\{g(\\cdot-t):t\\in\\mathbb{R}\\}$ for a fixed $g$. \n\nCould you cite [a] and describe your advantage over [a]? \n\nAlso, you present your results in CVAR and claim they hold for general OCEs at the beginning of Section 3.1. Can existing CVAR results like [12] also extend to general OCEs? \n\n**I would like to raise my rating if you can solve this concern.**\n\n[12] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18, 12 2015.\n\n[a] Kim D, Cho T, Han S, et al. Spectral-Risk Safe Reinforcement Learning with Convergence Guarantees, Neurips 2024.\n\n(2) To ensure Lemma 3.2 holds for general OCEs, what conditions should the function $g$ satisfy? \n\n(3) Could you define $\\widehat{\\nabla} _ {t,\\lambda}$ in Assumption 3.6? You could add the definition to your revised paper. \n\n(4) How can Theorem 3.12 indicate the quality of the output policy? \n\n(5) To make the experiments reproducible, could you explicitly define the state $s$, action $a$, reward function $r$ (in the objective)? \n\nMinor issues: \n\n(6) Abstract line 3: \"through\". \n\n(7) You could describe the Slater's condition, possibly by an equation.\n\nLIMITATIONS:\nYes, the authors have written their limitations in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a risk-aware constrained reinforcement learning framework that incorporates reward-based risk measures into the objective and constraints. It extends prior work by enabling constrained settings and establishes a strong duality result that allows for an exact partial Lagrangian relaxation. The authors develop a modular online algorithm compatible with various RL methods and provide theoretical convergence guarantees. Experiments on standard benchmarks demonstrate improved risk management and reduced constraint violations.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths\n  - The proposed method ensures an exact equivalence to the original constrained problem via parameterized strong Lagrangian duality, providing solid theoretical guarantees.\n  - The modular algorithm can be easily integrated with standard RL solvers such as PPO, making it practical and broadly applicable.\n  - The proposed method addresses tail risks and catastrophic events, which are critical in high-stakes applications.\n  - The evaluation is structured around three measurable goals—convergence of auxiliary variable to CVaR-beta, stabilization of the dual variable \\lambda, and safe yet reward-maximizing policy behavior, which align well with the theoretical contributions.\n  - The authors simulate realistic uncertainty by injecting Gaussian noise into agent actions in an otherwise deterministic environment, enabling a meaningful evaluation of risk-sensitive learning.\n- Weaknesses\n  - The experiments are limited to a single benchmark suite (Safety-Gymnasium), which restricts the ability to assess the proposed method's generalizability to other domains or task types.\n  - The paper lacks a comparative evaluation against baseline methods. While the authors provide a clear theoretical distinction from Bonetti et al. [7]—noting that [7] is limited to the unconstrained setting and assumes exact policy solvers—the practical impact of these differences remains unclear. Since both methods operate in similar reward-based risk-aware frameworks, a direct empirical comparison (even if limited) would strengthen the justification for the proposed approach.\n\nQUESTIONS:\nWhile the proposed method is theoretically distinguished from Bonetti et al. [7], particularly in its ability to handle constrained settings and inexact policy solvers, the lack of experimental comparison with [7] leaves some ambiguity regarding its practical advantages. Given that the Safe Velocity benchmark does not enforce hard constraints on dynamics (i.e., constraint violations do not terminate episodes), it may be feasible to apply [7] in this setting, at least in a limited form. Even a simple empirical comparison—e.g., reward, CVaR, or policy stability—would significantly enhance the reader’s understanding of how the proposed method performs relative to prior work. If direct implementation is difficult, an ablation study or visualization highlighting the behavioral differences would still provide valuable insight into the practical impact of the proposed approach.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a risk-averse constrained reinforcement learning framework that integrates Optimized Certainty Equivalents (OCEs) to address the limitation of traditional constrained RL, which neglects tail risks in high-stakes applications. By formulating objectives and constraints using reward-based risk measures, the framework achieves per-stage robustness in both reward values and time. It establishes a parameterized strong Lagrangian duality, ensuring exact equivalence to the original constrained problem under appropriate constraint qualifications. The authors design a modular algorithm that can wrap standard RL solvers and prove the convergence of the algorithm under common assumptions. Experimental results on locomotion tasks in Safety-Gymnasium demonstrate that the method can effectively reduce violations of risk constraints while maintaining competitive reward performance, validating its risk-aware properties.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n\n1. This work is technically rigorous with a solid theoretical foundation, providing detailed proofs of parameterized strong duality and non-asymptotic convergence guarantees, along with basic simulation experiments. The authors candidly acknowledge the limitations of this paper and validate the relevant claims with consistent results.\n\nWeaknesses：\n\n1. The constraint conditions for exact duality (Assumption 3.4) are relatively stringent, and their practical applicability across different environments has not been thoroughly examined. Furthermore, this paper fails to provide sufficient comparative experiments.\n\n2. The core ideas of this research (including OCE in reinforcement learning, Lagrangian relaxation of constraints, and CVaR in risk-aware optimization) are derived from existing concepts. Its originality lies more in integration rather than the proposal of entirely new concepts, which significantly undermines the innovativeness of this research.\n\n3. The connection between the partial Lagrangian relaxation method and the original constrained problem lacks the necessary intuitive explanation, which impairs the readability of the paper.\n\nQUESTIONS:\n1.The constraint (Assumption 3.4) is crucial for precise duality, yet its practical effectiveness across different environments remains unclear. Could the authors provide empirical evidence of violations of this assumption and discuss the algorithm's performance in such scenarios?\n\n2.This paper focuses on CVaR, but OCE encompasses other risk measures (e.g., entropy-regularized variants). How does the framework perform under these measures? Supplementary comparisons would highlight its flexibility and identify which OCEs are most effective for constrained reinforcement learning.\n\n3.The experiments exclude high-dimensional environments such as Ant. Could the authors test the method in such environments to verify its scalability? It would also be valuable to supplement comparative experiments with existing state-of-the-art baseline algorithms.\n\n4.How sensitive is the algorithm to β across different environments? Could sensitivity analysis be supplemented to clarify its impact on the risk-return trade-off?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMany existing safe RL works rely on expected cumulative constraints, which are not applicable to safety-critical RL problems. To this end, this paper proposes a risk-aware constrained/safe RL scheme. Several theoretical guanrantees such as the convergence and the parameterized strong duality are established. In addition, the practical implementation can be wrapped around the standard unconstrained RL algorithms like PPO, SAC etc. The proposed method is also substantiated by numerical experiments conducted on well-known Safety-Gymnasium benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. This paper is well-written and well-presented.\n\n2. The topic of this paper (risk-averse safe/constrained RL) is very significant for the community and real-world applications.\n\n3. The proposed method is supported by solid theoretical guarantees and experimental validations on well-known safe RL benchmarks.\n\n4. This work can be wrapped around the standard unconstrained RL algorithms such as PPO, TRPO, SAC, thus easy to implement.\n\n5. The convergence of rewards and dual variables are validated by empirical evidence.\n\n\n**Weaknesses**:\n\n1. This paper does **NOT** compare with any other risk-aware safe RL baseline methods. It is hard to evaluate the practical performance of this work without comparing with some baselines.\n\n2. No comments/discussions on the assumptions are provided, see e.g., Assumption 3.4. The proper discussions on the assumptions are necessary although I understand the concerns for main text space.\n\n3. The proposed method is only evaluated on one type of tasks, i.e., safe_velocity. Please see my suggestions later.\n\nQUESTIONS:\n1. Does the stepsize need to be carefully tuned? How can we select a proper stepsize for a new/unknown environment?\n\n2. The authors should include more references regarding probabilistic-constraints (a.k.a. chance-constraints) in safe RL.\n\n3. The authors should provide more experiments to validate the robustness of the proposed method on other type of tasks, e.g., Safe Navigation, Safe Vision, etc. They are all provided in Safety-Gymnasium.\n\n4. Please provide the exact formulations or details for the subgradients in Line 5 of Algorithm 1.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel framework for risk-averse constrained reinforcement learning (RL) by leveraging Optimized Certainty Equivalents (OCEs) to address the limitations of standard expectation-based RL formulations, which often fail to account for tail risks or catastrophic events. The core idea is to apply per-stage risk measures, rather than return-based ones—to both the objective and constraints using OCEs, such as Conditional Value-at-Risk (CVaR). They reformulate the constrained RL problem using a partial Lagrangian relaxation, which under suitable constraint qualifications yields an optimization problem that is provably equivalent to the original problem and amenable to a stochastic minimax solution. They show strong duality in this reformulation and provide a non-asymptotic convergence analysis of the resulting algorithm.\nIn the experiments, their algorithm wraps around any black-box RL solver such as PPO, enabling integration with existing pipelines. Experimental results demonstrate that the proposed method effectively manages risk while maintaining constraint satisfaction and stable convergence.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper addresses the crucial challenge of integrating risk-awareness into constrained RL, which is essential for deploying RL agents in many risk-averse problems in real-world.\n- The paper presents a method to optimize the OCE-constrained objective for risk-averse reinforcement learning with theoretical analysis.\n\n\nWeaknesses\n- Insufficient comparison with baselines and related work. The paper fails to adequately position its contributions within the existing literature through direct methodological and empirical comparisons (e.g. [1,2,3]). This is a significant weakness as it makes it difficult to assess the practical advantages and novelty of the proposed framework.\n- The paper's choice to constrain the per-step cost CVaR, rather than the more conventional return-based CVaR, is a major methodological decision that is not sufficiently justified. While this approach may be better suited for capturing instantaneous, catastrophic risks and offers theoretical tractability, it may also lead to overly conservative or myopic policies in scenarios where overall episodic safety is the primary goal. The paper lacks a discussion of this critical trade-off and fails to provide an empirical comparison against a return-based risk management baseline to demonstrate in which specific contexts its approach is superior. This omission makes it difficult to understand the true scope and limitations of the proposed framework.\n\n\n[1] Chow et al., Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach, 2015\n\n[2] Ying et al., Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Risk, 2022\n\n[3] Achiam et al., Constrained Policy Optimization, 2017\n\nQUESTIONS:\nQuestions\n- Please see the weaknesses section.\n- The proposed method uses a standard black-box RL algorithm in the inner loop to optimize a modified reward function for fixed dual variables. It is well-known that the optimal policy for a CMDP is generally stochastic. However, many RL solvers, when optimizing for a fixed reward function, will output a deterministic optimal policy. This seems to create a potential mismatch: the outer-loop problem requires a search over the space of stochastic policies, while the inner-loop solver might only explore a subset of near-deterministic ones. Could the authors comment on this? Does this mismatch potentially prevent the algorithm from finding the true stochastic optimum of the original CMDP?\n- Algorithm 1 specifies that the final policy is chosen by uniformly sampling an iterate j* from the entire training history. This seems counter-intuitive, as policies from early in the training (small j) are unlikely to satisfy the problem's constraints, given that the dual variables have not yet converged. Returning such a policy would be unacceptable in a safety-critical context. Could the authors clarify the rationale behind this uniform sampling strategy?\n\nLIMITATIONS:\n.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-11"}