Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents
Authors: Jane H. Lee
Baturay Saglam
Spyridon Pougkakiotis
Amin Karbasi
Dionysis Kalogerias
Venue: NeurIPS 2025 poster ()
Keywords: policy search
optimized certainty equivalents
risk aware learning
Primary area: reinforcement_learning
TL;DR: We propose a framework for solving constrained reinforcement learning which is risk-averse and exhibits a parameterized strong duality property under appropriate constraint qualifications.

Abstract: Constrained optimization provides a common framework for dealing with conflicting objectives in reinforcement learning (RL). In most of these settings, the objectives (and constraints) are expressed though the expected accumulated reward. However, this formulation neglects risky or even possibly catastrophic events at the tails of the reward distribution, and is often insufficient for high-stakes applications in which the risk involved in outliers is critical. In this work, we propose a framework for risk-aware constrained RL, which exhibits per-stage robustness properties jointly in reward values and time using optimized certainty equivalents (OCEs). Our framework ensures an exact equivalent to the original constrained problem within a parameterized strong Lagrangian duality framework under appropriate constraint qualifications, and yields a simple algorithmic recipe which can be wrapped around standard RL solvers, such as PPO. Lastly, we establish the convergence of the proposed algorithm and verify the risk-aware properties of our approach through several numerical experiments.

Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents
Jane H. Lee Department of Computer Science Yale University jane h. www.eeeellee.eee edu
Baturay Saglam
Department of Electrical Engineering Yale University baturay saglam@yale edu
Spyridon Pougkakiotis
Amin Karbasi
Department of Mathematics
Foundation AI
King's College London spyridon. www.kakiotisskkkl...... sprridbo..oouttacioc...kk...
Cisco Systems Inc.
ac. uk
com
Dionysis Kalogerias
Department of Electrical Engineering
Yale University
dionysis wal.:iiissaklgggraylllyl edu
Abstract
Constrained optimization provides common framework for dealing with con- flicting objectives reinforcement learning (RL). In most of these settings, the objectives (and constraints) expressed though the expected accumulated reward. However, this formulation neglects risky or even possibly catastrophic events at the tails of the reward distribution, and often insufficient for high-stakes appli- cations in which the risk involved in outliers is critical. In this work, we propose framework for risk-aware constrained RL, which exhibits per-stage robustness properties jointly in reward values and time using optimized certainty equivalents (OCEs). Our framework ensures an exact equivalent the original constrained problem within parameterized strong Lagrangian duality framework under ap- propriate constraint qualifications, and yields a simple algorithmic recipe which can be wrapped around standard RI. solvers, such PPO. Lastly, we establish the convergence of the proposed algorithm under common assumptions, and verify the risk-aware properties of our approach through several numerical experiments.
Introduction
Autonomous agents often used in settings where they must handle several confficting requirements while maximizing a main objective, such as navigating a maze without hitting any walls. These conflicting requirements are often modeled with constrained reinforcement learning (RL) (see Sutton and Barto [42]) and are either solved as a multi-objective problem with conflicting requirements controlled by weights, chosen manually through hyperparameter tuning (e.g., see Borkar [10], Achiam et al. [2], Mannor and Shimkin [28], Tessler et al. [45]. Mania et al. [27]), by using primal-dual methods which optimize over the weights [45, 5]. Others have approached constrained RL through reward-fie techniques, e.g., [30], or strategic exploration [54]. The work of Paternain al. [33] has given theoretical support to primal-dual approaches by proving that, despite their
Wook done while affiliated with Yale University.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

inherent nonconvexity, constrained RL models exhibit zero duality gap under mild assumptions. However, in the standard (risk-neutral) setting (including that of Paternain et al. [33]), the objectives and constraints are often formulated as expected values of a (discounted) sum of rewards. Nonetheless, in many practical scenarios, especially in the context of safety [24] and other high-stakes applications, the standard expectation may not be strict enough to describe the desired behavior. For example, instead of maximizing the on-average return (e.g., of an investment), one may want to learn policies that mitigate risk (e.g., of losing money).
Stochastic Optimization with Risk Measures Risk measures are often used to quantify this risk in problems with uncertain (random) outcomes (see, e.g., Shapiro et al. [41]). More broadly, stochastic optimization with risk measures the objectives and/or constraints has been studied in the online setting [26]. bandits [11, 37, 44], statistical learning [3, 15, 32, 48], non-convex resource allocation problems [21]. and others [20, 55]. These methods and analyses are not readily applicable in our setting either due to auu-convexity/concvviy in the objective and constraints or due to the fact that the risk envelope for our problem has dependence on the policy π.
RL In reinforcement learning specifically, there is a deep literature on risk-averse methods [31, 53, 13, 14, 29, 35, 49, 51, 50] with many applications in the area of robust control [52, 23]. While the most conservative approach consists of learning policies robust to worst- case scenarios [1, 18), we often prefer to take into account a set of events with significant probability, To do this, Huang al. [17]. Tamar et al. [43] consider the policy gradient method under coherent risk measures, where the risk measure replaces the expectation over the discounted sum of rewards. The work of Chow et al. [12] handles similar CVaR risk objectives with CVaR risk constraints by utilizing Lagrangian relaxation. Xu et al. [51], Wang et al. [50] study risk-sensitive RL with OCEs. Recently, Bonetti al. [7] propose an alternative risk-aware framework for (unconstrained) reinforcement learning which places the risk measure in the occupancy measure to derive what the authors call the reward-based conditional value risk (RCVaR) and mean-mean absolute deviation (Mean-RMAD), in contrast to the standard return-based formulations (like that of [12, 17, 43)).
Contributions
This work approaches risk-aware constrained RL through employing reward-based risk measures in both the objective and the constraints. Our contributions are as follows:
We extend the setting of Bonetti et al. [7] and, to the best of our knowledge, our work is the first to handle reward-based constraints, covering a large class of risk measures (OCEs);
We establish parameterized strong duality relation resulting in a computationally tractable partial Lagrangian relaxation (which is exact under certain constraint qualifications);
We propose an online algorithm which is modular, offering the user flexibility in model- ing (allowing for mix of risk-neutral or risk-averse objectives and/or constraints) and implementation (enabling the use of existing RL algorithms as a black-box). By rigorously reducing the underlying problem to certain instance of stochastic minimax optimization, we then establish convergence under common assumptions.
We demonstrate practical usefulness of our approach in extensive numerical experiments on standard benchmarks, showcasing its effectiveness in reducing risk constraint violations and improving stochastic stability through explicit risk management,
The setting of Bonetti et al. [7] is closely related to this work but is restricted to the unconstrained case, while our setting handles reward-based risk constraints as well. Our algorithm differs from the methodology of [7], which is based
block-coordinate descent scheme, the analysis of which does not readily extend to the type of optimization problems arising in the constrained setting. Further, [7] requires exact policy solvers but we work with inexact ones (cf. Assumption 3.6). Compared to other risk-averse constrained RL methods, our partial Lagrangian relaxation is exact under a certain constraint qualification, whereas no meaningful (Iet alone exact) relation between the original primal problem and the employed Lagrangian relaxation has been shown in other works (c.g.. [12]).
2 Background
Notation Given m E N. we let [m] = m........ Given x E R. we write (x) = maxfæ, 0). We denote by P(S) the space of probability distributions over S, equipped with its Borel o-algebra.
We provide the code at https: wgg.ttu...om/tttuuuuasssaaaaaaa/ wweeeeveeeeeeeeeeeeeeeeeeee conttriieedn RL.

---

2.1 Risk Measures
In the context of stochastic optimization, well-known that expectations are unable to capture "risky" events (related to statistical variability, dispersion, fat-tail behavior of the randomness). Risk-averse optimization aims to minimize the risk associated with such events, which is captured by certain functionals known risk measures. For an extended discussion risk-averse optimization, we refer the reader to Shapiro et al. [41, Chapter 6].
Optimized Certainty Equivalents (OCEs) Among different classes of risk measures, this work we are interested in those which are in infimal convolution form, i.c., p(2) = inf, + (a(Z Z t)]}, assuming that represents losses. A notable example is the so-called Conditional Value-at Satue-at-Riss at level (denoted by CVaR (Z)). which is retrieved by setting g(-) à (-)+, for some 3 E (0, 1).
If Z represents rewards (as often in RL., we consider risk measures in the supremal convolution form, Le., ((Z) sup, (t E [g +)]}- For example, we can consider the reflected - -CVaR (- (for reward maximization) by setting g(u) = - 3 (-=))+ In this notation, is called a utility function. Under certain conditions on the utility (e.g., satisfied by g(u) = (--u)+), these risk measures are known as optimized certainty equivalents (OCEs) (see Ben-Tal and Teboulle [4, Definition 2.2], where the reader can find many practical examples and properties). The class of OCEs is very rich (including many risk measures of practical value) and satisfies several important properties.
2.2 Risk-Neutra RL
Let NU U [0) denote the time instant and let S R" and C Rd be compact sets describing the possible states and actions of an agent. Reinforcement learning is often modeled as a Markovian dynamical system where given state and action pair at time т, say (8,, (,), the next state distribution only depends on the current aate-action pair (and is independent of (s<r: acr)). The agent chooses actions at each time instant using a policy E P(S), and the action taken by the agent results in rewards defined by the uniformly bounded function S x A + R. We next describe the standard RL objective, that is, to maximize the (discounted, infinite) expected sum of rewards.
Problem (Primal RL Formulation).
}
= sup TEP(S)
((T)= E
Σ Y (r(o,
where (s7, a denotes the state-action vector evolving under policy æ. It is well-known that the above can be reformulated in terms of the (discounted) occupancy (or occupation) measure of policy, defined as the discounted mixture d .) -1-7) Ex= 7 dd (., )), where Ph denotes the Borel probability measure induced by vector (s,, ).). Let R denote the convex and weakly
compact set of all occupation measures [8] induced by the policies P(S). We can then rewrite the RL problem as follows.
Problem (Convex Analytic Dual Form²).
V = sup STER
[((²)= 1
1-7 Eu-lr(s,a))
(2)
It is known that the occupation measure n is in a one-to-one relationship with policy æ, i.e., if two policies have the same occupation measure, they must be the same (Sutton and Barto [42]).
2.3 Risk-Averse RL Formulations
There several different frameworks for modeling "risk-awareness" and "safety" in RL. Existing works which explore risk-averse RL. formulations have often taken the approach of replacing the
"This notion of casting dynamic optimization problems into abstract "static" optimization problems over closed convex set of measures (as above) is referred to as the convex analytic approach by Borkar [9]. Thus, we will refer to these equivalent problems (1) and (2) as the "primal" and "convex analytic dual" formulations (not be confused with duality in the Lagrangian sense) throughout.

---

Table Comparison of the reward-based objective and the standard RI. objective for general OCEs.
FORMULATION
RISK-AVERSE (REWARD-BASED)
RISK-NEUTRAL.
PRIMAL
sup E 2,
7 g(r((7, (1+ 0))
sup E
(T a.)
-=0
DUAL
. 1 Byp (t g(((s, -t)
sup
B [r(s,a)]
expectations in the "primal" formulation (1) with risk measures, such as in the studies of Huang et al. [17], Tamar al. [43], Chow et al. [12], among others:
E En
sup TEP(S)
-P
Alternatively, one could generalize the tower property of expectations and iteratively evaluate a risk
measure at each decision stage (e.g., see Ruszczyńisi [36]). A less common formulation -of interest herein- herein-applies a risk measure on the occupancy measure, rather than the original probability space, as in Bonetti et al. [7]:
R
sup ER
1-7 Pu- (a.a) (=a())))))
where p L1 (MT .R) + R is a vanite-valued risk measure. This choice (4) enforces robustness over both space and time, capturing per-stage risk in contrast to (3); for details see Appendix A.1.
OCE Formulation By utilizing the relationship between (1) and (2) for risk measures induced by OCEs with utility g. given that the reward is bounded, obtain*
E (1 (1+ ger(s, 11)
8
supsup Evn |t [++g(((s, 1) p
sup E
This relationship is summarized in Table 1. As one can see, the risk-averse problem involves maximization over an additional variable which depends on (in this sequence). Importantly, given a fixed maximization over resembles problem of the form of (1) with modified reward function. Thus one can take advantage of several existing algorithms which solve (1). This has been explored in the unconstrained setting of Bonetti et al. [7, Algorithm 2].
2.4 Constraints
Given the discussion in the previous section for risk-aware objectives, we can similarly describe con- straints also of this form. It is common to address solving constrained problems through Lagrangian relaxation, which amenable to numerical optimization, but (generally) only offers an approximate solution the original constrained problem. We nonetheless show that the (partial) Lagrangian relaxation employed in this paper is exact under certain constraint qualifications.
Here, we highlight that Bonetti et al. [7] do not consider the constrained RL setting (which requires a dedicated analysis of the relation between the Lagrangian relaxation and the original constrained problem), while others using return-based risk objectives and constraints in the convex analytic "primal" space (c.g., Chow et al. [12]) are not able to show any meaningful relation between the primal problem and their employed Lagrangian relaxation.
3 Main Results (Risk Constrained Learning)
The framework we propose has favorable robustness properties, better capturing per-stage risk (see Appendix A.1) for applications which are also time-sensitive. Now, we: (1) show that this formulation
'Going forward, the expectation or risk without subscripts is taken over the initial state and sample path distribution induced by the state transitions of the system and the policy

---

allows us to solve the original constrained problem through a partial Lagrangian relaxation which practically implementable and (2) reduce the problem an instance of stochastic minimax optimization which lends itself to non-asymptotic convergence analysis.
3.1 Problem Deseription and Equivalent Reformulations
To avoid generalities and to facilitate exposition, hereafter we exclusively consider the CVaR
However, all results presented hold for general OCEs. We also use the same 3 for all constraints and the objective, but these need not be the same. We first write a constrained version of the problem (4):
CVaR (-VRR (-ro(s, s.t. 1-7
(-r: 2 ci Vi E [m], (5)
sup M ER - or, equivalently,
sup sup E TEP(S) E6
Σ r (ro(s;, at s.t. sup 4
Σ y (r((s),,7,tti))
2 ct. vi e [m].
(6)
where rl(8,,,t) == t - (t (r) and ri S A IR are the reward functions for i E {0} U [m]. Note that rf is written specifically for the CVaR here, but for general OCEs we
simply replace- --(-)) + with g()). Recall that, by standard convention, r; is bounded for all
Remark 3.1. Consider any OCE, and assume that ry are bounded for all i {0) U [m]. Then, by Ben-Tal and Teboulle [4, Proposition 2.1], we have that
8
20 Te (t g(ri(s,, (3) t1)
sup
E ER
Σ
r (t g(ri(s,, a.) 1))
sup E LET,
Σ
where To is a bounded interval containing the uniformly smallest and largest values of the reward (s(5, a.). Thus, can (equivalently) cast problem (6), by restricting t == [to, ti....,,t.. E T, with T C 1g=+1 some convex and compact set. This will be utilized throughout.
Lemma 3.2. Problem (6) equivalent to
sup TEP(S) FET
E
r" (ro(s), of of,
s.t. E Σ x of(s7,,2, 2 cr e [m].
(7)
The proof is based on Chow et al. [12], and given in Appendix A.2. Although may seem somewhat redundant, (7) is a maximization problem jointly over the variables I, r,t. subject to functional
inequality constraints. Thus, the Lagrangian associated to (7) reads L(T, () = BB[( A)], with
,(m,t, Σn xr (16(57, - A.C. + + (1166,
(8)
Then, the primal problem is p = sup,,t infx2o L(T, A). Note that the above, for fixed I, A, is the Lagrangian for risk-neutral constrained problem (similar those considered by Paternain al. [33)), parameterized by i, which controls risk-aversion (in harmony with the choice of (3).
Partial Lagrangian Relaxation
Next, we derive partial Lagrangian relaxation for (7) (which is exact under constraint qualification; cf. Assumption 3.4). In turn, this allows us to cast the problem a stochastic minimax optimization framework and derive an effective online algorithm for its solutions (cf. Section 3.3).
Partial Maximization Problem
Let us now consider problem (7) for fixed T. i.e.,
sup rd (r6 af, o0,4))
s.t. E
Σ Y (r((5,, (((77,07,14) 2 c, vi E [m]. L7=0
(9)
SEP(S)
This problem exhibits strong duality under usual conditions.

---

Proposition 3.3. Let be bounded functions for all i E {0} U [m]. Assume that Slater's condition holds for (9). Then, (9) exhibits strong duality, and thus
sup inf C(T, t, inf sup [TT, X). X20 n
T
The proof of Proposition 3.3 is given in Appendix A.3.
Maximizing over & From Remark 3.1, we know that t E T and T is a convex and compact set (since we assume bounded rewards). Note that Proposition 3.3 holds for each fixed E T, assuming Slater's condition is satisfied at this point. To proceed, it suffices to introduce another constraint qualification, as follows.
Assumption 3.4 (Constraint Qualification). Let t" be argument which attains the maximum the variable & = [lo, ,,,.,,,,,, fm]] in (7). There is a (non-singleton) convex and compact set I C T, with l" I, such that Slater's condition holds for (9), for every E I.
Under Assumption 3.4, we have that
supsup inf ((T,,, = sup inf sup L(T, EEI X20
teI X20
Parametrized Policies and Almost-zero Duality Gap In order to solve the (inner) policy opti- mization problem, we parametrize the policy by a vector 0 E ② C R', representing the coefficients of, say, neural network, assuming that O is compact shown that under Assumption 3.4, the proposed partial Lagrangian relaxation is exact. We argue that this remains almost true (in the sense described below) even if we utilize parametrized policies. In what follows, we assume that Ty is an e-universal parametrization of measures in P(S), according to Paternain al. [33, Definition l].
(which is the case in practice). We have
Theorem 3.5 (Almost-zero Duality Gap). Let Assumption 3.4 hold, assume that ne is e-universal for version of (7) is feasible. Then,
measures in P(S), and that the
P (t*) = sup sup inf C(T, A) 2 sup inf sup ((TT,t, 2 (t) - EET A20
rET A20
The proof is given in Appendix A.4. Theorem 3.5 states that solving the parameterized (partial) dual problem incurs negligible cost (depending on the universal parametrization error e >> 0) under mild assumptions and our usual constraint qualification (see Assumption 3.4).
The Practical Parametrized Partial Dual Problem In practice, we do not have access to the set (assuming that Assumption 3.4 is satisfied), and thus, may heuristically search over all T, noting that a reasonable initial guess can maximize our chances of landing in I (and thus solving the original constrained problem (7) (almost) exactly). Nonetheless, the proposed algorithmic framework does not rely on Assumption 3.4. Indeed, in the absence of this, we recover partial Lagrangian relaxation which yields approximate solution problem (7). However, we highlight that, unlike Lagrangian relaxations considered in the literature, our approach is exact under Assumption 3.4.
Additionally, from Proposition 3.3, we know that the optimal Lagrange multiplier is attained, assuming that Slater's s condition holds for (7). Thus, there is an optimal Lagrange multiplier associated to (7) within some sufficiently large convex and compact set A RT. In practice, the size of this set can be adjusted dynamically, if necessary. Hence, in what follows, we restrict the minimization over À E A. In light of the previous discussion, focus the following parametrized partial dual formulation:
D = sup inf sup L(To,t,)). IET XEA
(10)
blackbox RL
A Wrapper for Blackbox Unconstrained RL Algorithms Notice that C( I, A) is a linear combination of the expected discounted sum adjusted reward functions which take the original to (s a, t4) 1/ (t: (s a))+ for i = 0. m. Thus, the innermost problem in
"Slater's condition requires that there exists strictly feasible policy.

---

Algorithm 1 Reward-Based SGDA with Risk Constraints
Input: Bounded reward functions T; S A + I. discount factor 7 E (0, 1), step sizes
f), T)t > 0, batch size n.
2: Initialize: g(0) (
3: for each iteration j = 1, 2. do
4: Call the inexact oracle obtain Ta3) satisfying Assumption 3.6.
5: Collect batch of trajectories { using ToCo and compute sample (sub)gradients:
AeL(TOUS, t0), ((0)).
6: Update dual (A) and auxiliary (f) variables:
A(+1)_ IIA (10) 1/ v PAC(TToO 16), , (11)) ((+1) 11T (66) + nI V V.L((TTOO 160, (11))))))
7: end for
8:
Return: Tgo*), where sample jr Uni
(10) reduces an unconstrained RL problem with linear combination of terms defined by reward functions r, given and t. Hence, we have a practical algorithm for solving (10) which, if solved exactly for starting point t E I, is arbitrarily close the true primal solution P of (6), under Assumption 3.4. In practice, this problem is non-convex so we may only hope to find a locally optimal solution. Convergence to such solution is established next.
3.3 Algorithm and Analysis
Next, we present the algorithmic procedure for solving problem (10), in Algorithm 1. Consider a policy oracle that precisely solves the problem supace L(To, t, A), such that for every (t, X) E T x A, returns policy (t,(t,)) E arg maxage L(TT, 1, A). Using such an oracle, problem (10) reads
D sup inf 1ET XEA
((TT-(LA)), t, inf sup A(t TET AEA
(11)
=-f((())
Having access to a solver which returns (t, exactly is difficult. Instead, we assume the availability of inexact oracle, defined next. We justify the generality of this assumption in Appendix A.5 to save space in the main body.
Assumption 3.6 (Local Solutions). For any At T x A, let 0 (t, A) be maximum of L( t, A) and of (t, X) be returned by generic RL algorithm. Then, there exists a 5 > O such that V(t. A) T x A, Ve, (TTo
V.. A) + b(0", ot A), with [ ot f, A) < 6.
In what follows, we show that (10) can be reduced to a stochastic minimax optimization problem satisfying appropriate assumptions, and thus we can obtain a non-asymptotic convergence rate for Algorithm 1. For notational convenience, we define the function (1) == supxea [t, A). First, we show that is Lipschitz with respect to E gm-1 uniformly over À A.
Lemma 3.7. There exists a constant C > 0 such that
|sup, L(T,, t1, A) sup, L(T, 12, A) S Cl|t tell2
for all pairs 4,12 E RRm+1 and E A.
We prove Lemma 3.7 (which readily applies to parametrized policies well) in Appendix A.6. Following Lin et al. [25], we analyze our method under the assumption of Lipschitz smoothness of the function ((TT t, A). for 8* (t, A) E arg maxøee L(TT. t, X), which we state below.
Assumption 3.8 (Lipschitz Smoothness). The function f(t, A) = - -((= t, A) is e-smooth over T A, where 0* (t, A) E arg maxace L(TT, t, A) is an arbitrary selection.
Remark 3.9. We note that Assumption 3.8 is not particularly restrictive in our setting, and it holds under several general conditions, as outlined in Appendix A.7. Nonetheless, not satisfied, we can still enforce this assumption by making a slight algorithmic adjustment involving the appropriate addition of small uniform noise. For a comprehensive discussion we again refer the reader to Appendix A.7, where also present explicit formula for computing the (exact) sample gradients.
7

---

As appears in prior work [25], we make the standard (and, in our case, mild) assumption that the exact stochastic (sub)gradient oracles are unbiased and have at most o² variance.
Assumption 3.10 ((Sub)gradient oracles). For all (1, A) T x A, we assume that
C( (t. (t, X) + fA, t)] -0 Bll C(T A), t, A) + V fA, s aه
Given the imposed assumptions, we now present the convergence guarantees of Algorithm 1. In the context of nonsmooth weakly convex optimization, the Moreau envelope serves as measure of closeness to stationarity. With this in mind, we introduce the definition of an e-stationary point.
Definition 3.11 (e-stationary point). A point æ e-stationary if || 1 I 6, where, given
some 6 > 0, we define the Moreau envelope as (x) == infwER=+1 (f(1)) +
1 || - ==1²]- 28
When = 0, stationary point of the Moreau envelope (+)+) is also stationary for +(.). For c > 0, one can argue that an c-stationary point for $ 1/22/ is close to e-stationary point of ( as long as (-) is Ê-weakly convex (noting that, under our assumptions, (-) is Ê-weakly convex). We proceed with a bound on the iteration complexity of Algorithm 1.
Theorem 3.12.
Let the step-sizes T/r and тΙx in Algorithm be small enough as in Eq. (30) and (29), respectively, and batch size n = 1. Define the quantities Âς = D1/2e((10)) - min; D P1/2e(t) and An == (1))) _(X(0), (()).. If Assumptions 3.6, 3.8 and 3.10 hold, the iteration complexity for recovering an O( + de(diam(() + diamm()))sstatonnay sstationary point (Definition 3.11) is of order
- 0² 88) ) (diam(A)) D e ((diam(A)) ( +
Bo
max 1, o² +
})
6
e²
2
Note that this result says we only need single trajectories (n 1). Thus our algorithm can be run online. The proof, which is detailed in Appendix A.8, extends the developments of [25], by incorporating biased (sub)gradient samples, occurring due to the use of inexact oracles. If 5 = (c then we recover an c-stationary point and the complexity of [25].
4 Experiments
We conduct experiments on locomotion tasks from the Safety-Gymnasium benchmark [19].
4.1
Setup
We consider two constraints in the environment: safe navigation and safe velocity. The first involves simple problems with low-dimensional state and action spaces, where the agent must navigate under rules (e.g., along path or without hitting obstacles). The second is more challenging, involving standard nigh-dimensional control tasks based on MuJoCo-v4 agents [46]. At each timestep, the agent receives cost of +1 if it violates constraint and 0 otherwise. These violations do not affect environment dynamics- they neither terminate episodes nor alter transitions. Importantly, our experiments do not access this cost directly during training; instead, the agent manages risk by being risk-aware in its velocities.
To simulate risk and uncertainty in otherwise deterministic setting (where identical actions yield identical outcomes), we inject zero-mean Gaussian noise with a standard deviation of 0.05 (5% of the action range) into all agent actions during both training and evaluation. This adds stochasticity the agent's 's actions and introduces risk.
Experimental Goals and Motivation Our objective is to evaluate whether the proposed risk- sensitive method ensures stable convergence of the dual variable (A) and the auxiliary variable (t), while also inducing safe and meaningful behavioral changes to the agent. To emphasize conver- gence, we train agents for extended durations (5-18M steps) to provide thorough proof-of-work: Specifically, we aim to: (1) verify that / converges to the empirical 8 of the post-training constraint violation distribution (e.g., safe velocity violations), (2) confirm that stabilizes to enforce the constraint, and (3) assess whether the policy maximizes reward while maintaining safe velocities.
8

---

Table 2: Cumulative episodic evaluation costs and rewards of the converged agents, computed as the mean of the last 10 episodes. The PPO baseline unconstrained and risk-neutral, We use the simplest agent, Point at level difficulty, to avoid additional challenges and isolate the
effects of the constraints.
Cost +
Reward
Environment
# Training Steps
PPO
Ours
PPO
Ours
Button
5M
150.76
0.0
24.29
2.58
Circle
5M
206.74
0.0
60.18
39.19
Goal
10M
45.09
0.0
21.89
13.56
Push
5M
38.48
0.0
0.93
2.42
Optimization with CVaR Constraints Let 8 x A E
denote the constraint-quant quattriintquanttfyying
function (e.g., velocity), We formulate the following constrained optimization problem:
E 1rr क ((+)))
s.t. CVaR (v(s, a)) S.
sup xEP(S)
where is user-defined threshold and 3 the CVaR parameter. Our framework is general and allows risk-neutral objectives to be combined with risk-aware constraints. To solve this, we transform the reward at each timestep using the CVaR-based Lagrangian (see Appendix B.3.1 for the derivation):
r87, n (8+)) + X c+ll
u(ar, x (s+))))
In safe navigation, the constraint variable reduces to v(s, a) E {0, since navigation is not quantifiable in the same way as velocity. This makes the optimization problem more difficult, as the penalty term (the À multiplier) becomes sparse.
Proximal Policy Optimization (PPO) [39] is used as a black-box solver to optimize the policy m implementing line 4 of Algorithm The dual variable À and auxiliary are updated via stochastic gradient steps following the rest of Algorithm For further experimental details, see Appendix B.
4.2 Results
Results are shown in Table 2 for safe navigation and in Figure 1 for safe velocity.
Constraint Handling The performance in the navigation tasks shows us that our method can strictly prevent any constraint violations, whether quantified continuously or not. In fact, is the only method achieving strictly zero violations compared to other PPO-based constraint learning algorithms (see Table 5 in [19]). In some cases, constraint satisfaction also leads to higher rewards than the unconstrained vanilla agent (c.g., in the Push environment). This shows that constraints are not arbitrary but can guide the agent toward optimal task behavior.
tisk-Management On the other hand, the velocity tasks provide insights into the risk management capabilities. With properly tuned step sizes, the dual and CVaR variables stabilize and oscillate around consistent values. The post-training evaluations (the last column in Figure 1) extract the distribution of the trained agent's 's velocity over several evaluation episodes. We observe that the converged (-value matches the 3-upper quantile of the velocity distribution. This is what the CVaR aims to do: it captures the expected cost in the worst 3 fraction of outcomes.
Further, the evaluation reward becomes increasingly stable training progresses despite the uncer- tainty in the evaluation environment. Considering the number of data points in the curves-1000 IM steps (c.g., about 14,000 points in Hopper)- the robustness of our method in managing risk evident. This is also confirmed in post-training simulations: the agent moves forward cautiously compared to the unconstrained (risk- PPO agent, demonstrating that the learned policy adheres to the task while remaining interpretable from the perspective of the constraint.
We recognize the extended training duration and the need to tune the step sizes of À and I. Nonetheless, once these variables converge, algorithm is effective not only toy tasks but also in realistic control settings, making patience during training important practical consideration.

---

Evaluation reward
Velocity threshold
B Bppper quantile
Converged t-valuc
Exalustion Rewnd
Pmogresa of CVaR Variable (O)
Progress of Dual (arialle
Veleeity Distriution
Epioalo Rowend
0.02
1.35
Froggencc
1,40
0.00
15
16
Toiniig Skepe (LM)
Number Rollon Opdates (IK)
Namber of Rollont Updates (IK)
Pe-Suup Milncty
(a) Environment: HalfChectah
1.450
8 8-upper quantile: 1.419
Converged t-value: 1.417
Evaluation Reward
Progress CVaR Variable
Pregress Dual Variable
Velocit Distribution
-220
1000
Faalue
-2.5
A Value
8,00
-0.15
0.00
Training Stepe (IM)
Number of Roliont Updates (IK)
Number Roilout Updates
Per-Step Vilocity
(b) Environment: Hopper
0.373
8-upper quantile: 0.370
Converged t-value: 0.370
Evaluction Reward
Progress of CVaR Variable (*)
Propress Deal Variable (A)
Velocity Distribttion
100
Epitodd Revard
0.10
VVan
Valne
Prequece
50
0,05
-0.2
0.00
Training Steps: (LM)
Number Rollout Updates (K))
Number Rollout Updates (1K)
Per-Sep Welcctt
(c) Environment: Swimmer
= 0.228
U-upper quantile: 0.248
Converged t-value: 0.207
Evaluction Reward
Progress of CVaR Variable
Progress of Dunl Variable (A)
Velocity Distibution
Upisode Reeand
2000
-1.00
[Válue
Valae
0.010
Froquetyy
1000
-1.10
0.005
1.12
25
0.75
1.00
1.25
Triining Steps (IM)
Number of Rollout Updates (IK)
Number Rallout Updates (1K)
Per Step Velocity
(d) Environment: Walker2d
1.171
B-upper quantile: 1.133
Converged f-value: 1.122
Figure 1: Learning curves of the agent trained to convergence. We report episodic evaluation rewards, the progression of optimization variables and x, and the velocity distribution the trained agent evaluated over 100 post-training episodes without further learning. Curves are not smoothed.
5 Conclusion
We introduce a training framework for solving constrained reinforcement learning (RL) problems with risk-awareness which has desirable properties. The problem exhibits parameterized strong duality under constraint qualifications, which lends itself to an algorithm that is theoretically-suporteee and flexible to implement; the inner problem can be solved using any black-box RL algorithm, while the other variables are updated using SGDA. The framework handles general OCEs beyond the CVaR, and captures both risk-neutral (e.g., by setting 3 = for the CVaR() and risk-aware objectives and constraints, combination of the two, with empirical evidence that, in the context of risk-aware constrained RL, it effectively manages risk.
Limitations An open problem resulting from this work asks whether or not full strong duality holds unconditionally. Also, while our framework is inherently risk-aware, is more computationally intensive compared risk-neutral methods (as each update of À and variables requires solving for an approximately optimal policy), is also true for other risk-aware methods, e.g., [7, 12).
10

---

References
[1] Mohammed Amin Abdullah, Hang Ren, Haitham Bou Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust reinforcement learning, arXiv preprint arXiv: 2019.
[2] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. CoRR, abs//105, 10528, 2017. URL httt:://arxii.oorgabb////// 10528.
[3] Amrit Singh Bedi, Alec Koppel, Ketan Rajawat, and Panchajanya Sanyal. Nonparametric compositional stochastic optimization for risk-sensitive kernel learning, 2020.
[4] Aharon Ben-Tal and Marc Teboulle. An old-new concept of convex risk measures: The optimized certainty equivalent. Mathematical Finance, 17(3):449 2007. doi: https: 1ttp:///..111111116-111100 2007. 200..0037 1.x. URL https: http://////mimmmbbrrrrrrr .elee..wlllly.......... com/ doi/abs/10.. 11111iijaa. 14467-9665 2007 00311.x.
[5] Shalabh Bhatnagar. An online actor-critic algorithm with function approximation for constrained markov decision processes. Journal of Optimization Theory and Applications, 153, 06 2012. doi: 10.10077/10057-012-99999
[6] Lorenzo Bisi, Luca Sabbioni, Edoardo Vittori, Matteo Papini, and Marcello Restelli. Risk- averse trust region optimization for reward-volatility reduction. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, ISCAI- 20, pages 4583-4589, International Joint Conferences on Artificial Intelligence Organization, 2020. doi: 00.24963/jccii222200 Special Track on Al in FinTech.
URL htttps://oo org/ 10. 200.249633// cai 2020/2022/22
[7] Massimiliano Bonetti, Lorenzo Bisi, and Marcello Restelli. Risk-averse optimization of reward- based coherent risk measures. Artificial Intelligence, page 103845, 2023. ISSN 0004-3702. doi: http:// 103845. URL https: //www sciencedirect. com/
[8] Vivek S. Borkar. A convex analytic approach to markov decision processes. Probability Theory and Related Fields, 78:583 602, 1988.
[9] Vivek S. Borkar. Conver Analytic Methods in Markov Decision Processes, pages 347-375. Springer US Boston, MA, 2002. ISBN 978-1-4615-0805-2 doi: 10.1007/978-114615-00000 11. URL https: org/ 4ttp://......0000000000999111-00
[10] V.s Borkar. An actor-critic algorithm for constrained markov decision processes, Systems & Control Letters, 54:207-213, 03 2005. doi: 10. 10.10166/ http://.yyscoomee2000.000000
[11] Adrian Rivera Cardoso and Huan Xu. Risk-averse stochastic convex bandit. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statisties, volume of Proceedings of Machine Learning Research, pages 39 47. PMLR, 16-18 Apr 2019. URL https: https:///rroceeding.. mlr. httpes//vv//carrolll11oo html.
[12] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18, 12 2015.
[13] Yingjie Fei, Zhuoran Yang, Yudong Chen, Zhaoran Wang, and Qiaomin Xie. Risk- sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 22384-22395. Curran As- sociates, Inc., 2020. URI. https://proceeddnng: wwweiiinnnnnn.uisseeeeeeeeee pttuurpps.ccppppe////eee ww/ppaper/0020//1222. pdf.
[14] Peter Geibel and Fritz Wysotzki. Risk-sensitive reinforcement learning applied control under constraints. Journal of Artificial Intelligence Research, 24:81-108, 2005.

---

[15] Jun-ya Gotoh and Stan Uryasev. Support vector machines based on convex risk functions and general norms. Annals of Operations Research, 249, 02 2017. doi: 10.1007//104499-16--33333
[16] Hassaan Hashmi, Spyridon Pougkakiotis, and Dionysis Kalogerias. Model-free learning of two-stage beamformers for passive irs-aided network design. IEEE Transactions on Signal Processing, 72:652-669 2024. doi: N0.1109/TSP.22233344446
[17] Audrey Huang, Liu Leqi, Zachary C Lipton, and Kamyar Azizzadenesheli. On the convergence and optimality of policy gradient for markov coherent risk. arXiv preprint arXiv:2103.02827 2021.
[18] Wenjie Huang and William B. Haskell. Risk-aware q-learning for markov decision processes. In 2017 IEEE 56th Annual Conference Decision and Control (CDC), pages
2017. doi: N0.1109/CDC.2217..2266 82643888.
[19] Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforce- ment learning benchmark. In Thirty-sevnnth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL htttp:://openrevier/ www.earwwwwwwweeeeeeeeeeeeeeeeee net forum? id=WZm1xIuIGR.
[20] Daniel R. Jiang and Warren B. Powell. Risk-averse approximate dynamic programming with quantile-based risk measures. Mathematics of Operations Research, 43(2):554 - 579, 2018. doi: 10. 10.1287/moo:20117.0777
[21] Dionysios Kalogerias and Spyridon Pougkakiotis. Strong duality in risk-constrained nonconvex functional programming, 2023. htt.httss:/azzii.or////ae
11948v3.
[22] Drew P. Kouri and Thomas M. Surowiec. Epi-regularization of risk measures. Mathematics of Operations Research, 45(2):774-7955 2020. doi: 10.1288/mooo:221991111 URL https:
org/0.org/0.... 1287 moor 2019. 1013.
[23] Thanh Lam, Arun Verma, Bryan Kian Hsiang Low, and Patrick Jaillet. Risk-aware reinforcement learning with coherent risk measures and non-linear function approximation. In The Eleventh International Conference on Learning Representations, 2023. URL https: http::://oppnrevviee net/forum? id= R=ZOVybbj.
[24] Jan Leike, Miljan Martic, Viktoriya Krakovna, Pedro Ortega, Tom Everitt, Andrew Lefrancq. Laurent Orseau, and Shane Legg. Ai safety gridworlds. 11 2017.
[25] Tianyi Lin, Chi Jin, and Michael Jordan. On gradient descent ascent for nonconvex-come minimax problems. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th Inter- national Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6083-6093, PMLR, 13-18 Jul 2020. URL https:/ htaps://proceedinga mlr. press/v119 1 html.
[26] Avinash N. Madavan and Subhonmesh Bose, A stochastic primal-dual method for optimization with conditional value at risk constraints. Journal of Optimization Theory and Applications, 190(2):428 460, August 2021. ISSN 0022-3239. doi: 10. 10.10007//10957221--0101
[27] Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies is competitive for reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grau- man, N. Cesa- Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: / /pwpc//ed/n/o/e//sss/eeee www.eeeddinnn.u.uis..... neurips. Paper .df.
[28] Shie Mannor and Nahum Shimkin. A geometric approach multi-criterion reinforcement learning. J. Mach. Learn. Res., 5:325-360, dee 2004. ISSN 1532-4435.
[29] Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning, 49(2):267-290, 2002.
12

---

[30] Sobhan Miryoosefi and Chi Jin. A simple reward-free approach to constrained reinforcement learning. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pages 156666 PMLR, 17-23 Jul 2022. URL https: https://proceedd www.mm...............t..w....... watpsssvvv22mmmryyrme
mlr
22a. html.
[31] Somayeh Moazeni, Warren B. Powell, Boris Defourny, and Belgacem Bouzaiene-Ayari Parallel nonstationary direct policy search for risk-averse stochastic optimization. INFORMS Journal on Computing, (922):32-349, 2017. doi: 10.1287/jjo::2116.7735.
[32] Matthew Norton, Alexander Mafusalov, and Stan Uryasev. Soft margin support vector classifi- cation buffered probability minimization. Journal of Machine Learning Research, 18(68):
1-43, 2017. URL http:/ / jmmrm
peapprs/v18//// 111/665-6661666 html.
[33] Santiago Paternain, Luiz Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained reinforcement learning has zero duality gap. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: h mng.uurriinnnn.nnn.s........ neurips. cc cc/paaper/219/ff1ee htt.a////e.............111111100
pdf.
[34] Spyridon Pougkakiotis and Dionysis Kalogerias. A zeroth-order proximal stochastic gradient method for weakly convex stochastic optimization. SIAM Journal on Scientific Computing, 45(5):A2679-A2700 2023. doi: 10. 10.1137/22M14444200 URL https:/ /doi. org/ wtg/1...////0 1137/ 22M1494270.
[35] A. PrashanthL. and Michael C. Fu. Risk-sensitive reinforcement learning: A constrained optimization viewpoint. ArXiv, abs/1810.09122 126, 2018.
[36] Andrzej Ruszezyúski. Risk-averse dynamic programming for markov decision processes. Mathematical programming, 125:235-261, 2010.
[37] Amir Sani, Alessandro Lazaric, and Rémi Munos. Risk-aversion in multi-armed bandits. In F. Percira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Ad- vances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL bttps://pproce https:://proceeddings. neurins cccpaper_ffiieeeeeeeeeeeee fnmeerfffllss files/ pttll//pppprr/22///iieee pdf.
[38] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arxee wwerxiv..orgaaan 0t////15600022220
[39] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL https: htttp:://arxiv.orr/abb//// /1007 .06477.
[40] Alexander Shapiro. Second-order derivatives of extremal-value functions and optimality condi- tions for semi-infinite programs. Mathematics of Operations Research, 10(2):0072119 1985. ISSN 0364765X, 15265471. URL http:://ww. httt://wwwj.jtttrrr wwwgstaable/369377
[41] Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczyúski. Lectures on Stochastic Programming: Modeling and Theory, Third Edition. Society for Industrial and Applied Mathematics, Philadelphia, PA, 2021. doi: 10. 1111377197881619965599 URL //stps::/epubs.
si siam. wtt//oo/ats/////eoottotoogtotaat 10. 1111/11199999 91777/1.7886119966555
[42] Richard S. Sutton and Andrew G. Barto. Reinforcemeett Learning: An Introduction. The MIT Press, second edition, 2018. URL http::/ httt:://iincompeeeiidesss httpeeeeiiiiiieeeeeeoeoeeeeeeeee www.bbooooooooooeeeeeeeeeeeeeeee the-book 2nd. html.
[43] Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Policy gradi- ent for coherent risk measures. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Cur- ran Associates, Inc., 2015. URL https: http:///ppoocecooneeeecccsssssea wwwripp..c/pppprrreeeeeee cc//paperr/015//ilee 12447884ff111dd11888550000013331
Paper, pdf.
13

---

[44] Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Sequential decision making with coherent risk. IEEE Transactions on Automatic Control, PP:1-1, 12 2016. doi: 10.1 101.9TAAA.221.224488777
[45] Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. CoRR, abs/1805. 1074, 2018. URL httpt////aaxr/ htt//bb//11000 180 11074.
[46] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012. doi: 10. 10.1109/ROOS. 2012,636619.
[47] Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez- Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Hannah Tan, and Omar G. Younis. Gymnasium: standard interface for reinforcement learning environments, 2024. URL https://arxil: org/ abs/2407. 17032.
[48] Constantine Alexander Vitt, D. Dentcheva, and Hui Xiong. Risk-averse classification. ArXiv, abs/1805.00119 19, 2018.
[49] Kaiwen Wang. Nathan Kallus, and Wen Sun. Near-minimax-oper risk-sensitive reinforce- ment learning with evar. In Proceedings of the 40th International Conference Machine Learning, ICML'23. JMLR.org, 2023.
[50] Kaiwen Wang, Dawen Liang, Nathan Kallus, and Wen Sun. A reductions approach to risk- sensitive reinforcement learning with optimized certainty equivalents, 2025.
[51] Wenhao Xu, Xuefeng Gao, and Xuedong He. Regret bounds for markov decision processes with recursive optimized certainty equivalents. In Proceedings of the 40th International Conference on Machine Learning, ICMI.23.. JML.. 2023.
[52] Chen and Andre Rosendo. Risk-aware model-based control. Frontiers Robotics and AI, 2021. ISSN 2296-9124 doi: 10.3389/frobb..221..7777 6617839. URL attpps///wwr www.tiersinnnn.. org/articles/11 10. 33t9/33380000 2021 617839.
[53] Pengqian Yu, William B. Haskell, and Huan Xu. Approximate value iteration for risk-aware markov decision processes. IEEE Transactions on Automatic Control, 630)):313--314,2000 2018. doi: 10. 10.1199A...201822999000
[54] Tiancheng Yu, Yi Tian, Jingzhao Zhang, and Suvrit Sra. Provably efficient algorithms for multi-objective competitive rl. In Marina Meila and Tong Zhang, editors, Proceed- ings of the 38th International Conference on Machine Learning, volume 139 of Proceed- ings of Machine Learning Research, pages 12167-12176. PMLR, 18-24 Jul 2021. URL https://proceedinggs mlr ctress/v139/yu22/ html..
[55] Lifeng Zhou and Pratap Tokekar. An Approximation Algorithm for Risk-Averse Submod- ular Optimization, pages 144-159, 05 2020. ISBN 978-3-030-44050-3, doi: 10.1007/ 978- 978-3-030-44051-0.99
14

---

A Supporting Details for Theoretical Results
A.1 Robustness in Value and Time
The so-called reward-based formulation in (4), as coined by Bisi et al. [6], Bonetti et al. [7], captures per-stage risk, contrast to the return-based approach (3), which captures aggregate discounted risk. To see this, first note that the occupation measure can be reexpressed as
dv Σ (1-7))"dp T=0
dd (T),
T=0
where 7' is N (0)-value geometric random variable such that P(T =) =: =: "),", E E N [0), and p now the conditional state-action distribution relative to T. other words, // may be seen the marginalization of the disintegration dp, "o, (opp"(r) (r) relative T, the latter interpreted as a random time. One can show that
-VVaR (-r(s,a) = - (-
where (+7,, denotes the state-action vector under policy and evaluated at random time T. Specifically for the case CVaR (and similarly for other OCEs), can be readily seen that for 3 = 1 recover the objective of (1) which can be equivalently written as (TT) (1 (1-?) -11-:)-EEr((5 a
whereas if 1 O, we obtain
lim C- BLU
= inf ess (r,)));) r(s,a).
which shows in particular that such reward-basse RL formulations enforce joint aversion in reward value and the time for which the reward takes each value. For instance, for sufficiently small 3, the functional -CVaRR(-r(( (--(sss, a7)) will be more sensitive very small rewards happening far in the future (and with low probability (1 >)""), as compared with (1) and (3), which both consider reward time averaging, heavily discounting future reward contributions.
A.2 Proof of Lemma 3.2
We will work with the convex analytic dual form for this proof, but replacing the occupancy measure with the expected discount sum will give the same result.
Statement
The problem
p sup sup (s,(s, HER EoE
to (to ro(s,a))+
(12)
s.t. sup (o eER
14 - z (1i r((,,a))++ 2a 4
Vi 1,2, 1,2,....,
and that of
1
sup
Ep (s.a)
(to r)(5
8
(13)
1 (t: r)(8,a))+
s.t.
E (s,a)
t,
=i= = 1,2,...,m
equivalent.
Proof. We need to prove that it valid to move the supremum over each t; to the objective. The proof follows that of Chow et al. [12].
(Case (6) < (7).) Let T² be a feasible policy for the problem (6). Then define
1 (t: r((ssa))+ ci
t! arg max E By (s.a) E.ER
Vi = 1,2,..., m,
that (T ()) is feasible for (6). Then this tuple (x {.) is clearly also feasible for (7).
15

---

(Case (7) - (6).) On the other hand, for any we clearly have that
- -VVR Z| = sup Eyn ( -1 2 Eug i f- 2)+
(t
2)
for any f. So, if the quantity the RHS of the inequality is lower bounded by c2 (that is, feasible for (7)), then the LHS is clearly also lower bounded by a (thus, feasible for (6)).
A.3 Proof of Proposition 3.3
Statement Let r; be bounded functions for all E {0) U [m]]. Assume that Slater's condition holds for (9). Then, (9) exhibits strong duality, and thus
sup inf L(T, = inf sup 1, A
A
Proof.
Paternain et al. [33, Theorem 1] proved strong duality holds for problems of the form
sup TEP(S)
E
Σ x ro (s7 o=)
s.t. E Σ r' af) 2 vi E [m],
(14)
which involve the expectation of the discounted sums of some bounded reward functions rs, for i {0} U [ms] Thus, the problem (9) is exactly in the form of Paternain et al. [33], with the reward
functions (s, a, tz) = 4 3 (t+ - ri(ss ()),,)))+ for E {0} U [m]] Since the original ry are bounded
and 4 are fixed, we have that rf are bounded and we can readily apply Paternain et al. [33, Theorem 1].
A.4 Proof of Theorem 3.5
Under Assumption 3.4, we have shown that the proposed partial Lagrangian relaxation is exact, i.e., that
supsup inf L(T, f, A) = sup inf sup L(T, EE a A20
tE A20
We assume that тe, for 0 E θ C Rº (with è some compact set), is an e-universal parametrization of measures in P(S). according Paternain et al. [33, Definition 1] and that the parametrized-polt -policy version of (7) is feasible. We proceed to show that there is almost no price to pay for the policy parametrization, terms of duality gap.
Indeed, assuming that the parametrized optimization problem is feasible, we can utilize Paternain et al. [33, Theorem 2] and Assumption 3.4 to deduce that, for all E I, the following inequalities hold:
(*()) sup inf [(T, t) X20
inf sup C(oe, t, (t) G(t)e/(1-n))
X20
where G(l) O(1) is constant depending on and e > 0 is arbitrarily small con- stant (depending on the employed parametrization). Let to E arg max (*(t) and to e arg max,ez infx2o sup, L(Te,t, A). Let also G max
Then, we obtain that
P*(t") 2 inf sup L(T, t,, A) P*(t)) Gc/(1 - -7,
X20 x
(*))))= 2 inf sup L(TO, t/ A) > PM (1%) - Ge/(1 (1-7). X
Since, by definition, P*(1*) 2 PM (17), while sup, inf A20 Limg, 1" >) S sups inf x20 C( 17, A),
we obtain that
P*(t)) P*(t) = sup inf L( (,,,,,) 2 sup inf sup 2 0 (1--)
1 * x
tEī A20
which completes the proof.
16

---

A.5 Discussion on Assumption 3.6
Below, we provide an insight as to why the inexact oracle utilized in Assumption 3.6 reasonable under minimal conditions (under which the sample gradients used in Algorithm satisfy Assumption 3.6). Specifically, building upon the discussion given later in Appendix A.7, we note that the most general condition guaranteeing Assumption 3.8 is given in the second scenario of Appendix A.7, which requires that the function [( (t, A), I, A) satisfies the Lojasiewicz inequality with uniform constants over (t, X), for any selection Tg- (t, X) E arg maxøge [(TO, i, A) (again, see Appedinx A.7 for the definition of the Lojasiewiez inequality in this context). Since the (parametrized) policy optimization problem maxege [[TT, t, A) is nonconvex (and is approximately solved using solver like PPO), we can only expect to obtain an approximate solution 0 (t, X) such that
[( (t, 2)) (TTO) 1, A) S "00", ,,",,,, A) S E,
for some > (uniformly bounded in A)). Then, if ((TT- (t. A), satisfies the Lojasiewicz inequality for some uniform positive constants C, 17, we obtain that
||0º - 0 S Ce".
Thus, under the mere assumption that the sample gradient function V t, A) is L-Lipschitz continuous with respect to (which can be enforced, if necessary, by appropriate smoothing of the utility function g; a general smoothing strategy for generating smooth OCEs from non-smooth utilities can be found in Kouri and Surowiec [22, Example 2)), obtain that
VexC(Toraas) A) - Ve,2 t, 2) s iijt - 8- 5 iC="
Upon noting that V..x (TTge A) is an unbiased sample of - rt,,f(t, A) (cf. Appendix A.7), we can easily deduce that under the said minor conditions, the sample gradients utilized in Algorithm I satisfy Assumption 3.6. Thus, we can see that our oracle condition in Assumption 3.6 is justified and can be shown to hold under minimal conditions (without requiring convexity or uniqueness of the solution of maxage L( L A)).
A.6 Lipschitzness of the Lagrangian
First we show a technical lemma, and then we proceed with the proof of Lemma 3.7. Lemma A.1. For any pair tf, E 2m,+22 and for any RT it true that
sup L(T, E, A) sup [xx, 7, A) sup (( C(x, A)1
(15)
Proof of Lemma A.1 For any pair 7, E IR it is true that
sup C(T,T,,) sup C(x, I, >) I(=,Z,A) + L(T, A)
(16)
S sup C(T, T, A) -C(m,t,) A) + sup L(TT, F, X) =
(17)
sup F(,, sup L(x, F, < sup C( F, A) -(m,T, - sup L( T, 2) - -(T, (18)
Similarly, we can show that sup [(T, 7, A) sup [l, S sup [(T, 1, L(m, T, sup L L( A))-
(19)
The last two displays give the inequality (15).
Proof of Lemma 3.7 For any pair (f, t) R22+2 we find upper bound on the term L(T T, 2) [(TT, f, A)] as follows
[L(T, F, A) - L(T, A,)l
E
8
T0
z 3(o [o - ro((sr a+)) - to - 3 (lo -o((sr (0(s).aa1)).)
1/
1
17

---

m
7
Σ X
( - r((o,)
3.
+ E
m
1
Σ
8
31
(B: "1(5);
3
E
00
to
max too ro(s,, az).0))
80 max {To ro(or,
at): ro(s)rat))0)
B
00
n
7
X
t
max { 1hi-rin (((s))
B.
+ E
m
Σ
max { re((5, (..(),.)))))))
3
80
E
7 (10
-bol+ + 3
max {To ro((=;a7),00 a)) max (To ro(s,, az).0))))
0C
m
1
+ E
Σ 7"
>x((T - Tel + B:
max {T: - ri - max {T+ re(0).
20
$ E
7
(1oo - tol + 3 max {1oo - ro(s); - to - ro0(
00
m
1
+ E
?
>x((T= - Tul +
max {1r - rr(( - - -ri(0r: arr)rll)30 0})
B:
T==
1=1
80
x
m
_ i7 - E01
1
EEP
Σ
7" (1To - dol + 3 [00 tol)
+ E
7
Ae( - +4++ +
33
L==
i=1
(1-7)) (1 + 1 1
m Σ
Tol (1 --
17: 7.1
3
1 ) + Σ 12 + 3 $ (1-7) t|| / (1+ +
m
J2
Then, apply Lemma A.I get
C(T, - sup [(T, F, X) sup [ A) -(T,T,
(20)
s <1-7)--118 - (1+ (
m
12 +
1
)2
(21)
3
Thus the function ( is Lipschitz, with Lipschitz constant
m
8,7, (1 (1-))
(1 (1+ + )
Σ X2
3, )2
(22)
The result then follows since interested in multipliers À E A, with A a convex and compact set (and thus, there exists a constant C' 2 c(y, 3, A), independent of X and t, for which L(T, : A) is Lipschitz continuous for all At T x A).
A.7 Conditions That Guarantee Assumption 3.8
Let us note that Assumption 3.8 is not particularly restrictive in our setting. In what follows, we discuss some general conditions under which this holds readily, as well as a simple methodology of
18

---

enforcing this assumption. if necessary, by slight algorithmic adjustment. Before we do this, let us first note that [(TT, ;, X) is not necessarily differentiable (unless the probability of t, - (s, a) = is 0), because of the plus function (+)+- Nonetheless, adding small random noise to the reward following a random variable with a smooth mollifier density ensures that the function is infinitely differentiable with respect to both & and x. Note that, since we assume that the reward is bounded, we cannot (in theory) add Gaussian noise, but any smooth and compactly supported mollifier-based noise is admissible. Thus, we assume (without loss of generality) that [(TT," A) is differentiable (in fact, to an arbitrary degree).
Next, we focus on the task of assessing the differentiability properties of ftt, A). Specifically, we identify three general scenarios that guarantee Assumption 3.8 in our case.
1. The first scenario, which is somewhat restrictive, relies on the assumption that, for each (t, >) E T x A, the maximization problem, with respect to 8, admits a unique solution. In that case, we can readily utilize Lemma 2.2 Shapiro [40], which states that [(TT (t.A)s L, A) is twice-continuously differentiable. Since T x A is a compact set, this immediately implies that the function is E-smooth over T A, for some constant & O.
2. The second scenario is significantly more general, and relies on the strong second-order sufficient optimality conditions for the maximization over @ and the assumption that the function ((,)) t, A) satisfies the Lojasiewicz inequality with some constant uniform over (t, X), for any (t.A) arg max L(To, I, A), that is, for each (t, A) E T x A, there exist constants C and 17 > such that
ar. arg max Limo, 2, A) S C max ((TO, L(Too 1,) " BEE
dist (,
BEe
We note that the previous holds in several cases (typically with uniform exponent 77). For example, this is true when maxøee [(mo, ..-) is sub-analytic (which, for example, is implied if C(TT, t, A) is analytic, noting that this can easily be enforced our setting). Since T x A is compact and convex set, we readily utilize Theorem 11 from [16] to deduce that under this general assumption, the function L(T (t,A), t, A) is E-smooth over T x A, irrespectively of the selections 6" (t, A) E arg maXaçe [(TT, t, A).
3. Under certain qualification conditions, laid out in Section 4 of Shapiro [40] (that do not require the second-orde -onder strong sufficient conditions for the maximization problem over f), Ł-smoothness of maXage L( t, A) is also guaranteed. We refer the reader to Shapiro [40] for additional details, since this analysis directly applicable in our setting.
If any of the above conditions holds, then Assumption 3.8 is true, and we can readily compute the gradient of f as
Veas(1,)) V L, A) (..))))>
where 0 (t, E arg maxace C(TT, t, A) an arbitrary selection.
If none of the above is true, we can still guarantee that Assumption 3.8 holds by slightly altering Algorithm 1. Specifically, for some small f O, we can define the surrogate function
(TT (4 A) = BU, {
L + μU1, À + nU2)).
(23)
where U1, U2 follow a uniform distribution over the unit ball (of appropriate dimension in each case). Then, this new surrogate function can be made arbitrarily close to maxage L( t, A) (where the proximity is uniform in the constant μ1), while at the same time being f,-smoothh with ł,. O(1/u).
Then, can substitute the original Lagrangian with the smoothed Lagrangian given in (23) and Algorithm 1 on the surrogate problem. Following standard zeroth-order optimization techniques (e.g., see Pougkakiotis and Kalogerias [34]), we note that, in this case, the only algorithmic adjustments that need to be made relate to the computation of the sample gradients of f. which would require two (instead of one) evaluations of the policy oracle. In light of this discussion, we conclude that Assumption 3.8 can be utilized (almost) without loss of generality.
A.8 Proof of Main Theorem
In this section, we provide the proof of Theorem 3. 12. To prove the main theorem we will need the following auxiliary lemmas. The proof extends the analysis given in [25], since the presented algorithm only assumes having access to biased and inexact (sub)gradient samples.
19

---

Denote by b1 (6", St,t, A), 62(0", (t) the components of b(0", 07, tt, ,t,,, X) corresponding to t and X from Assumption 3.6.
Lemma A.2. VeL(Tor X)) and & ( X)) have bounded bias and variance.
Proof. By Assumption 3.6 and by linearity of expectation, have
E Ve [, E [ V (LA)). t, A)) + b(0", 0 2,,) -V A) + E(b(0", ,',r, A)]
(by Assumptions 3.6, 3.10)
The bias of the gradient is controlled by 6(0", 01, t, X), whose norm is at most 6, Furthermore,
V (TTr ADll
E
| ( (Tx (,) (( N)) + (EA t, &,))| A
=E
- 2E ||0(or, oT, t, 10]l + +2
$ (A))()
s /2/ + =2 + +0²)
(by Lemma A.2 of [25] with M = and Assumptions 3.6, 3.10)
Lemma A.3. Let A (k)
E ( - f((()), 1()))). The
The following holds for Algorithm 1:
E Φ 1 (t(K))
S 1 1 (t + 2n, (x-1)
Tt
(E(K (
4
+ 2n, 68 diam(T) + 3nf (C² + +50
Proof. Let 1) = prox, p (1(k-1)) and fix Tg. = (gi 1) 11 where l" E
arg maxsce L(TTO, ((--1), ((k-1)), and गgh 1 satisfying Assumption 3.6. Then,
[1-11) - 100) 12 = | - -TT (1(--1)- - VeL(Teer (k-1)) (0 ((=-1))) 1
$ g 11 + me V.L( ((k-1) 1 2
(projection is non-expansive)
-1 - ((x || + 1/2 2. V. (kk-1) 1
+ +21 (zih-1) - 11)-1) 1) V V.L(TTO ((k-1) x
Taking expectations, conditioned on (tx- Ax-1). 1). and using Assumption 3.6 we have
E | 16o0 ( -) i
((k |
p -r | 1 (1) ((t
< E
(1--1)) ( it V Vel(1l (1 it ((-- 00, ,r , 11,((1111))
+ 2m
b1 (0', , (1-1)) +t (اا | + Fi8 1. (1-1))) ((-1 ?
+ 3/
( (k (1=1)) V | 1 -لد! / (1( 1) 1 (0-
+ 31/2B E
(using || + y 2|I s s31|ا|2 + l1rl + +3112113)
20

---

Taking expectations of both sides and using Lemma A.2 of [25], that A( is C-Lipschitz (cf. Lemma 3.7), and Assumption 3.6,
B (1--1) - ,100
= E 1 - f(k-1) 2
2m (210-1) - 110-11,, ([(((t-1), ((0-11)) b(0", ot, (0,001)1,, a(b-11)) 33 (C² 0² + 8))).
Since T is compact, define diam == maX |> =ll. Using Cauchy-Schwarz Schwarz on the term
21)r (11h-1) - [k 1) bi (0", 01, (10-1), )((-1))
S 2m ( - )( | 10,100, 10 f(a-1) )((11))|-
and substituting, yields
B 1 - (10) 110) S E
S Th-1) - 21}, (T(k-1) - ((k-1), ( (111)))))))))
(24)
+ 2n.6 diam + 3 (C2 + 0² + +84).
From [25, Eq. (20)), obtain
$ 1/22 (t I $ (1)22(t(h-1)) + 1 _ )(h0)
-
(25)
while, from [25, Enq. (23), (24)), we obtain
() -(k-)) (ef((((-1), (
S S1(((-11), 1 11=11)) f((((--), 10(0--1,((0011 (1(0-1)) + 8
((o-1) )
(26)
2
(( - f(t(k-1) ( (11-11)
< (10((111) - f(t(l-1),
11k-1)) -((()))))))))) A (k-1)
( -(t-)))
(|=-
(27)
Plugging in (24) to (25), then combining (26), (27), and the fact that | - 1(R-121| 1 (1(k-1))|| /2,, finally gives
E (1)))
E (1/20(((4111) + 22, (k-1) The V (
+ 01 diam(T) + E(22 + -8)
which completes the proof.
Lemma A.4. Let A (k) E [0(1(()) - ((100), [[k)]]] and (²) E arg S(,, A). The follow ing holds for all s S k - 1:
A kk- 1) < 2n CV / C + +2( -2 -1)
1
-
-x((6))) /|² -E 1
E
2na
((E()), ((0)) f(((t-1) [((t)))]]] +6 diam(A) + 202 + +2)).
E
Proof. For any X E A, the update of (k) and convexity of A imply that
(A-21)) (a(4) ((- - 7/ VA (k
((--1))) 1 2 0.
Then, we have
- |x~>000²
100 s (1 - 2) (x (() (1=11))
21

---

+ - Vaf(E(E-1) 111-11
-
+11 - 12 11A(2) 1/
S 2 1) - TV, ( 116-11, )1
(1 11_ 1)VVA Vaf(t 111-11)
(11)
+ (m(()() - -1 (xf(e(H-1)) (1=11) + E2ā(TTr, ((t-1), ((0-1))
+ a - 1 1/2- || (k) - 11
where we let Toz (112- 1) 1)) (satisfying Assumption 3.6). Using Young's inequality,
7A (1(K)) - T( (Vas(c(H-1) 1) + V (T8 1k--1) 1(1-1)))
$ -
- \
2 || + nr Pi[((Trr ((1-1), ((=11)) + (xf(t((- 1), 20
Taking expectations on both sides, conditioned on (1(t-1),((A111) gives
E ||a-xco00 (1
2nn (orr) - - (-VAS(OLE-D), 1(k-1)) + =20 @t, g(k-1), 100-),,;000)) ,)1-111) (1-4-1)) - ->) V Vislele ( ((6-11 (cll |
(Assumption 3.6)
Mili 4
(x) 1 - /llceaa
( ( ((t
+ 1B
|
| , (ا= + VA 11 | (icaap ((--))
+ E
+E ||ا-~ x ( E - ? (rrt))
Taking expectation on both sides (and
E x (2)
$ 2 E
100-1--(11)) 1
2(0-1))+ + (((k-1)-))
V Vas(ElH-1) 1) ((r-a
) 2 1 | - x1-11//²= 21x - | 1), ((
12
+ EE
+ 2 | ( ((k-1), ( + Vaf(t(l-1), ,
From Lemma A.2, [25, Lemma A.2], and since diam(A) is finite, we get
E A (())
S 2 -((1)) T
V FAS(((-1), 2(0-1)) + (A(A-1) [
+ | - |(=
[1B11 11= + 2n diam(A) + 417(002 + -22). 2
1
Since f(te-1, -) is concave and A is convex (take a S 1/22),
B B|A- =|>->(001 SE/// - x(-1 2 + (1) ((0)) - -1((K-1),)))
22

---

+ 2nx diam(A) + 4n2 + +8))
Substituting X = (t(ss) (where s S k 1),
E
f(tlk-1)
(((()) f(((k-1),
1
- -
E |1/00 - x(((o)l(²) + 8 diam(A) + 21/(0²2 + 8²)
E
2n1
By the definition of A (k-1)
1(- & E f(t(H-1), X²) (-((--1))) - f(tlh-1), 1), (((16))
+ (1(111),()) s(1(R-1), (1)- + (11001)),(0))) 1(100
(B|1aa1) - -Xi(((0) 1² - -E| - xx((6))) - + diam(A) + 20² + +82).
27/x
Following the steps in [25, Lemma D.4), using that f(; X) is C-Lipschitz (by Lemma 3.7) and Lemma A.2, we have
E [sCO xx((l - X² x (x((((-11))
s 2 02
E [[r (((())) - [((()-)),,)))))) ((((0)))(((0)))
s
02 C2 60
f(((t-1) (()))) - f(t ((1h0,100)) {())]
+2 52
Using [25, Eqn. (25)].
A (k-1)
2n, C / C2 +
02 + 82. (2t 2s -1)
1
| ('((())) 1 1|a --B
E ((() - x(((
+ 2
E
+ E (((6(),2(*)) - f(((A-1)) ((11111 + 8. diam(A) + 20 + 52)..
Lemma A.5. Let (R) = =E [+(t(()) - ,(((0),((1)]] Let B JJ+1 be such that (J + 1)/ is an
integer. The following holds
1
Σ A
s
J+1
diam(A)2
Bo
+ diam(A) + 2n2 +8²)+
2Byx
J J+1
Proof. We divide {A()))
/ k=u into blocks where each block contains at most B terms:
A
(b)
B-1
A
(4)
2B-1
A (R)
k=
K=B
K-J-B+1
Then,
(J41)/B-1
(1+1)8-1
]
1
Σ
A
(k)
B
Σ
Σ
A (4)
(28)
J J+1 I
J+I
k=
1=0
k=
23

---

Letting = 0 and applying Lemma A.4,
B
1 E 110) - x xx(00))
(k)
22 2m. 2m.C 02 +.2 02 B² + 277x
k=
E ( ((3)) - f(((0), f(t(0),((1)) {
8B diam(A) + 27, B(² +
2n, o2 + 82 B² E [(1 (1) s(1 )(11))110))
diam(A) (A)² 8B. diam(A) 2n, B(o² + +
2m
Letting 8 = iB and applying Lemma A.4,
(+11)--1
Σ
A
22. C2 + +2 + + (ollBBB)),((B+1000 f(1(t)), {(oli)
k=
diam(A)2
5B diam(A) + 211 B(o² + +22)
2713
Substituting these into (28):
1
A (k)
s 2n,CV C2 + 02 8
J+1 E ((t1)) - (to:Ao))
J+1
k=0
diam(A)2
+ diam(A) +n2x(02 (o² + +22).
2B11
By the Lipschitzness of f(,
1(1411, - (o, () 1) +
This yields
Σ Ax
J+1
k==
+ diam(A)² + 8 diam(A) 20² +88)+ + J J+1
Ao
2ByA
Proof of Theorem 3.12
Summing up the inequality from Lemma A.3, over k = 1,..., + 1, yields
E [1/22(( < (t)))
+ 2nd Σ (k)
Th
E
V 1 (1(t)))
4
k=
k=0
+ (2 diam(T)
3 1C22 + + +0)) (J+
Applying Lemma A.5,
E Φ [01/2(((U433)]
1 2 (2()))(
+ +
( + 1)
diam(A)2
+ 2BTx
+ 5- diam(A) + 27) + +00)
24

---

2m
7h
Σ E v (0))))(
- (2mol diami() + 3n + +f (J +
By the definition of Âg, we obtain
V (2e
J+1
k=0
4 +
diam(A)² 2BTa
8e 22 02 +B++1++
+ odiam(A) + 2n + +82)
ne(J + 1)
gp
+ J+1 12m + + 0²) + 868 diam
Letting B diam(A)
we obtain
4
Σ E V
248 diam(A) v ICCCC4022
J+1
ne(J +1
8f.D
16 +8²)+ + J+1
+ 12n, + 52) + 828 (Tiam(T) + diam(A))
With the choice of step sizes
1
2
7 = min
(29)
166 + +22) { 26
8(0 +2 (2))
8
V/ min
O
CV C2C + 0² - 82
(30)
Adiam(A)]
e
(A)3 (2) CVC2 +
have
Σ E V 4
/ (t2
J + 1
A==
4 b
+ 868 (diam(T) + +(())
(J +1)
1+1
Thus, have iteration complexity of
[(C2 + t
82)
Ao
max {I. (diam(A)2 (diam(A))(22 (00o2++++)) +89) P008
2
for recovering an e2 + de(diaam (T) + (iamm()) (aaam(()-sstatinnn) -stationary point (cf. Definition 3.11). By simplifying, we obtain the desired iteration bound:
o (( ( ((110*+ + o² + 82) (diam Do @(diam(A)) :Bo { o² + 82
})
6
2
25

---

B Experimental Details
B.1 Simulations
Constraint tasks central class of real-world RL problems with applications in robotics, tonomous vehicles, and industrial control. To simulate realistic conditions, we use the locomotion tasks from the Gyymas--Gymmmiimm suite [19]. Further details, including visualizations of both con- strained and unconstrained agent behaviors, are available on their website
Velocity Cost The velocity cost is defined as:
cost = bool (veleuren >> velthreshold),
where the velocity thresholds velarrshold for the tested environments are listed in Table 3. As reported by Towers al. [47], these thresholds set 50% of the maximum velocity achieved by each agent after PPO training for 10? steps. At each time step, the agent's instantancous velocity is computed as:
veleureett vel" vel cuureet + vel
(31)
y.cumentt
where velg, and vely, are the agent's instantaneous velocities along the x- and y-axes, respectively, provided by the simulator.
Environments We use the latest version (v1) of Sefety-Gymnasium without modifying the state, action, or reward space. Constraint violations do not alter the agent's behavior or environment dynamics. As a result, vanilla PPO converges to high reward solutions with large cumulative costs effectively ignoring the constraints. All actions are normalized to the range [-1, 1]. To isolate the effects of our method, we reset the agent to the same initial state after every termination common practice in robotics.
Episode Terminations Episodes terminate either when the time limit is reached or when the agent fails (e.g.. by falling). Notably, HalfCheetah, Swimmer, and all safe navigation environments have no failure condition; their episodes always end due to the time limit.
Table Environment-speecffe parameters used in our experiments.
Variable
Safe Navigation
HalfCheetah
Hopper
Swimmer
Walker2d
Threshold
0.0
3.2096
0.7402
0.2282
2.3415
Failure condition
x
Time limit (steps)
500
1000
1000
1000
1000
Initial CVaR variable fmmi
0.0
-1.3
-0.1
-0.0
-0.975
Stochasticity for Risk Management The environment fully deterministic iddnticall actions from the same state always yield the same rewards and transitions-makine it difficult to evaluate risk-sensitive behavior. To simulate uncertainty without altering the environment's 's internal dynamics, we inject zero-mean Gaussian noise (std. 0.05, i.e., 5% of the action range) into all agent actions at every step during both training and evaluation. This controlled perturbation introduces stochasticity in action execution, enabling us to assess how well the agent manages risk under uncertain conditions while maintaining consistent environment behavior.
Evaluation Agents are evaluated every 1000 time steps by averaging the undiscounted sum of rewards over 10 episodes. The PPO agent uses the mean action, ensuring consistency in evaluation. Evaluations are entirely separate from training- no data stored, and no network updates are performed.
'nttps://safety httpa//aw..aacooo.....e/eeeeeeee
html
26

---

B.2 Proximal Policy Optimization -olver
We use Proximal Policy Optimization [39] as solver learn policy (i.e., line 4 in Algorithm 1). Our method serves wrapper, described next, that modifies the agent's 's raw reward to incorporate risk measures and constraints.
PPO first collects rollouts of state-action-reward sequences using the current policy, storing them as trajectories. Once sufficient data is gathered, it applies minibatch learning, splitting the rollout data into smaller batches and iteratively updating the policy over multiple epochs. During this process, the dual and CVaR variables remain fixed.
Neural Networks The value function and policy are approximated by neural networks, each with two hidden layers of 64 neurons using the (anh activation function. The value network processes states s and outputs scalar value. The policy network takes states as input, extracts hidden features, and passes them to Gaussian distribution with learnable mean and standard deviation parameters. The action a then sampled from this distribution.
Table 4: PPO hyperparameters used in the experiments.
Hyperparameter
Value
Optimizer
Adam 3 10 /
Learning rate (all networks)
Linear learning rate decay
Adam
10-6
Adam aE
0.99
# rollout steps
2048
# minibatches per rollout
32
# epochs
10
Discount factor 7
0.99
GAE X
0.95
Entropy coefficient
0.0
Value loss coefficient
0.5
Maximum gradient norm
0.5
Clip parameter
0.2
Hyperparameters We employ Generalized Advantage Estimation (GAE) [38] estimate advan- tages in PPO. The hyperparameters used by the PPO agent is provided in Table 4.
B.3 Reward-Bard Beward-Based SGD with Risk Constraints wrapper
We follow the same rollout strategy to optimize the dual and CVaR variables. First, the policy is updated using collected rollouts while keeping À and t fixed. Then, a new rollout is collected with the updated policy and used to update À and f, while keeping the policy parameters frozen.
B.3.1 Implementation
To balance reward maximization with constraint handling, we frame the problem as standard risk- neutral reward maximization subject to constraint that regulates violations through the conditional value-at-risk (CVaR) of the constraint quantity.
r S x A + R be the reward function and v x A + IR a constraint-quantifiing function, e.g., velocity function in (31). We want to solve:
sup E
Σ r' m(s)))
s.t. (o Sc.
TEP(S)
==0
This constraint can be equivalently written as
CVaR (v(s,a)) Sc = -CVaR (-u(s, cc,
27

---

Table 5: Common hyperparameter values used across all environments.
Variable
Value
VaR level 8
0.3
# trajectories used compute gradients of and
8
Initial dual variable Anni
0.0
Step size T/A Learning rate decay on 1JA
5 x 10
X
Step size
5 x 10-5
Learning rate decay on T)t
which aligns with the supremal convolution form of the reflected CVaR. Rewriting the constraint:
-CVaR (-u(s,
sup
E tE
o Σ 7"
(
3 (1 v(8r. ((0r:
2-1-2
Thus, we solve the following constrained problem:
sup TEP(S),EE
E
Σ (((sr,m(-)))
(1-J
E
We implement this by modifying the reward at each time step using the Lagrangian:
r(sr, ((r))(s-))++ >; (c+l- j (e u(s m(s))))-))
(32)
where PPO is used as a black-box solver for the inner maximization over æ,, while À and are updated using single stochastic gradient descent and ascent steps as in Algorithm 1.
B.3.2 Hyperparameters
All hyperparameters used in algorithm are listed in Tables and 5.
The parameter controls which quantile mean constrained. Higher values (close to enforce more risk-neutral constraints, while lower values focus on rare events. We chose 3 = 0.3 to strike balance: strong enough risk control avoid constraint violations, while keeping the problem solvable.
Setting 3
Number of Trajectories for Gradient Computation We tested n = {2, 8, 16] (in Algorithm 1) and concluded that offers the best trade-off between runtime and gradient smoothness.
Step Sizes 7 and N Step sizes were extensively tuned on Hopper and Walker2d. We tested values from 10-3 to 10-7. The best-performing configuration on the convergence
5aa 7) = 5 x 10 was selected based
and within 15M time steps.
Initial Value of Since must take values in the negative real range (due to the supremal form), we expect it to converge to the negative value-at-risk. Thus, we initialized such that, with step size ?/ x 55 its magnitude could reach the velocity threshold over training.
Initial Value of X Because scales the penalty term added to the reward in (32), we initialized neutrally with x 0.0 avoid overly aggressive penalties at the start.
28

---

B.4
Computational Resources
All experiments were performed on computing system powered by an AMD Ryzen processor with 64 cores and 512 GB of RAM. A single NVIDIA RTX A6000 GPU with 48 GB VRAM was used for neural network training.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims make in the abstract and introduction are that propose new framework for risk-aware RL with constraints, that exhibits per-stage robustness, admitting exact equivalent partial Lagrangian relaxation under appropriate constraint qualifications, convergence analysis, and experiments demonstrating such. These are supported by the related works (Section 1), Section 2.3 and Appendix A.1, Section 3.1-3.2, Section 3.3, and Section 4, respectively.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include separate paragraph addressing the limitations at the end of the manuscript. In addition, throughout we qualify our results and claims (e.g., when introducting assumptions like Assumption 3.8). We also clearly state the environments on experiments, emphasizing that simulations are not exhaustive and
which we serve as a proof-offwoor,, with computational resources outlined in Appendix B.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
30

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: made efforts to clearly state assumptions and claims using the appropriate environments to make them obvious. All proofs are provided either in the main body or in the appendix to save space.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our results reprodueible and provide the hyperparameters, training procedure, and the (anonymized) code to facilitate review and reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
31

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The data/enviromment we use is synthetic and open source. We provide our code in an anonymized repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ public/ Seaaddde//ooeSuumiiee ciiioooooiiiy Policllyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. wwwpppugguuuuua.p.ppppppppapaeep gewidde/Coedeuuaiiiii hwt////ooebmmssiiiooio.. swisssoliccyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
These experiment details are all provided in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
32

---

Justification:
This note applies indirectly and exclusively to the histogram shown in Figure 1. Our intention to estimate the probability distribution specific variable, aggregated over 100 trials as noted in the figure caption.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We detail this information in the last subsection of Appendix B (in B.4).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : hww.rnuu....//.ee//eee/eee/..u.m cc/
Answer: [Yes]
Justification:
We have read the Code of Ethics and believe our work adheres to it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
33

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is mostly theoretical and the experiments are done on synthetic environments, thus we believe our work as minimal societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Similar to the previous section, our experiments are done in clean synthetic environments thus this section does not seem applicable to our work. We do list limitations, as mentioned before.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite [19] and include the URL as well as the license in the main text.
34

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not include human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
35

---

Justification:
Our work does not include human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is required.
Answer: [NA]
Justification: There was no usage of LLMs in this work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https htwpr//pssc///ooteeeenne//////// for what should or should not be described.
36


Reviews:
[{"review_id": "LSsWJVLBQU", "reviewer": "Reviewer_uZsP", "summary": "This paper proposes a novel framework for risk-averse constrained reinforcement learning (RL) by leveraging Optimized Certainty Equivalents (OCEs) to address the limitations of standard expectation-based RL formulations, which often fail to account for tail risks or catastrophic events. The core idea is to apply per-stage risk measures, rather than return-based ones—to both the objective and constraints using OCEs, such as Conditional Value-at-Risk (CVaR). They reformulate the constrained RL problem using a partial Lagrangian relaxation, which under suitable constraint qualifications yields an optimization problem that is provably equivalent to the original problem and amenable to a stochastic minimax solution. They show strong duality in this reformulation and provide a non-asymptotic convergence analysis of the resulting algorithm.\nIn the experiments, their algorithm wraps around any black-box RL solver such as PPO, enabling integration with existing pipelines. Experimental results demonstrate that the proposed method effectively manages risk while maintaining constraint satisfaction and stable convergence.", "questions": "Questions\n- Please see the weaknesses section.\n- The proposed method uses a standard black-box RL algorithm in the inner loop to optimize a modified reward function for fixed dual variables. It is well-known that the optimal policy for a CMDP is generally stochastic. However, many RL solvers, when optimizing for a fixed reward function, will output a deterministic optimal policy. This seems to create a potential mismatch: the outer-loop problem requires a search over the space of stochastic policies, while the inner-loop solver might only explore a subset of near-deterministic ones. Could the authors comment on this? Does this mismatch potentially prevent the algorithm from finding the true stochastic optimum of the original CMDP?\n- Algorithm 1 specifies that the final policy is chosen by uniformly sampling an iterate j* from the entire training history. This seems counter-intuitive, as policies from early in the training (small j) are unlikely to satisfy the problem's constraints, given that the dual variables have not yet converged. Returning such a policy would be unacceptable in a safety-critical context. Could the authors clarify the rationale behind this uniform sampling strategy?", "limitations": ".", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The paper addresses the crucial challenge of integrating risk-awareness into constrained RL, which is essential for deploying RL agents in many risk-averse problems in real-world.\n- The paper presents a method to optimize the OCE-constrained objective for risk-averse reinforcement learning with theoretical analysis.\n\n\nWeaknesses\n- Insufficient comparison with baselines and related work. The paper fails to adequately position its contributions within the existing literature through direct methodological and empirical comparisons (e.g. [1,2,3]). This is a significant weakness as it makes it difficult to assess the practical advantages and novelty of the proposed framework.\n- The paper's choice to constrain the per-step cost CVaR, rather than the more conventional return-based CVaR, is a major methodological decision that is not sufficiently justified. While this approach may be better suited for capturing instantaneous, catastrophic risks and offers theoretical tractability, it may also lead to overly conservative or myopic policies in scenarios where overall episodic safety is the primary goal. The paper lacks a discussion of this critical trade-off and fails to provide an empirical comparison against a return-based risk management baseline to demonstrate in which specific contexts its approach is superior. This omission makes it difficult to understand the true scope and limitations of the proposed framework.\n\n\n[1] Chow et al., Risk-Sensitive and Robust Decision-Making: a CVaR Optimization Approach, 2015\n\n[2] Ying et al., Towards Safe Reinforcement Learning via Constraining Conditional Value-at-Risk, 2022\n\n[3] Achiam et al., Constrained Policy Optimization, 2017", "was_revised": true, "final_justification": "I appreciate the theoretical contributions of this work, and the theoretical rigor justifies raising the score. However, the lack of experimental comparison with established safe RL algorithms limits the impact of the work. Therefore, I am only raising the score to borderline reject. A more thorough experimental evaluation including comparisons to relevant safe RL baselines would be necessary for full acceptance.", "rebuttal": "Thank you for your questions and observations! We hope that we are able to address the points of weakness and related questions below. \n\n---   \n### **Weaknesses**   \n\n1. Thank you for pointing out these additional references. We will include these comparisons in the related work section.  \n\n    - **Reference [1]:** We cite the later work of Chow et al. (reference [12] in our paper), which addresses constraints, whereas [1] focuses on replacing the expectation in the objective with CVaR in a return-based formulation without constraints.     \n\n    - **Reference [2]:** It applies risk measures to constraints in the return-based setting and appears to focus on a risk-neutral objective with a CVaR constraint (see Eq. (11) in [2]). \n\n    - **Reference [3]:** It proposes solving CMDPs via a local policy search algorithm, where the constraint enforces proximity to the previous policy iterate using a distance metric. In contrast, our method is not based on local policy search and does not impose constraints based on policy distance.   \n    \n \n2. Our goal is to position this work primarily as a theoretical contribution, with the following key strengths:\n    - a parameterized strong duality,\n    - an interesting interplay of convex analytic and Lagrangian duality,\n    - a modular algorithmic framework that accommodates a broad class of OCEs,\n    - and the ability to handle per-step cost CVaR constraints, which differs from most prior work that focuses on return-based CVaR.\n\nNotably, strong duality is a property that does not hold for many constrained problems, which are often solved via relaxation. We provide a complete theoretical treatment, including a novel convergence analysis and detailed discussion of assumptions and their practical relevance (see Appendices A.1, A.6, and A.7). In this regard, our approach aligns with prior theoretical works such as the given reference [1], which similarly emphasizes rigorous analysis over extensive empirical comparisons.    \n\nThat said, we acknowledge the importance of this trade-off and will aim to include additional experiments and return-based baselines, if time permits and the paper is accepted.  \n\n---    \n### **Questions**\n\n1.\tThe PPO solver employs a stochastic policy, modeled as a Gaussian distribution with learnable mean and standard deviation, as noted on line 727. Importantly, our framework does not require the final policy to be deterministic. As you pointed out, recovering the true stochastic optimum of the original CMDP requires maintaining stochasticity, which our algorithm preserves through the use of such policies.\n\nWe will add 1–2 clarifying sentences to the paragraph starting on line 281 to make this point explicit.\n\n2.\tIn the min-max optimization literature, theoretical guarantees are typically established for average or randomly selected iterates, as last-iterate convergence is generally difficult to prove without additional structural assumptions. Uniform sampling of an iterate is a standard approach in nonconvex optimization, including for gradient descent and stochastic gradient descent, to ensure convergence to stationary points. In practice, one usually returns the iterate when the learning curve stops changing significantly.\n\nThat said, in our practical implementation—given the use of a deep reinforcement learning solver in high-dimensional environments—we return the final policy observed when the learning curve stabilizes. This is noted on line 235 in our wrapper code (`wrapper > RA_C_RL.py`).\n\nThank you for bringing this observation; we will clarify this point with 1–2 sentences in the text."}, {"review_id": "whnBfEhA51", "reviewer": "Reviewer_WADh", "summary": "Many existing safe RL works rely on expected cumulative constraints, which are not applicable to safety-critical RL problems. To this end, this paper proposes a risk-aware constrained/safe RL scheme. Several theoretical guanrantees such as the convergence and the parameterized strong duality are established. In addition, the practical implementation can be wrapped around the standard unconstrained RL algorithms like PPO, SAC etc. The proposed method is also substantiated by numerical experiments conducted on well-known Safety-Gymnasium benchmarks.", "questions": "1. Does the stepsize need to be carefully tuned? How can we select a proper stepsize for a new/unknown environment?\n\n2. The authors should include more references regarding probabilistic-constraints (a.k.a. chance-constraints) in safe RL.\n\n3. The authors should provide more experiments to validate the robustness of the proposed method on other type of tasks, e.g., Safe Navigation, Safe Vision, etc. They are all provided in Safety-Gymnasium.\n\n4. Please provide the exact formulations or details for the subgradients in Line 5 of Algorithm 1.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n1. This paper is well-written and well-presented.\n\n2. The topic of this paper (risk-averse safe/constrained RL) is very significant for the community and real-world applications.\n\n3. The proposed method is supported by solid theoretical guarantees and experimental validations on well-known safe RL benchmarks.\n\n4. This work can be wrapped around the standard unconstrained RL algorithms such as PPO, TRPO, SAC, thus easy to implement.\n\n5. The convergence of rewards and dual variables are validated by empirical evidence.\n\n\n**Weaknesses**:\n\n1. This paper does **NOT** compare with any other risk-aware safe RL baseline methods. It is hard to evaluate the practical performance of this work without comparing with some baselines.\n\n2. No comments/discussions on the assumptions are provided, see e.g., Assumption 3.4. The proper discussions on the assumptions are necessary although I understand the concerns for main text space.\n\n3. The proposed method is only evaluated on one type of tasks, i.e., safe_velocity. Please see my suggestions later.", "was_revised": true, "final_justification": "The rebuttal looks good given the limited time. I would encourage the authors to include more advanced safe RL baselines in Safe Navigation for comparison, at least in the camera-ready version. At this stage, I will maintain my positive score.", "rebuttal": "Thank you for the careful review and thoughtful suggestions! We address the identified weaknesses and respond to your specific questions below.\n\n---    \n### **Weaknesses**\n\n1. Please see our responses below.\n\n2.  We included some discussions of assumptions in Appendix A.6 and A.7 (for Assumptions 3.6 and 3.8, respectively). We will take care to add a brief discussion of Assumption 3.4 as well but in the main body to improve readability. \n\n3. Please see our responses below.\n\n---    \n### **Questions**\n\n1.\tWe did not carefully or extensively tune any of the step sizes.\n\n- **PPO agent:** We did not experiment with alternative step sizes for the PPO solver. We adopted a value previously found effective in our other projects using the same MuJoCo environments.\n\n- **CVaR and Dual Variable:** For the CVaR and dual variables, we used a smaller learning rate relative to that of the networks (e.g., the policy), based on the common practice of using lower step sizes for scalar parameters due to their increased sensitivity to gradient noise and lack of averaging effects present in high-dimensional updates. We tested three magnitudes: $5 \\times 10^{-4}$, $5 \\times 10^{-5}$ (chosen), and $5 \\times 10^{-6}$. The middle value provided better stability than the first and faster convergence than the last.\n\n2.\tWe will add additional references and relevant discussion on probabilistic (chance) constraints in safe RL. If the reviewer has specific examples of related work they believe are particularly relevant to our approach, we would be happy to consider and include them.\n\n3.\tWe primarily chose Safety-Velocity tasks because they are based on MuJoCo environments, which are widely used benchmarks in deep RL. These tasks are among the most realistic and challenging within the Safety-Gymnasium suite, involving high-dimensional state and action spaces, 3D spatial configurations, and complex physics dynamics. In contrast, many other tasks in the suite, such as those in Safe-Navigation and Safe-Vision, tend to be simpler—often limited to 2D or relying on lower-dimensional inputs.\n\nThat said, in response to your suggestion, we have begun experiments on Safety-Navigation. We did not include Safety-Vision due to the additional need for CNN-based image processing, which may not be feasible within the current timeline. If completed in time, we will include Safety-Navigation results in the camera-ready version.\n\n4.\tThank you for the suggestion! The subgradient in Line 5 of Algorithm 1 can be computed using Equation (8) along with the definition of $r'$ provided on line 149. Still, we will make sure to include this formulation explicitly in the paper for clarity."}, {"review_id": "uHbenpY2xk", "reviewer": "Reviewer_6DSm", "summary": "This work proposes a risk-averse constrained reinforcement learning framework that integrates Optimized Certainty Equivalents (OCEs) to address the limitation of traditional constrained RL, which neglects tail risks in high-stakes applications. By formulating objectives and constraints using reward-based risk measures, the framework achieves per-stage robustness in both reward values and time. It establishes a parameterized strong Lagrangian duality, ensuring exact equivalence to the original constrained problem under appropriate constraint qualifications. The authors design a modular algorithm that can wrap standard RL solvers and prove the convergence of the algorithm under common assumptions. Experimental results on locomotion tasks in Safety-Gymnasium demonstrate that the method can effectively reduce violations of risk constraints while maintaining competitive reward performance, validating its risk-aware properties.", "questions": "1.The constraint (Assumption 3.4) is crucial for precise duality, yet its practical effectiveness across different environments remains unclear. Could the authors provide empirical evidence of violations of this assumption and discuss the algorithm's performance in such scenarios?\n\n2.This paper focuses on CVaR, but OCE encompasses other risk measures (e.g., entropy-regularized variants). How does the framework perform under these measures? Supplementary comparisons would highlight its flexibility and identify which OCEs are most effective for constrained reinforcement learning.\n\n3.The experiments exclude high-dimensional environments such as Ant. Could the authors test the method in such environments to verify its scalability? It would also be valuable to supplement comparative experiments with existing state-of-the-art baseline algorithms.\n\n4.How sensitive is the algorithm to β across different environments? Could sensitivity analysis be supplemented to clarify its impact on the risk-return trade-off?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths：\n\n1. This work is technically rigorous with a solid theoretical foundation, providing detailed proofs of parameterized strong duality and non-asymptotic convergence guarantees, along with basic simulation experiments. The authors candidly acknowledge the limitations of this paper and validate the relevant claims with consistent results.\n\nWeaknesses：\n\n1. The constraint conditions for exact duality (Assumption 3.4) are relatively stringent, and their practical applicability across different environments has not been thoroughly examined. Furthermore, this paper fails to provide sufficient comparative experiments.\n\n2. The core ideas of this research (including OCE in reinforcement learning, Lagrangian relaxation of constraints, and CVaR in risk-aware optimization) are derived from existing concepts. Its originality lies more in integration rather than the proposal of entirely new concepts, which significantly undermines the innovativeness of this research.\n\n3. The connection between the partial Lagrangian relaxation method and the original constrained problem lacks the necessary intuitive explanation, which impairs the readability of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and insightful comments. We hope the responses below effectively address the identified weaknesses and answer your questions.\n\n---    \n### **Weaknesses**\n\n1. Please see our response to Question 1 below.\n\n2. Thank you for your feedback. While our work builds on established tools, we believe the contributions lie in their nontrivial integration and the resulting theoretical framework. The conclusions we draw and the guarantees we establish are not straightforward applications of existing results but require careful analysis and novel synthesis across these domains.\n\n3.\tWe will revise the exposition to make this connection clearer. Specifically, the problem in Eq. (6) corresponds to the left-hand side of the equation on line 180. We will add a reference to the earlier constrained problem and briefly remind the reader of the Lagrangian duality framework to improve clarity and readability.\n\n---    \n### **Questions**\n\n1. This is a valid point—thank you for raising it. Although this condition may be hard to check in practice, the framework still serves as a valid Lagrangian relaxation—even when this assumption is not met and in the absence of strong duality—as is often the case in constrained optimization. \n\n2. We focus on CVaR as it is one of the most widely used and well-understood risk measures, which we believe helps with clarity of exposition. That said, our framework is designed to accommodate a broad class of OCEs. We can add explicit equations and explanations in the Appendix to illustrate how the formulation and algorithm would adapt to other OCEs to further highlight the flexibility of our approach.\n\n3. Thank you for the suggestion! We will aim to include additional experiments, including comparisons with state-of-the-art baselines, if the paper is accepted. As noted in the limitations section, we anticipate that computational cost may become a bottleneck in high-dimensional settings.     \n\nIn our preliminary experiments, we evaluated a vanilla PPO agent in a deterministic, non-constrained Ant environment. Across five seeds, the agent failed to converge to meaningful rewards, indicating that the solver was insufficient for this setting. Consequently, we excluded Ant from our main experiments.     \n \nWe also ran vanilla PPO on the reported environments as a baseline for comparison. However, to better isolate the practical effectiveness of our proposed risk-aware method, we chose not to report those results. The vanilla PPO agent violated constraints in 85–90% of episode steps, and the rewards were highly volatile due to noise injected into the actions.      \n\nIf preferred, we can include the vanilla PPO results in the final version. We will also add a brief note in Appendix B.1 explaining our findings in the Ant environment.\n\n4. While the algorithm itself is not directly sensitive to the $\\beta$ parameter, the constraint-adhering behavior of the final policy—i.e., the average frequency of constraint violations—is. By definition, $\\beta$ specifies the risk level by setting the percentile of the worst-case outcomes over which the expected cost is computed. For instance, if we use a larger $\\beta$, and assuming the dual and CVaR variables converge, we would generally observe more constraint violations, as the policy is optimized over a broader range of outcomes and is less focused on avoiding the worst-case scenarios."}, {"review_id": "u3BXikm3g8", "reviewer": "Reviewer_enYi", "summary": "This paper proposes a risk-aware constrained reinforcement learning framework that incorporates reward-based risk measures into the objective and constraints. It extends prior work by enabling constrained settings and establishes a strong duality result that allows for an exact partial Lagrangian relaxation. The authors develop a modular online algorithm compatible with various RL methods and provide theoretical convergence guarantees. Experiments on standard benchmarks demonstrate improved risk management and reduced constraint violations.", "questions": "While the proposed method is theoretically distinguished from Bonetti et al. [7], particularly in its ability to handle constrained settings and inexact policy solvers, the lack of experimental comparison with [7] leaves some ambiguity regarding its practical advantages. Given that the Safe Velocity benchmark does not enforce hard constraints on dynamics (i.e., constraint violations do not terminate episodes), it may be feasible to apply [7] in this setting, at least in a limited form. Even a simple empirical comparison—e.g., reward, CVaR, or policy stability—would significantly enhance the reader’s understanding of how the proposed method performs relative to prior work. If direct implementation is difficult, an ablation study or visualization highlighting the behavioral differences would still provide valuable insight into the practical impact of the proposed approach.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strengths\n  - The proposed method ensures an exact equivalence to the original constrained problem via parameterized strong Lagrangian duality, providing solid theoretical guarantees.\n  - The modular algorithm can be easily integrated with standard RL solvers such as PPO, making it practical and broadly applicable.\n  - The proposed method addresses tail risks and catastrophic events, which are critical in high-stakes applications.\n  - The evaluation is structured around three measurable goals—convergence of auxiliary variable to CVaR-beta, stabilization of the dual variable \\lambda, and safe yet reward-maximizing policy behavior, which align well with the theoretical contributions.\n  - The authors simulate realistic uncertainty by injecting Gaussian noise into agent actions in an otherwise deterministic environment, enabling a meaningful evaluation of risk-sensitive learning.\n- Weaknesses\n  - The experiments are limited to a single benchmark suite (Safety-Gymnasium), which restricts the ability to assess the proposed method's generalizability to other domains or task types.\n  - The paper lacks a comparative evaluation against baseline methods. While the authors provide a clear theoretical distinction from Bonetti et al. [7]—noting that [7] is limited to the unconstrained setting and assumes exact policy solvers—the practical impact of these differences remains unclear. Since both methods operate in similar reward-based risk-aware frameworks, a direct empirical comparison (even if limited) would strengthen the justification for the proposed approach.", "was_revised": true, "final_justification": "Since additional experiments with the baseline methods have not been included at this point, I will maintain my original score. The strengths and weaknesses remain as previously stated.", "rebuttal": "Thank you for your thoughtful review and for raising this question, which we had considered during the development of the manuscript. We hope the following response effectively addresses the identified weaknesses and provides a clear answer to your question.\n\n---    \n### **Weaknesses**\n\n1. We primarily selected Safety-Velocity tasks as they are based on MuJoCo environments, which are widely recognized benchmarks in deep RL. These tasks are also the most realistic and complex within the Safety-Gymnasium suite, featuring high-dimensional state and action spaces, 3D spatial structure, and intricate physics dynamics. In contrast, many of the other tasks in the suite are either toy problems, Atari-style environments, or limited to 2D settings.\n\nNonetheless, we have initiated experiments on Safety-Navigation as well and will include the results if they are completed in time for a camera-ready version.   \n\n2. Please refer to our response to your question below.\n\n---    \n### **Question**\n\nThank you for the thoughtful question! Our experiments were primarily designed to demonstrate the theoretical properties and practical behavior of our algorithm, rather than to benchmark against prior methods. That said, we acknowledge the value of such comparisons.\n\nWe were unable to implement the approach of Bonetti et al. [7], as their block-coordinate descent algorithm as written in the paper is missing implementation detail and is not accompanied by publicly available code. (Our attempts to obtain it from the authors were also unsuccessful.) Nonetheless, our framework can be adapted to approximate their setting—for example, by fixing the Lagrange multiplier to zero and disabling updates—allowing us to mimic an unconstrained optimization scenario similar to [7]. If the reviewer believes such a comparison would improve the clarity or impact of our presentation, we are happy to include it.\n\nWe also note that our approach, which leverages stochastic gradient methods, is conceptually distinct from the optimization scheme in [7]. Even in the absence of a direct implementation, we will consider including ablations or visualizations to highlight behavioral differences, should space and time permit."}, {"review_id": "LHdmpJbsi6", "reviewer": "Reviewer_uAkR", "summary": "This work proposes a risk-aware constrained RL problem with both reward-based objective and constraints, for which this work established parameterized strong duality relation, online primal-dual algorithm and its convergence result, and verified the convergence by experimental results.", "questions": "(1) **My major concern** is in the setting novelty. \"Our work is the first to handle reward-based constraints, covering a large class of risk measures (OCEs)\" in the introduction looks like an overclaim, since [12, a] use reward-based constraints. \n\nYour setting seems to be a special case of the risk measure-constrained RL (RCRL) [a], by restricting the function $g$ in Eq. (2) of [a] to $\\{g(\\cdot-t):t\\in\\mathbb{R}\\}$ for a fixed $g$. \n\nCould you cite [a] and describe your advantage over [a]? \n\nAlso, you present your results in CVAR and claim they hold for general OCEs at the beginning of Section 3.1. Can existing CVAR results like [12] also extend to general OCEs? \n\n**I would like to raise my rating if you can solve this concern.**\n\n[12] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18, 12 2015.\n\n[a] Kim D, Cho T, Han S, et al. Spectral-Risk Safe Reinforcement Learning with Convergence Guarantees, Neurips 2024.\n\n(2) To ensure Lemma 3.2 holds for general OCEs, what conditions should the function $g$ satisfy? \n\n(3) Could you define $\\widehat{\\nabla} _ {t,\\lambda}$ in Assumption 3.6? You could add the definition to your revised paper. \n\n(4) How can Theorem 3.12 indicate the quality of the output policy? \n\n(5) To make the experiments reproducible, could you explicitly define the state $s$, action $a$, reward function $r$ (in the objective)? \n\nMinor issues: \n\n(6) Abstract line 3: \"through\". \n\n(7) You could describe the Slater's condition, possibly by an equation.", "limitations": "Yes, the authors have written their limitations in the conclusion section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: I can understand this work clearly. The risk-aware RL problem is popular and important for safe reinforcement learning. The algorithm and theoretical results are clear and reasonable based on my knowledge about minimax optimization and reinforcement learning. \n\nWeaknesses (See my questions below for details): The major concern is that problem novelty is not as strong as claimed. The convergence result does not directly imply the quality of policy that we are interested in. Some points need clarification.", "was_revised": true, "final_justification": "My questions including my major concern are well solved. \nI also read the comments and rebuttal for reviewer uZsP who gave rejection rating, and feel that the rebuttal has also solved reviewer uZsP's comments. \n\nTherefore, I raised my rating to 4.", "rebuttal": "Thank you for your detailed review and thoughtful comments. We hope the responses below address your questions clearly.\n\n---   \n### **Questions**\n\n1.\tThank you for the question! We originally included a table clarifying this distinction, but omitted it due to page constraints. We will take care to make this distinction clearer in the text. The key difference is that the constraint in [12] applies the risk measure to the discounted return of a cost function (see Eqs. (2) and (3), and the definition of $\\mathcal{J}^\\theta(x^0)$ at the bottom of p.5 in [12]). We refer to this as the return-based formulation, in contrast to the reward-based formulation used in our work. In our framework’s notation, the return-based formulation (like in [12, a]) can be written as:\n\n$$\\max_{t} \\left(t - \\frac{1}{\\beta} \\left(t - \\mathbb{E}\\left[\\sum_{\\tau=0}^\\infty \\gamma^\\tau r(s_\\tau, a_\\tau) \\right]\\right)_+ \\right),$$\n\n\nwhile the reward-based formulation (our work), which places the risk measure inside the occupancy measure, becomes:\n\n$$\\max_{t} \\mathbb{E}\\left[ \\sum_{\\tau=0}^\\infty \\gamma^\\tau \\left(t - \\frac{1}{\\beta}(t - r(s_\\tau, a_\\tau))_+ \\right) \\right].$$\n\nSimilarly, in [a], the spectral risk measure is applied to $G_{C_i}^\\pi$ (Eq. (5)), which is also a return (defined at the bottom of p.2). This places it within the return-based framework of [12], though extended to more general spectral risk measures.\n\nWe hope this clarifies the distinction and will make these points more explicit in the manuscript. There is more that we can say and we are happy to further elaborate or follow up as needed.\n \n\n2.\tWe will add this to the background section. The conditions on $g$ appear as Definition 2.1 in Ben-Tal and Teboulle [4]. Specifically, $g$ is a utility function that is proper, closed, concave, and non-decreasing, and it satisfies $g(0) = 0$ along with $1 \\in \\partial g(0)$, where $ \\partial g(\\cdot)$ denotes the subdifferential map.\n\n3.\tAs noted in our response to Reviewer WADh, we will make sure to include this explicitly. The subgradient can be computed using Equation (8) and the definition of $r'$ provided on line 149, and we will ensure that this is clearly stated in the manuscript.\n\n4.\tThis is a good question! Like many nonconvex problems that arise in DL/RL, we can usually only guarantee convergence to different notions of local stationary points rather than true global optima. In our setting, we use the Moreau envelope which is standard in nonsmooth weakly convex optimization. As written in Theorem 3.12, the quality of the policy depends on the scale of $\\delta$ to be small (we want Assumption 3.6 to say that the RL solver returns nearly the optimal policy for the inner problem) and we’d ideally want the non-convexity to be such that all $\\epsilon$-stationary points have values close to the global optima. We could state Theorem 3.12 in the main body more simply (e.g., assuming $\\delta = O(\\epsilon^2)$) if it would help the presentation.\n\n5.\tThe objective function—that is, the modified reward implementing the risk-aware constrained objective—is explicitly defined in terms of the state, action, and original reward function in Appendix B.3.1, Equation (25).\n\nThe state and action spaces are defined by the joint positions, velocities, and other physical properties of the agents in the MuJoCo simulator. These elements vary across environments and are governed by the physics engine, as documented on the MuJoCo website. The Safety-Gymnasium environments we use are built directly on MuJoCo v4, and we do not modify the state, action, or reward definitions provided therein. Therefore, loading any environment from the suite automatically applies the default state-action-reward configuration.\n\nGiven the complexity and high dimensionality of these environments, we were unable to include some of these details in the main body. However, we will point readers to the simulator's official documentation and, if the paper is accepted, elevate the content of Appendix B.3.1 to the main body using the additional page allowance.\n\n6.\tThank you! We now fixed it.\n\n7.\tWe will define Slater’s condition in the Appendix and reference it accordingly. Slater’s condition, a strict feasibility constraint, is often necessary to establish strong duality. It states that there exists a policy which is strictly feasible (i.e., there exists some policy which satisfies the constraints with strict inequality)."}]

Author rebuttal:

Meta-review: **Summary:** This paper investigated risk-averse constrained reinforcement learning (RL) using Optimized Certainty Equivalents (OCEs) as a unified risk-measure framework. It formulated constrained MDP objectives that (i) optimize an OCE-based performance criterion and/or (ii) enforce safety-style constraints (e.g., on costs or tail risks). This paper also derived policy-gradient estimators for OCE objectives and presented a primal–dual (Lagrangian) actor–critic scheme to handle constraints. Theoretical performance guarantees on convergence to  stationary points under some standard assumptions were provided. Finally, some experimental results were provided. 


**Strength:**
- The problem itself is important and general. This paper brings a general risk-measure family, OCE to the constrained RL setting. Such a measure subsumes other popular choices like CVaR. 
- The proposed methodologies are clearly presented, for instance, the derivation of OCE policy gradients and the primal-dual training loop are clean and likely reproducible in standard actor–critic codebases. 
- This paper is guaranteed with strong theory. It provided convergence guarantees under some standard assumptions, and clarifies how constraint penalties and OCE parameters enter the updates.

**Weakness:**
- One key limitations or weakness pointed out by the reviewers is the limitations of the experimental evaluations. It is better to add some stronger risk-aware and constraint-satisfaction baselines. 
- The theoretical analysis focuses on asymptotic/stationary-point convergence. There are no finite-sample or sample-complexity bounds, which are often provided in constrained RL literature. 

**Reasons for acceptance:** This paper offers a general risk-averse formulation for constrained RL with a sound derivation and implementable algorithm. Theoretical analysis on the convergence was conducted and analyzed. The performance was also supported by experiments, including some additional experiments conducted and added during the rebuttals. 

**Discussion \& rebuttal assessment:** The authors addressed most of the concerns raised by the reviewers during the rebuttal. A major concern was on the experimental evaluations. Given some additional experimental results provided during the rebuttals, the authors are highly encouraged to include them into the final version.
Decision: Accept (poster)