Embedding Safety into RL: A New Take on Trust Region Methods
Authors: Nikola Milosevic
Johannes Müller
Nico Scherf
Venue: ICML 2025 poster (2025)
Keywords: Constrained MDP
Information Geometry
Safe RL
Primary area: reinforcement_learning->deep_rl

Abstract: Reinforcement Learning (RL) agents can solve diverse tasks but often exhibit unsafe behavior. Constrained Markov Decision Processes (CMDPs) address this by enforcing safety constraints, yet existing methods either sacrifice reward maximization or allow unsafe training. We introduce Constrained Trust Region Policy Optimization (C-TRPO), which reshapes the policy space geometry to ensure trust regions contain only safe policies, guaranteeing constraint satisfaction throughout training. We analyze its theoretical properties and connections to TRPO, Natural Policy Gradient (NPG), and Constrained Policy Optimization (CPO). Experiments show that C-TRPO reduces constraint violations while maintaining competitive returns.

Embedding Safety into RL: A New Take on Trust Region Methods
Nikola Milosevic 12
Johannes Müller Nico Scherf'
Abstract
training, posing significant risks in real-world applications. Many methods have been proposed to improve the safety of policy gradients, often based on the Constrained Markov Decision Process (CMDP) framework. However, existing methods either struggle to minimize constraint violations during training or severely limit the agent's 's performance.
Reinforcement Learning (RL) agents solve di- verse tasks but often exhibit unsafe behavior. Con- strained Markov Decision Processes (CMDPs) address this by enforcing safety constraints, yet existing methods either sacrifice reward maxi- mization or allow unsafe training. We introduce Constrained Trust Region Policy Optimization (C- TRPO). which reshapes the policy space geometry to ensure trust regions contain only safe policies, guaranteeing constraint satisfaction throughout training. We analyze its theoretical properties and connections to TRPO, Natural Policy Gradi- ents, and Constrained Policy Optimization. Ex- periments show that C-TRPO reduces constraint violations while maintaining competitive returns.
This work introduces simple strategy to enhance constraint satisfaction in trust region- safe policy optimization methods without compromising performance. We propose a novel family of policy divergences, inspired by barrier func- tion methods in optimization and safe control, that modify the policy geometry to ensure that trust regions consist only of safe policies. Our approach is motivated by the observa- tion that TRPO and related methods base their trust region on the state-average Kullback-Leeible (KI.) divergence. It can be derived as the Bregman divergence induced by the negative conditional entropy on the space of state-action occupancies (Neu et al., 2017).
1. Introduction
The key insight of this work is that safer trust regions can be obtained by modifying this function to account for cost constraints. This leads to provably safe trust region-based policy optimization algorithm that preserves TRPO's guar- antees, while simplifying existing methods and reducing constraint violations during training, without sacrificing re- ward performance.
Reinforcement Learning (RL) has emerged as a highly suc- cessful paradigm in machine learning for solving sequential decision and control problems, with policy gradient (PG) algorithms as popular approach (Williams, 1992; Sutton al., 1999; Konda & Tsitsiklis, 1999). Policy gradients are especially appealing for high-dimensional continuous con- trol because they can be easily extended to function approxi- mation. Due to their flexibility and generality, there has been significant progress in enhancing PGs to work robustly with deep neural network- based approaches. PG-based policy optimization methods such as Trust Region Policy Optimiza- tion (TRPO) and Proximal Policy Optimization (PPO) are among the most widely used general-purpose reinforcement learning algorithms (Schulman et al., 2017a;b).
Related Work Classic solution methods for CMDPs rely on linear programming techniques, see (Altman, 1999). However, they struggle to scale, making them unsuitable for high-dimensional or continuous control problems. While there are numerous scalable approaches solving CMDPs, we focus model-free, direct policy optimization methods. Model-based approaches (Berkenkamp et al., 2017; As et al., 2025), are attractive due to their stability and safety guar- antees, but require learning a model, which is not always feasible, or imposes additional assumptions on the model space.
While flexibility makes PGs popular among practitioners, it comes at a cost: the policy can explore any behavior during
'Ma Planck Institute for Human Cognitive and Brain Sciences, Leipzig "Center for Scalable Data Analytics and Artificial Intel- ligence (ScaDS.AD), Dresden/Leipzig 'Institut für Mathematik, Technische Universität Berlin, 10623 Berlin, Germany. Correspon- dence to: Nikola Milosevic <nmilosevic ebs @ct.riiddcsb..m.e......
Lagrangian methods are widely adopted approach, where the optimization problem is reformulated as weighted ob- jective that balances rewards and penalties for constraint violations. This is often motivated by Lagrangian dual- ity, where the penalty coefficient is interpreted as the dual variable. Learning the coefficient with stochastic gradient
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Embedding Safety into RL
descent presents a popular baseline (Achiam et al., 2017:
Rethinking Safe Trust Region Methods We adopt trust region approach that constructs trust regions exclusively within the safe policy set, eliminating the need for projec- tions or constrained optimization in the inner loop. Trust region methods retain TRPO's 's update guarantees for both reward and constraints but often underperform compared to barrier penalty methods in terms of constraint satisfaction. To address this, we replace the state-average KL-divergence with policy divergences that act barrier functions, see Fig- ure 1. This modification encourages updates of the resulting trust region method to move more parallel to the constraint surfaces rather than directly toward and thereby improves constraint satisfaction, simplifies optimization, and achieves competitive returns by maintaining policies within the safe set for longer, see also Figures 5 and in the Appendix.
Ray et al., 2019; Chow et al., 2019; Stooke et al., 2020). However, a naively tuned Lagrange multiplier may not work well in practice due to oscillations and overshoot. To address this issue, (Stooke et al., 2020) uses PID control to tune the dual variable during training, which achieves less oscillation around the constraint and faster convergence to a feasible policy. While Lagrangian approaches are becoming increas- ingly popular, it is not entirely clear how update the dual variables during training, which has attracted significant research interest, see e.g. (Sohrabi et al., 2024).
Penalty methods such as IPO (Liu et al., 2020) and P30 (Zhang et al., 2022) propose weighted penalty-based policy optimization objectives, where the penalties are weighted against the reward objective using a weighting ryper-parameter instead of a learnable one. This simpli- fies the Lagrangian approach since the penalty coefficients don't 't have to be optimized during training, which results in improved stability. More recently, the approach to use (smoothed) log- bagrierssss (Usmanova et al., 2024; Zhang et al., 2024a; Dey et al., 2024) became more popular, which keeps the algorithm simple due the penalty approach, but can offer certain constraint satisfaction guarantees, see e.g. (Ni & Kamgarpour, 2024). However, working with an explicit penalty produces suboptimal policies w.r.f the original con- strained MDP and thus introduces an additional error, which has to be controlled; see for example (Geist et al., 2019; Müller & Cayci, 2024) for treatments of the regularization error in the unconstrained case, and (Liu et al., 2020) for an example of an optimization gap in safe policy optimiza- tion. In contrast, combining trust region-basee based updates in TRPO (Schulman et al., 2017a) with constrained optimiza- tion techniques does not change the objective function and the set of optimizers, and therefore does not introduce an additional error.
Toak Begies
Prodly
On the left, vanilla trust regions (dotted) and safe trust regions (dashed) are shown: on the right, a schematic visualization of common failure modes in CMDPs is shown based on optimization trajectories; here, vanilla trust re- gions can suffer from oscillations around the constraint, and penalty methods might introduce a bias. Lagrangian meth- ods (not shown) can exhibit both issues.
Figure 1:
Contributions
We summarize our contributions:
In Section 3, we introduce a modified policy divergence such that every trust region consists of only safe poli- cies. We introduce an idcalized TRPO update based on the modified divergence, a tractable optimization algorithm for deep function approximation (C-TRPO), and corresponding natural gradient method (C-NPG).
Trust region methods are closely related to our approach, in particular Constrained Policy Optimization (CPO; (Achiam et al., 2017)), which extends TRPO by restricting updates to the intersection of the trust region and the safe policy set, ensuring safety during training. While CPO guarantees constraint satisfaction in the infinite sample limit, in prac- tice it tends to oscillate around the constraint boundary with high overshoot, because it relies on noisy cost advantage estimates, and because the constraint only enters the opti- mization problem when the iterate is close to the boundary of the safe policy set. To address constraint satisfaction, projection-basee based CPO (PCPO; (Yang et al., 2020)) projects updates into the safe policy space between updates, im- proving stability but further hindering reward maximization. Building on PCPO, (Zhang et al., 2020) and (Yang et al., 2022) also introduce projection-based approaches based first-order updates.
We provide efficient implementation of the pro- posed approximate C-TRPO method, see Section 3.2, which comes with a minimal overhead compared to TRPO (up to the estimation of the expected cost) and no overhead compared to CPO. We demonstrate exper- imentally that C-TRPO yields competitive returns with smaller constraint violations compared to common safe policy optimization algorithmss see Section 5.
In Section 4, we introduce C-TRPO's 'ss improvement guarantees and contrast to TRPO and CPO. Further, we show that C-NPG is the continuous time limit of C-TRPO and provides global convergence guarantees towards the optimal safe policy; this is in contrast to penalization or barrier methods, which introduce a bias.
2

---

Embedding Safety into RL
2. Background
where [x], max 20, (No, Ti, and K is the number of training iterations. The cost regret represents the cumulative sum of the expected constraint violations throughout training. A similar metric has been used in related online optimization settings, see (Efroni et al., 2020; Müller et al., 2024). It is our goal to design a method that produces solutions of (1) of similar quality compared to existing method, while achieving minimal cost regret.
We consider the infinite-horizon discounted constrained Markov decision process (CMDP) and refer the reader to (Altman, 1999) for a general treatment. The CMDP is given by the tuple (S, A, P. pr, H4 7/s C), where S and A are the finite state-space and action-space respectively and we refer to Appendix B.3 for a discussion of continuous state and action spaces. Further, P: S x A + As is the transition kernel, r: S x A - R is the reward function, fr E As is the initial state distribution time
The Dual Linear Program for CMDPs Any stationary policy induces discounted state-action (occupancy) mea- sure d E ASxA. indicating the relative frequencies of visiting a state-action pair, discounted by how far the event lies in the future. This probability measure is defined as
O, and y E 1) the discount factor. The space As is the set of categori- cal distributions over S. Further, define the constraint set C {(G, b,) ) where c;: S A + R are the cost functions and b; E IR are the cost thresholds.
An agent interacts with the CMDP by selecting a policy π E II from the set of all Markov policies, i.c an element from the Cartesian product of |S| probability simplicies on A. Given such a policy n, the value functions V,*, V S R. action-vall value functions O. O functions A,, AA S A + IR associated with the reward r and the i-th cost C are defined as
d=(s,)) = (1-7) (1-1-) Σ
PP = s)t(als).
(3)
E
where P, (s = 8) is the probability of observing the envi- ronment in state s time given the agent follows policy m. For finite MDPs, it is well-known that maximizing the expected discounted return can be expressed as the linear program
SxA- + R, and advantage
(1s)==(1-0)
Er 80
maximizoa r Td subject to d E 2,
where the function / is either r or c, and the expectations are taken over trajectories of the Markov process, meaning with respect to the initial distribution so μt, the policy a- (-1ss) and the state transition 8141 P())s,,;.) Analogously, we set
where 9 is the of feasible state-action measures, which form a polytope (Kallenberg, 1994). Analogously to an MDP, the discounted cost CMDP can be expressed as the linear program
maximizea r subject to E Dsate:
(4)
a) = (1- 7)
Ez E Σ 7 Hs oz) 00 = 8, 0, =
where Danto - {d: cT d b;} n 2 is the safe occu- pancy set, see Figure in Appendix A.
and AJ a) = (s) VF (s). Constrained Markov
Information Geometry of Policy Optimization Among the most successful policy optimization schemes are natural policy gradient (NPG) methods variants thereof, such as trust-region and proximal policy optimization (TRPO and PPO, respectively). These methods assume convex geom- etry and corresponding Bregman divergences in the state- action polytope, see (Neu et al., 2017; Müller & Montúfar, 2023) for more detailed discussions. general trust region update is defined as
decision processes address the optimization problem
maximizeren (u) subject to V (n) < b; (1)
for all i = 1,..., m. where (1) are the expected values under the initial state distribution (u) = E=~=(VF((s)) We will also write V = V (n), and omit the explicit dependence on μ for convenience, and we write (T) when want emphasize its dependence on æ. We denote the m
set of safe policies by 1laate = 0mm { Ve, (T) 6.) and
always assume that it is nontrivial.
Tk+1 E arg max ATH (T) NElT
sbj. to Dip (ooddylld S 6, (5)
Cost Regret Depending on the task at hand, it is manda- tory to solve the constrained optimization problem Equa- tion 1 in a safe way, meaning with method that respects the constraints during optimization. This motivates the use of the cost regret
where Do: Φ + R is the Bregman divergence induced by a convex Φ: int + R, and
T Talss) To (T(ss
A (T)) E, ar
A (s, (s.a)
(6)
K-
COSTREG (TT, K,C)= =
Σ 1=0 k=0
v - bil4,
(2)
is called the policy advantage or surrogate advantage. We can interpret A. as a surrogate optimization objective for the
3

---

Embedding Safety into RL
expected return. In partieular, for parameterized policy ne. 1000, VoVi (8x). see (Kakade it holds that V A (Tr
To this end, consider mirror functions of the form
m
& Langford, 2002; Schulman et al., 2017a).
$c(d) = + +
Σ Bib(b; - C d),
(9)
TRPO and the original NPG assume the same policy ge- ometry (Kakade, 2001; Schulman et al., 2017a), since they employ an identical Bregman divergence
where Ox is the conditional entropy defined in Equation (7),
and Φ; Rz0 - IR is convex function with d'xx) + -00
Dx Di(dm ( = Σ
for x '> 0. This ensures that de: int (ate)) + R is
strictly convex and has infinite curvature at the cost surface
b = 0, which means [c + +oc, when 6 - c d2 > 0. Possible candidates for á are dbx)) = log(s) and Φ(x) log(z) corresponding logarithmic barrier and entropy, respectively.
We refer to Appendix A for details on Bregman divergences. We call Dx the Kakade divergence and informally write Dx [ dzz). This divergence can be shown to be the Bregman divergence induced by the nega- tive conditional entropy
( т2)
The mirror function &C induces the Constrained KL- Divergence given by
ΦK (dx) = (dsss,))oogT(
8,0
De(dillda) DR(dillda) + Σ (,Do,dilli (10)
see (Neu et al., 2017). It well known that with a parameter- ized policy Ta. a linear approximation of A and quadratic approximation of the Bregman divergence Dk at 8g, one obtains the natural policy gradient step given by
where
(dilldd))) @(b, Va, (TT)) - o(b - Va, (T2))
+ d' (b: - Va, (T2) Ve, (T2)).
0 02 + (..
(8)
(11)
where Gk(0)* denotes a pseudo-inverse of the generalized fesher-nfformation matrix of the policy with entries given by Gk (f);, Ög dg &2 00, daa see (Schulman et al., 2017a; Müller & Montúfar, 2023) and Appendix A for more detailed discussions.
The corresponding trust-region scheme is
Tk+ E arg max A. (T) sbj. to Do(dayllds S 6,
TEll
(12)
where A, is defined in Equation (6). Note the constraint is only satisfied if d1 d2 E inte( and the divergence approaches +00 as d2 approaches the boundary of the safe set. Thus, the trust region {d 9 Dc(delld S 63 is contained in the set of safe occupancy measures for any finite 6. Analogously to the case of unconstrained TRPO the corresponding natural policy gradient scheme is
3. A Safe Geometry for Constrained MDPs
To prevent the policy iterates from violating the constraints during optimization, we construct policy divergences for which the trust regions are contained in the safe policy set.
3.1. Safe Trust Regions
Bregman divergence is induced by a mirror function that dictates the behavior of the divergence, see Appendix A. Take for example the mirror function for TRPO and NPG in
0k+1 = 0g + ExC
(13)
where Go(D)+ denotes an arbitrary pseudo-inverse and
Equation (7). The divergence is defined when both policies are in the interior of 2, and either one of the policies approaches the boundary of the state-action polytope, the di- vergence approaches infinity. Hence, TRPO and NPG don't allow their policy iterates to become entirely deterministic during optimization.
Gc(()) = -o dJ V=Lc(do)0, do-
3.2. Constrained Trust Region Policy Optimization
If we could solve the optimization problem in Equation (12) exactly, we would obtain a provably safe trust region policy optimization method with zero constraint violations, as long start with a safe policy. However, the exact trust re- gion update Equation (12) cannot be computed. Firstly, the divergence depends on expected cost values, which we can only estimate. The resulting estimation errors of the diver- gence might cause the policy iterates to leave the safe set, in which case the divergence becomes ill-defined. Further, the
Since the behavior of Bregman divergences is dictated by the shape of its mirror function, we first construct family of safe mirror functions, that induce policy divergences that are finite only in the safe occupancy set Peate instead of the entire state-action polytope Z. Safe policy divergences, in turn, let us derive safe trust region and natural policy gradient methods.

---

Embedding Safety into RL
divergence also depends on the expected cost value of the proposal policy, which is not available during the updates. To address these issues, we propose an update based on surrogate divergence, similar to how surrogate objectives are used in policy optimization. We propose the following update, which we call Constrained TRPO (C-TRPO).
in the practical implementation, &g and the policy cost ad- vantage are estimated from trajectory samples using GAE- À (Schulman al., 2018). The consequences of the substi- tution in the surrogate will be discussed in Section 4.
Comparison with CPO This approach is similar to the update in CPO (Achiam et al., 2017), but incorporates the constraint into the design of the trust region, with an in- fluence controlled by the parameter B. This yields more conservative updates within the safe set without introducing bias in the optimal solution. Additionally, it simplifies the inner-loop constrained optimization: C-TRPO approximates single quadratic constraint, rather than solving for the in- tersection of quadratic and linear constraint as in CPO, see also Appendix C.4.
FA++1 = arg max AT (iT) sbj. Do(T]|T)) S 8. (14) Tel
Here, De is a surrogate for De, which we define in Equa- tion 16 and Equation 17. Algorithm I shows the imple- mentation of C-TRPO, which performs a constrained trust region update if the current policy is safe or recovery step that minimizes the cost if the policy is unsafe. For the trust region update, we follow a similar implementation to the original TRPO, estimating the divergence, using a linear approximation of the surrogate objectíve, and quadratic approximation of the trust region.
Multiple Constraints C-TRPO naturally extents to multiple constraints, by introducing the divergence (TT(TT]TTR) + Σ, D,D (=1[ma). where each Dd. is defined according to Eq. 17 but with the re- spective G. In section 3, we discuss that this divergence approximates natural policy gradient (C-NPG) on the safe state-action occupancy set, where Theorem 4.5 implies that the optimal feasible solution aa satisfies as few constraints with equality as required to be optimal.
Surrogate Divergence For the sake of clarity, we first focus on the case with a single constraint, but the results are easily extended to multiple constraints by summation of the individual constraint terms, as discussed in the respective paragraph below. In practice, the exact constrained KI.- Divergence De cannot be evaluated, because depends on the cost-return of the optimized policy V.(TT). However, can approximate it locally around the policy of the k-th iteration, TR, using surrogate divergence. This surrogate can be expressed as a function of the policy cost advantage
(T
Recovery with Hysteresis The iterate may still leave the safe policy satee either due to approximation errors of the divergence, or because we started outside the safe set. In this case, perform a recovery step. where we only minimize the cost with TRPO as by (Achiam et al., 2017). In tasks where the policy starts in the unsafe set, C- TRPO can get stuck at the constraint surface. This is easily mitigated by including hysteresis condition for returning to the safe set. If π the previous policy, then T& E I with IT (mo E TT, and Va(mo) S bu) where bn = b if T-- 11 and user-specified fraction of b otherwise.
AT (3,11) [a)(
T(als)
Th (T) Edine
(15)
which approximates Vz(TT) - V up to first order in the policy parameters (Kakade & Langford, 2002: Schulman et al., aa; Achiam et al., 2017). Assume n E Igaat and define the constraint margin 5, b b- V which positive if Th E ISAAEE. Further, define the surrogate divergence De(T]]TT)> = DKL( (T) + 3D,((TT))) [[T]) where
(T = = dr (eL
(16)
SES
Computational Complexity The C-TRPO implementa- tion adds no computational overhead compared to CPO, since De is function of the cost advantage estimate, and is added to the divergence of TRPO. Compared to TRPO, the cost value function must be approximated.
and
(AAA))
if 8g AT E do
(T(TTl(TeL)
+00
otherwise
(17)
where
3.3. Constrained Natural Policy Gradient
V (A 065b- A.
(((6)) + 0
+0'(6))A (T). (18)
Practically, the C-TRPO optimization problem in Equa- tion (14) is solved like traditional TRPO: the objective is approximated linearly, and the constraint is approximated quadratically in the policy parameters using automatic dif- ferentiation and the conjugate gradient method. This leads
The surrogate Do is closely related to the Bregman di- vergence Dø- They are equivalent up to the substitution Ve(TT) - V((Tx)) + A (I), see Appendix B.I. The surro- gate can be estimated from samples of the CMDP, where
5

---

Embedding Safety into RL
Algorithm 1 Constrained TRPO (C-TRPO); differences
4. Analysis
from TRPO in blue
Here, we provide a theoretical analysis of the updates of C-TRPO and study the convergence properties of the time- continuous version of C-NPG, All proofs are deferred the Appendix C.
Input: Initial policy To E Hg, safety parameter 3 > 0,
recovery parameter 0 < bu < b
2: for k == 0, 1,,....... do
3: Sample set of trajectories following R m.
4: if .TA E I then
5: A + A,; D t- De = DBD+ [Constrained
4.1. Properties of the C-TRPO Update
trust region }pdate}
The practical C-TRPO algorithm is implemented using the surrogate divergence introduced in Equation (14), which is identical to the theoretical divergence De introduced in Equation (12) up to a mismatch between the policy advan- tage and the performance difference. The motivation for substituting the policy cost advantage for the performance difference is their equivalence up to first order and that we can estimate the advantage from samples of dg. Similar to CPO, can guarantee almost improvement of the return (Achiam et al., 2017), despite the new divergence. Proposition 4.1 (C-TRPO reward update). Set e,
6: else
7: A Ac; D t DKL [Recovery)
8: end if
9:
Compute Th+11 using TRPO with A as advantage estimate and with D as policy divergence.
10: end for
to the policy parameter update
28
0.+1 02 a²
H gk:
(19) max, [Bawte4: AF* (s, a)l. The expected reward of a policy updated with C-TRPO is bounded from below by
V
g H gk
where
V28
[+(T+1) 2 Vr(TT) 1-7
(23)
gk VoA. (To))((TTT) (Te))
(20)
and
Constraint violation, however, behaves slightly differently for the two algorithms. To see this, we establish a more concrete relation between C-TRPO and CPO. As > 0, the solution to Equation (14) approaches the constraint surface in the worst case, and recover CPO, see Figure 5.
H = Hc(()) = ViDo - 1o=0
(21)
are finite sample estimates, and H- is approximated us-
ing conjugate gradients. The a' E [0 are the coefficients for backtracking line search, which ensures De(Tel(Te)) x
Proposition 4.2. The approximate C-TRPO update ap- proaches the CPO update in the limit as 3 2 0.
5.
We show in Appendix B.2.3 that the Hessian
Intuitively, solving the C-TRPO problem with successively smaller values of B, would be similar to CPO with the interior point method using (-((T( as the barrier function. However, C-TRPO is more conservative than CPO for any 3 > 0 and as 3 - +oo the update is maximally constrained in the cost-increasing direction.
(Oc(Os) (0+) Bb
is equivalent to the Gramian Gc(Oz) of the natural gradi- ent update in Equation (13). We call the resulting policy gradient
Proposition 4.3 (C-TRPO worst-case constraint viola- ( I 8 with d > 0 and set e.
Ou+1- 02 - - ex (a.e,e
(22)
tion). Let (o( max, E [Ba 11 A (8, a)). It holds that
the Constrained NPG (C-NPG). In particular, this shows that the C-TRPO update can be interpreted as natural policy gradient step with an adaptive step size, see Appendix A. We emphasize that the idealized safe trust region update in Equation (12) and the C-TRPO update of Equation (14) agree up second order in the policy parameters. This justifies the surrogate divergence in C-TRPO and motivates the discussion of the C-NPG flow in Section 4.2. We show in Theorem 4.4 that int saate) is invariant under the dynamics of the C-NPG. This implies that if the trust region radius 5 is small, and the advantage estimation is accurate enough, the iterates under C-TRPO never leave the safe set.
Vc(T++1) S Vc(TTK) + AT* (T&+1) + 228(3)7ec
(24)
1-
5 is decreasing in 3 0. lim3 5(3) = 5, and 8(3) - for + 8 8D ( mx))
where 5(3) 5 DDB (
This result is analogous to the worst-case constraint viola- tion for CPO (Achiam et al., 2017, Proposition 2). where the term 6(3) is replaced by 5. As 6(3) < for all / > 0, the bound for C-TRPO is higher than the corresponding
6

---

Embedding Safety into RL
guarantee for CPO. For 8 - the bound converges to the CPO bound, where for B3- + +00, the bound becomes Vo(TT+11 & Vo(Tes), see Appendix C.I.
C-NPG (8 = 110-00)
C-NPG (3 = 1)
4.70
1.50
(as (22)
1.26
4.2. Invariance and Convergence of Constrained Natural Policy Gradients
1.00
0.5
It is well known that TRPO is equivalent to natural policy gradient method with an adaptive step size, see also Ap- pendix A. We study the time-continuous limit of C-TRPO and guarantee safety during training and global convergence. In the context of constrained TRPO in Equation (12), we study the natural policy gradient flow
a(a1|s1)
T(ai(s))
Figure 2: Shown is the policy set IT = [0, 1J2 for an MDP with two states and two actions with heatmap of the ex- pected reward V,; the constraint surface is shown in black with the safe policies below; optimization trajectories are shown for 10 safe initialization and for 3 = 10 10-2,1.
0.0.- = Gc((,)++VVv
(25)
where Gc(f)+ denotes a pseudo-inverse of (0 = Oo, dg V2 (de (d, de and 0 + v differentiable pol- icy parametrization. Moreover, we assume that 0 1> To is regular, that it is surjective and the Jacobian is of maximal rank everywhere. This assumption implies overparametriza- tion but is satisfied for common models like tabular softmax, tabular escort, or expressive log-linear policy parameteriza- tions (Agarwal et al., 2021a; Mei et al., 2020a; Müller & Montúfar, 2023).
In case of multiple optimal policies, Equation (26) identi- fies the optimal policy of the CMDP that the natural policy gradient method converges to as the projection of the ini- tial policy my to the set of optimal safe policies I with respect to the constrained divergence Dc. In particular, this implies that the limiting policy Taae satisfies as few constraints with equality required to be optimal. To see this, note that II safe forms face of Danto and that Bregman projections lie at the interior of faces (Müller et al., 2024, Lemma A.2) and hence satisfy as few linear constraints as required.
We denote the set of safe parameters by Sste == {0 E R²; Te E ]. which is non-convex in general and say that Ssate is invariant under Equation (25) if 0u E Sate implies Or E Saaee for all I. Invariance is associated with safe control during optimization and is typically achieved via control barrier function methods (Ames et al., 2017; Cheng et al., 2019). We study the evolution of the state-action distributions de d= as this allows linear programming formulation of CMDPs and we obtain the following convergence guarantees.
5. Computational Experiments
employ the
Setup and main results We benchmark C-TRPO against 9 common safe policy optimization algorithms (CPO (Achiam et al., 201 PCPO (Yang et al., 2020), CPPO-PID (Stooke et al., 2020), PPO-Lag and TRPO-Lag (Achiam et al., 2017; Ray et al., 2019), FOCOPS (Zhang et al., 2020), CUP (Yang et al., 2022), IPO (Liu et al., 2020) and P3O (Zhang al., 2022)) on 8 tasks (4 Navigation and 4 Lo- comotion) from the Safety Gymnasium (Ji et al., 2023) benchmark.' The locomotion tasks reward distance traveled, while penalizing high velocities, and the navigation tasks reward goal reaching and penalize certain unsafe states. For the C-TRPO implementation we fix the convex generator Φb x)) x log(z)) motivated by its superior performance in our experiments, see Appendix 1.2. and 6y = 0.86 and 8 = 1 across all experiments. Each algorithm is evaluated by training for 10 million environment steps with 5 seeds each, and the cost regret is monitored throughout training for every run. To get a better sense of the safety of the algorithms during training, we take an online learning per- spective and include as metric the cost regret introduced in Equation 2 (Efroni et al., 2020; Müller et al., 2024) For
Theorem 4.4 (Safety during training). Assume that d: Ry + Ih satisfies (x()) + +o0 for x: > O and con- sider regular policy parameterization. Then the set Osate invariant under Equation (25).
A visualization of policies obtained by C-NPG for different safe initializations and varying choices of 3 is shown in Figure for a toy MDP. We see that for even small choices of 3 the trajectories don't cross the constraint surface and the updates become more conservative for larger choices of 3.
Theorem 4.5. Assume that d'(x:) + 4rx > O, set Vio = maxxellaata Vx(TT) and denote the set of optimal constrained policies by II = {x E Igaate VV(TT) = Vic). consider a regulur policy parametrization and (0. )22 solve Equation (25). holds that V, (T(TT)) -> ViC and
lim n. Taate = arg (io( (m)) I- 1-0
(26)
Code: https: / thub. com/mi losen/ctrpo
7

---

Embedding Safety into RL
Final Cost
Reward
Cost Regret
0.8
C-TRPO (oun)
TRPO-Lag
0.7
3.0
730
Normalizol IQM
POPO
0.6
2.5
0.5
0.4
2.0
0.4
1.5
0.3
0.0
1.0
-0.2
0.1
0.3
0.0
probably safe
probably unsafe
probably safe
probably wisafe
probably safe
probably unsafe
Figure 3: Comparison of safe policy optimization algorithms based on the 25% Inter Quartile Mean (IQM) across 5 seeds and 8 tasks. From left to right, the following metrics are shown measured at 10 million training steps: the final cost, i.e. the mean return the cost at the last iterate fbreshold-normalizne (normalized with the performance of unconstrained PPO), and the mean cost regret (normalized by CPO's cost regret), The algorithms are sorted into probably safe and probably unsafe, based their final constraint violation (negative is probably safe), and by expected reward within each group. Note that cost regret is different from the final cost, since it sums up all constraint violations throughout training.
and centered at zero), the mean return of the reward
completeness, we also report Deviromment-wise sample effi- ciency curves and the results of Figure 3 in a tabular format in Appendix D.4.
Figure 8 in the appendix. For higher values, the expected return starts to degrade, partly due to D& being relatively noisy compared to DKL. and thus we recommend the choice 8
Discussion In Figure the interquartile mean (IQM) of normalized expected reward, cost, and cost regret, includ- ing their stratified bootstrap confidence intervals (Agarwal et al., 2021b) is shown. It can be observed that C-TRPO is competitive with the leading algorithms of the benchmark in terms of expected return, while being safe on the last iterate as opposed to CPO and CUP, see Figure Furthermore, achieves notably lower cost regret throughout training than the high-return algorithms. TRPO-Lag, which is also safe at convergence, has notably higher cost regret than the other safe methods, meaning it oscillates more around the threshold during training, see also Figures 13 and 12 in the appendix. In general, methods that, in practice, rely on gegraaggaaninssiree optimization routines (TRPO-Lag.. FOCOPS, and CUP) perform well in terms of reward, but poorly in terms of cost regret. C-TRPO's 's regret performance is comparable to the more conservative PCPO algorithm, but is not as low as that of P3O. The low cost regret achieved by P3O comes at the price of expected reward, which is due to it's wide margin to the threshold at the last iterate.
Further, observe that in most environments constraint vi- olations seem to reduce as the algorithm converges, meaning that the regret flattens over time. This behavior suggests that the divergence estimation becomes increasingly accurate over time, potentially allowing C-TRPO to achieve sublin- ear regret. However, we leave regret analysis of the finite sample regime for future research.
We attribute the improved constraint satisfaction compared to CPO to a slowdown and reduction in the frequency of oscillations around the cost threshold, which mitigates over- shoot behaviors that could otherwise violate constraints. The modified gradient preconditioner appears to deflect the parameter trajectory away from the constraint, see Figure 2. This effect may also be partially attributed to the hysteresis- based recovery mechanism, which helps smooth updates by leading the iterate away from the boundary of the safe set. Employing a hysteresis fraction 0 < 6y < 6 might also be beneficial because C-TRPO's divergence estimates tend to more reliable for strictly safe policies. The effect of the choice of by is shown in Figure the appendix. Finally, we present ablations in Appendix D.2, which support our claims that both components the modified trust region and hysteresis-ar are effective in reducing safety violations.
Our experiments reveal that C-TRPO's performance is closely tied to the accuracy of divergence estimation, which hinges the precise estimation of the cost advantage and value functions. C-TRPO's 's behavior w.r.t noisy cost func- tion estimates is analyzed in Appendix D.3. The safety parameter 3 modulates the stringency with which C-TRPO satisfies the constraint, and can do so without limiting the expected return on most environments at least for 8 < 1, see
6. Conclusion and outlook
We introduced C-TRPO and C-NPG, two novel methods for solving CMDPs. C-TRPO extends Trust Region Policy
8

---

Embedding Safety into RL
Optimization (TRPO) by embedding constraint handling into the policy space geometry, while C-NPG provides provably safe natural policy gradient method for CMDPs. Our experiments showed that C-TRPO reduces constraint violations while maintaining competitive returns compared to state-of-the-art constrained RL algorithms. Despite these advances, challenges remain. Estimating the proposed diver- gence is difficult, and we did not analyze its samile properties. Additionally, CMDPs constrain average cost re- stseewise safety constraints harder to model. Future work includes integrating C-TRPO with model-based methods (As et al., 2025), leveraging mirror descent (Tomar et al., 2022), and considering alter- native formalisms for risk-sensitive RL like distributional RL (Dabney al., 2018) or Sauté RL (Sootla et al., 2022).
Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., and Bellemare, M. Deep Reinforcement Learning the Edge of the Statistical Precipice. Advances in Neural Information Processing Systems, 34, 2021b.
Altman, E.
Constrained Markov Decision Pro- CRC Press, Taylor & Francis Group.
cesses. 1999. URL https: /atp::///ppp... semanticschola. org/ htt//CorusID::4966222
turn, making trajectory-win
Alvarez, F., Bolte, J., and Brahic, O. Hessian Riemannian gradient flows in convex programming. SIAM journal on control and optimization,, 43(2):47-5011, 2004.
Amari, S.-i. Information geometry and its applications, volume 194, Springer, 2016.
Overall, the proposed algorithms, C-TRPO and C-NPG, present a step forward in general-purpont CMDP algorithms and move us closer to deploying RL in high-stakes, real- world applications.
Ames, D., Xu, X., Grizzle, W., and Tabuada, P. Control barrier function based quadratic programs for safety critical systems. IEEE Transactions on Automatic Control, 62(8):3861-3877 August 2017. ISSN 1558- 2523. doi: 10. 10.1109//ac.206...202 2638961.
URL http: / //dx doi orooorgg///0 11..0........... 201 20006.2630000 2638966.
Impact Statement
This paper presents work whose goal is advance the field of constrained Markov decision processes and safe rein- forcement learning. There are many potential societal con- sequences of our work, none of which we feel must be specifically highlighted here.
As, Y. Sukhija, B., Treven, L., Sferrazza, C., Coros, S., and Krause, A. ActSafe: Active Exploration with Safety Con- straints for Reinforcement Learning. In The Thirteenth International Conference on Learning Representations, /ttttp::://ppeneevaa hwwp//eeeeneeeeeeeeeeeeeeeeeeeee net forum? i id=aKRADWBJ11.
2025. URL https:
Acknowledgements
Berkenkamp, F., Turchetta, M., Schoellig, A., and Krause, A. Safe Model-based Reinforcement Learning with Stability Guarantees. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:: www.eiiiinnnnnnnnnneeeeeee...eee cc/sapprrfilles/ppprr///////////
N. M. and N.S. are supported by BMBF (Federal Ministry of Education and Research) through ACONITE (011S22065) and the Center for Scalable Data Analytics and Artificial In- telligence (ScaDS.AL) Leipzig and by the European Union and the Free State of Saxony through BIOWIN. N.M. is also supported by the Max Planck IMPRS CoNI Doctoral Program.
76 pdf.
References
Cheng, R., Orosz, G., Murray, R. M., and Burdick, J. W. End-to-end safe reinforcement learning through barrier functions for safety- continuous control tasks. In Proceedings of the AAAI conference on artificial intelli- gence, volume 33, pp. 3387-3395, 2019.
Achiam, J.. Held, D., Tamar, A., and Abbeel, Constrained Policy Optimization. In Precup. D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Ma- chine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 22-31. PMLR, 06-11 Aug 2017. URL https:t////. http///peccccc..cceeeee mlr, press/v70/ achiami html.
Chow, Y., Nachum, O., Faust, A., Duunnz-Guzzanl E., and Ghavamzadeh, M. Lyapunov- Safe Policy Opti- mization for Continuous Control, 2019. URL https: / wwwrxii..ogg/aaa /990..
901 10031.
Agarwal, A., Kakade, S. M., Lee, J. D., and Mahajan, G. On the Theory of Policy Gradient Methods: Optimality, Ap- proximation, and Distribution Shift. Journal of Machine Learning Research, 22(98):1-76, 2021a. URL http: html.
Dabney, W., Rowland, M., Bellemare, M., and Munos, R. Distributional reinforcement learning with quantile re- gression. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
//jmlr wttp.//rppp///////111r

---

Embedding Safety into RL
Dey, S., Dasgupta, P., and Dey, S. P2BPO: Permeable Penalty Barrier-Based Policy Optimization for Safe RI. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 38, pp. 21029 21036, 2024.
Liu, Y., Ding, J., and Liu, X. IPO: Interior-Poinl Policy Optimization under Constraints. Proceedings of the AAAI Conference on Artificial Intelligence, 34:4940-4947,, 04 2020. doi: 10.1609aaai. http/aaai..444..555. v34.04.5932.
Efroni, Y., Mannor, S., and Pirotta, M. Exploration- Exploitation in Constrained MDPs, 2020. URL https: //arxiv. org/ a203 02189.
Mei, J., Xiao, C., Dai, B., Li, L., Szepesvári, C., and Schu- urmans, D. Escaping the Gravitational Pull of Softmax. Advances in Neural Information Processing Systems, 33: 21130-21144 140, 2020a.
Feinberg, A. and Shwartz, A. Handbook of Markov decision processes: methods and applications, volume 40.
Mci, J., Xiao, C., Szepesvari, C., and Schuurmans, D. On the Global Convergence Rates of Softmax Policy Gradient Methods. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Ma- chine Learning Research, pp. 18 Jul 2020b. URL https:/ httpc//prrooeeeeee wwwceediiiioooooccc ngs .mlr. press/vi19 9 /e 20b. html.
Springer Science & Business Media, 2012.
Geist, M., Scherrer, B., and Pietquin, O. A Theory of Regu-
larized Markov Decision Processes. In International Con- ference on Machine Learning, pp. 2160-2169. PMLR, 2019.
PMLR, 13
Ji, J., Zhang, B., Zhou, J., Pan, X., Huang, W., Sun, R., Geng, Y., Zhong, Y., Dai, J., and Yang, Y. Safety Gym- nasium: A Unified Safe Reinforcement Learning Bench- mark. In Shirty-sevntth Conference Neural Informa- tion Processing Systems Datasets and Benchmarks Track, 2023. URL httpt::tt//..........ttt http://pppeeeeeeeeeeeeeeeee net/ forum? id=WZmlxIuIGH.
Müller, A., Alatur, P., Cevher, V., Ramponi, G., and He, Truly No-Regret Learning in Constrained MDPs. In Salakhutdinov. R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Pro- ceedings of the 41st International Conference on Mu- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 36605-36653, PMLR, 21-27 Jul 2024. URL httphhttpp:/pppccceeeeeee v235 1233/muulest httm//mll122.... html.
Kakade, S. and Langford, J. Approximately Optimal Ap- proximate Reinforcement Learning. In Proceedings of the Nineteenth International Conference on Machine Learning, ICML '02, pp. 267-274, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc. ISBN 1558608737.
mlr. press/
Müller, J. and Cayci, S. Essentially Sharp Estimates on the Entropy Regularization Error in Discrete Dis- counted Markov Decision Processes. arXiv preprint arXiv:2406.0411 2024. doi: https://doi.or//1..4453 arXiv.2406.04163.
Kakade, S. M. A natural policy gradient. In Di- etterich, T., Becker, S., and Ghahramani, Z.
Müller, J. and Montúfar, G. Geometry and convergence of natural policy gradient methods. Information Geometry, pp. 1-39, 2023.
(eds.). Advances in Neural Information Pro- cessing Systems, volume 14. MIT Press, 2001. URL oc hcc/pappr_filleeeppper////////// https:/pproeediige..eerttt 4b8 BBGGaeesaaseee1119cccee113333333 pdf.
Müller, J., Çaycι, S., and Montúfar, G. Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Pol- icy Gradients. arXiv preprint arXiv:2403. / 19448, 2024.
Kallenberg, L. C. Survey of linear programming for stan- dard and nonstandard Markovian control problems. Part I: Theory. Zeitschrift für Operations Research, 40:1-42, 1994.
Neu, G., Jonsson, A., and Gómez, A unified view of Markov decision processes. arXiv preprint arXiv: / 1705007798 2017.
Ni, T. and Kamgarpour, M. A safe exploration approach constrained Markov decision processes. In ICML 2024 Workshop: Foundations of Reinforcemeni Learning and Contoo-Conmections and Perspectives, 2024.
Konda, V. and Tsitsiklis, rithms. In Solla, S., Leen, T., and Müller, K. (eds.), Advances in Neural Information Pro- cessing Systems, volume 12. MIT Press, 1999. URL sttps://prcceetingg.co ce/paper_fei [haaper_fiees/ppper//999///LL 4449444a1222088444669ddd00239 Papa.. pdf.
Actor-Critic Algo-
Pirotta, M. Restelli, M., Pecorino, A., and Calandriello, D. Safe Policy Iteration. In Dasgupta, S. and McAllester, D. (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28, pp. 307-315, 2013.
10

---

Embedding Safety into RL
Ray, A., Achiam, J., and Amodei, D. Benchmarking Safe Exploration in Deep Reinforcement Learning. arXiv preprint arXivvI990.011
van Oostrum, J., Müller, J., and Ay, N. Invariance proper- ties of the natural gradient in overparametrised systems. Information geometry, 6(1):51-67, 2023.
7(1):2, 2019.
Schulman, J., Levine, S., Moritz, P., Jordan, M. I., and Abbeel, P. Trust Region Policy Optimization, 2017a.
Williams, R. J. Simple statistical gradient-following algo-
rithms for connectionist reinforcement learning. Machine learning, 8:229-255 1992.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and
Klimov, O. Proximal Policy Optimization Algorithms, 2017b. URL https:/ /ttps://arxvv../ http:ii.wrg//bb//1111//
Yang, L., Ji, J., Dai, J., Zhang. L., Zhou, B., Li, P., Yang, Y., and Pan, G. Constrained Update Projection Approach to Safe Policy Optimization. Advances Neural Informa- tion Processing Systems, 35:91 1 11-9124, 2022.
06347.
Schulman, J., Moritz, P. Levine, S., Jordan, M., and Abbeel, P. -igennimesional Continuous Control Us- ing Generalized Advantage Estimation, 2018. URL https: /ttps///aavvv..//aa wwrriiiorr//iee.. abs///b/////// /1061.66.. 02438.
Yang, T.-Y., Rosca, J., Narasimhan, K., and Ramadge, P. J. Projection- constrained policy optimization. In International Conference on Learning Representations, 2020. URL https: /oppneeveeeeeeeeee/eeeeeeeeeeeee net/ net//frum? id=rke31JrtPS.
Sohrabi, M., Ramirez, J., Zhang, T. H., Lacoste--ulien S., and Gallego-Posadaa J. On PI controllers for updating lagrange multipliers constrained optimization. In Forty- first International Conference on Machine Learning, norrm
Zhang. B., Zhang, Y., Frison, L., Brox, T., and Bödecker, J. Constrained Reinforcement Learning with Smoothed Log Barrier Function. arXiv preprint arXiv:2403.14500 2024a.
2024. URL httpps:///pee.ee id=1khG2xf1yt
Sootla, A., Cowen-Rivens A. I., Jafferjee, T., Wang, Z., Mguni, D. H., Wang, J., and Ammar, H. Sauté RL: Almost surely safe reinforcement learning using state augmentation. In International Conference on Machine Learning, pp. 14423-0044. PMLR, 2022.
Zhang, L., Shen, L., Yang, L., Chen, S., Wang. X., Yuan, B., and Tao, D. Penalized Proximal Policy Optimization for Safe Reinforcement Learning. In Thirty-First Inter- national Joint Conference Artificial Intelligence, pp. 3719-3725, 07 2022. doi: 10. 00.24963/jcaa..2022/517
Stooke, A., Achiam, J., and Abbeel, P. Responsive Safery in Reinforcement Learning by PID Lagrangian Methods. In III, H. D. and Singh, A. (eds.), Pro- ceedings of the 37th International Conference on Ma- chine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9133-9143 9143. PMLR, 13-18 Jul 2020. htlLtttss://proeeeeeee v119 /t119sstooke22.hht..
Zhang, Y., Vuong, Q., and Ross, K. First Order Constrained Optimizationn in Policy Space. Advances in Neural Infor- mation Processing Systems, 33:15338-15341 2020.
Zhang, Y., Pan, J., Li, L. K., Liu, W., Chen, Z., Liu, X., and Wang, J. On the properties of Kullback- diver- gence between multivariate Gaussian distributions. Ad- vances in Neural Information Processing Systems, 36, 2024b.
mlr press/
Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In Solla, S., Leen, T., and Müller, K. (eds.), Advances in Neural Infor- mation Processing Systems, volume 12. MIT Press, 1999. URL http::///pppceeeigg.g..eeeeeeeee ec //lpppee_flle//pppp/////////111 2odd22883500eee3eee0aaeeea222223 Paper. pdf.
Tomar, M., Shani, L., Efroni, Y., and Ghavamzadeh, M. Mir- ror Descent Policy Optimization. In International Confer- ence on Learning Representations, 2022. URL https: w//pperevvew.neete wtw.:e/e/.oommmmmooo.oaa0..
Usmanova, I., As, Y., Kamgarpour, M., and Krause, A. Log Barriers for Safe Black-box Optimization with Applica- tion to Safe Reinforcement Learning. Journal of Machine Learning Research, 25(171): 2024.

---

Embedding Safety into RL
A. Extended Background
We consider the infinite-horizon discounted Markov decision process (MDP), given by the tuple (8, AS,A,P, P.r. [4, 7). Here, S and A are the finite state-space and action-space respectively. Here, we make the restriction finite MDPs as this simplifies the presentation. For discussion of continuous state and action spaces, we refer Appendix B.3. Further, P: S A As is the transition kernel, r: S x A + R is the reward function, μ As is the initial state distribution time = 0. and 7 E [0, 1) is the discount factor. The space As is the set of categorical distributions over S.
The Reinforcement Learning (RL) protocol is usually described as follows: At time t = 0, an initial state so is drawn from μ.. At each integer time-step t, the agent chooses an action according to it's (stochastic) behavior policy a ~ (-(ss)). A reward r, = r(*,, at) is given to the agent, and a new state 8t+11 P(-[st, (t) is sampled from the environment. Given a policy x, the value function V S - IR, action-value function Q S x A I, and advantage function AZ: S x A R associated with the reward are defined as
(T(s)= = (1 (1-7) E
Σ 0
y
rr(s az) =8
(s, = (1-7) E
Σ
y (rrs ar) 80 = 8, 00 = a and A7 a) = (s, -
- (s).
where and the expectations are taken over trajectories of the Markov process resulting from starting at s and following policy n. The goal is to
maximize V (p)
(27)
where Vi (μ)) is the expected value under the initial state distribution V, (n) = Eg (F))), We will also write V V. (u), and omit the explicit dependence on for convenience, and we write Vi( when we want to emphasize its dependence on π.
The Dual Linear Program for MDPs Any stationary policy æ induces a discounted state-action (occupancy) measure d, E ASxA, indicating the relative frequencies of visiting a state-action pair, discounted by how far the visitation lies the future. It probability measure defined as
8
as 7)) Σ
r ' s)m((ss).
(28)
where (st = 8) is the probability of observing the environment in state at time given the agent follows policy æ. For finite MDPs, it is well-known that maximizing the expected discounted return can be expressed as the linear program
max TT subject to d 9, d
(29)
where 2 is the set of feasible state-action measures (Feinberg & Shwartz, 2012). This set is also known the state-action polytope, defined by
2 = fdf RSAA (d) for all s E SS.
where the linear constraints (d) are given by the Bellman flow equations
(d = d(s)) - Σ dl -11
where d(s,a) denotes the state-marginal of d. For any state-action measure d we obtain the associated policy via conditioning, meaning
d(s,a) a) E. d(s,aa)
7 (als)
(30)
in case this is well-defined. This provides a one-to-one correspondence between policies and the state-action distributions under the following assumption.
12

---

Embedding Safety into RL
r(s, a)
d safe
(s,a)
safe
Figure 4: The dual linear program for CMDP of two states and actions.
Assumption A.1 (Exploration). For any policy π E A we have (s) > 0 for all s S.
This assumption is standard in linear programming approaches and policy gradient methods where it is necessary for global convergence (Kallenberg, 1994; Mei et al., 2020b). Note that d OD if and only if d(s, a) 0 for some 8, and hence the boundary of D is given by
0 = { dz <aass) = for some & E ES E A).
Constrained Markov Decision Processes Where MDPs aim to maximize the return, constrained MDPs (CMDPs) aim to maximize the return subject to number of costs not exceeding certain thresholds. For general treatment of CMDPs, we refer the reader to (Altman, 1999). An important application of CMDPs is in safety-critical reinforcement learning where the costs incorporate safety constraints. An infinite-horizon discounted CMDP is defined by the tuple (S,A,P,r M, 7,C), C), consisting of the standard elements an MDP and an additional constraint set = {(c,, b,)] where c; Xxxxxx A are the cost functions and h; are the cost thresholds.
In addition the value functions and the advantage functions of the reward that defined for the MDP, define the same quantities Ve,, Qe and Ae, w.r. the ith cost ç; simply by replacing r with q. The objective is to maximize the discounted return, as before, but we restrict the space of policies to the safe policy set
1Isafo n IT: Vo) s b1).
(31)
where
(n) = Egnp V (s)]
(32)
the expected discounted cumulative cost associated with the cost function G, Like the MDP, the discounted cost CMPD can be expressed the linear program
max Td sbj. E Deates
(33)
where
Deate
n 1=1
dd SSA
ng
(34)
the safe occupancy set, see Figure 4.
13

---

Embedding Safety into RL
Bregman divergences Here, we give short introduction the concept of Bregman divergences, which is required for the formulation of trust region methods. For this, we consider convex subset of Fuclidean space C C TR with a non-empty interior iC) and strictly convex function Φ: C' + R which we assume to be differentiable on the interior int(C'.. Then, the Bregman divergence induced by & is given by
Do(rllv) = - Vo(v) (I y),
y),
(35)
which is well defined for E C,y E int(C'.. Intuitively, the Bregman divergence measures the difference between 6 and its linearization at y. The strict convexity of á ensures that (x 2 O and D yy) = 0 if and only if x = y. Therefore, Bregman divergences are commonly interpreted as generalized measure for the distance between points, however, important notice that is not generally symmetric. An important example the Euclidean distance D& || - 8 which arises from the choice <(x) = ||=13. Another important Bregman divergence is the Kullback-Leibler (KL.) divergence
(KL(P)lg) Σ
Pt log
Pi qi
Pr
4
(36)
where we use the common convention 0 log 0 == 0. Then, the KL divergence is defined for E IR and g E R$o which 0 absolutely continuous with respect to p. meaning that ps = implies qi O. Note that both p and q are probability vectors, meaning that E, P. E. L, we obtain
d
DKL(p)lg) = > Pi log Pt
(37)
i=1 4i
Information Geometry of Policy Optimization Among the most successful policy optimization schemes are natural policy gradient (NPG) methods variants thereof like trust-region and proximal policy optimization (TRPO and PPO, respectively). These methods assume convex geometry and corresponding Bregman divergences in the state-action polytope, where we refer to (Neu et al., 2017; Müller & Montúfar, 2023) for more detailed discussion.
In general, a trust region update is defined as
Tk+1
E arg max A (R) sbj. to De(dx,lldz) S TEll
(38)
where Do: 2 x ② >> R is a Bregman divergence induced by a suitably convex function Φ: int(2) - R. The functional
A Th (T) = Exwde, a (-1s) [AT (s,
(39)
as introduced in (Kakade & Langford, 2002), is called the policy advantage. As a loss function, it is also known as the surrogate advantage (Schulman et al., 2017a), since we can interpret A as a surrogate optimization objective of the return. In particular, it holds for a parameterized policy Ty. that VaA" (Te))l=8, (0v) see (Kakade & Langford, 2002: Schulman al., 2017a). TRPO and the original NPG assume the same geometry (Kakade, 2001; Schulman et al., 2017a), since they employ an identical Bregman divergence
T (als)
Dx(dz,]ldsg = Σ dm (8, a) log
Σ dry (<mm
Tz (a)
3,a
We refer to Dr as the Kakade divergence and informally write (M( (2) DK(dm, duz). This divergence can be shown to be the Bregman divergence induced by the negative conditional entropy
(d(d)) = Σ [dd(s,a)loggT(ls))
(40)
6,0
see (Neu et al., 2017). is well known that with parameterized policy na, a linear approximation of and a quadratic approximation the Bregman divergence DR one obtains the natural policy gradient step given by
0x+1 02 Cr. (.)(O))))
(41)
14

---

Embedding Safety into RL
where (k) denotes a pseudo-inverse of the Gramian matrix with entries equal the state-averaged visher-informatiin matrix of the policy
Do (ol mo(a|s)
Gk (k(0)a = Es Ex-ddz
(42)
Ta(als)
Ed [0p. log (s)a,, log Ta (a|s)],
(43)
where we refer to (Schulman et al., 2017a) for more detailed discussion.
Consider convex potential Φ: Ø - IR or x Daate + R and the TRPO update
0.+1 arg max Arr non (me) sbj. to Do (do, |do)) S
(44)
In practice, one uses a linear approximation Ar A (TT) and a quadratic approximation of Dz to compute the TRPO update.
This gives the following approximation of TRPO
0k+1 E arg max Vod, 00) (0 0%) sbj. to 1|0 0% 0xll2(0.) S c,
(45)
where
G(O);; 0o, (do)
(46)
Note that by the policy gradient theorem, it holds that
VO th (0)lo=8, = VV,(0u).
(47)
Thus, the approximate TRPO update is equivalent to
0x+1 = 02 +
(48)
where
Vc
C [G(())+ (OW(lo))
(49)
Hence, the approximation TRPO update corresponds to natural policy gradient update with adaptively chosen step size.
B. Details the Safe Geometry for CMDPs
B.1. Safe Trust Regions
The safe mirror function for a single constraint given by
oc(d) = 0x Σ 8 cTd).
(50)
and the resulting Bregman divergence
Do(di = (c((1)) vc((2) (VDc(ds), - -d)).
(51)
linear operator in Φ; hence
(s Dox (dilld2) 3DD(dilldd))
(52)
where
Do(dilld2) ø(b c di) o(b c d2) c d2)
(53)
= ø(b - di) - @(b c d2) Φ(bb et d2)(c di - c d2)-
(54)
ø(b - Vo(TT) - o(b V((T2)) + o Vv(m)))(V((T)) Vc(TT)).
(55)
The last expression can be interpreted as the one-dimensional Bregman divergence Db Vc(T))lb Vc(Tx)), which is a (strictly) convex function in Vz(Tr) for fixed m, Φ is (strictly) convex.
15

---

Embedding Safety into RL
b)
TRPO
CPO
C-RRPO (Ø 10
C-TRPO (Ø =
Figure Illustration of policy divergences (dashed) close to the constraint (red). a) TRPO (dotted for reference) and CPO. b) C-TRPO's divergence depends on the syper-parameter 3, which modulates the strength of the barrier towards the constraint surface. For B ' 0 we obtain update equivalent CPO, and more conservative updates for larger values (3 = 2). The plots were generated with the toy MDP in Figure 2. c) Shown are the quadratic approximations of the divergence in parameter space, which is obtained by mapping the policy onto its occupancy measure, where a safe geometry can be defined using standard tools from convex optimization (safe region in white).
B.2. Details C-TRPO
B.2.1. SURROGATE DIVERGENCE
In practice, the exact constrained KL-Divergence De cannot be evaluated, because it depends on the cost-return of the optimized policy VT( Therefore, the surrogate divergence
(T(TT me.) = <(b V A (m²) Φ(b VoTk) + (V--
(56)
which is obtained by the substitution VT(TT) - V + A (T) in De-
When we center this divergence around policy ne and keep this policy fixed, it becomes a function of the policy cost advantage.
Do(TTol ToL) = @bb VTT - o(b -(-
+ 00(b 6'(b-Vo666 = 000 - A (x))) + A (T)
(A ).
Note that (T(TTO(TTo.) ((AA((m)) where (xx) 06,, - - ((6)) - (6(6)) x is a (strictly) convex function if (strictly) convex, since is equivalent to the -in-dimensional Bregman divergence Do el)6)) on the domain of o(b x)), Figure 6.
Example B.1. The function (xx) log(x) induces the divergence
8 5, A (m²))
Da (TO.) = A. (TTO) (56 - AA (TT))
(57)
16

---

Embedding Safety into RL
2
log(z)
Φ log(z)
Do
0
0
66
Cost Advantage
Figure 6: The surrogate Constrained KL- KL-Divergence as a function of the policy cost advantage.
B.2.2. ESTIMATION
In the practical implementation, the expected KL--divergence between the policy of the previous iteration, Tk. and the proposal policy π is estimated from state samples s; by running Th in the environment
Σ dr (O
1/N
Σ DEL (KL(T((s))(Ta((sss)
(58)
where DKL can be computed in closed form for Gaussian policies, where N the batch size. For the constraint term, estimate d, from trajectory samples, well the policy cost advantage
(ails:) (a:(s))
(-) = h
N Σ
A.
(59)
where A. is the GAE-À estimate (Schulman et al., 2018) of the advantage function of the cost. For any suitable ø. the resulting divergence estimate is
D& 0(6. - -) -(66) (
(60)
and for the specific choice d(x) x lo)(x))
8b
Do = A -66 - A)log log
(61)
88
A.
B.2.3. DETAILS ON C-NPG
In showing that TRPO with quadratic approximation agrees with a natural gradient step, see Appendix A, we have used that Ao (0) 1==0,- = (V,)),), which holds although A, is only a proxy of V,. We now provide a similar property for the quadratic approximation of the surrogate divergences De.
Proposition B.2. For any parameter 9 with n E IIaae it holds that
viD0(010))0 = V3Do(0110)
(62)
and hence
V 2x + 9V3
Gc(
(63)
where Go( denotes the Gramian mutrix of C-NPG with entries
Gc((),) = Ob dI (2
(64)
17

---

Embedding Safety into RL
Proof. Let (0) VzD and (o) V;Do(0110))=bb One can show that HKL Gk(0) (Schulman
al., 2017a). Further, we have
(o) - "AA"000000000000000000""0000"000
(0)0" (.)
where a) follows from (' (0)) 0 since (0) 0, v 2 0 and A2)0) O. Further, follows because
T (")x))x= (66). Thus, He is equivalent to the Gramian
Gc(();; db V (2
(65)
= Gk(O); + B6" cx da) do de cc 0g
(66)
HKL (V)o V.(0))
(67)
= HKL + 3Hg.
(68)
Again, for multiple constraints, the statement follows analogously.
In particular, this shows that the C-TRPO update be interpreted natural policy gradient step with adaptive step size and that the updates with De and De are equivalent if we use a quadratic approximation for both, justifying Do as surrogate for De.
B.3. Beyond finite MDPs
For the sake of simplicity and this is required for our theoretical analysis, have introduced C-TRPO only for finite MDPs. However, C-TRPO can also be used for problems with continuous state and action spaces as we discuss here. In this case, the state-action and state distributions are defined as
00
ds A) (1-7) Σ
7 P E S, A) and
ds(S) = (1 77) Σ
(s ES)
t=0
for any measurable subsets A A and S SC S. Further, the Kakade divergence is then given by
Dx (d = Egnd (1(s)
(69)
which is well defined mı (-1s) is absolutely continuous with respect to T2(-|s) for d almost all 8 E S. The Bregman divergence that C-TRPO builds on is just as in the finite case given by
m
De(dil/d2) DK(d) (d2) +
BDD (dilld))
(70)
where
Do. (dilld2) = 0bb; - Val (TT)) o(b. - Va. (TZ)) + @'bb Ve, (T))) - Val (TZ)).
(71)
Like in the finite case, the policy advantage is defined
T(als) (als))
A (s, (9.
nh (TT
E Bs
(72)
18

---

Embedding Safety into RL
where Aă (s, (s, a) (s) denotes the advantage function, which is defined analoguously to the finite case. Now, the plain trust region update given by
0k+1 E arg max Ar (m) sbj. to De(day [ S 6.
(73)
Just like in the finite case, we use a surrogate divergence De and obtain the formulation of C-TRPO
T = arg max A, (T) sbj. to (e(T)(TT) S 8.
(74)
TEll
Here, the differences De are that we use samples from the state distribution d and use a surrogate for the cost advantage estimate the divergence Dæ, described in Section 3.2. Further, we parametric policy model my and linear approximation of A TH as well as quadratic approximation of (T( for our practical implementation.
Expression for Gaussian policies We test C-TRPO various control tasks where we use Gaussian policies. More precisely, the state and action space consist of Euclidean spaces S Rd and = R Then, we consider a policy network :e: S + which predicts the mean action and assume parameterized but state independent diagonal Gaussian noise, meaning that To(+ss) N (No (s), Eo), where Eg is diagonal. Consequently, we can closed-fore expression for the KL divergence
DKL. (Fo, (-1s)]|TO, (-s)) ( tr -2 Eo.) dg +1|po, (s) poo (s)())) In
det Eo²
det 20, ))
see (Zhang et al., 2024b).
C. Proofs of Section 4
C.1. Proofs of Section 4.1
Our theoretical analysis of C-TRPO is built on the following bounds the performance difference of two policies.
Theorem C.1 (Performance Difference, (Achiam et al., 2017)). For any function [(s, a), the following bounds hold
2
Va(a1) - (T( $ A
Eanday DKL
(75)
(1-7)
where Ef max, [ (s,
Theorem C.1 can be interpreted bound on the error incurred by replacing the difference in returns Vy( VT(T)) of any state-action function by its policy advantage A 22 (m1).
Proposition 4.1 (C-TRPO reward update). Set Gr = max, [BuNnaRY ( a)) The expected reward of a policy updated with C-TRPO is bounded from below by
V2ōyer 1-
Vx(T+41) 2 V((TA)
(23)
Proof. It follows from the lower bound in Theorem C.I that
V=(Tx+1) VT(TT) 2 A (TT+1)
ye, (1 1-77)
2D DD
(76)
where we choose The bound holds because D& 2 O. and thus De 2 EDKL. Further, 8 2 De and A (T&+1) by the update equation, which concludes the proof. See Appendix C.3 for a more detailed discussion.
2 0
Proposition 4.2. The approximate C-TRPO update approaches the CPO update in the limit as >
19

---

Embedding Safety into RL
Proof. Let fix a strictly safe policy To int see)- In both cases, approximate the expected cost of a policy using Ve(i) Va(mo) + Ago (T),, which is off by the advantage mismatch term in Proposition 4.1. Hence, maximize the surrogate of the expected value A (T)) over the regions
Pero = f E II: DK(T, TO) S 6, Vc(To) + A (T) - bb
in the case of CPO, and
Ps = ( EI: De(a, s 10)
with C-TRPO for some B 0. Note that
De( To) DK( mo) (A7(
(77)
and V (-o0,66) 66) + (0, +oo) and (t) - to0 for 7 6y. where 8, = - Va(To). Denote the corresponding updates by ACPO and the C-TRPO update by ng. Note that we have Pa C Py C PCPo for 3 2 8". Further, we have
U P P P: 8, V.(To) + A (T) < b). 0>0
Hence, the trust regions P3 grow for 3 0 and fill the interior of the trust region PcPo-
Remark C.2. Intuitively, one could repeatedly solve the C-TRPO problem with successively smaller values of 3, which would be similar to solving CPO with the interior point method using V as the barrier function.
Proposition 4.3 (C-TRPO worst-case constraint violation). max, E [1 AT (s, a)). It holds that
Let De(Ts+il|TH)
6 with 5 > 0 and set
Ve(Tx+1) S Ve(TTK) + (T<+1) + / (3(
(24)
1-7
where 6(3) 8- D--DDo(T44imm) S 8 is decreasing in 3 0, l
8, and 6(B) + 0 for B +
Proof. Setting f = c: in the upper bound from Theorem C.1 we obtain
27cc
(o(Te+1) < V((T)) + AT A (Tx-1) + (1- V
Bond., DKL (KL(Te41(1)))l
Note now that we have
Eandin DKL. (TI.(TE44( (T)) Do(TT -B (T+ mb) 56-1 (Do(TTL 6(3).
C.2. Details on the results in Section 4.2
Recall that we study the natural policy gradient flow
0,0, Go(OO)+VV..CC)
(78)
where G denotes a pseudo-invers inverse of (0) with entries
Ge(A).g = 0o dJ V Gk(().g + Brrd"bb Cg do)oo, db ctck Go
(79)
and 0 - no is differentiable policy parametrization.
Moreover, we assume that + Te is regular, that is surjective and the Jacobian is of maximal rank everywhere. This assumption implies overparametrization but is satisfied for common models like tabular softmax, tabular escort, or expressive linelinear policy parameterizations (Agarwal al., 2021a; Mei et al., 2020a: Müller & Montúfar, 2023).
20

---

Embedding Safety into RL
We denote the set of safe parameters by Saate = {0 IRP : me saaee); which is non-convex in general and say that O sade invariant under Equation (25) if 0g E safe implies 0g saate for all t. Invariance is associated with safe control during optimization and is typically achieved via control barrier function methods (Ames al., 2017; Cheng et al., 2019). We study the evolution of the atatenaction distributions de d as this allows us to employ the linear programming formulation of CMPDs and we obtain the following convergence guarantees.
Theorem 4.4 (Safety during training). Assume that Φ: Ry0 - T. satisfies (x(x) - +000 for x ' 0 and consider regular policy parameterization. Then the set eate is invariant under Equation (25).
Proof. Consider solution (0t) of Equation (78). As the mapping π + d is a diffeomorphism (Müller & Montúfar, 2023) the parameterization ssafo + Dante 0 + d is surjective and has a Jacobian of maximal rank everywhere. As Gc(O),) = dg, dg V&cOo de this implies that the state-action distributions de = d solve the Hessian gradient flow with Legendre-type function &e and the linear objective + T see (Amari, 2016; van Oostrum et al., 2023; Müller & Montúfar, 2023) for a more detailed discussion. It suffices to study the gradient flow in the space of state-action distributions dz. It is easily checked that &e is Legendre-type function for the convex domain Zc, meaning that it satisfies [[V(( too for dn + d E 29 Since the objective is linear, it follows from the general theory of Hessian gradient flows of convex programs that the flow is well posed, see (Alvarez et al., 2004; Müller & Montúfar, 2023).
Theorem 4.5. Assume that d' (x)) - too for x > O, set Vic max VA(T) and denote the set of optimal constrained policies by M = { E Ilaafo (T( = Vi consider a regular policy parametrization and let (0t)
solve Equation (25). It holds that (To.)) + Vic and
lim xe = safe = arr ( (D((=" ,T)) : n²* E a +
(26)
Proof. Just like in the proof of Theorem 4.4 we see that de = t solves the Hessian gradient flow with respect to the Legendre type function &c. Now the claims regarding convergence and the identification of the limit lim,->+- To, follows from the general theory of Hessian gradient flows, see (Alvarez et al., 2004; Müller et al., 2024).
C.3. Performance improvement bounds and choice of divergence
In a series of works (Kakade & Langford, 2002; Pirotta et al., 2013; Schulman et al., 2017a; Achiam et al., 2017), the following bound on policy performance difference between two policies has been established.
27Cr B (1-7) Bond, (iv(r'(lm)(s)
Vy(T') (y
T5 A (T)) +
(80)
where Dry is the Total Variation Distance. Furthermore, by Pinsker's inequality, we have that
Drv(T s V (xa((())
(81)
and by Jensen's 's inequality
Ee-d, DTV (T)) S - DKL DKL((T) (T)))))(s),
(82)
It follows that we can not only substitute the KL-divergence into the bound but any divergence
Do (d', [dz) 2 Eawd, DrI. (T'((=))))
(83)
can be substituted, and still retains TRPO's and CPO's update guarantees.
C.4. Comparison with CPO
In the approximate case of C-TRPO and CPO, where the reward is approximated linearly, and the trust region quadratically, the constraints differ in that C-TRPO's constraint is
(0 - 0zx) (FTk.()))))))))))))))))))))))))) + (0) 0x) <
21

---

Embedding Safety into RL
π
CPO
0
0
n
D
+
π
C-TRPO
Optimization Path
Pram, Update (Theory)
Pram, Update (Approx.)
Figure 7: Pictorial illustration of conceptual and practical differences between CPO and C-TRPO. The local approximation of C-TRPO's s trust region results in a single quadratic constraint, which compressed in the direction the closest cost surface, depending on the hyper-parameter 3 (blue dashed lines on the right). This is in contrast CPO, where the local approximation of the update results in quadratic constraint which is not affected by the cost, and linear constraint which only takes effect upon contact with the cost surface. Intuitively, this results in smoother optimization path for C-TRPO that remains on the interior of the safe policy space for longer.
whereas CPO's is
(0 - (K))) 0x) < and Voe + (VoA 0x) s
Figure illustrates the differences between CPO and C-TRPO.
22

---

Embedding Safety into RL
D. Additional Experiments
D.I. Effect of hyper-parameters
To better understand the effects of the two hyperparameters / and by, we observe how they change the training dynamics through the example of the AntVelocity environment.
The safety parameter B3 modulates the stringency with which C-TRPO satisfies the constraint, without limiting the expected return for values up 3 1, see Figure 8. For higher values, the expected return starts degrade, partly due to Do being relatively noisy compared to DrL. and thus recommend the choice 3 1.
100
3000
400
Cost Regret
300
Return
2000
Cost
200
40
TRPP (d) 0.05)
100
C-TRPO (8 0.2)
20
0-TRPO (2
rist limit
0.4
0.6
0.4
0.6
0.6
0.8
Steps
x107
Steps
x100
Steps
x10"
Figure 8: Influence of on C-TRPO's performance.
Finally, employing a hysteresis fraction 0 < bu < b seems beneficial, possibly because leads the iterate away from the boundary of the safe set, and because divergence estimates tend to be more reliable for strictly safe policies. The effect of the choice of by is visualized in Figure
100
C-TRPG Par 11
C-TRPG 0.2)
Seait
80
3000
Cost Regret
Return
Cost
60
2000
440
20
0.0
0.2
0.1
0.6 Steps
0.8
1.0
0.0
0.2
0.4
0.6 Steps
0.8
1.0
0.0
0.2
0.1
0.6
0.8
1.0
Steps
Figure 9: Influence of the hysteresis fraction my C-TRPO's performance.
23

---

Embedding Safety into RL
D.2. Ablation Study: CPO vs. C-TRPO
We conduct an ablation study rule that our improvements of C-TRPO over CPO only due hysteresis. For this, run both CPO and C-TRPO with and without hysteresis with the same hysteresis parameter in other experiments. We that the hysteresis improves safety for both algorithm. Further, find that the hysteresis slightly reduces the return of C-TRPO. Overall, we clearly see that C-TRPO itself is much safer compared to CPO as even C-TRPO without hysteresis achieves lower cost regret compared to CPO with hysteresis.
Reward
Cost
Cost Regret
1QM Normalized Score
1.2
1.0
C-TRPO (apet.)
0.6
CPO (last.)
1.0
CPU
0.5
0.4
0.8
0.2
0.6
10
10
2
4
6
10
Env. Steps (millions)
Env. Steps (millions)
Env. Steps (millions)
Figure 10: Ablation study on the core components of C-TRPO: Safe trust region (C-TRPO no hyst.) and recovery with hysteresis (CPO hyst.). Evaluation is based on the Inter Quartile Mean (IQM) normalized scores across 5 seeds and 8 tasks. From left to right: episode return of the reward (PPO normalized), episode return of the cost (threshold normalized), and cumulative cost violation (CPO normalized).
D.3. Noisy Cost Estimates
To evaluate the sensitivity of C-TRPO to noisy or inaccurate estimates the cost value function V., we train multiple policies with C-TRPO on the Anteelocity task and corrupt each policy's value estimate by varying levels of noise, i.c. white noise with varying standard deviations σ, Figure 11.
Safety Ant Velocity-vl
Reward
Cost
Cost Regret
IQM Score
3000
30
500
C-TRPO (- 0.6)
C-TMP (e 0.1)
20
250
2000
10
2
6
8
10
10
2
6
10
Env. Steps (millions)
Env. Steps (millions)
Env. Steps (millions)
Figure 11: Performance of C-TRPO on the AntVelocity task as function of cost value noise. The final reward decreases slightly for noise with a = 0.5 and the cost regret increases with the amount of noise added to the value estimates.
24

---

Embedding Safety into RL
D.4. Performance on individual environments
Here, we compare C-TRPO relevant baseline algorithms on all individual environments in terms of their sample efficiency curves. To improve readability of the plots, only the algorithms that are, average, safe in the last iterate included.
Safedy Aut Melel
SafidyHopppeee Hilper
Rewad
Cou
Oue
Ropert
Rowed
Cuul
Chu Regeet
200
1gkt Sam
2000
Shrpps
Steps
Strppe
Stepe (catilone)
Strpp
Sadetylitmmmoolaaaettyy
Ropec
Ropert
ToM Sooe
1024
1000
-000
10
Exs. Sigg (taillamm))
Eac. Stepo (aiilien)
Bas Sloge (allismn)
Bus. Stoge (aillamm))
Enc. Stago (tailiin))
Ban. Stego (aalliona)
Figure 12: Benchmark on the locomotion environments.
CalityCaBuutttall vp
Paltty vD
Boward
Cas Hegret
Howed
Oan
Cur Hoger
300
1ow Seor
9301
3331
Ear. Stape (tillonn)
Stepe (uilikana)
Supp (sallano)
(aillion))
Stage [sailaen]
Ban. Stepp (ailliona)
SulreeSstutPullll
Resanl
Cood Regiet
Bewwed
Cod Regiet
R2M Sare
F2M Sooe
5000
Ear. Stepe (ualloon)
Eas. Stepe (aallbina)
Ear. (callano)
Stepe Caalliionl
Ear. Stupe [mallaeo)
Bar. Stepp (nailiona)
Figure 13: Benchmark on the navigation environments.
25

---

Embedding Safety into RL
Table I: Final performance per task for 5 seeds each, where expected reward V,, expected cost Ve, and cost regret Reg+ shown, including lower and upper confidence intervals. We highlight the best performance with respect to the IQM of the return V, in bold, and box the lowest cost regret Reg+.
(san0) OOBLL
DV
1 520000
OMI
C0P
+1 OLD
11 STEIE +4 old
+20g
+20
+20g
+20g
+
+000
PA
2/
44
2/
PA
P4
2/
4A
5'0592 990 - 119951
1S/ F 2962
1'M00S VOC + STOLL
5099999 T FANTEE
FUIW 1NE
9 F2
S 00091
L.0AT #02200
699211 501- T LY21
S1ISE F1'ST
ST2 FLMMEE
6238999- FROSE
1W +OE1
EWED9 N9911
TO 3 F
999099991 T EOOOEE
29110001 F900
WILLIS 98- 6 TOME
5'09/9 FO'.
SWEE: = KX1
5'90 F01600
LW09 T
5
L'T19 VILL
11142011 1992- TE 15606 0'29118 CORS CO0NOSSE 2152
1'2100 FVMARE
L' VAL T LI28 2 1996 1001 FUUM
D' T OOLYS 8 LIAV F SUMIE
0' DOCCIE TeaHl
WIIIL EVERE
11001 218)
L'DWALL WEEI
99901 REI - 1882
STUNT FST0IC
6'LLMOIS TI F FYYEE
LODE 90 F 19900E
WELOO TRIS- PSYPEE
(122IN2211 6 6 64/11A OONCE
6009 52- S0092-FF8E5 F
0/22/ 2011 66P1S 1199
696
09899 FF12
SWES-FOL FOL
1UV2- OLL
ZULT 5992
6816- FOS
0500000 F.500
1STW 000
O' 54C 89988
6'5395 VIS T FOACE
2 #6
6'0/21S F6911
S'OME WE- 1 9
F LES 18S"+ UTEEP
LNI1OSS FIZ00S
7636
8'200'00-- F6
2'0W FINI
S'ODIEL FOME
9'8IL'S #291
YOULDE F4SS
9 F16
0 =00
099999 F
C9103' 2ZZZ
6011 #221
01891
SWIP
6'HIC
69111 USS L 25651
0'6C10 1INE
DIEA LOC 1999
EB/ 91091
FINE W0 80691
9'90618 TOSSE
POIS9 1019919992 99991
L'KEEWWORG FVES
1991 5992
0'8P1 F53911
F'S9 WH0T F995K
0' 190 59 5590
ST 16 +65009
2'9900 XOP 3T0OL
6111115 1OP: 19991
TSS/ F009
SOVE 6532
L'199- 632
FNES L.ST
L'SL WZZ
Pa00 9XL
11860 9O1
9
6'RLOOL VENDY F FIOSII
9 199 199 7 0U
990707 VOLL- D 188111
L' 100) FOREL
S'O0BE 9X8I21
PTIEB FOUEE
F'ENNG YAL" * 13PP11
198955 59p 66682
6129 F SINE
SWO 5
L'IEE'I- 2'
WON''- FED
F'N2D- 61
11 (2001
OLSIC NE- 5112
6'D F
0'116 9H5
L'W901 FSSE
2W-90 E1
FWELI-FFI FEID
C9IW 66
EWEOO 220
9WS0 CTT
LSSIT 29A F99
9WS5 61
6/WR-- F s/95
S 01
F'BEYOO
L'DE
L'006 F 1999
V'NCUOLNNI 991666
0'3SN' = SPICI
VOSUIL LONOP
6'0299 3MP 9'655
POCAOO FLSS
699561155 1992
F' = 9/12
5'002 F 952
0'N 99
9UTE 5 9
LWVS- FES
11 F 638
L'US 1US1-888
CONU SS
W 19
20000 7
00200 =
L'OIL
Y 19991199 900- F59225
FIZUAAY F F22ZN
52017 LEPI- F111--611000 1099
90996 FRA00
LAMILLLE STAST
WP9LLP 200 FO0ORST
PISSIL S9C- 88991
6627 + 9Y9
CWTN 1Z12
1985--991 9972
FUEV 9952
SISINS'SI
5166 1/22
09999- T NST
FOTTD 60
E I'8
FIOS-- 9.0
9063 - 999
Ch0C- 2
POCO-TS
9HITTIIE
CCC FOWEE
8991/111 1O 2919
E 1ST OOMA
596662 61 F SY0R1
6.9902 10-------- +1 1288
T'UNCIT E1N61
16660 901- 901 C909C
COCCO 190
OOVED 190
111I11-99. T955
TVEE +50
ROMUN
000A08-- FOST
011099- COC
FSYS-FIM
601111= WEZ
11190--01 F0
9ES90 WIY
FWEN + 1'2
POC0-F60 6.0
26


Reviews:
[{"review_id": "X5xl3UbiTj", "reviewer": "Reviewer_gPum", "summary": "The paper proposes C-TRPO and C-NPG for solving CMDPs. Mirror functions are used to define policy divergences that are finite only for safe policies. This divergence metric is then used to reshape the policy shape geometry to ensure that trust regions contain only safe policies. The algorithms are analysed theoretically, and it is empirically shown that they lead to fewer constraint violations while obtaining similar rewards.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback! We have addressed your specific comments in detail below.\n\n> Log barrier baselines are not considered in the experiment.\n\nWe considered **IPO** as a practical log-barrier baseline, along with **P3O**, a proximal adaptation.\n\nRegarding the specific works mentioned:\n\n- LB-SGD (Usmanova et al.) is discussed in our related work section. However, our focus is on practical deep RL algorithms, whereas LB-SGD primarily offers theoretical insights. Given the lack of large-scale empirical evaluations and implementation details for deep function approximation in LB-SGD, we did not include it as a baseline in our experiments.\n\n- Saute RL presents an interesting comparison but tackles a subtly different problem formulation than standard CMDPs. As seen in Definition 4 of their work, Saute RL imposes stricter constraints by disallowing stochastic policies from occasionally violating constraints and assuming non-negative safety costs. While this formulation is reasonable for safety, it does not always correspond to a general CMDP. A key strength of C-TRPO is its reliability in solving standard CMDPs, making it applicable to broader tasks like diverse policy optimization (e.g., [1]). A rigorous theoretical and empirical comparison would require additional work beyond the current scope. However, we will mention Saute RL in the manuscript and highlight its comparison with C-TRPO as an important direction for future research.\n\n- ActSafe is a newly accepted work that we were not previously aware of. Since it falls under model-based safe RL, which we briefly discuss in our related work section, we will add a reference in the final manuscript. However, we want to highlight key differences from our approach. Specifically, in Definition 4.7, ActSafe indeed defines a distance measure between policies, which is informed by cost continuity. In contrast, our Bregman divergence explicitly captures differences in the cost values of the policies, representing a key conceptual distinction in how safety is enforced.\n\n[1] Zahavy, Tom, et al. \"Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality.\" The Eleventh International Conference on Learning Representations.\n\n> I am not sure about analysing cost-regret with trust region methods given that they are rarely applied for learning in the real world due to sample inefficiency\n\nThis is an important point. While near on-policy approaches like TRPO tend to be sample inefficient, they remain valuable due to their stability and well-established theoretical foundations in the function approximation setting. While they may not be ideal for online real-world learning, understanding how to minimize cumulative constraint violation regret is essential, as any algorithm used for fine-tuning on a real system must consider it.\n\nWe believe that the insights from C-TRPO’s regret analysis will contribute to the development of future algorithms that are both sample-efficient and safety-aware.", "claims_and_evidence": "The claims in the paper are clear and convincing.", "methods_and_evaluation_criteria": "The experiments are thorough and the benchmark considered is widely used for safe RL methods. I wonder why log-barrier-based approaches such as (1 or 2) are not used as baselines. Furthermore, I would have also expected Saute RL (3) to be a baseline.\n\n\n1. https://www.jmlr.org/papers/volume25/22-0878/22-0878.pdf\n2. https://arxiv.org/abs/2410.09486\n3. https://arxiv.org/pdf/2202.06558", "theoretical_claims": "I think the theoretical claims are correct.", "experimental_designs_or_analyses": "The experiments are evaluated on a well-established safe RL benchmark and are sound, in my opinion.", "relation_to_broader_scientific_literature": "The problem addressed is important and relevant. The insights drawn for trust region methods are also broadly important. The only thing I am unsure about is that trust region methods are generally very sample-inefficient and, therefore, rarely applied for direct learning in the real world. The common strategy is to train the policy in simulation. In this case, what is the benefit of analysing the cost-regret? Effectively, what we care about is that the final policy is safe. I would appreciate it if the authors could discuss this in the paper.", "essential_references_not_discussed": "From my knowledge, I think the paper covers the essential references well.", "other_strengths_and_weaknesses": "**Strengths**:\n\n-- The paper tackles an important problem of safe RL using widely applied trust region approaches.\n\n-- The paper is well written and easy to follow.\n\n-- The experiments are thorough.\n\n**Weaknesses**:\n\n-- Log barrier baselines are not considered in the experiment. \n\n-- I am not sure about analysing cost-regret with trust region methods given that they are rarely applied for learning in the real world due to sample inefficiency.", "other_comments_or_suggestions": "No additional comments.", "questions_for_authors": "See strengths and weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aFpgZ1GnAI", "reviewer": "Reviewer_kedz", "summary": "The paper introduces Constrained Trust Region Policy Optimization, a new Constrained RL algorithm based Trust Region Policy methods. The trust region is made to only contain safe policies for the update step. The new algorithm enjoys good theoretical properties, with improvement and safety guarantees similar to Constrained Policy Optimization while having provable safety and convergence properties with some additional assumptions. Because of constraining the trust region to safe policies only, the algorithm is designed incur less constraint violations during training than the popular lagrangian approaches. The algorithm is benchmarked on different experiments, where it is shown to be competitive with state-of-the-art algorithms like CPO or CUP.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment of our work!\n\nWe agree that a regret analysis would further strengthen the theoretical contribution and see this as a valuable direction for follow-up work.", "claims_and_evidence": "The practical claims are that the algorithm is competitive with state-of-the-art algorithms, with a focus on the number of safety violations during training while still providing good optimal reward. The claim is supported by the experiments where the algorithm is benchmarked against other state-of-the-art algorithms in Safety Gym environments.", "methods_and_evaluation_criteria": "The method for evaluating the performance of the algorithm is benchmarking the algorithm on Safety Gym environments, and the criteria for the evaluation of the algorithm is the number of violations during training, and the cost and average reward at the end of training.", "theoretical_claims": "The theoretical claims are mainly Proposition 4.1 (reward improvement) and Proposition 4.3 (worst-case constraint violation), Theorem 4.4 (safety during training) and Theorem 4.5 (convergence).", "experimental_designs_or_analyses": "The experimental designs are evaluating the influence of the hyperparameters, comparing C-TRPO to CPO with hysteresis, and more generally evaluating C-TRPO versus other state-of-the-art algorithms on Safety Gym environments", "relation_to_broader_scientific_literature": "C-TRPO fits into a growing body of primal and primal-dual methods for solving constrained MDPs, and is a primal approach closely tied to CPO and TRPO. To the best of my knowledge, the approach in the paper is new and different from e.g. PCPO.", "essential_references_not_discussed": "I do not see any essential references that are not discussed in the paper.", "other_strengths_and_weaknesses": "The paper's strength is designing a new approach that has very good theoretical properties that are shown (maybe a regret analysis would strengthen the paper even more). The experiments are also quite convincing: even though the improvement in practice is not extreme, it is still significant. The paper is well-written.", "other_comments_or_suggestions": "The paper provides good background and intuition, and the subtleties of the links with NPG are well-explained.", "questions_for_authors": "I have no questions for the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2PvcUN7ePa", "reviewer": "Reviewer_D8iN", "summary": "In this paper, authors present idea of solving Constrained Markov Decision Processes (CMDPs) using trust regions that obey the constraints strictly and allow for return maximization. Earlier approaches work with KL-divergence based trust regions and try to recover the policy if the constraints of CMDP are violated using hysteresis. The paper works in the state distribution space, where KL divergence is expressed as Bregman divergence and the constraints of the MDP are converted to barrier functions that make the divergence go infinitely high for violating the constraint. This creates a \"safe\" trust region under which a policy would be optimized using natural policy gradient-style updates. If due to empirical approximations, the constraint gets violated, the hysteresis approach would be used to bring policy back into safe region. The paper presents important theoretical results for designing safe trust region, guarantees for reaching the optimal policy for the given CMDP and safety guarantees during training. The methods proposed are compared against other baselines in safe RL literature on safety gym environments. The results demonstrate that the proposed approach provides high returns while keeping the violations minimal.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments and suggestions! We respond to each of your points in detail below.\n\n> I had a problem understanding the results depicted in figure 3. Why are the results aggregated together across multiple tasks? How can we compare the average of normalized rewards and constraint violations across 8 tasks together?\n\nWe follow a standard procedure from unconstrained RL to provide a concise summary of results, adopting the methodology from [rliable](https://github.com/google-research/rliable):\n\n- Each algorithm is trained across multiple seeds per environment.\n\n- Performance is normalized per environment relative to a reference algorithm.\n\n- Normalized results are then aggregated across seeds and tasks to enable a high-level comparison.\n\nTo ensure robustness, we use interquartile mean aggregation and report bootstrapped confidence intervals, following best practices in RL evaluation. While this approach provides an informative summary, we acknowledge that individual task-level variations are important, which is why we also include per-task sample efficiency curves in the appendix.\n\n> The barrier function over state distributions is a clever way of inducing such a safe trust region.\n\nThank you for the positive assessment!\n\n> C-TRPO inherits methodology from CPO’s hysteresis idea that allows for safe policy updates.\n\nWe would like to clarify that hysteresis was introduced in our work, not in the original CPO paper. However, our results show that applying hysteresis to CPO improves its performance, as reported in our ablation studies. To achieve the full performance, C-TRPO’s safe trust region is necessary.\n\n> On line 233 (left column), the paper discusses divergence before it is formally defined (Equations 16 and 17).\n\nThank you for this suggestion! There are two key divergences:\n- The theoretical divergence $D_C$ (Equation 10).\n- The approximated divergence $\\bar{D}_C$ (Equation 16).\n\nWe will clarify this distinction earlier in the manuscript.\n\n> Algorithm 1 should be moved after the full methodology description to improve readability.\n\nWe agree and will restructure the manuscript accordingly.\n\n> Question regarding Theorem 4.4, Isn't CPO also safety invariant? It would be sufficient to show the CPO is invariant and then, C-NPG with any choice of beta would be invariant too.\n\nNeither CPO nor C-TRPO are strictly safety-invariant as defined in Theorem 4.4 due to approximation and estimation errors. However, Proposition B.2 shows that in the limit of small step sizes, C-TRPO converges to the safety-invariant C-NPG.\n\nIn contrast, CPO does not follow a natural gradient update, making its small-stepsize limit unclear. While a bound similar to Proposition 4.3 exists for CPO, it is less conservative than C-TRPO’s. Importantly, both methods provide only a bounded approximation of the ideal safety invariance property of C-NPG, even with perfect value function knowledge. When value functions are estimated from finite samples, these bounds are further affected.\n\nIn practice, C-TRPO offers a more conservative safety guarantee with minimal computational overhead or performance loss. Since estimation errors also impact safety, we plan to explore the finite-sample safety properties of both methods in future work.\n\n> I feel the property that needs to be more emphasized in the text is the conservativeness of the policy update under higher beta values w.r.t. CPO.\n\nThank you for this suggestion. We will highlight this aspect in the manuscript.", "claims_and_evidence": "The contributions claimed in the work are supported with theoretical and empirical evidence.", "methods_and_evaluation_criteria": "- The algorithm proposed in the paper is closely related to Trust Region Policy Optimization (TRPO) [1] and Constrained Policy Optimization (CPO) [2]. The components added to TRPO and CPO are justified properly and make sense.\n\n- The paper uses safety gymnasium [3] for their experimentation. The choice of environments is apt. As for metrics, expected returns and constraint violations are compared across multiple algorithms.\n\n[1] Schulman, J., Levine, S., Moritz, P., Jordan, M. I., and Abbeel, P. Trust region policy optimization, 2017\n\n[2] Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained policy optimization, 2017.\n\n[3] Ji, J., Zhang, B., Zhou, J., Pan, X., Huang, W., Sun, R., Geng, Y., Zhong, Y., Dai, J., and Yang, Y. Safety gymnasium: A unified safe reinforcement learning benchmark. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023", "theoretical_claims": "I have not checked the proofs of propositions 4.1, 4.2 and 4.3 and the proofs of theorems 4.4 and 4.5.", "experimental_designs_or_analyses": "- I had a problem understanding the results depicted in figure 3. Why are the results aggregated together across multiple tasks? How can we compare the average of normalized rewards and constraint violations across 8 tasks together?", "relation_to_broader_scientific_literature": "Unlike earlier methods, the paper presents a way to design a safe trust region and provides a practical algorithm to optimize for returns within this trust region. The barrier functions over state distribution is a clever way of inducing such a safe trust region. \n\nThe paper provides C-NPG, a natural policy gradient variation with safe trust region, and C-TRPO, TRPO under safe trust region, algorithms which naturally inherit the methodology from earlier prominent works with CPO's hysteresis idea that allows for safe policy updates.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "Strength of the paper is in its theoretical rigor and innovation behind design of safe trust region. The paper takes more principled route of adhering to the safe trust region while updating policies.\n\nWeakness: I don’t see any major weakness of this work, although I have not checked the proofs in this work which can have mistakes.", "other_comments_or_suggestions": "I had a few suggestions regarding writing: \n- On line 233 (left column) the paper starts talking about divergence $D_C$ which is said to be defined below and there is no definition until it has been used multiple times before it is formally defined in equations 16 and 17.\n- Also, algorithm 1 can be moved after the description of entire methodology that way it avoids unnecessary introduction of variables and steps that are not talked about in the text until algorithm 1 starts in the text.", "questions_for_authors": "Question regarding Theorem 4.4, Isn't CPO also safety invariant? It would be sufficient to show the CPO is invariant and then, C-NPG with any choice of beta would be invariant too. I feel the property that needs to be more emphasized in the text is the conservativeness of the policy update under higher beta values w.r.t. CPO.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UEsIvPKN8l", "reviewer": "Reviewer_seAB", "summary": "This paper considers the problem of constrained MDPs. The key idea is to modify the geometry of the policy space to ensure that trust regions contain only safe policy space. This is achieved by introducing a new family of policy divergence that incorporate certain mirror functions. The author provide theoretical guarantees on the convergence of CNPG and theoretical properties of the C-TRPO updates. In addtion, the author also present empirical evaluations across 8 different tasks to show that C-TRPO reduces constraint violations while maintaining competitive returns.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> How sensitive is C-TRPO to the accuracy of the estimated cost function? Could you provide an analysis or experiments showing how performance degrades with increasingly noisy or misspecified constraints?\n\nThis is an excellent question. While we do not yet have a comprehensive theoretical analysis, we are running experiments to assess C-TRPO’s robustness to noisy or misspecified cost function estimates and will report results in the updated appendix.\n\n> How does C-TRPO handle multiple, potentially conflicting constraints? Could you provide theoretical insights or empirical results for scenarios with multiple safety criteria?\n\nC-TRPO naturally extends to multiple constraints. The benchmark environments we consider already include multiple cost signals, which are aggregated into a single constrained objective. While our current experiments adhere to this setup, it is possible to modify the benchmark tasks to enforce separate constraints on individual cost functions.\n\nWe will expand our discussion on handling multiple constraints in the final version of the paper. Theorem 4.5 already implies that the optimal constrained policy $\\pi^*_{safe}$ found by C-NPG satisfies as few constraints with equality as required to be optimal, which is discussed in the respective paragraph. We expect similar results to hold for C-TRPO as well.\n\nAdditionally, we recognize the importance of empirical evaluation in scenarios with distinct, potentially conflicting constraints, and we consider this an important direction for future work.", "claims_and_evidence": "Yes. The theoretical properties (convergence and optimality) are well-supported by the proofs in section 4 and appendix C. \nThe empirical results demonstrate reduced contraint violations. The comparison is comprehensive and fair.", "methods_and_evaluation_criteria": "Yes, the proposed methods C-TRPO makes sense as it provides a theoretically sound way to incorporate safety constraints into the policy geometry. The evaluation framework comprehensively assesses both theoretical claims and practical performance while maintaining relevance to real-world safety constraints.", "theoretical_claims": "Yes, I examined several key theoretical proofs in the paper, particularly in Section 4 and Appendix C.I did not find any obvious errors in the proofs, though some proofs in the appendix (particularly for technical lemmas) could benefit from more detailed explanations of intermediate steps.", "experimental_designs_or_analyses": "8 tasks from safety gymnasium benchmark are included in the experiment for empirical analysis. Overall, the experimental design is sound.", "relation_to_broader_scientific_literature": "This paper synthesizes ideas from multiple areas, applying concepts from optimization and control theory to address practical challenges in safe reinforcement learning. It represents a step forward in making RL more applicable to safety-critical real-world scenarios.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. the idea of combines trust region with barrier functions is novel, and it provides better solution for solving constrained MDPs.\n2. This paper provides strong theoretical guarantees and clean math framework connecting trust regions and safety constraints.\nWeaknesses:\n1. The paper has limited evaluation on larger-scale problems.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. How sensitive is C-TRPO to the accuracy of the estimated cost function? Could you provide an analysis or experiments showing how performance degrades with increasingly noisy or misspecified constraints?\n2. How does C-TRPO handle multiple, potentially conflicting constraints? Could you provide theoretical insights or empirical results for scenarios with multiple safety criteria?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces C-TRPO, a novel approach to safe reinforcement learning by reshaping policy space geometry so that trust regions inherently contain only safe policies. This theoretically grounded method extends trust region policy optimization with safety guarantees, ensuring constraint satisfaction throughout training without sacrificing return. Reviewers consistently praise the paper for its strong theoretical contributions, rigorous empirical evaluation across Safety Gym environments, and thoughtful handling of reviewer concerns. While some noted the lack of certain baselines and deeper empirical tests for constraint robustness, the consensus is that the work presents a significant step forward in safe RL, justifying its acceptance.
Decision: Accept (poster)