SafeMPO: Constrained Reinforcement Learning with Probabilistic Incremental Improvement
Authors: Alexander Mattick
Dominik Seuß
Christopher Mutschler
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Constrained Reinforcement Learning
Primary area: reinforcement learning
TL;DR: We provide a novel view on the Constrained Reinforcement Learning problem via bayesian inference and incremental, rather than greedy improvements towards the feasible set

Abstract: Reinforcement Learning (RL) has demonstrated significant success in optimizing complex control and planning problems. However, scaling RL to real-world applications with multiple, potentially conflicting requirements requires an effective handling of constraints. We propose a novel approach to constraint satisfaction in RL algorithms, focusing on incrementally improving policy safety rather than directly projecting the policy onto a feasible region. We accomplish this by first solving a nonparametric surrogate problem which is guaranteed to contract towards the feasible set, and then cloning that solution into a neural network policy. As a result, our approach improves stability, particularly during early training stages, when the policy lacks knowledge of constraint boundaries. We provide general theoretical results guaranteeing convergence to the safe set for this class of incremental systems. Notably, even the simplest algorithm produced by our theory produces comparable or superior performance when compared to highly tuned constrained RL baselines in challenging constrained environments.

Published as a conference paper at ICLR 2026
SAFEMPO: CONSTRAINED REINFORCEMENT LEARNING WITH PROBABILISTIC INCREMENTAL IMPROVEMENT
Alexander Mattick', Dominik S Christopher Mutschler
1 Fraunhofer Institute of Integrated Circuits IIS, Fraunhofer IIS. Nürnberg, Germany
2 Technical University of Applied Sciences Würzburg-Schue Schweinfurt, Germany
3 University of Technology Nuremberg (UTN), Germany
f firstname.. assr.ame) @iis wtt.:riii...assaoooeoe.aaoma fraunhofer.de
ABSTRACT
Reinforcement Learning (RL) has demonstrated significant success in optimizing complex control and planning problems. However, scaling RL to real-world ap- plications with multiple, potentially conflicting requirements requires an effective handling of constraints. We propose a novel approach to constraint satisfaction in RL algorithms, focusing on incrementally improving policy safety rather than directly projecting the policy onto a feasible region. We accomplish this by first solving nonparametrie surrogate problem which is guaranteed to contract to- wards the feasible set, and then cloning that solution into a neural network pol- icy. As result, our approach improves stability, particularly during early training stages, when the policy lacks knowledge of constraint boundaries. We provide general theoretical results guaranteeing convergence the safe set for this class incremental systems. Notably, even the simplest algorithm produced by our theory produces comparable superior performance when compared to highly tuned constrained RL baselines in challenging constrained environments.
INTRODUCTION
Reinforeement Learning (RL) offers significant potential for improving performance in a wide range of control and planning problems by directly optimizing the behavior policy through interactions with black-box environment (Sutton & Barto 2018). However, RL can pose safety challenges due to an untrained policy that accidentally damages equipment or even harms users during the learning process. Further, even if a policy has been safely trained to completion (c.g. in simulator), one can generally not be sure whether the policy can be considered safe during deployment in the real world.
To address these issues, there is growing trend towards Safe Reinforcement Learning (SRL) which explicitly introduces safety considerations into the training process. One of the most popular formu- lations for SRL is the Constrained Markov Decision Process (CMDP), which extends the traditional Markov Decision Process (MDP) by incorporating constraints C(a, s) on the policy's 's behavior:
max R(T)
s.t. Ci(TT) < B, Vi = O,..,
where R(T) Ex - yfre] represents the expected discounted future rewards, and C,(T)) E, D2 7.cas the expected discounted future costs for the i-th cost function. For the sake of this work, we define C(s, a) as the cost associated with executing action a in state & and following the policy after (analogous to the Q-function). Constrained Reinforcement Learning (CRL) refers to the process of finding a policy that is optimal and is within the fcasible set defined by Eq. [
It is worth noting that CRL is of wider interest than just safety. Constraints can be used to encode a wide range of different requirements, such as ensuring an autonomous quadcopter has enough battery return to its charging station, preventing policies from abusing known inaccuracies in simulation of the real-world environment. Indeed, many reward shaping problems can be efficiently reformulated as constraint satisfaction tasks.
Methods that optimize directly for safety can struggle when the set of feasible policies is difficult to find when the closest feasible policy is very suboptimal. Fig. shows a robot that attempts

---

Published as a conference paper at ICLR 2026
to throw a ball onto one of three platforms, each of them separated by chasms. The initial (unsafe) policy throws the ball onto the middle platform (R:0, C:O) and potentially into the holes to the left (R:1, C. and right (R:0, C:1). Since the initial policy has never reached the left platform, a greedy projection of the policy onto the feasible set would mean the policy only hits the central column, terminating with a reward of O and cost of O. Policy optimization methods that rely on greedy recovery (Achiam et al. 2017| Xu et al. 2020) or projection (Narasimhan) 2020) Yang et al.) 2022) will naturally find themselves generally, an arbitrarily bad reward), due to a lack of exploration.
local optimum that is safe, but do not obtain reward (or, more
Such situations are common in practice as they may also arise from inherent randomness in sampling from stochastic environments with stochastic policies. A suffi- ciently small probability of reaching high-reward, safe ar- eas can effectively eliminate potentially feasible policies from consideration. Ensuring a sufficiently close policy Tk+1 using KL-ball constraints K L(( ((Tk) < E, in- troduces challenge, as the intersection of feasible policies and policies within the e-ball might be empty.
Cook
Gestred policy
policy
support
oupoort
This is known problem: Constrained Variational Policy Optimization (CVPO) (Liu et al. 2022) aims to directly solve nonparametric surrogate problem. However, there is no guarantee that safe policy within a KL-ball exists as the initial policy can be arbitrarily far away from the feasible set. As a result, Liu et al. (2022) decay their cost upper bound B from some large value Bmaz - B over the course of training. The limitation of this is obvious: Too fast decay may lead to unsatisfiable problems, while too slow decay requires a large number of iterations exceeding B. This means that the decay speed is crucial hyperparameter that is also difficult to tune in practice, since it largely depends on the exploration performance and the specifics of the constraints and the environment.
Figure 1: A robot throws a ball a dif- ferent platform. The initial policy (gray) either hits the central platform (R:0) or one of the two chasms (C:1). The opti- mal policy throws the ball directly to the platform the left (R:1, C:O).
One might challenge this notion with the proof provided by Paternain et al. (2019) who show that constrained RL has zero duality gap_ Indeed, this is theoretically true: Assuming the occupa- tion measure has nonzero probability to reach every state, one could discover every possible pol- icy. However, this relies on the assumption that the occupation measure has a nonzero probability of reaching every state, which is unrealistic in practical algorithms with finite evaluations. Low- probability states effectively disappear from the sampled distribution and are never optimized. Since RL over occupation measures is linear (see (Paternain et al. 2019), program PT"), the distribution will never get optimized increased occupation in those states.
We propose an alternative approach: instead of greedily projecting onto the safe set, we aim to solely ensure that policy T is safer than the previous policy ти- We formulate this as a constrained Bayesian optimization problem, seeking a monotonically increasing likelihood of a trajectory being safe P(S = 1). Being Bayesian approach this is typically more robust to approximation errors and local optima. This way we can design algorithms that achieve high safety, while being more robust against approximation errors and local optima.
2 RELATED WORK
Contemporary methods for achieving constraint satisfaction on CMDPs often rely identifying and maintaining safety once a safe policy has been found. For instance, Constrained Policy Opti- mization (CPO) (Achiam et al., (017)) bounds the increase in costs ensuring that given safe policy и the updated policy x' will remain safe. During training, whenever a policy is considered un- safe, most-minimizing recovery steps are performed. Constrained Update Projection (CUP) (Yang) eet al., 2202) utilizes a similar bound to project the updated policy back onto the feasible set de- fined by generalization of the inequality used in (Achiam al. 2017).
Policy
'Indeed the proof by Paternain et al. Paternain et al. 2019 rediscovered a more general notion: every optimization problem max, I.(x) can be turned into linear program over measures: max, [[x:) )(r)dd,,
with the minimum being obtained by e.g. the Dirac impulse the optimal x' (see e.g., (Laasserre 2009
2

---

Published as a conference paper at ICLR 2026
Optimization (CRPO) (Xu et al.) 2020) updates the policy towards the optimum as long the policy is safe, reverting to cost minimization to regain safety otherwise. Reward Constrained Pol- icy Optimization (RCPO) (Tessler et al., 2018) combines the reward and cost signal into a single value function (s A) parameterized by the dual allowing for more principled updates than a traditional Lagrangian approach. This method is limited by how quickly À can be updated under exploration noise. Projection-Basee Based Constrained Policy Optimization (PCPO) (Narasimhan) (020) directly projects the policy onto the feasible set in every iteration, but requires Hessian calculations to perform accurate projections. Furthermore, projections may lead to large jumps in policies espe- cially during early training as the estimate of the feasible set may be very noisy and the policy may be far from feasible. Stooke et al. (2020) adopt PID perspective for tuning the Lagrangian À which gives rise to a host of different traditional methods combined with penalty function. Constraint recovery steps (either via hard projection as in CUP or descent-basee as in CPO) easily lead to the scenario from Fig. [ This is because the projection onto the space of safe distributions may not be exact (e.g., due to sampling errors) which prematurely reduces exploration for states that may be safe in practice. Our method solves this by not overcommitiing to any individual cost estimate.
Outside of CRL, maleki et al., 2018). MPO first optimizes a surrogate objective using a nonparametric approximation q of the poliey-improvement objective. The result of the surrogate then cloned to the policy neural network by minimizing (T(llgg) This explicit surrogate means that, from the point of view of neural network optimization, reinforcement learning becomes a supervised problem of matching to a known target distribution.
notable earlier work is Maximum Posteriori Optimization (MPO) (Abdol-
RL has long relied on incremental objectives, such as advantage functions, to reduce variance and improve convergence. It stands to reason that a similar treatment would benefit constrained RL methods. Indeed, most convergence proofs used in reinforcement learning rely on some notion of relative improvement compared to the previous policy (e.g. (Sutton & Barto (0088 Xiao (022)). We would argue that such guaranteed improvement constraints are more natural the iterative "explore then optimize" framework commonly used in RI..
3 SAFE MAXIMUM A POSTERIORI OPTIMIZATION (SAFEMPO)
We build upon Maximum a Posteriori Optimization (MPO) Abdolmaleki et al. 2018), extending their EM-style RL update to CMDPs. MPO sees policy updates inference problems with the sumption that the probability of a trajectory being optimal p(O = 1) follows a Boltzman distribution
p(O 1/T) x exp(2, rt/a). This leads the following ELBO
log Po = 1) = log (r(())(0=1)) = 1|r)dr 2 f
/ ((())
logp(O = 1/T) + log Pa(TT) dr
g(T)
= E. Σ r1/a
K (L(((T())))))))) (l)()))))=())),,)))))))))))(()))
which is then solved in two phases, This is a traditional Bayesian inference problem which can be solved using the Expectation-Maxiiiaza 1998) solves Squationn]by by iteratively maximizing of MPO, we first solve a nonparametric approximation derived directly from a batch of samples from the replay buffer to obtain g:
(EM) algorithm. The EM algorithm (Neal & Hinton
while keeping the other fixed. In the case
max B(a.s)~/(((a, (a)
(3a)
s.t. KL(g](T) SE
(3b)
f (q(a,s)dade s)dadas =
(3c)
where the KL-constraint is introduced to automatically construct suitable temperature a. In second step, one matches the policy neural network to the nonparametrically optimal
min [L(]))) KL(mmaa /r) < E).
This step is not dependent on any potentially nonstationary objective, but only has to clone the information of the optimal distribution found in eq. (3)

---

Published as a conference paper at ICLR 2026
MPO now proceeds to alternatinly solve Equation (3) and Equation In every iteration, the current policy т is approximated empirically by drawing states from the replay buffer and predicting [a]s)) The samples used solve Equation 33 for the optimal d which is then cloned into the
policy network using Equation (4)
We believe this be appealing structure for CRL as the nonparametric model eq. ( can easily accomodate additional constraints, which can then be handled by efficient optimizers (in our case Sequential Least Squares Programming).
3.1 SAFEMPO
We generalize this to CMDPs by introducing a second "policy is safe" event S and then optimizing the resulting joint distribution:
log P=(O 11,5=1 =
log p=( p=(r)p(0= 117000011-))p---00 1/7)dr
(5a)
a log
= 1/7) + logp(s 1/r) + log P+(TT)
>
a(T))
dr
(5b)
q(T)
Σ En/a + log 1|r) - (K(g(T))lPp(l
Bg
(5c)
Following Abdolmaleki et al. (2018), we define the probability of safety as an exponential law, but with the notable change that truncate the law while the policy is safe
exp
(C(a.s))B) 9
C(a, s) 2 B
p 1/7)
(6)
otherwise
Notice that this truncation automatically makes sure that all feasible policies have equal likelihood to be safe, which is exactly the behavior we want in CRL. We also define the cost-log-likelhhodd function:
G(a, s) p( 1|7) max (C(a,)) ((a.s))-B.1) (C(u,s)-B.))
B P...
This is the canonical choice for mapping the costs to a distribution, just as exp( R(T)) is the canon- ical choice for returns (Abdolmaleki et al. 2018). Specifically, the Gibbs distribution is the unique distribution that weights higher It higher while keeping maximal entropy (Jaynes) 1957). Our trun- cation adds the third property that values above B are considered equal, which is the core difference between an optimization objective, and constraint.
Notice that Eq. 5 allows for wide variety of optima: Any weighting for safety against optimality is possible. As we are interested in CRL (instead of multi-objective RL), we try to select an optimum such that the new policy q is least as safe as the old policy Like Abdolmaleki et al. (2018), we replace the temperature B with a constraint, just that we want to use an improvement bound (af I (a)
Remark 1. Note that our proofs would work, even if we choose different distribution/tikeliiodd function G since our proofs only rely on a safer policy having a higher likelihood. Therefore, any function that is monotonically decreasing for values C((T) < B, will provide feasible safety likelihood. We choose G as a truncated exponential, due to its connection with the reward function used in (Abdolmaleki et al. 2018 Haarnoja el al. 2017. However, this choice has disadvantages, which we will discuss in Section
In the following, we will show that with very limited assumptions, a simple improvement constraint (a) safety(g) is insufficient guarantee improvement, and how can create consis- tently improving policy by explicitly forcing improvements every iteration. First, to formalize the notion of safety (x) safety (g), introduce the notion of a safety order preservation function:
In practice one solves the dual problem which is equivalent due to Equation being convex problem with nonempty interior
4

---

Published as a conference paper at ICLR 2026
Definition (Safety Order Preserving Functions). A function K (p. q) is called safety order preserv- ing if. for all distributions a and all functions Gia, 8) we have
1. R (a. (G(a, 8)] B (a (G(a, 8)] = K(g,T) > 0,
G(aa,
2. E ( (a s)] E (a,s)) (G(a, [)) T(q,T) <
3. E (a G(a, s)] ((a.a)n( (G(a,
Informally, Safety Order Preserving function one that defines a "direction" towards the feasi- ble set. This definition will allow later to make general monotonic improvement guarantees by defining "improvement" in terms of a general (-,,)) function, which may allow individual states to increase cost, while decreasing the cost all, facilitating greater freedom of exploration. We will now give examples for Safety Order Preserving Functions, starting with the naive choice of linear improvement expectation:
Example 3.1 (Linear Safety Function (LSS)).
K(q,T) = / (Gla,) s)dads /
(a, s)G(a, s)dads
safety order preserving. Example 3.2. Priors conditional on safery
f(4)
I s)G(a, s)dads (((a,))(( 8 s)dads =
K(g, x)
0
g(g)
otherwise
for f is positive, is a negative function, are safety order preserving.
Flexibility in the safety order preserving function allows us to c.g. favor safe policies with lower variance, or to favor policies deeper in the interior of the feasible set by choosing suitable f. also allows a "hard" improvement boundary which simply counts the number of safe trajectories by setting f = 1 and g = We will keep a more thorough analysis of good choices for K for future work. For now, the framework of safety order preserving functions allows us make very general statements about how incremental improvement constraints can used in CRL.
Theorem 1. Let Q(a 8) be Q-function, G(a, 8) the cost-likelhoood function, т the current policy, and K (p. q) safety order preserving function. Then the optimal nonparametric policy q solving the improvement equation
max ((a,))n (Q(a, 8)]
(8a)
9
s.t.
KL(g](+) S
(8b)
K(q, T) 20
(8c)
f (gla,s)ddde sdddds
(8d)
either has the same safety as the original n, or ignores and solves the unconstrained MPO
problem (Eq. 3
Proofs can be found in Appendix B This theorem immediately gives rise two corollaries:
Corollary 1. Whether a policy ends up being safe under the incremental model completely depends on whether Qa, s) is higher in safe regions than in unsafe ones.
Corollary 2. Choosing the naive improvement constraint
B(e,s) (G( ((a,))) 2
Ela,s) s)]
for Equation ② is not contraction over the safety likelihood G.
5

---

Published as a conference paper at ICLR 2026
3.2 GUARANTEED IMPROVEMENT E-STEP
The core issue behind the method outlined in Theorem T is that they do not force the model to pick a point in the interior of the feasible set of Eq. 8 This means cannot guarantee zero improvement in safety. To solve this issue, we borrow techniques from interior point methods, namely logarithmic barrier functions.
In interior point methods, the log-barrier function is used to maintain constraints g(r) S 0 by converting them barrier of the form μ =l))(= This ensures that once a point is feasible, will always stay feasible due to the singularity at log(0). The downside for classical optimization that points close to the barrier g(x) 0 are hard reach, which is why barrier methods slowly reduce the barrier weight μ + 0 during optimization approximate the true feasible region. This way interior point methods slowly walk closer towards the barrier, but can never jump over g(x) due to the singularity of the logarithm.
Luckily, as we expressly want points in the interior, we can pick any arbitrary μ and a logarithmic barrier to ensure nonzero improvement in safety constraints. Specifically, with K being sadety-order preserving function, we can formulate our incremental approach as
max (( (Q(a, s)] + wlog
x
(9a)
s.t. (L(g|(=) S
(9b)
K (q. 2x
(9c)
f g(a, s s)dads = 1
(9d)
Assuming E[Q(a, s)] is bounded, this method is guaranteed to improve the safety margin in every iteration by construction and provides geometric convergence towards the feasible set (see theo- rem 4). For any ^ > 0, every with strictly positive improvement x >> 0 will be favored over all other q since for x < 0 we have log(x) -oo Importantly, neither the improvement final optimum when this is iterated depends on the choice of K, only the local per-iteration tradeoff between reward maximization and constraint improvement changes as changes!
the
For example, when choosing K as the LSF in Example 3.1 k + produces a hard projection onto {q) [L] xx) S E, Cg(g) S B,), and for x -> 0 one obtains the MPO update (see also Theorem 3). We simply pick x = 10 for all of our experiments. This value is not tuned for our experiments, which corresponds to the real-world usecase where the optimal (or even good) may not be known.
One can easily solve Eq. 9 by optimizing over the dual variables X, in
/ 1" ((Qs), s)dads is loo À x (f q' (a, (a,a, s)dads- - / T(a, ((a s)dads
((),))
(10a)
(10b)
+ v (e-/ 9" s) log
(a,s))
dads)
(10c)
()
where the optimal q" can be computed in closed form as
1
Q(a,s)) + AG(a,
a' T(a, 8) exp
(11)
V
Note that the optimization problem (Eq. 10 is convex and only has 2 variables To guarantee a
solution to Eq. 10 corresponds to solutions of Eq. 9 one needs a constraint qualification and strong duality. In general, solving the dual problem can be seen favorable for constrained RI., as the dual problem's complexity scales with the number of constraints, rather than the number of state-action pairs in the batch. The latter might be very large
may need lot of samples to accurately track the expected values in Eq. 9 Convex optimization commonly uses Slater's condition:
We make the usual assumption convex optimization that undefined values receive a -oo penalty Boyd,
Stephen & Vandenberghe, Lieven (2004).
More generally, for problem with N constraints, we have + N variables.

---

Published as a conference paper at ICLR 2026
Proposition 1 (Slater's condition). If a convex program with constraints fu...., S has point in the relative interior fo....: fo O, strong duality holds
Slater's condition acts as both a strong duality certificate, and a constraint qualification. Verifying Slater's condition can be somewhat tricky for very general constraint sets such as the ones used for Eq. 10, Fortunately, for Eq. 9 we can give an easier way to check condition for strong duality
Let (AxS, E, p)) be measure space. (f, AxS - R is not almost everywhere constant, ı is fully supported and O, and K is concave, then Eq. has solution and strong duality holds.
Theorem 2.
The proof can be found in Appendix B Using this theorem we can conclude the following:
Corollary 3. Let O 7
be an improvement operator induced by Equation 9 Then O is contraction wrt the pseull-metric K( K(:)).
Remark 2. Theorem 2 still holds for multiple constraints Gi...., Gy if one replaces the condition G(aa is not almost everywhere constant" with "there exists subset A' S' A x S with ju(A' x 5") > O where none of the G; are constant."
This means that, in practice, one only has to ensure that G(a, s) is not constant, and that æ has nonzero probability for all outputs (which is standard assumption both inference and RL), and that whatever K is chosen has to be concave and safety order preserving. For simplicity, we will choose K as the LSF (Example B.I.. p. since, without further assumptions, this can be seen as the canonical choice for K. Choosing different K might further improve performance by e.g. favoring high-entropy solutions within the safer set. Choosing K choice has an interesting consequence: even if G(a,s) s) is constant, we can define reasonable notion of the optimal solution in Eq.
Theorem 3. Assume sufficiently large upper bound M > on the dual variable associated with the improvement constraint. If K is the LSF (see Example 3.1. then solving Eg. 9 with a constrained
dual produces solution Eq. 9 if G is not constant, and solution of Eq. G is constant.
The result in Theorem provides very natural way of implementing the nonparametric sur- rogate: We optimize the dual variables under some sufficiently large constraints (we choose E 1-16; 10%). If the batch are optimizing over is constant (e.g., all (a, 8) are already safe, random sampling). we automatically make an MPO step, while in all other cases we perform a safe improvement step. We present the complete method in Algorithm
Combining all previous theorems, we obtain a very general convergence guarantee
Theorem 4, Assuming Eqa.))~a( ((a. s)] is bounded in [-M, M), and the feasible set is not empty, eq. (9) converges towards the feasible set, and K is sequentially continuous on the nonnegative component (cases 1., 3. for definition 7 and not constant. If we additionally choose K as the LSF (example 3. 3.7, then we obtuin geometric convergence speed.
3.3
M-STEP
After the optimal nonparametric q" is computed, the resulting distribution has to be "cloned" into the policy neural network. Just like MPO (Abdolmalcki et al. 2018), we factor π (a, s) 7 and q(a,s) q(a ())(()) where (u(s) is the stationary distribution estimated using the replay buffer. As in MPO, we use split KL-difference constraint as the prior in our M-step. Therefore our M-step
C(g,aj, 02) E,u(e) (Eyz [log (a)))) a1 Ciu) + azles Cz),
(12)
is the split KL-constraint. We update both Q(a, s) and C'a, s) using RETRACE (Munos et al. 2016 which has empirically good performance and is guaranteed converge under small assumptions.
where Cs = Ep (od) - -
X
and C₁ = (u (M (ota)"" y (H- Pota)]
Dard
Generally speaking, our E-step written to allow for different factorizations, such as the one used in V-MPO (Song al. 2019), which has higher ppr-ue efficiency but comes the cost of not being able to use off-policy data. For the sake of this work, we utilize the traditional MPO estimator, but V-MPO style updates should be plug-in replacement.
7

---

Published as a conference paper at ICLR 2026
Algorithm SafeMPO update step
Require: Policy safety function K, replay buffer D, dual multipliers a1, 02 maximal dual Mx
Ensure: π > 0
sample Batch B DD
Update Q(a, s) using RETRACE on B
Update C(a,s) using RETRACE on B
solve Equation (10) for E [0, MA). v E [0, oc) on B
Compute q" (a(a|s) using Equation 11
for 0 < i <<< N do
4- using Equation
[2 descend ा
Q1 f using Equation
12 ascend on
022 using Equation
12
ascend on dg
end for
3.4 IMPLEMENTATION
Algorithm I shows the prototypical implementation of single-batch update our policy. We first fit both the Q and C function to the sampled trajectories using the RETRACE (Munos et al.] (016) estimator. We can then solve the dual problem Equation (TO) and compute the nonparametric refer- ence q- (a) which we use the target for our neural network. For the experiments in Section 4 we sample 20 batches of size 1024 and set N 8.
4 EXPERIMENTS
We compare against state-of-the-art algorithms implemented in Omnisafe Ji et al. (2024). For all experiments, we choose x 10 and budget of B = 25. Our benchmarks are performed three tasks from the car in randomly generated 3d environment towards "goal" position while avoiding stationary hazards. Once a goal has been reached, a new goal is sampled. The reward is defined as the distance reduction towards the goal. The cost is +1 when obstacle is hit, otherwise. SafetyPointGoal is the same task as the previous, but instead of navigating differential drive car, the point agent directly decomposes rotation and movement direction, making control much easier. The agent is also physically smaller, further improving the navigation performance. Safety CarButton is the most challenging environment: Instead of driving the car towards the goal and having to avoid stationary obstacles, the car has to avoid both stationary and moving obstacles, while navigating towards the correct goal button. Rewards and costs are defined the same in the Goal environments.
(Ji et al. 2023) benchmark set: SafetyCarGoal aims to drive
Since CRL has two, often diametrically opposed objectives (constraint satisfaction and reward max- imization), it is often difficult compare whether one algorithm outperforms another. Further, one considerable issue when comparing different algorithms are the hypeparameters: This is of course limitation for all MI. algorithms, but in CRL we have the additional problem that the optimal solu- tion to the CRL problem provably relies only the Lagrangian parameter (Paternain et al. 2019). Therefore even a naive algorithm with fixed dual parameter could reach state-of-the-an when that parameter is known. This complicates benchmarking since over known benchmark environments practitioners know appropriate values for the dual, in contrast to real world problems where such ranges unknown. This holds even for some dynamic methods, such 2020) which implicitly has environment information inside their PID-values.
(PPOPII""(Stoke al.
To give existing methods the best possible opportunities, utilize the highly optimized hyperpa- rameters provided by Omnisafe (Ji et al. 2024). We would, however, argue that the fact that we only have single hyperparameter which does change the asymptotic behavior is significant advantage of our method that is hard to show in benchmarks.
When analyzing the results in Table we observe that our method is competitive reaching state- of-the-an art results in both the "SafetyCarGoal" and "SafetyCarButton" environments. Specifically,
Note: follow the nomenclature adopted by Omnisafe Ji et al. 2024 and call the combination of PPO (Schulman et al. 2017) and PTD presented in (Stooke al. 2020) "CPPOPID".
8

---

Published as a conference paper at ICLR 2026
Table 1: Safety Car Goal. All methods are evaluated after 10 Mio steps. Following (Ji we set the cost budget B 25 for all algorithms unless otherwise mentioned.
2024
SaferyCarGool
SafetyCarBution
SafetyPointGooal
Method
Reward
Cost
RCPO Tessler al.
2018
18.71 +: 2.72
23.10
12.57
2.04 + 2.98
43.48 + 31.52
15.27 + 4.05
30.56 4 19.15
PCPO Narasmban
2020
21.56
2.87
38.42
8.36
0.36
0.85
40.52
1: 21.25
18.37 1: 1.71
22.98
6.56
FOCOPS
2120
15.23
10.76
31.66
93.51
0.21
2.27
31.78
47.03
17.97 4 9.01
33.72
42.24
Achiam el al. 2017
25.52
2.65
43.32
14.35
0.82
1.60
37.86
27.41
20.46
1.38
28.84
7.76
CPPOPID Slonke
2020
10.00
2.51
30.66
7.53
-1.36 0.68
14.62
9.40
8.43 -.43-3.43
25.74
7.883
CUP Yang el 2022
6.14 6.97
36.12
89.56
1.49 2.84
103.24
123.12
14.42
6.74
19.02 1: 20.05
Mafe
21.43 +- 5.10
32.23
7.43
0.67
0.61
30.87
4.47
13.09 + 2.79
32.92
+ 3.43
SafeMPO 8 20 (ours)
20.24
2.24
26.08
+ 2.97
0.94
0.12
33.37
2.46
18.36 3.41
23.00
1.56
400
9CP0
150
TRAOLOg
200
100
300
75
200
50-
100
25-
returss
20
10
10
0-
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
step
step
step
(a) SafetyCarGoal
(b) SafetyCarButton
(c) SafetyPointGoal
Figure 2: Results throughout training. We adopt the same evaluation procedures as described in Ji et al. 2024). The shaded area corresponds to l deviations across independent runs. To maintain readability, we only plot a subset of the algorithms we compare against in Table
our agent manages reach higher reward than CUP Yang al. 2022) and CPPOPID (Paternain et al., (201)), while producing substantially lower cost. The only method that produces policy below the cost threshhold of 25 RCPO (Tessler al.) 2018), which also substantially drops in observed reward compared to SafeMPO. The major downside of RCPO compared to our method becomes apparent the more challenging SafetyCarButton environment. SafeMPO produces a slightly lower constraint violation than FOCOPS (Zhang et al. 2020) while providing almost 3x more reward. The only method to stay fully within the constraint set (CPPOPID) does so at the loss of performance, producing highly negative reward. RCPO the best model from SafetyCarGoal produces the worst results on SafetyCarButton: RCPO has the lowest reward and the second highest cost across all methods.
The pattern of inconsistent performance continues to SafetyPoinGoal the easiest of the en- vironments here find the best performing model be CPO (Achiam et al., 2017) PCPO (Narasimhan) (020), depending on how one values + cost overrun/undershoon against reward gain of points. This is the worst performing benchmark for SafeMPO, but we remain close to SOTA. Generally, our method has the advantage of behaving consistently, providing low run-to-run variance across independent runs.
Another interesting observation is the robustness provided by our method regarding costs: Our method consistently has the lowest run-to-run variance among all other approaches, which we hy- pothesize is due to the lack of "edge cases" introduced by other methods.
DISCUSSION
One interesting point to mention is that all methods have cost of roughly 30 compared to the cost limit of 25. This suggests that this is a limitation of how our likelihood function for safety is defined: since we only penalize policies outside the feasible set, the solution Eq. 9 might end

---

Published as a conference paper at ICLR 2026
up simply "eaatig" the cost for states that are almost safe already, leading effective shift in boundary due to sampling variance inside the approximations of the expectation. To validate our hypothesis, we re-run the SafetyCarGoal experiments with artifically lowered cost threshhold of The results can be found in table T As one can see, both SafetyPointGiall and Safe- tyCarGoal return feasible policies SOTA reward levels (for feasible policies). Nevertheless, our method fails to find a feasible policy for SafetyCarButton, which is not unexpected considering that no method aside from CPPOPID can produce a feasible policy
B == -0.
66 Of course, tuning the boundary B is not a feasible solution but a smaller barrier leading to superior results suggests that our method is able to theoretically hit better threshholds, but is unable to reach the points due the log- liglikklihoodd G' we chose. Our method can technically handle arbitrary notions of safety, including e.g. Condi- tional Value at Risk (CVaR). Since this problem goes deeper than SafeMPO and touches the question of "What definition of safety is appropriate for Reinforcement Learning", we leave the analysis of alternative safety likelihood functions for future work.
One important aspect to note is the general behavior of Reinforcement Learning in Constrained sce- narios and the "RL as inference" notion employed this paper. is generally impossible for RL agents regardless of Value, policy, model based to be 100% safe in unknown environments. This is due to the inherent uncertainty when estimating the constraints: To choose an extreme exam- ple, one cannot know the safety of states that are never sampled. This means any well designed CRL method should have the property that action probabilities are nonzero (but perhaps very low) during finite exploration, and only converge zero in the limit of infinite data. Otherwise, one cannot guarantee that an action is "banned" just due to poor initialization of the cost function or missing data. Indeed, this is the behavior we observe in our proof of Theorem High cost states decrease in likelihood but only hit zero in the limit of infinite steps where one has certainty on the cost function.
6 CONCLUSIONS AND FUTURE WORK
We propose a novel framework for Constrained Reinforcement Learning that relies on relative im- provements in the likelihood of policy being safe, rather than primal-dual projection based updates. We first analyze these types of relative improvement algorithms theoretically, proving that naive improvement bounds may not lead to contraction towards a feasible set. However, we prove that with small changes inspired by interior point barriers, we can make broad statements of the con- vergence properties under relative improvement constraints. Finally, we demonstrate our method's high performance by comparing against highly tuned state-of-the-an algorithms.
Our method leaves wide room for future work both on novel Safety Order Preserving Functions, and algorithmic improvements. For instance, we currently only consider the off-policy MPO algo- rithm (Abdolmaleki et al., 2018), but our framework is also compatible with other updates, such as (Sogg et al. 2019). As already mentioned in Section 4 we also noticed opportunities to improve our method further, by choosing better likelihood function. Another interesting possibility is framing ordinary MDP RL. problems as CMDPs where the objective is given as improvement constraint. This way one could design RL methods that by construction guarantce improvement.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenherg, Yuval Tassa, Rémi Munos, Nicolas Man- fred Outo Heess, and Martin A. Riedmiller. Maximum a posteriori policy optimisa- tion. ArXiv, 100000000 2018. ott.tttss//ppp.emmatcccccoooaaa www.smmmmcccsccolaa...oo CorpusID:4931075:
Joshua Achiam, David Held, Aviv Tamar, and P. Abbeel. Constrained policy optimiza- tion. ArXiv, abs/1705..15520 2017. URL https: wtw..i..eetttttccolslsss..a org/
CorpusIDe ID: 10647707
Boyd, Stephen and Vandenberghe, Lieven. Conver Optimization. Cambridge University Press,
March 2004. ISBN 1804441. doi: 141017//hh9788511884488 URL http://dx. doi.
10. 10.7/1111/BBB00 11110//...1007181777111111000110
'In fact, inspecting some of the trajectories created by CPPOPID, it turns out that CPPOPID learns to walk outside the area which the simulation takes place in, yielding low reward but high safety.
10

---

Published as a conference paper at ICLR 2026
Tuomas Haarnoja, Haoran Tang, P. Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, 2017. URL https: //api. :.api.semannisschoo..... org wwwpooooowsi::1112.....
Edwin T. Jaynes. Information theory and statistical mechanics. Physical Review, 106:620 630,, wtt.pccho.......oooooouuauaaaa..
1957.
Jiaming Ji, Borong Zhang, Xuchai Pan,
Jiayi Zhou, Juntao Dai, and Yaodong Yang.
Syfety-GGymmsiumm May 2023. URL Itttps://githus: http://gtttuu.c..oeeec
safety-gymnasium
Jiaming Ji. Jiayi Zhou, Borong Zhang, Juntao Dai, Xuchai Pan, Ruiyang Sun. Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. Journal of Machine Learning Research, 25(285):1-6, 2024.
Jean Bernard Lasserre. Moments, Positive Polynomials and Their Applications. IMPERIAL COL- LEGE PRESS, 2009. doi: 10.1 142/p665. URL httph::t/////..... htwp::::/wwwwww.. wor http://wwwwwll.ii........ ffc.comm
http//ab/....112/////
Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Zhiwei Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. ArXiv,
abs/2201.119277, 2022. URL httpp::/ap..semmatttt..ee http://aa..seaanccccccooooor.aes httpechhooaa..oo/oorsosslllla 246411597
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare G. Bellemare. Safe and efficient off- policy reinforcement learning. Neural Information Processing Systems, 2016. URL https:
7/api. www.semmannisccieeee wttphoolaarrroooooussssssasss :49
Karthik Narasimhan. Projection-based constrained policy optimization. ArXiv, abs/2010.03152, 2020.
Radford M. Neal and Geoffrey E. Hinton. A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants, pp. 355-368. Springer Netherlands, Dordrecht, 1998. ISBN 978-
94-011-5014-9 doi: 10.1007/978 0000000997799-01------11 178-54-011-501422
94-011-500149.11 URL https:://ei.org////////
Santiago Paternain, Luiz F. O. Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained reinforcement learning has zero duality gap. In Neural Information Processing Systems, 2019. URL htt.hhttp://appp semant i csscolarr http://orpps111:2444222222
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proxi- mal policy optimization algorithms. ArXiv, abs//7777.6347,, 2017. URL https://apt:
semanticscholes wtt.:/cccccoaaa..oooooooooooee wttgg//oousslai 286
H. Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W. Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, Nicolas Manfred Otto Heess, Dan Belov, Martin A. Riedmiller, and Matthew M. Botvinick. V-mpo: On-policy maximum a posteri- ori policy optimization for discrete and continuous control. ArXiv, abs/1909, 12238, 2019. URL httpps://apl semant innntiiscooe cho 1 ar www.oorg/oooooo TD:
Adam Stooke, Joshua Achiam, and P. Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, 2020. URL https:
//api. semant www.hccccaacooo. hrtphoolll...rr/ooootttt::: 2200P000TD:2044228888
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Chen Tessler, Daniel Jaymin Mankowitz, and Shie Mannor. Reward constrained policy opti- mization. ArXiv, abs/1805.1 11074, 2018. URL pttttp:://api.eeee hstts//ppp..eeaaatccccc......... orrg/
EorpusID:4409553
Lin Xiao. On the convergence rates of policy gradient methods. ArXiv, abw/a/2/./2000443,. 2022. URL https://api semanticscho ar. http

---

Published as a conference paper at ICLR 2026
Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning, 2020.
URL httph://appi.. sew.iiissmmmmcmllllal.... ott.oaaattto/aaaaoooaoooooooooor org/ wttp:a/oooo..oo....22333300.
Long Yang, Jiaming Ji, Juntao Dai, Linrui Zhang, Binbin Zhou, Pengfei Li, Yaodong Yang, and Gang Pan. Constrained update projection approach to safe policy optimization. ArXiv, abs/2209.07089, 252280735
2022. URL https://app
Yiming Zhang, Quan Ho Vuong, and Keith W. Ross. First order optimization in policy space for constrained deep reinforcement learning. ArXiv, abs/2002.06506, 2020. URL https://aapt
A
LLM USAGE
We used LLMs improve the writing and presentation of the work, well aid in translation. We manually ensured that any alteration in wording did not change the meaning of the work.
B PROOFS
Proof for Theorem 7
Constructing the Lagrangian yields
+ x) (1-1) g(a, s)dads
C(g, 11, X, v) = ((a,))) s)] r LL(g]l=) + AK n)
(13)
for n, A 2 Solving for the optimal q (after absorbing the normalisation constraint) yields
Q(a, s) +
(K(q.x)
8g(a,s)
(a, s) z x(a s) exp
(14)
with Z = ((, 8) exp (aa,)))
4k(g. Agia
dads and
6K(g.)) the functional derivative wrt g. ōg(o,s)
Notice that for the of 1). A. Specifically we know that either ((q. T) 0 or A = 0.
conditions hold, we also need complementary slackness
In the former case, we have that the solution q is no safer than the solution क, be definition. In the latter case, the solution reduces to
(a z T(a, 8) exp Q(a,s))
(15)
which is the optimal solution to the unconstrained model.
L
Proof for theorem] 2 G(a, s) is not almost everywhere constant, then this means that there exists
pair of disjoint sets X, X2 with 0 < p(X1) (xX2) such that for all ((a,, 82), (aj, 8;))
X1 x X₂ we have G(a;, 8:) > G(aj, 85).
Choose g such that q(a,, 8;) E1 n na, s;) and g(aj, m(aj; 8,). For any
0 < < min s(),, 1 2\ (())XX))
s
Keep all other g(az, sk) < 8k) Vkk # i
The Kl--constraint is upheld due to Pinsker's inequality, i.e for any g. ग we have
2Dvv(g. L(g][=) S..
Our distributions only differ on the set X1 U X2. meaning the total variational distance is less than when choose E < 1 2 p(X(Ux)))
12

---

Published as a conference paper at ICLR 2026
Trivially, the equality constraint f q(,, s)dads is valid (E1 deviations cancel over pairs). The improvement constraint is nonzero:
JAnS s)dads
=/x T(a,s)G(a, ()d
*fx T(a, ()a,s) s)dads JAASSICKUUX) T(a, s)G(a, s)dads X2
I(a, s)dads
I (unSpicx.ux) ((a,))((a, s)dads (T(a, E1)G(a, s)dads
T(a, ()
X2
X1
(5(a, (la,s))siil =i)G (alaassddie s)dads J J(AKS)NCXIUNS) g(a, s)G s)dads
X2
q(a, s)G(a, s)dads
X1
g(o, s)G((, s)dads SAANSINONION
g(a, s)G((, s)dads
X2
- A a(a,s) SGG, s)dads
where we use the fact that values G(a, 8) in X, are strictly better than values G(a, 8) in X2, and the fact that the sets X1 and X2 are not of measure zero to ensure strict inequality, Since K preserves the strict inequality, _AXS T (a, s) S(, s)dads < SAxx a s)G(a, s)dads = K K(g, x) > 0.
Therefore there exists a with nonzero x, which means log(z) is finite. which means feasible interior point exists. The problem also convex as both the KI. constraint and the constraint are convex (assumption). This also directly implies Slater's condition, which implies strong duality. L
Proof for Theorem
Consider the Lagrangian
/ g(a, s)Q(a, s)dads + log
Elg,z, A,
(ʃ ((a,)))( / T I(a,s)G(a s s)dads x)
v (E-) a(a, s) log( g(a, s)
Idads) + μ (f (qla,s)dads -
T(a, s)
The optimal is given by
(16)
while the optimal (after absorbing the normalization constant) is given by
1 z T(a, 8) exp (((a,)) 8) + AG(a, 8)
a(a, s) =
(17)
is not constant eq. 99) has a solution, which can be found by optimizing over the dual variables X, v. This follows directly from theorem
13

---

Published as a conference paper at ICLR 2026
If G is constant, but À is bounded, the produced by optimizing eq. (
have any finite X.
However, since G constant \G s) can be absorbed into the normalizer
1 q(a, ((a, 8) exp
Q(a, s) + AG((,
Z
V
1
Q(a,
(GG(,
π (a, exp
exp
2
1
((a,s)
* (a, 8) exp
Z2
which is the solution to MPO
C
CONVERGENCE
Proof of theorem f
To simplify the notation, we will define P(q) ((a.s)~ (Q(a, s]].
We will show that the sequence i yo, (r,... defined by recursively applying eq. contraction towards the feasible set. First, consider the slightly simplified problem
leads to
K (4(+1.97)
4 argmax P(gs+1) + log
(18)
Note that this corresponds to eq. 99) without the KL-divergence constraint.
Let's pick some reference point of within the feasible set. This acts as a lower bound for the maxi- mization in eq. 18). By definition, we have that for
P(1[+1) + K log K(
((G++1.0)
P(d(+1) + K log
K
rearranging yields
K ((9 > (P(d+1) P(u[+1]] + x log
K (((4+1995))
log
K
Due to boundedness of P(g.+1) < M and finiteness of P(g.+1) we get P(g(+1) P(qi+1) 2 P(u(+1) M =: -it1
This give us the rate
K ((( 9i)
-i+11 + +og
(K((++,.9))
K [((6+1.9]*
-iit1
log
log
exp
K
K
K
R
K
which yields the inequality
Ait1
K (((+114))) 2 K ((++1:9)) exp
K
One possible choice for q' is any feasible point within the original CMDP (a, (C(a, s B. Since this is feasible in every iteration, we get a fixed constant = exp( - P(1))/K) e (0, such that
K K( 2 (K(g'
which is valid for all iterations. Therefore, have real decrease K (9 q ) in every iteration. By definition of Equation (18), we have lower bound on K, which means converges.
We now have to show that it converges to safe in the limit. For this consider
K K(q x 2 cK(g.. q.) 0
Assume that converge unsafe q, then have fixed point where q 4. =4441 via sequential continuity we have
K K(g.g) OΛ cK cK >
def. K
improvement
14

---

Published as a conference paper at ICLR 2026
Figure 3: A comparison between different K values.
which is a contradiction. Therefore, we converge to a safe point the limit.
Adding the KL-divergence bound complicates analysis, but the idea is the same. Picking strictly feasible q wrt the original CMDP constraints is not enough, since that might be outside the KL-ball. The fix is simple: Let
4 = projKz( (e))
be the projection of the selected point onto the KL ball. Assuming 0 can perform exactly the same proof as above, just that instead of having fixed rate (0, 1]. we have
M P((+1)) E (0, 1] (-M-
C exp
This still gives us an (iteration dependent) lower bound
K K( 2 cK(g',
where we can apply the same argument for convergence.
Now assume K is the LSF. Let's call Cmin inf we get geometric convergence via standard squeeze argument:
K K(g+t1.9) 2 Cmin K (4',
Meaning we have
K( 4.) 2 Cmin ((''
for every step. Since the LSF telescopes K ba, K(b,c) c) -E.(G]-E,[GG + E..G] K (a, c). we have
((d1))+1)) K(d (((t)199)= S K(d. 96) Cmin K(d's (1 Cminn (4',
By induction we get
K(d' 9h) (1 Cmin " "(q', (0 (%)
which proves the geometric rate.
Remark 3. Note that other choices for K may equally lead geometric convergence rate, but for the sake of conciseness we only cover the convergence of the LSF used in our experiments.
D ABLATION OF HYPERPARAMETERS
We ablation the impact of on the overall performance. We generally observed similar performance between the different values across the entire interval tested.
We found statistically significant difference in changing e slightly (evaluated across 10 runs).
15

---

Published as a conference paper at ICLR 2026
E VALUE AND COST ESTIMATION
We use the RETRACE (Munos et al, 2016) algorithm to estimate the returns of Q" and C RE- TRACE works by computing importance-weighter updates of the Q function(s) over a replay buffer. Specifically
Qnew (s, = Q(s, + Bp Σ y I + + 1Br
120
where yu is arbitrary behaviour policy (e.g. previous version of the policy) and is the current policy. RETRACE can be implemented analogously for C(s,a) RETRACE sets
C2 = min(1, {(as(81)
pla: (s))
RETRACE, is designed to give high quality estimates of the current policy by exploiting the similar- ity of previous policies to get lower variance estimates.
F HYPERPARAMETERS
We use nearly identical network layout for policy, cost, and Q-function with MLP layers with leakyReLU activations, followed by either a mean and covariance prediction layer (for the policy) or value prediction head for cost/value networks. We use width of 256 for all latent dimensions.
For the policy network we parameterize the mean and the cholesky factors of the covariance matrix.
We use 10 and a standard 0.1
G ENVIRONMENTS
Safety Gymnasium uses "composable environment" format where every environment contains an "Agent" (in our case "Point" and "Car") and a task (in our case "Goal" and "Button").
The "Goal" tasks give as input 48 dimensional state which corresponds to the relative distance 64 dimensional state which corre-
towards obstacles and the goal. The "Button" task consists sponds to the distance towards the goal, the buttons, the static obstacles, and the moving obstacles.
In addition to the task-intrinsic state, every agent also adds its own state, which for point an ac- celerometer, velocimeter, gyroscope, and magnetometer (12 dimensions), and for the are Quater- nions the rear wheel which are turned into 3x3 rotation matrices, Angle velocity the rear wheel, accelerometer, velocimeter, gyroscope, magnetometer (24 dimensions).
Both "Car" and "Point" supply a dimensional action space corresponding to a differential drive platform.
16


Reviews:
[{"review_id": "t5QQFxlw30", "reviewer": "Reviewer_y7UB", "summary": "This paper introduces a constrained policy optimization algorithm based on Maximum-a-Posteriori Policy Optimization (MPO), providing theoretical guarantees for convergence to the feasible set of the constrained problem. The approach extends MPO’s control-as-inference formulation by adding a “policy is safe” event alongside the “policy achieves goal” event. The authors argue this improves conditioning, as safety can be improved even outside feasibility. Experiments in Safety-Gymnasium (Ji et al., 2023) show partial empirical gains.", "questions": "- Why can the most likely posterior policy not be non-feasible?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel idea of redefining MPO using a safety probability event  \n- Engaging and well-motivated introduction  \n- Theoretical results appear sound (not fully verified)", "weaknesses": "- Experimental results are not fully convincing  \n- Expected stronger sample efficiency gains since MPO is off-policy  \n- Theoretical clarity: Corollary 3 seems central but is not well explained  \n- Mathematical presentation issues: $ C(a,s) $ is estimated but never defined or used; unclear whether this refers to $ G(s,a) $ or $ K(s,a) $. Clarify how these are estimated via Retrace.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dViTEkDZzJ", "reviewer": "Reviewer_3C3X", "summary": "This paper proposes SafeMPO, a novel algorithm for Constrained Reinforcement Learning (CRL) that extends Maximum a Posteriori Policy Optimization  to the CMDP setting. Instead of using primal–dual updates or projection-based recovery, SafeMPO enforces *incremental safety improvement* via a *log-barrier constraint on a safety-order-preserving function \\(K\\)*. The method yields a closed-form E-step distribution \\(q^*(a|s)\\) and theoretical guarantees that the iterates contract toward the feasible set under mild assumptions. Empirical results on **Safety-Gymnasium** tasks demonstrate competitive or superior reward-cost trade-offs compared to strong baselines like CPO, PCPO, RCPO, and CPPOPID, with notably low variance across runs.", "questions": "Please address the points I raised in the Weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper offers a novel probabilistic viewpoint on safe RL: treating safety improvement as a likelihood maximization problem rather than a constraint projection.  \n - The barrier-regularized E-step and its EM-style update are well derived and integrate naturally with existing MPO framework.\n- Theoretical contributions are well-structured. The paper first proves that naive improvement bounds do not necessarily contract toward feasibility, then introduces a log-barrier to guarantee convergence (Theorems 2–4). I did not check all the theoretical results but they look correct and rigorous \n- The algorithm looks simple.\n- Experiments across three standard safety environments show SafeMPO achieving near-SOTA returns with moderate constraint violations.", "weaknesses": "- **Experimental scope is limited.**  \n  Only three tasks from the Safety-Gym benchmark are evaluated, which is insufficient to demonstrate the generality of the approach. The study would be strengthened by including a broader range of tasks to better assess scalability and robustness.  \n\n- **Ablation studies are underdeveloped.**  \n  The authors should conduct sensitivity analyses on key hyperparameters—specifically the step-size parameter \\(\\kappa\\), the KL divergence radius \\(\\varepsilon\\), and the maximal dual variable \\(M_\\lambda\\)—since the theoretical convergence guarantees explicitly depend on these quantities.  \n\n- **No exploration of alternative safety likelihoods.**  \n  Although the paper acknowledges that other likelihood formulations beyond the truncated exponential are theoretically possible, no experiments evaluate such alternatives. Empirical validation of different safety likelihoods would be necessary to justify the exclusive use of the truncated exponential form.  \n\n- **Baselines are outdated.**  \n  The experimental comparison omits several recent state-of-the-art constrained RL algorithms, such as **Constrained Variational Policy Optimization (CVPO)** [1] and imitation-based constrained methods [2]. Including these baselines is essential for establishing the competitiveness of the proposed approach.  \n\n**Refs**\n\n- [1] Liu, Z., Cen, Z., Isenbaev, V., Liu, W., Wu, S., Li, B., & Zhao, D. (2022). *Constrained Variational Policy Optimization for Safe Reinforcement Learning*. In *Proceedings of the 39th International Conference on Machine Learning (ICML 2022)*, PMLR 162, 13644-13668. [https://proceedings.mlr.press/v162/liu22b.html](https://proceedings.mlr.press/v162/liu22b.html)\n\n - [2] Hoang, H., Mai, T., & Varakantham, P. (2024). Imitate the Good and Avoid the Bad: An Incremental Approach to Safe Reinforcement Learning. In Proceedings of the 38th Annual AAAI Conference on Artificial Intelligence (Vol. 38, No. 11, pp. 12439-12447). AAAI.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TyFdji4eq4", "reviewer": "Reviewer_MFV7", "summary": "This paper presents SafeMPO, a constrained reinforcement learning algorithm derived from the maximum a posteriori policy optimization (MPO) framework. The method aims to guarantee asymptotic safety constraint satisfaction with contiguous policy improvement. The authors formulate an expecation maximiation algorithm with a constrained optimization problem in the E-step (Theorem 1), introduces a modified policy update that includes a safety barrier (Eq. 8a), and provides theoretical results relating the algorithms contraction and monotonicity properties to safety performance. Empirically, SafeMPO is evaluated on standard continuous-control safety benchmarks (SafetyGym) and compared to existing constrained policy optimization baselines.", "questions": "- Could you clarify in why the optimal solution to Theorem 1 does not guarantee safety improvement? Does this follow from a characterization through KKT conditions? If so, do these condition allow a general statement as done about the global optimal solutions? \n- How does the behavior of the barrier function in Eq. (8a) differ from a standard Lagrange or penalty term? Why is the non-zero improvement of constraint satisfaction permitted in the primary constraints in Eq. (8a) if it appears to be the authors wish to ensure non-zero improvement? \n- In what sense is the E-step mapping a \"contraction\"? What operator or function is being iterated, and in which norm?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- I find the idea interesting and novel. Combining MPO-style policy updates with constrained optimization is a natural direction that should provide some interesting insights. The attempt to incorporate constraint satisfaction directly into the policy updates of an MPO-style algorithm is conceptually nice.\n- The derivation of the constrained optimization E-step are motivated optimization. The authors include a variety of additional theoretical results (contraction and convergence) to their algorithmic contribution. \n- The motivation for combining the probabilistic policy optimization framework with constrained objectives is clear, and the paper structure (E-step / M-step / theory / experiments) is easy to follow.\n- The paper is overall well-written.", "weaknesses": "**Conceptual and theoretical clarity**:\n\n- I found the E-step section, particularly around Theorem 1, quite hard to follow. It is not clear to me why an optimal solution to Theorem 1 would not guarantee a safety improvement, as constraint (7c) seems to directly enforce this. Any solution violating safety improvement should be infeasible under that constraint, so the subsequent introduction of the barrier function in Eq. (8a) is hard to comprehend from my position. Going to the appendix reveals that the authors likely mean that the first order KKT conditions permit points that are not feasible, but these conditions in my understanding are not a characterization of global optima. Either way I believe this should be explained more explicitly, as it lays the foundation for the motivation of subsequent algorithmic components. \n- The barrier function added as an additional regularizer in (8a) makes the algorithm less \"pleasing\" to me. If I am correct, the introduced term behaves similarly to a Lagrange multiplier or penalty term, which raises the question of whether the same problem is being solved twice - once in the primal constraints and again in the added regularizer. In general I find it slighly confusing that the authors seem to pursue a \"a nonzero improvement in our safety constraints\" but do not formalize their primary constraints as such, instead permitting zero-improvement (7c). \n- The authors’ usage of the term \"contraction mapping\" (Corollary 2 and 3) confuses me a bit. Contractions are properties of operators, yet the text seems to apply the notion to entire inequalities. It is a bit unclear to me what mapping is claimed to be contractive, and with respect to which norm or metric.\n\n**Conceptual tension with the control-as-inference view**:\n\nThis is a discussion on a more philisophical level, but I believe there is a tension between soft control formulations (where policies emerge from defining optimality from a lens of energy-based exponential distributions) and hard safety constraints. The paper incorporates constraints onto a probabilistic inference objective, but what does the introduction of hard constraints imply for the soft nature of the probabilistic treatment of control? In the classical control-as-inference setting, maximum entropy solutions naturally emerge as solutions to the probabilistic problem, which appear to be incompatible with hard constraints if left untreated. This conceptual issue perhaps deserves some discussion.\n\n**Experiments**:\n\nThe experiments look executed well but their results look rather inconclusive to me. The differences between SafeMPO and existing constrained methods are minor across most benchmarks, and it is unclear in which scenarios SafeMPO’s particular constraint handling makes a significant difference. The authors should better articulate where SafeMPO is expected to excel (e.g., in nonconvex or highly stochastic safety settings) and provide corresponding experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EvyfsDDh4i", "reviewer": "Reviewer_4cCG", "summary": "This paper presents a novel approach to constrained reinforcement learning, referred to as SafeMPO: Instead of relying on projection based approaches, which project the policy onto the constraint-satisfying set at every step, SafeMPO relies on incremental improvements by ensuring that each policy update monotonically increases the likelihood of safety, rather than enforcing hard constraint satisfaction at every step.\n\nThe motivation for this is that projection based approaches can be overly restrictive, effectively preventing the discovery of optimal, constraint satisfying solutions by enforcing strict constraint satisfaction at every step.\n\nThe paper builds on MPO and the EM algorithm, but extends MPO to handle additional safety constraints. In the E step, the method computes the improved, nonparametric distribution (in terms of return **and** constraint violation) $q^\\ast$ over states and actions in closed form, subject to a KL constraint for $q^\\ast$ to remain close to the current policy. Then, in the M step, the policy is fitted to the improved distribution $q^\\ast$.\n\nThe paper provides theoretical guarantees showing that the incremental improvement process forms a contraction toward the feasible set, yielding geometric convergence under mild assumptions. \nSafeMPO is evaluated empirically on three environments from the OmniSafe benchmark and compared with a number of appropriate baselines.", "questions": "1. In equation 8a, the term $\\kappa\\ \\log (\\frac{x}{\\kappa})$ seems to be constant in (s,a). How does it affect maximization over $Q(s,a)$?\n\n2. Why do all almost all of your baselines fail to find feasible solutions, even on the simplest environment? Might it be that there is insufficient capacity in the estimators for the cost function or policy networks?\n\n3. Why does the performance of SafeMPO in terms of return increase on 2/3 environments when a stricter cost coefficient is use?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The proposed incremental approach to constrained reinforcement learning is, to the best of my knowledge, novel and well-motivated. Rather than enforcing strict constraint satisfaction at every update via projection of the policy onto the feasible set, the method ensures that each policy is at least as safe as the previous one. In settings where incremental progress toward the feasible set is acceptable, this formulation is intuitively less restrictive and may enable the discovery of higher-return, constraint-satisfying solutions that projection-based methods could fail to discover.\n\nThe proposed SafeMPO method is principled, and the paper's theoretical analysis provides guarantees for a practical constrained RL algorithm, including monotonic safety improvement and geometric convergence toward the feasible set under mild assumptions.\n\nFinally, by leveraging a nonparametric surrogate optimization in the E-step and a supervised M-step, the method inherits the stability and low-variance optimization behavior of MPO-style algorithms while extending them naturally to constrained settings.", "weaknesses": "The proposed approach is principled and I very much agree with the motivation and would like to see (an improved version of) this work published. However, the following points strongly impact my score:\n\n**(1) Clarity and exposition**\nWhile the paper uses typical notation from variational inference, the presentation of key background material should be improved substantially. The paper assumes strong familiarity with MPO and the EM algorithm, but neither is introduced sufficiently, making the paper not self-contained.\n\nIn particular, the introduction of MPO (lines 120–124) is confusing and somewhat misleading. What exactly is \n$q$ here? Unlike what is stated, and if I am not mistaken, $q$ is not an approximation of the RL objective, but rather an auxiliary, implicit distribution over states and actions used to optimize it.\n\nNitpick: There are also a few other statements that lack clarity, IMO. For example, the feasible set is never properly defined. The $\\epsilon$-ball on line 67 is not defined. Referring to “the neural network” as part of SafeMPO is problematic, since there are multiple neural networks in RETRACE alone (RETRACE is also mentioned only briefly and not explained in a self-contained manner).\n\n**(2) Experimental validation**\nThe empirical results are limited and somewhat contradictory. In all reported environments, the standard SafeMPO variant fails to find a policy satisfying the cost constraint (B=25), as the asymptotic costs after 10M steps remain above this threshold. The found policies are therefor **not** part of the feasible set. The chosen environments are relatively simple navigation tasks, making this result somewhat concerning. Lowering the cost bound (to B=20) to achieve feasibility (on 2/3 environments) is not an adequate solution, as it effectively alters the optimization objective in a way similar to tuning the relative weights of reward and cost. Thus, the experiments do not convincingly demonstrate that the proposed algorithm converges to constraint-satisfying solutions in practice.\n\n**(3) Disconnect between motivation and findings**\nThe authors argue that the incremental approach in SafeMPO is less restrictive and can therefore discover solutions that are hard to find using projection-based methods, as nicely illustrated in the didactic example in Figure 1. It would strengthen the paper’s claims considerably if the authors demonstrated an environment where such exploration challenges due to constraints exist, and where SafeMPO successfully overcomes them.\n\n**(4) Lack of environment and implementational details**\nThe environments are not sufficiently explained. The state and action spaces are not stated, making it hard to know how easy or hard these environments truly are. Similarly, the hyperparameters of SafeMPO aren't stated, limiting the interpretability and reproducibility of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)