Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?
Authors: Hannah Markgraf
Shambhuraj Sawant
Hanna Krasowski
Lukas Schäfer
Sebastien Gros
Matthias Althoff
Venue: Accepted by TMLR ()

Abstract: Projection-based safety filters, which modify unsafe actions by mapping them to the closest safe alternative, are widely used to enforce safety constraints in reinforcement learning (RL). Two integration strategies are commonly considered: Safe environment RL (SE-RL), where the safeguard is treated as part of the environment, and safe policy RL (SP-RL), where it is embedded within the policy through differentiable optimization layers. Despite their practical relevance in safety-critical settings, a formal understanding of their differences is lacking.
In this work, we present a theoretical comparison of SE-RL and SP-RL. We identify a key distinction in how each approach is affected by action aliasing, a phenomenon in which multiple unsafe actions are projected to the same safe action, causing information loss in the policy gradients. In SE-RL, this effect is implicitly approximated by the critic, while in SP-RL, it manifests directly as rank-deficient Jacobians during backpropagation through the safeguard.
Our contributions are threefold: (i) a unified formalization of SE-RL and SP-RL in the context of actor-critic algorithms, (ii) a theoretical analysis of their respective policy gradient estimates, highlighting the role of action aliasing, and (iii) a comparative study of mitigation strategies, including a novel penalty-based improvement for SP-RL that aligns with established SE-RL practices. Empirical results support our theoretical predictions, showing that action aliasing is more detrimental for SP-RL than for SE-RL. However, with appropriate improvement strategies, SP-RL can match or outperform improved SE-RL across a range of environments. These findings provide actionable insights for choosing and refining projection-based safe RL methods based on task characteristics.

Published in Transactions on Machine Learning Research (03/2026)
Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?
Hannah Markgraft
Shambhuraj Sawant
Hanna Krasowski
Lukas Schäfer!
Sébastien Gros
Matthias Althoff
Corresponding outhor: hannah. mankgnaanrkk de
Techhical University of Munich, University of California, Berkeley, Norwegian University of Science and Technology
Reviewed on OpenReview: https: openreiee..........mm net/ forum? id- DDrGSEYaGU
Abstract
Projection- based safety filters, which modify unsafe actions by mapping them the closest safe alternative, are widely used to enforce safety constraints in reinforcement learning (RL). commonly considered: Safe environment RL (SE-RL), where the safeguard is treated as part of the environment, and safe policy RL (SP-RL), where it is embedded within the policy through differentiable optimization layers. Despite their practical relevance in safety-critical settings, a. formal understanding of their differences lacking. In this work, we present a thcoretical comparison of SE-RL and SP-RL. We identify a key distinction how each approach is affected by action aliasing, a phenomenon in which multiple unsafe actions are projected the same safe action, causing information loss in the policy gradients. In SE-RL, this effect is implicitly approximated by the critic, while in SP-RL, it manifests directly rank-deficient Jacobians during backpropagation through the safeguard. Our contributions are threefold: (i) unified formalization of SE-
Two integration strategies
RL and SP-RL in the context of actor-critic algorithms, (ii) a theoretical analysis of their respective policy gradient estimates, highlighting the role of action aliasing, and (iii) a comparative study of mitigation strategies, including a novel penalty-based improvement for SP-RL that aligns with established SE-RL practices. Empirical results support our theoretical predictions, showing that action aliasing is more detrimental for SP-RL than for SE-RL. However, with appropriate improvement strategies, SP-RL can match outperform improved SE-RL across a range of environments. These findings provide actionable insights for choosing and refining projection-based safe RL methods based on task characteristics.
1 Introduction
For safety-critical environments, reinforcement lcarning (RL) policies have to be verified or safeguarded to ensure safety specifications at all times. Provably safe RL through closest-point projection (Krasowski et al. 2023), also often called safety filtering, widely used approach that provides safety guarantees during both training and deployment. The projection operation adjusts unsafe actions the closest safe action by solving optimization problem. This operation is usually diffcrentiable (Gros et al. 2020), allowing for two different formulations:
(a) Safe environment RL (SE-RL): The projoction is treated as part the unknown environment dynamics, requiring the critic to understand its eflect indirectly. Intuitively, the agent learns unsafe policy that acts in a safeguarded environment (Hunt et al. 2021).

---

Published in Transactions on Machine Learning Research (03/2026)
(b) Safe policy RL (SP-RL): The projection is integrated into the policy itself, meaning that gradients are backpropagated through the safeguard. In SP-RI, the objective is to approximate the optimal safeguarded policy for the original unsafe environment (Pham et al. 2018).
Figure D illustrates the structural differences between SE-RL and SP-RL. A key advantage of SE-RL is that it leaves the underlying RL algorithm unchanged, preserving any existing theoretical guarantees (Hunt ct al. 2021p. On the other hand, SP-RL promises more direct informing of the agent about the impact of the safeguarding by including the sensitivity of the safe action with respect to the unsafe action in the policy gradient estimator. However, SP-RL requires embedding the safegnard within the policy, typically using differentiable optimization layers (Agrawal et al. 2019), and incurs additional computational cost due to sensitivity analysis. These trade-offs lead to a central question: should one safeguard the environment or the policy?
Although empirical comparisons exist (Pham et al. 2018 Kasaura et al. 2023 theoretical foundation for understanding the trade-offs between SE-RL and SP-RL is still lacking. In this work, we close this gap by developing a formal framework that clarifies their relationship, identifying both equivalences and key differences. As discussed later in section 6.3 a central difference lies in how both approaches are affected by what we refer to as action aliasing: In closest-point projection, multiple unsafe actions are mapped to the same safe action, resulting in identical returns. Action aliasing has been well studied in the context of SP-RL. It has been shown to lead to rank-deficient Jacobian of the safeguard (Gros et al. 2020), also referred to the zero-gradient problem (Lin et al. 2021). The zero-gradient problem has been associated with degraded performance (Pham et al. 2018 Bhatia et al. 2019 and is commonly addressed through
alternative loss functions (Bhatia et al.) 2019; Chen et al. 2021 or modified policy update rules (Pham
ct al. 2018 Similarly, performance issues linked to action aliasing have been reported for SE-RL (Krasowski 2023), , and are oflen mitigated by introducing penalty terms in the reward function that penalize the
et al.
distance between the original and projected actions (Wabersich & Zeilinger 2021 Wang 2022 Markgraf &
Althoff 2023 Bejarano et al., 2025). However, formal comparison of the action aliasing effect on learning in SE-RL versus SP-RL, and of the differences in improvement strategies, does not yet exist.
Our contributions are as follows:
We develop a unified formalization of SE-RL and SP-RL in terms of the underlying Markov decision process (MDP), value functions, and policy gradient estimators;
We prove that the optimal value functions of SE-RL and SP-RL coincide under mild assumptions;
We formalize the impact of action aliasing on SE-RL, highlighting how it differs from the known zero-gradient problem induced by action aliasing in SP-RL;
We identify fundamental differences in how action aliasing is mitigated in SF-RL and SP-RL,, and propose novel remedy for SP-RL that aligns more closely with the renalty-based strategy used in SE-RL;
We perform an empirical comparison of SE-RL and SP-RL, emphasizing the importance of adapta- tion strategies for handling action aliasing in both deterministic and stochastic policy settings.
The remainder of this study is structured as follows: In section 2 we provide an overview of the related literature. The theoretical background for SE-RL and SP-RL is established in section 3 We then formalize the problem statement in section 4 Sections 3 and 0 provide a formal definition and a comparative analysis
of the two approaches, respectively. We discuss mitigation strategies for action aliasing and suggest new alternative for SP-RL in section
7 before conducting a thorough experimental evaluation in section g We
discuss our findings in section before concluding in section 10
2 Related Work
Safe RL augments RL algorithms with mechanisms to increase the probability of learning and deploying safe policies, to ensure hard safety guarantees for the policy (García & Fernández) 2015). Specifically,
2

---

Published in Transactions on Machine Learning Research (03/2026)
differentiated
not differentiated
Target policy
Target policy no (uf x1))
Parameter update
Parameter update
Vex
Safe environment et
VoT
Environment e
Policy me)
u
Safeguard +
uf
System
Policy Tal (u,
Safeguard
u
System
(T,)
1++
(a) SE-RL: Safeguard is part of the environment.
(b) SP-RL: Safeguard is part of the policy.
Figure 1: In provably safe reinforcement learning, we can either consider the safeguarding as part of the environment (figure lab or as part of the policy (figure lbb.
provably safe RL encompasses approaches for which hard safety guarantees are provided during training and deployment (Krasowski et al.) 2023). Provably safe RL can be further categorized by means of ensuring that only safe actions are exccuted. The first category relies on pro-characteriiing the set of safe actions through specific structural properties, as it is often necessary to compute center or boundary points efficiently. It comprises methods such as mapping unsafe actions to the interior of the safe action set (Tabas & Zhang 22022 Kasaura et al. 2023) or sampling from within it (Stolz et al. 2024). The second category of safeguarding mochanisms works dircctly with the safety constraints themselves, definod, for example, through control barrier functions (Marvi & Kiumarsi 2022: Wang 2022) or predictive filters (Selim et al. 2022 Wabersich
et al. 2023 Markgraf & Althoffi 2023; Gros et al., 2020 Within this group, closest- projection is
particularly prevalent in continuous control settings, mapping unsafe actions to their nearest safe counterpart by solving a constrained optimization problem.
While effective and widely used, action projection introduces a key limitation that we term action aliasing: All unsafe policy actions that lie within the normal cone to the boundary of the safe action set are projected to the same safe action (see lemma 2 Consequently, they all incur the same reward, no matter how close the policy action was to the safe action. In the context of SE-RT, action aliasing has not been theoretically analyzed, although it has been empirically acknowledged by some studies (Wang, 2022, Krasowski et al. Markgraf & Althoff 2023). In contrast, in SP-RL, the impact of action aliasing is well understood. Here, the projection is commonly integrated into the policy using differentiable optimization layers ( gAgaaal ct al. 2019) as the last layer of the policy network to retain gradient flow through the safeguard (Pham et al., Dalal et al. 2018 Bhatia et al.) 2019 Chen et al., 2021 Kasaura et al., 2023). Consequently, action aliasing directly affects the policy gradient computation, eliminating components in the normal direction of the mapping (Gros et al., 2020 Walter et al. 2025)).
2023
2018
The approaches for addressing action aliasing differ in SE-RL and SP-RL. A common remedy in JE-RL is to augment the reward with a penalty proportional to the action adjustment (Wabersich & Zeilinger 2021
Markgraf & Althoff 2023 Stanojey et al.] 2023 Bejarano et al.] 2025 Kasaura et al. 2023 Dawood et al. 2025j. As shown in Markgraf & Althoff (2023). agents trained with proportional penalties often outperform
those trained with constant or no penalties. Recently, [Bejarano et al. (2025) confirmed this observation
in quadrotor hardware experiments. In SP-RL, additional policy loss terms are often usod. For example, Bhatia et al. (2019) employs loss term that is proportional to the safety constraint violation. Similarly, Chen et al. (2021) proposes a loss term that is proportional to the Euclidean distance between the unsafe and the corresponding safe action. Pham et al. (2018) proposes two-step gradient step approach in which a first update step is calculated for the unsafe action (with penalty, following SE-RL), followed by an update step on the safe action.
Most existing studies treat projection safcguarding scparately in the context of either SE-RL or SP-RL, leav- ing a theoretical comparison of the two approaches unexplored. Kasaura et al. (2023) provides valuable empirical comparison of both basic and improved versions of these methods, focusing deterministic poli-
3

---

Published in Transactions on Machine Learning Research (03/2026)
cies with static safety constraints. Building their empirical insights, our work contributes a comprehensive theoretical framework that explains the fundamental differences between SF-RI, and SP-RL, including their respective improvement strategies for handling action aliasing. We extend the empirical analysis to in- clude both deterministic and stochastic policies, consider benchmark problems with state-dependent safety constraints, and focus on appropriately scaling penalties SE-RL and additional loss terms in SP-RL.
3 Preliminaries
Policy-based RL algorithms augmented with safety measures have proven successful in safety-critical tasks [Kassoski et al.) 2023). Therefore, we provide an overview of the basic concepts of these algorithms before detailing how to differentiate between SE-RL and SP-RL for projection cogeetiion-baed safeguarding section
3.1 Reinforcement Learning
An MDP is a tuple (X, Pr Ps. Pa: 7); where X and U are the state space and action space. We assume x ut and r to be continuous random variables (CRVs) modeling the state, action, and reward time step t, respectively. The transition function Ps X x U x X + Rzo denotes the probability density Pa(2444 [z uz) of transitioning to state #/+1 when applying the action u, in state æ, (Van Hasselt 2012 section 1.1). The reward for a transition is defined through the probability density pr Rt; uz), where pr X x U x R - Rg (Van Hasselt 2012 section 1.1). The discount factor 0 S y S is used to discount long-term rewards and serves as a simplistic model of the probabilistic lifetime of the MDP.
We define the discounted return at time t as Sutton Barto 2018 equation 3.8)
g+
The goal of RL to find the policy ı associated with a probability density function ruu (x)) over actions given state It, that maximizes the expected return (Sutton & Barto 2018 equation 13.4)
((T) E, Mo u, (-11,,))
(2)
where E+ refers the expected value of random variable given that the agent follows the policy æ. Note that in the deterministic policy case, states are directly mapped to actions, such that u, (2²2).
To find the optimal policy x' arg max [xx [Gros et al. 2020 equation 2). most RL algorithms use various forms of value function estimation. A state value function (x,) = E. Ig, X2 = xz. (Sutton & Barto 2018 equation 3.12) expresses how good it is to be in a certain state æt provided that policy æ is use. A statc-action value function qu (x,, uz) = E, [gs X; = Iz, u = u. Sutton & Barto 2018 equation 3.13) expresses how good is to use action u, in state æt, provided that policy æ is in use after that. Finally, the advantage of taking an action u, in state Φ, is commonly defined as a. (xe, ue) = qu (2 ue) - Uy (xx).
All optimal policies x² satisfy the same so-called optimal value functions
v" (x:) Un (x:) Vas. X, q" (æ,, Uz) gx- u.) Vz, E X, E. U.
(3)
(4)
3.2 Policy Gradient Algorithms
The policy be inferred directly from the value function, central concept of so-called value-based RL algorithms. However, these algorithms are mainly designed for discrete action spaces. In continuous and high-dimensional action domains, it is more common to directly learn a policy ao parameterized by 0, also referred to as policy- based RL. Most modern algorithms use actor-critic algorithms, combining concepts from value-based and policy-based RL by simultancously learning policy (the actor) and a value function (the critic) parameterized by &. To find the optimal policy parameters 6*, actor-critic algorithms perform repeated updates using 0 0 (TTO) (Van Hasselt, 2012 equation 13) where is learning rate. The
4

---

Published in Transactions on Machine Learning Research (03/2026)
gradient (TT( cannot be computed analytically since the reward function and the transition probability distribution are not known explicitly. RT algorithms thus estimate the gradient from data, more specifically from tuples (Ix, Un 22+1, rz). According to the policy gradient theorem, an unbiased estimate of ( is
given by (Silver et al. 2014 theorem 1)
(ae( = Es (at Vu, On (ar, un)l
(5)
for deterministic policies and paramcterized critic gr and Van Hasselt 2012 equation 15)
(Te( = Ex (Z ut) Ve log (e [x,]]
(6)
for stochastic policies, where v differs depending ( uz); ( u,) (Shhllaa et al. 2015), which vary in the variance of the gradient estimate.
the algorithm. Possible choices include
Actor-critic algorithms with stochastic policies such as Advantage Actor Critic (A2C) (Mnih et al. 2016 or Proximal Policy Optimization (PPO) (Schulman el al. 2017) that learn a state value Un use V/ = a(æ,, ut),
where is an estimate of the true advantage function. The most common estimator is the generalized advantage estimation (GAE) (Schulman al., 2015 equation 16)
GAE
(7A)%
(7)
1=0
where À is a hyperparameter and of is the temporal difference residual (Schulman et al. 2015 equation 10)
8 = rt + (# Ux (x:))
(8)
To find the optimal parameters o= of the value function t, actor-critic algorithms minimize the expected squared error between the current value function estimate and target y using (Lillicrap et al. 2015 equation 4)
L = E (re Ml, ( o uc+1) In
In o ( (,)), u,)
(9)
the equivalent equation for a state value function Un Here, In is target critic that is slowly updated using Polyak averaging. Alternative choices for the target include using the return & directly, or more advanced formulations that address overestimation bias using the minimum value of two target critic networks Fujjimoto et al. 2018).
4 Problem Statement
We consider a system with dynamics defined through the transition function Px. that is subject state- dependent safety constraints of the form
9( u.) S 0
(10)
which must be satisfied at all times. To enforce these constraints, we define state-dependent safe action set Ue U, such that for any Ug there exists a policy that, when applied consecutively from time + onward, ensures satisfaction of equation 10 at all times t' 2 t. We assume that Up can be represented as
(11) where constraints could, for example, be formulated using control barrier functions, predictive filters, or, as in our experiments, robust control invariant (RCI) sets (see appendix A.1 Furthermore, we define X X as the set of states for which U2 is non-empty.
U = {u s(x,, u.) < 0),
Given the safe action set, we introduce a safeguard X x + Uy into the interaction between the RL policy and the system that ensures that unsafe inputs are projected to the constraint boundary by solving
( u:) = arg min
2 a. uz 11
(12a)
R.ED
s.t. s(x,, ú.) S 0,
(12b)
5

---

Published in Transactions on Machine Learning Research (03/2026)
where ue .[ [1). As result, only safe actions ui = ( ut) can be applied to the system, and the return in equation [ depends on sequence of safe actions and states. Note that the mapping in equation 12 only modifies unsafe actions. To ensure that & is a deterministic operator, we assume that Ug, is convex. This is a mild assumption since safe sets are commonly constructed as convex inner approximations in practice due to their simple parametrization (e.g., polytopes, ellipsoids); computational tractability, and the uniqueness of closest-point projections.
As shown in figure I we use two different architectures to integrate Tb into the RL training loop to maximize the expected return in equation 2 We can consider the safeguard to be part of the black-box environment dynamics and aim to approximate the optimal unsafe policy x² E II for the safeguarded environment. Or we can learn a safe policy nー E III,, where II- C II is the set of safe policies. Note that for any safe policy, the probability density function π (uf xx) is zero for all unsafe actions. In this work, we investigate whether one solution approach dominates the other, both theoretically and empirically.
Notation: In the remainder of this work, we will omit the subscript t when possible and use æ, x', u, instead of Tt 2441, ue re for brevity. Furthermore, we will omit the subscripts A and Φ to improve rcadability.
5 Perspectives on Safe Reinforcement Learning
We develop a unified theoretical framework for both approaches to safe RL with action projection by spec- ifying the corresponding value functions and policy gradient estimates. An overview is provided in table [] For brovity, we limit our analysis to RL algorithms derivod directly from the policy gradient thcorem, as including methods with additional modifications (e.g., PPO, Soft-Actor Critic (SAC)) would require exten- sive theoretical derivations beyond the scope of this paper. Similar to the work in Gros et al. (2020), we distinguish between stochastic and deterministic policies. We then discuss the equivalences and differences of both approaches in section 6
5.1 Safe Environment Reinforcement Learning (SE-RL)
In SE-RL, the safeguard is considered part of the system dynamics that are unknown to the RL agent. This changes the MDP from section 3.1 to MSE = (X, U. SE p :7), with p X x U X + R2o and p XxUXR-UR 22.- Please note that the support of these densities changes compared to the original MDP introduced in section 3.1 Here, ph
(2e ar, $(xr, and (r æ., (1x u)) are defined the composition of the safeguard mapping & with the original transition functions introduced in section 3.1 Accordingly, the environment receives (potentially) unsafe action u ~ л that is projected internally. Consequently, the value functions are learned for the unsafe policy such that
4
SE (x. u) Es [ge
Xx
x, U = u..
(13a)
SB (x) E
BB((2)
Ige
X xc]
(13b)
The policy gradient estimate is computed as given in equation 5 and equation 6 Note that the critic forming this estimate is not aware of the safeguard and has to assess its impact through the data alone.
5.2 Safe Policy Reinforcement Learning (SP-RL)
If we consider the safeguard to be a part of the policy, sampled actions are adjusted by differentiable optimization layer, such that the final policy action is a random variable u,a == 4(x,, uz). The transition function and the probability density of the reward of the MDP MSP for SP-RL are then defined as p % x P(U+) x X + R20 and SP X x IP(U)) x R + R20, respectively, where the action space P(UF) is the power set of all Us. We aim to learn value function for the safe actions such that
SP (xx, u²) = E [ge
X+ = xc, uuf = u
(14a)
SP (xx B [ge Xx
xc]
(14b)
6

---

Published in Transactions on Machine Learning Research (03/2026)
Table 1: Overview of the MDP formulations and learning equations for SE-RL and SP-RL with stochastic and deterministic policies, respectively.
SE-RL
SP-RL
M
SE :P' .7) U.P
M = (X, Pr P2z :7)
MDP
Th x
R20
D x P x X R20
x x
U x R
R20
pr X x PP x R - R20
Policy
x
R
20:
m X P(U+) R R20;
X - P((U))
Value functions
(z, u) = E gg
X =
SP (x, u²) = E [gt X2 = x. u. = =
SE (x) = 1x 1gt X = x
SP (x) = 1 1s, X++ = x:|
Policy
VOJ(T) = Ex S az Vo log
VoJ(T )- E8 / (æ,, u²) Vo log ग. (2 (2/ [
gradient
estimates
VoJ(T) = Ex
Ve ( V v (x, (x,u) w)
Vol(T+ = E Ven (x) Ver
9rd (x, 26)
SP
Note that this requires us also project the action u' when computing the stato-action value 9n (a', ux To obtain the policy gradient estimates, we must distinguish between
for the target y given in equation g deterministic and stochastic policies.
5.2.1 Deterministic Policies
The deterministic policy gradient estimate in SP-RL is defined by
g VoJ(T = E, Ven (x) ur Ix- (x, u²)
(15)
Computing the gradient VoJ(x=) requires differentiating through the safeguard Φ. We can apply the chain rule to obtain
(o)+ v.t(a, u) (o)(
(16)
The sensitivity of the safeguard Vu 2) can be obtained using the implicit function theorem as described in appendix A.2
5.2.2 Stochastic Policies
To find the optimal parameters of the safeguarded policy - in the stochastic policy case, we use the policy gradient estimate
VoJ(m')= =T, [v.ST (xr, u Vo log x (ut") [x)]]
(17)
which depends on the probabilities of safe actions given states as well on the value function estimate for the safe policy - While the unsafe policy ( x)) features a bounded probability density, the safe policy Dirac-like structure on the boundary of the safe set as illustrated in Gros et al. (2020)
I (u*) x) takes figure 2). In general, no closed-form expression of x+(=" ax) exists, making it difficult to provide gradient estimate for a projected stochastic policy. However, in Gros et al. (2020) Proposition 2), the authors show that we obtain an unbiased gradient estimate using
voJ(xt)= E. SP (x. u V log (u x]]
(18)
An alternative for SP-RL with stochastic policies proposed by Chen al. 2021). Given an unsafe policy represented by Gaussian distribution (He, Eg), they first project the mean using Mg (x) @(uo x)) and then sample uz ~ A( Eg). However, since this action is not guaranteed to satisfy the constraints, they project again obtain u² 0(u", Since this approach does solve the problem of finding closed-form solution for - (u² (a), not further considered here.
7

---

Published in Transactions on Machine Learning Research (03/2026)
6 Comparative Analysis of SE-RL and SP-RL
Having established SE-RL and SP-RL two distinct solution paradigms for the same task, we now ex- amine their theoretical relationship and practical differences. Since the learning equations differ between approaches, identical initial policy and critie parameters will yield different updates, raising fundamental question: do both frameworks converge to equivalent (sub-Joptimal solutions?
6.1 Theoretical Equivalence of Optimal Solutions
We first establish that, in principle, both approaches target the same optimal value function.
Theorem 1. For any given task, let u be a value function for policy a interacting with the MDP MSE and SP value function for policy n- interacting with MSP. Then, a value function uSE that is oplimal for ME is also an optimal value function ySP for MSP.
The proof is provided in appendix A.3 Theorem I states that, given specific task, a policy n² that is optimal in the SE-RL framework yields the same expected return as optimal policy xt in the SP-RL framework. While this theoretical result provides important insight, it offers limited practical guidance for continuous state and action spaces where optimal policies are rarely found (Vnn Hasselt 2012). Subsequently, we derive a more practical result for a certain group of algorithuns.
6.2 Practical Equivalence for Stochastic Policies
A Algorithms employing stochastic policies and GAE (e.g., A2C) learn state value function Ue (xx) to compute GGAE The following lemma shows that in this case, the differences between SE-RL and SP-RL disappear.
Lemma 1. Let ne, To and u SP n- denote the parameterized policies and value functions for a given task when employing SE-RL or SP-RL, respectively. Define the advantage estimates used the policy gradient estimates SE-RL (equation 6 and SP-RL (equation 18 as
S x (r. u) = à GAE u) and
TT (ax, u²) = a. GAB n²),
Then, for any initial parameters O, and du, the parameter updates 0x41 < Oz + a J(0x) and db+1 db V L((s) are identical in both the SE-RL and SP-RL framework at every ileralion 2 0.
respectively, where à GGAE computed using equation
The proof is provided in appendix A.4
Corollary 1. Under the conditions of Lemma the sequences of parameters (0x, Øu) generated by SE-RL and SP-RL are identical, and consequenily both frameworks converge to the some policy and value function parometers.
In contrast, for algorithms using deterministic policies, SE-RL and SP-RL have different learning equations and may thus converge to different locally optimal policies shown next.
6.3 Differences for Deterministic Policies
As summarized in table the first difference consists in the scope of the critic. Most algorithms using deterministic policies learn a state-action value function q. In SP-RL, is evaluated only safe actions and, thus, does not learn a meaningful approximation outside of the safe action set. In SF-RT,, q is also evaluated on unsafe actions. However, any value function in SE-RL that is conditioned on actions is affected by what we refer to as action aliasing:
Lemma (Action aliasing). Let OUg be the boundary of a conver safe action set Up and Nug (u²) {n n²" (u -u)) < 0 Vu E (2) the normal cone at u² E OUr (Boyd Vandenberghe, 2004). Furthermore, let U (u())) = {u U u²= u un+ 0,n E Nue(u²)) be the set of unsafe actions u the normal cone of a given u² OUS Then, any action u U²(x") will be projected u', cousing transition the same neat slate æ' and yielding the same reward b pr xx, u²).
8

---

Published in Transactions on Machine Learning Research (03/2026)
1:2
120
12
5.0
277
2
277
132
Pestile
13
Poottlee
1.5
Feodtlee
Sude
Poemeii tagecoory
Paramoser trojecory
Panmero injectery
Projection
Valce frncor sotmooe
Vahee froction odimatt
Volue dunction ertmmat
2.30 133
1:5
8.
7.20
X10
710
6.3
438
(a) Without safeguarding, the critic (b) SE-RL: Policy converges to an (c) SP-RL: Policy with differen- approximates the true objective func- unsafe and suboptimal action that lies tiable projection safeguard does not tion, and the policy converges to the in the direction normal to the safe improve in the direction normal to optimal safe action.
timal action and the safe bound- the projection and therefore does not reach the optimal safe action.
Figure 2: Effect of action aliasing on SE-RL and SP-RL algorithms using deterministic policies. We illustrate the policy improvement step for a given state x in the quadrotor balancing task. The deterministic policy x (z) updated for 50 steps using a loss function based on the learned state-action value function q(æ, u) in the SE-RL case and q<z, u²) in the SP-RL case.
This means that the critic cannot distinguish between the actions u" E (u"",, potentially hindering learning. We formalize this flat-lining critic phenomenon in the following lemma, which is proven in appendix AA.5
Lemma 3 (Flat-lining critic). Following lemma the state-action value function, the advantage function, and the GAE adhere to
In (x, u²) = qo (2, u²) Vu E U (x u²) a a2, (2) Vu E US, and
a (x, u²) a CAAB( ul) Vu e
GAE
respectively-
Note that the flat-lining critic issue also applies to algorithus using stochastic policies in SE-RL. Essentially, lemma states that gradient information along the normal directions will be eliminated by using perfect value function approximation equation 5
We illustrate this using a simplified example. Consider the quadrotor task described in section we assume deterministic system dynamics and compute the safe action set Uy for fixed state æ given the RCI set. We use supervised learning and a random behavior policy to train a critic network to approximate the state-action value function 4x (z, u) for the given state. Then, we use a deterministic target policy represented by a tensor of the same dimension as and perform policy update steps using stochastic gradient descent to maximize the Icarned objective. Note that for this environment, task performance (reaching the equilibrium state) and safety well-aligned since the equilibriun is considered a safe state. Therefore, the optimal action is often also safe, as shown in figure 2 In appendix A.6 provide second example where task performance and safety are not aligned.
A.8 Here,
Figure 2a shows a non-safeguarded setup, where the critic has learned good approximation of the true objective function. The policy improvement steps lead to the optimal safe action. However, such a learning process is only possible in computer simulation or non-safoty-critical environments, as unsafe actions might be executed. In SE-RL, the safeguarding takes place in the environment such that the projection is not visible in figure 2b
We observe that the policy actions never converge to the safe action set, and would
9

---

Published in Transactions on Machine Learning Research (03/2026)
not reach the optimal safe action even after the projection. This is caused by the flat-lining critic, which eliminates gradients in the direction normal to the boundary of the safe action set.
SP-RL avoids the flat-lining critic by restricting evaluation to safe actions. We visualize this in figure 2c
Instead, the impact of action aliasing is shifted to another part the policy gradient estimation. When computing the sensitivity of the safeguard (a, u), any components in the normal direction to the boundary vanish [Gros et al., 2020), leading to rank-deficient Jacobian. This is termed the zero-gradient problem in existing work {Lin et al., 2021 Kasaura al. 2023 As shown in figure (6c| the prooggradient problem can be particularly challenging for safe action sets with non-smooth surfaces, as multiple constraints are active on vertices, further reducing the rank of the Jacobian. This is also visible to lesser extent for the flat-lining critic in figure 6b
We would like to highlight that while the sero-gradient problem SP-RL always exists, the flat-lining critic problem in SE-RL depends on the quality of the Iearned value function. An imperfect value function approximation may, in fact, alleviate the issue, shown in other works highlighting the advantages of sampling sampling-based gradient estimation (Suh et al. 2022). We examine the practical implications in section
7 Addressing Action Aliasing in SE-RL and SP-RL
We extend our comparative analysis to variants of SE-RL and SP-RI. that aim to address the shared issue of action aliasing. First, we establish how existing approaches differ, impairing the thcoretical equivalence established in theorem [ Then, we propose an allernative approach for SP-RL that re-establishes the validity.
7.1 SE-RL
In SF-RL, action aliasing is commonly addressed by adding penalty h to the reward each time the safeguard has to intervene (Wabersich & Zeilinger 2021
Markgraf Althoff 2023 Stanojev et al. 2023 Bejarano
et al.
2025 Kasaura et al., 2023: Dawood al. 2025). This changes the MDP reward function from section
5.1
SE,sug X
x
R
R 20 where
SE,a (x, u) = Sx, u) h.
(19)
A common choice for the penalty function is the squared Euclidean distance between the safe and the unsafe action
if u E Uy fo - 11 otherwise,
h = E(u u) -
(20)
wilu
where is a hyperparameter. Introducing penalty changes the optimization goal of SE-RL as shown in figure Bal To enable a better comparison with the improvement strategies for SP-RL presented in section 7.2
we quantify the impact the penalty has on the value function and policy gradient estimate. The state-action can rewritten as
value function (2 u)
gx
aug
I u) = Ex
SE,aug Fe+k+1
Xi = E. =
E
k SE
(1++4+1 h(+k+1)
Xi I. U
u
0==
E
k SE 1t+k+1
X1 H, U.
E
b14k+1
σ, U
k=
SP (a.uu)
pen (an,z)
Then, the policy gradient estimate for deterministic policy can be expressed as
(oJ(m) E, (o)(w)) V sug (x, u)) E,, [o) (g, (a. u. gee (x)
(21)
10

---

Published in Transactions on Machine Learning Research (03/2026)
12
120
1.2
277
2'50
g
122
Peatble
132
Poottbee
1.2
Feostle
Sude
Poemeii tagecoory
Paramoser trajecooy
Panmerx injectery
Projection
Valce frncor sotmooe
Valee fraction odimat
Volue Seection ecttaat
1:5
770
635
(a) SE-RL: Adding a penalty to the (b) SP-RL: An additional policy loss (c) SP-RL: Learning an additional reward addresses the flat-lining critic term that penalizes the distance be- penalty critic very similar to adding problem and improves convergence tween unsafe and safe action improves penalty to the reward in SE-RL. toward the optimal safe action.
convergence the optimal safe action Note that the penalty critic con- but stops the boundary of the safe ditioned unsafe actions, display the objective over the entire action space.
action set.
Figure 3: Effect of improvement strategies when using a differentiable safeguard during policy updates for a given state æ in the navigation task. The deterministic policy = is updated for 50 steps using a loss function based on the learned state-action value function q(z, u) in the SE-RL case and q(x, u²) in the SP-RL case.
showing the impact of the penalty. The term (x, u) steers the policy away from actions that frequently trigger the safeguard, such that policy actions that are inherently safe or closer to the safe region are preferred. The strength of this effect is controlled by the hyperparameter w in equation 20 A similar derivation can be done for stochastic policies, resulting
VoJ(n) E, Ta u) 1og a:) E,, [T (x. u) Tppen (z. Vo
Vo (ugg((ul m))
(22)
7.2 SP-RL
In SP-RL, action aliasing manifests as rank-deficient Jacobians the safeguard that directly enter the policy gradient computation through backpropagation. Unlike in SE-RL, where penalties can influence the value function approximation to mitigate action aliasing effects, reward penalties in SP-RL cannot eliminate the underlying rank-deficiency in the sensitivity of the safeguard. Instead, existing literature in SP-RL suggests direct modification of the policy loss to improve performance (Chen et al. 2021 Bhatia et al. 2019). For deterministic policies, this results in combined policy gradient estimate
vaJ(x=)= =,, (Von~(x) Vur
SP (xx u²) Vad(-))
(33)
where d(-) commonly depends on the unsafe and the safe action. For example, in Chen et (021), the authors suggest using the squared distance between the unsafe and the projected action
additional loss such that d = (x), (x))) as defined in equation 20 Others use loss term proportional to the constraint violation, which is theoretically similar (Bhatia et al. 2019 We illustrate the effect of the squared distance loss in figure 3b for a deterministic policy. Compared to figure 2c the policy evolves Loward the boundary of the safe action set, even though it does not reach the optimal safe action. For stochastic policies, the vanishing gradient problem does not apply as discussed section 6.3 Nevertheless, we can apply this additional loss the mean a of the Gaussian policy N Σ) and its projection μ such that
(=)(n E [V (xc, u²) Vo log n (u [)) E. Vo)(u)())) (x))
(24)
11

---

Published in Transactions on Machine Learning Research (03/2026)
Using an additional loss term in equation 23 and equation 24 captures only myopic effects of unsafe actions. In contrast, penalties added to the reward are embedded into the value function estimate, providing information regarding their long-term consequences. Consequently, the per-sample loss in SP-RL targets a different optimal value function than the improved SE-RL version presented in section 7.1 breaking the equivalence established theorem | Furthermore, the per-sample loss can only push unsafe actions to the boundary of the safe action set, but not its interior.
Therefore, suggest an alternative that consists of training additional critic conditioned on the unsafe policy such that
(az, u) E lgte
X = u, = u):
(25)
where gi -Eto 1thh h++k+1. This yields the deterministic policy gradient estimate
VoJ(T=) - Ex- Ven (o) w 92-
SP (z, u² - E, V (x, u)]
(26)
which is similar in structure to the one from equation 21 The main difference to SE-RL is that the first term
in equation 26 depends on the safeguarded policy and involves the sensitivity of the safeguard. Since the
second term is computed for the unsafe policy, convergence behavior compared to the vanilla loss function
is improved as shown in figure 3c. In comparison to the per-sample loss, the policy reaches the interior of
the safe action set.
In principle, this approach can also be applied to algorithms with stochastic policies, resulting in the policy gradient estimate
VoJ(m=)) = FT, [ (x, u Vo log tu æ:) - E,, [Tren (æz, Vo log ( a=)-
(27)
However, as discussed in section 5.2.2 the distinction to SE-RL with additional penalties vanishes when the algorithu is based on learning & state value function u (a) that does not depend the action. More details on the penalty critic implementation can be found in appendix A.7
8 Experiments
We design our numerical experiments] to answer the following questions: (1) Without any modifications, which approach achieves better returns empirically SE-RL or SP-RL? (2) How do the approaches for addressing action aliasing in SE-RL and SP-RL compare to each other? For SP-RL, we consider the per- sample squared distance loss (PSL) and our proposed penalty critic (PenC), and for SE-RL proportional penalties.
We restrict our investigation to actor-critic algorithms based on the policy gradient theorem, Gros et al.) (2020) provide derivation of unbiased policy gradient estimates under the SP-RL framework for these algorithms. We choose Twin-dclayed Deep Deterministic Policy Gradient (TD3) (Fujimoto et al. 2018 and A2C (Mnih et al. 2016 as RL algorithms using deterministic and stochastic policies, respectively.
We use cvxpylayers (Agrawal et al. 2019) to integrate the safeguard into the policy for SP-RL as automates the computation of the sensitivitics of the projection in the backward pass. To ensure a fair comparison, we run all experiments on the same CPU (Intel Core 19-14900K). Furthermore, we conduct hyperparameter tuning for the unsafe baselines of each algorithm and then use this set of hyperparameters for all experiments. The weighting factor is an important hyperparameter. Therefore, we do not tune it, but instead test for different choices: w E {0.1, 0.5, 1.0, 2.0}. We report the final performance using the interquartile mean and 95% bootstrapped confidence intervals of the undiscounted return over 7 training runs evaluated on 10 random seeds, respectively.
8.1 Benchmark Problems
We perform the evaluation using a stabilization task for two classic control examples - a pendulum and a quadrotor - as well as a navigation task, and an energy management system (EMS) optimization task. For
2Qur implementation is available https///gthaahccmm//TMupppp/ppp
12

---

Published in Transactions on Machine Learning Research (03/2026)
Pendulum
Quadrotor
Seeker
EMS
SE-RL
SE-RL Penally
SP-RL-
RL PSL
RL
20
10
G0
40
-60
20
Interquartile Mean
Intenquartile Mean
Interquartile Mean
Interquartile Meun
(a) Results for TD3 (deterministic policies).
Pendulum
Quadrotor
Sceker
EMS
SE-RL
SE-RL Penalty
SP-RL
SP-RL PSL.
SP-RL PenC
20
10
10 Interquartile Mean
60
400
20
5
Interquartile Mcan
Interquartile Mean
Interquattil Mean
(b) Results for A2C (stochastic policies). For the pendulum, clip the returns obtained with vanilla A2C enable better comparison. The actual returns are much lower as shown in table 7
Comparison of vanilla and modified SE-RL/SP-RL. approaches (PSL: per-sample loss, PenC: penalty critic). We provide the interquartile mean and the 95% confidence interval of the return achieved at test time. For the improved SE-RT//SP-R, approaches, we only show the result for the best-performing choice of (0.1, 1.0, 2.03. The full results are listed in appendix A.10
Figure 4:
the first two tasks, we compute RCI sets using the approach from Schäfer et al. (2024). Since the RCI sets are centered around the equilibrium point, task performance is closely aligned with safety. This is different in the navigation task, where a simple point mass seeker has to find the shortest path to a goal while avoiding obstacles, such that the optimal safe policy operates in close proximity to the unsafe regions. Here, instead of an RCI set, we use state-dependent safe action set avoid collisions. In the EMS task, there are two competing objectives: maintaining the indoor temperature close set point and minimizing electricity cost. Consequently, the alignment of safety and performance depends which objective dominates at given point in time. Further details on the benchmark problems are provided in appendix A.8
8.2 Results
8.2.1
Comparison of SE-RL and SP-RL Without Modifications
As shown in lemma 1 for algorithms such as A2C that employ stochastic policies and learn state value function, SE-RL and SP-RL result. in the same policies. Consequently, they deliver the same return at test time as shown in figure 4b
For TD3, figure 4a shows that vanilla SE-RL often outperforms SP-RL, especially for more complex environ- ments. For the pendulum task, the final performance is very similar, with slight advantage for SP-RL. For the quadrotor, the EMS, and the seeker task, the differences are more pronounced, and vanilla SE-RL clearly dominates SP-RL. Figure shows large variance in the training performance for TD3-SP-LL caused by several non-convergent runs, which explains the poor performance at test time.
8.2.2 Comparison of Improved SE-RL and SP-RL
In figure | we only report the results for the best pest-performing choice w, while the full results are shown in appendix 10 For the quadrotor and the seeker environment, the choices of w that perform well are largely consistent for one algorithm. For the pendulum, larger in SE-RL can deteriorate performance significantly, while SP-RL with the per-sample loss is very robust..
13

---

Published in Transactions on Machine Learning Research (03/2026)
Pendulum
Quadrotor
Seeker
EMS
-60
-15 -30
-00
-02
-40
Return
-50
-20
140
-60
-00
160
-70
S000
7500
10200
S0000
100200
15:000
200000
2:00
5000
15000
1000
20000
40000
Training Steps
Training Steps
Training Steps
Training Sleps
200
Safety Interventios
130
175
100
125
80
100
10
83
40
59
40
20
200003
Training Stops
Training Steps
Training Steps
Training Steps
SE-RI
SE-RL Penaly
SP-RI
SP-RL
PSL
SP-RI, PenC
Figure 5: Interquartile mean and 95% bootstrap confidence interval of the return and safeguard interventions over environment steps during training with TD3. We compare the vanilla SE-RL and SP-RL versions and the strategies for mitigating action aliasing.
For TD3-SP-RI, both the per-sample loss and the penalty critic improve test performance across all four environments (see figure [aa. As shown in figure 5 the convergence issues of vanilla TD3-SP-RL in the quadrotor, the EMS, and the seeker can be mitigated. Which of the two strategies for action aliasing performs better depends on the type of enviromment: the two balancing tasks and the EMS, the penalty critic delivers better results, while the per-sample loss has the edge in the navigation task. Statistical analysis using Kruskal-Wallis and Dunn's tests with Bonferroni correction (p 0.05) confirmed that compared to TD3-SE-RL with penalties, one of the improved SP-RL versions always performs par better.
For A2C, the penalty in SE-RL and the penalty critic in SP-RL are again equivalent. They significantly improve performance for the pendulum and quadrotor enviromments, and slightly for the EMS. In the seeker environment, performance slightly deteriorates for all improvement strategies in both SE-RL and SP-RL.
9 Discussion
For actor-critic algorithms using stochastic policies and GAF, both theoretical and empirical results confirm that SE-RL and SP-RL are equivalent. While the per-sample loss on the policy mean can sometimes improve performance, it is usually outperformed by penalty-based approaches. The seeker environment proved more challenging for improvement strategies due to the difficulty of weighing task performance with safety and the already strong performance of vanilla. A2C.
As reported previously (Pham et al. 2018), we observe convergence issues when using SP-RL with determin- istic policies, particularly in complex environments, while SE-RL shows such issues. This supports theoretical finding that critic flat-lining only hinders learning under perfect value function approxinations. In contrast, rank-deficient Jacobians always affect policy updates in SP-RL, making action aliasing more detrimental. Following Bhatia et al. (2019): Chen et al. (2021), we find that these convergence issues can be mitigated by adding loss terms proportional to the distance between safe and unsafe actions.
The relative effectiveness of our proposed penalty critic versus the per-sample loss depends on the alignment between safety and task performance. In the balancing tasks and the FMS, where these objectives align, the
14

---

Published in Transactions on Machine Learning Research (03/2026)
penalty critic outperforms the per-sample loss because it can converge to optimal actions within the safe set, while the per-sample loss is limited to the boundary (figures 3b and 3cl. In the seeker environment, where optimal task actions typically lie outside the safe set, both methods converge to boundary actions, making the per-sample loss more efficient.
Overall, improved SP-RT variants typically match or exceed SE-RL performance for deterministic policies, but require 3 12 times the computation time of SE-RL due sensitivity computations via cvxpylayers Our wall clock time analysis in appendix A.9 shows that the increase in computation time scales with the
complexity of the projection problem 12 We observe the smallest difference between SE-RL and SP-RL for the socker task, where the constraints 12b reflect a point containment problem given an unsafc action and a safe action set. This is much simpler than the set containment problem that has to be solved for the pendulum, the EMS, and the quadrotor task (see appendix [A.1). Since the state and action space dimensionality of the quadrotor and the EMS are higher than that the pendulum, they feature the highest gap between SE-RL and SP-RL wall clock times. While for such environments the performance gains in SP-RL have to be weighed against computational differentiable optimization layers (Nguyen & Donti Grontas et al. 2026 Frey et al. 2025) to mitigate this issue in the future. Furthermore, it is often possible to decouple high-dimensional safety-critical dynamics into several subproblems, which can then be solved in parallel during both the forward and the backward pass Chen et al. 2021).
complexity, we expect recent. advances in the field of
10 Conclusion
We present a comprehensive theoretical and empirical comparison of SE-RL and SP-RL, two prominent approaches for integrating projection-bann based safeguards into actor-critic RL. Our unified formalization enables us to prove that both approaches share optimal value functions but differ their learning dynamics. For a specific subclass of RL algorithms that use stochastic policies and GAE, we show the theoretical and practical equivalence of SE-RL and SP-RL. For algorithms using deterministic policics, the approaches diverge depending
whether the sensitivity of the safeguard mapping is used explicitly in the backward pass of the policy loss (SP-RL.,, or whether it is learned implicitly through the value function approximation (SE-RL). A central concept when analyzing these differences is action aliasing, where multiple unsafe actions are mapped to identical safe actions. Our analysis reveals that the effect of action aliasing is more detrimental to SP-RL. than SF.-L,, potentially leading to convergence issues. We propose a novel penalty critic for SP-RI. that estimates discounted cumulative penalties proportional to the distance between safe and unsafe actions, providing a principled mitigation strategy aligned with penalty-based approaches in SE-RL. Our theoretical findings and empirical analysis provide practitioners with clear guidance: Vanilla SE-RL presents strong bascline, particularly for environments where scaling additional penaltics or loss terms is challenging, while improved SP-RL variants should be considered when performance gains justify the additional computational cost.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Acknowledgments
The authors gratefully acknowledge the partial financial support of this work by the German Research Foundation through the SAFART project (grant no. 458030766) and the SFB 1608 (grant no. 501798263), the Research Council of Norway (grant no. 300172, SARLEM), and the research training group ConVeY, funded by the German Research Foundation under grant GRK 2428/2. Furthermore, thank Jonathan Külz for his ideas on how to visualize the action aliasing phenomenon for the different learning paradigms.
References
Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and J Zico Kolter. Dif- ferentiable convex optimization layers. Advances in Neural Information Processing Systems, 2019.
Federico Pizarro Bejarano, Lukas Brunke, and Angela P Schoellig. Safety filtering while training: Improving the performance and sample efficiency of reinforcement learning agents. IEEE Robotics and Automation Letters, 10(1):788-795, 2025.
Dimitri P. Bertsckas. Dynamic programming and optimal control Athena Scientific, 2018.
Abhinav Bhatia, Pradeep Varakantham, and Akshat Kumar. Resource constrained deep reinforcement learn- ing. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 29, pp. 610 620, 2019.
Mikael Andreas Bianchi. Adaptive modellbasierte prädiktive Regelung einer Kleinuürrmeeuuppennnlaae PhD thesis, ETH Zurich, 2006.
Stephen Boyd and Lieven Vandenberghe. Conver optimization. Cambridge University Press, 2004.
Bingqing Chen, Priya L Donti, Kyri Baker, J Zico Kolter, and Mario Bergés. straints through differentiable projection for energy optimization. In Proceedings of the ACM International Conference on Future Energy Systems, pp. 199-21 210, 2021.
Enforcing policy fcasibility con-
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in contimuous action spaces. arXiv preprint arXic:1801.08757 2018.
Murad Dawood, Sicong Pan, Nils Dengler, Siqi Zhou, Angela P Schoellig, and Maren Bennewitz. Safe multi- agent reinforcement learning for behavior-based cooperative navigation. IEEE Robolics and Automation Letters, 1066) 6263, 2025.
Jonathan Frey, Katrin Baungärtner, Gianluca Frison, Dirk Reinhardt, Jasper Hoffmann, Leonard Fichtner, Sebastien Gros, and Moritz Diehl. Differentiable nonlinear model predictive control. arXiv preprint arXiv:2505.0135. 2025.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pp. 1587 1596. Proceedings of Machine Learning Research, 2018.
Javier García and Fernando Fernández. A comprehensive survey safe reinforcement learning. In Journal of Machine Learning Research, volume 16, pp. 1437-1480, 2015.
Kasra Ghasemi, Sadra Sadraddini, and Calin Belta. Compositional synthesis via convex parameteriza- tion of assumo-guarantee contracts. In Proceedings of the International Conference Hybrid Systems: Computation and Control, pp. 10, 2020.
Panagiotis D Grontas, Antonio Terpin, Efe C Balta, Raffaello D' D'Andrea, and John Lygeros. Pinet: Opti- mizing hard-constrained neural networks with orthogonal projection layers. In International Conference Learning Representations, 2026.
Sebastien Gros, Mario Zanon, and Alberto Bemporad. Safe reinforcement learning via projection on safe set: How to achieve optimality? IFAC- IFAC-PagersOnTinee, 53(2):8076 88066688811110 2020.
16

---

Published in Transactions on Machine Learning Research (03/2026)
Nathan Hunt, Nathan Fulton, Sara Magliacane, Trong Nghia Hoang, Subhro Das, and Armando Solar- Lezama. Verifiably safe exploration for end-to-end reinforcement learning. In Proceedings of the Interna- tional Conference on Hybrid Systems: Computation and Control, pp. 1-11, 2021.
Kazumi Kasaura, Shuwa Miura, Tadashi Kozuno, Ryo Yonetani, Kenta Hoshino, and Yohei Hosoe. Bench- marking actor-critic deep reinforcement learning algorithms for robotics control with action constraints.
IEEE Robolics and Aulomation Letters, 8(8):4449 4456, 2023.
Steven George Krantz and Harold R Parks. The implicit function theorem: history, theory, and applications. Springer Science & Business Media, 2002.
Hanna Krasowski, Jakob Thumm, Marlon Müller, Lukas Schäfer, Xiao Wang, and Matthias Althoff. Provably safe reinforcement learning: Conceptual analysis, survey, and benchmarking. Transactions on Machine Learning Research, 2023.
Adrian Kulmburg and Matthias Althoff. On the co-NP-commletennss of the zonolope containment problem. European Journal of Control, 62:84-92 91, 2021.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.020 2015.
Jyun-Li Lin, Wei Hung, Shang-Hsuan Yang, Ping-Chun Hsich, and Xi Liu. Escaping from zero gradient: Revisiting aetion-constrainee reinforcement learning via Frank- policy optimization. In Uncertainty in Arlificial Intelligence, pp. 397 407. Proceedings of Machine Learning Research, 2021.
Hannah Markgraf and Matthias Althoff. Safe multi-agent reinforcement lcarning for price-based demand response. In JEEE PES Innovative Smart Grid Technologies Europe, pp. 1-6, 2023.
Zahra Marvi and Bahare Kiumarsi. Reinforcement learning with safety and stability guarantees during exploration for linear systems. IEEE Open Journal of Control Systems, 1:322-334, 2022.
Volodymyr Mnih, Adria Puigdomenech Badia, Mchdi Mirza, Alex Graves, Timothy Lillicap,, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Inler- national Conference on Machine Learning, pp.
2016.
Hoang T Nguyen and Priya L Donti. FSNet: seasini neural network for constrained optimization with guarantees. Advances in Neural Information Processing Systems.
Tu-Hoa Pham, Giovanni De Magistris, and Ryuki Tachibana. Optlayer practical constrained optimization for deep reinforcement learning in the real world. In IEFF International Conference on Robotics and Automation, pp. 6236-6243, 2018.
Lukas Schäfer, Felix Gruber, and Matthias Althoff. Scalable computation of robust control invariant sets of nonlincar systems. IEEE Transactions on Automatic Control, 69(2):755-770, 2024.
Mark J Schervish. Theory of statistics. Springer Science & Business Media, 1995.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. flig--dmensiooal continuous control using generalized advantage estimation. arXiv preprint 1rXiv:1506.02438 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint (lo:1707.06347 2017.
Mahmoud Selim, Amr Alanwar, M Watheq El-Kharashi, Hazem M Abbas, and Karl H Johansson. Safe rein-
forcement lcarning using data-driven predictive control. In International Conference on Communications, Signal Processing, and their Applications, pp. 1 6, 2022.
David Silver, Guy Lever, Nicolas Heoss, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deter- ministic policy gradient algorithms. In International Conference On Machine Learning, pp. 387-395. Proceedings of Machine Learning Research, 2014.
17

---

Published in Transactions on Machine Learning Research (03/2026)
Ognjen Stanojev, Lesia Mitridati, Riccardo de Nardis Di Prata, and Gabriela Hug. Safe reinforcement learn- ing for strategic bidding of virtual power plants in day-ahead markets. In IEFE International Conference on Communications, Control, and Computing Technologies for Smart Grids, pp. 1 1-7, 2023.
Roland Stolz, Hanna Krasowski, Jakob Thumm, Michael Fichelbeck, Philipp Gassert, and Matthias Althoff. Excluding the irrelevant: Focusing reinforcement learning through continuous action masking. Advances in Newral Informalion Processing Systems, 37:95067 95094, 2024.
Hyung Ju Suh, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake. Do differentiable simulators give better policy gradients? In International Conference Machine Learning, pp. 20668 20696. Proceedings of Machine Learning Research, 2022.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Daniel Tabas and Baosen Zhang. Computationally efficient safe reinforcement learning for power systems. In American Control Conference, pp. 3303-3310, 2022.
Hado Van Hasselt, Reinforcement learning in continuous state and action spaces. In Reinforcement Learning: State-of-the-Art, pp. 207-251. Springer, 2012.
Kim P. Wabersich, Andrew J. Taylor, Jason J. Choi, Koushil Sreenath, Claire J. Tomlin, Aaron D. Ames, and Melanie N. Zeilinger. Data-driven safety filters: Hamilton-Jacobi reachability, control barrier functions, and predictive methods for uncertain systems. IEEE Control Systems Magazine, 43(5):137-177) 2023.
Kim Peter Wabersich and Melanie N Zeilinger. A predictive safety filter for learning-based control of con- strained nonlinear dynamical systems. Automaticn, 129, 2021.
Tim Walter, Hannah Markgraf, Jonathan Kiilz, and Matthias Althoff. Leveraging analytic gradients in provably safe reinforcement learning. IEEE Open Journal of Control Systems, 4:463-481, 2025.
Xiao Wang. Ensuring safety of learning- motion planners using control barrier functions. TEEF Robotics and Automation Letters, 7(2):4773 4780, 2022.
A Appendix
A.1 Action Projection Using Zonotopes
One option for defining safe action sets is to consider control invariant sets as safe state sets X where E X ensures that there exists admissible action u, E U such that equation 10 can be satisfied for all times. The constraints in equation 12b can be defined using C C X where C is the set of all reachable states at the next time step under the system dynamics and bounded noise w, E W. The precise formulation of these constraints depends on the chosen set representations for approximating reachable and safe sets. In this work, we adopt zonotopes to enable efficient computation. Consequently, we have to replace equation 12b with the necessary constraints for verifying zonotope-in-zonote containment. Consider a zonotope Z g in generator representation that is given by
2 = (=ER" : B3, | 1)
(28)
with the center E R and the generator matrix B E x.,,)x(2) where (2 E No denotes the number of
generators of Z. The inequality in equation 28 has to be applied elementwise. A more compact notation is
given by Z (c, B)z. Consider two zonotopes Z Bi) Zz (c2, Bz)z. Then, 2, is contained in
E2, i.e., 2, C 2,, if there exist P E (2((2)))(Zz,, E RR((22) such that (Ghasemi et al. 2020 Lemma 1)
B B
(29a)
= Byw,
(29b)
T
w
1000 < 1.
(29c)
18

---

Published in Transactions on Machine Learning Research (03/2026)
The resulting optimization problem involves both equality and inqquality constraints, which is an important consideration when analyzing the sensitivity of the solution, as discussed in appendix A.2
If the safe action set is given directly as a zonotope = B.)z, we only have to verify that ū E Ug.
This can be achieved by replacing equation 12b with (Kulmburg & Althoff 2021
B = i Cu
(30)
| S 1.
(31)
A.2 Differentiating the Safeguard Using the Implicit Function Theorem
If the projection safeguard + integrated into the policy as shown figure Tb we require the sensitivity of output (the safe action) with respect its imput (the unsafe action) for the backward pass of the policy optimization. To oblain the sensitivity of u² (ar, u) with respect u, let us first. consider the Lagrange function associated with the projection problem
12
[(x, u. u, KE, M(x u. u) K² D(x, u. û) v² H(x, u, u,ü)
where Q is the safeguard objective, and the equality and inequality constraints resulting from the zonotope containment problem in equation 29 are represented with D and H. The corresponding dual variables are k and v, respectively. Let us further consider the Karuuh-Kuhhn Tucker (KKT) conditions
V Valie FaC( u, K)
<(x, u. ū. U)
u. v (1(
(32)
At the KKT point, we have
<(x, u, ú, K. v)l ,=v' 0.
where [u0, _,K""""" w²) are the primal-dual solution of the safeguard. Then, with the implicit function theorem Krantz & Parks 2002), the sensitivity of the safeguard with respect to u is
V ((r = V (x, u. ū. K. - ' Vuelar ut v) jä= li=u:x==et,o=o
(33)
A.3 Proof of Theorem 1: Equivalence of Optimal Value Functions
To prove theorem 1 let us first recall the definition of optimal policy. A policy is called optimal if its expected return is greater than or equal to that of all other policies in all states. Therefore, all optimal policies satisfy the same optimal value function (Sutton & Barto
2018
("(x)= = Ux (x) 2 Un (x)
Va E X, π E II.
(34)
If we can show that SE-RL and SP-RL share the same optimal value function, this implies that any optimal policy in cither framework will yicld the same expected return. Note that policies in both frameworks, æ and - are parameterized by 0, though this dependence omitted for readability.
To demonstrate the equivalence of the optimal value functions, we begin by comparing the conditional probability densitics of SE-RL- - namcly, the state transition density g and the reward density n with those of SP-RL, denoted SP and p.P. At the core of both approaches, have sequence of mappings,
u (x',r),
where each arrow indicates either sampling from conditional distribution applying the deterministic transformation Φ. However, the mappings are grouped differently in SE-RL and SP-RL, resulting in the sequences
SP SP-RL: u
(w', r),
BE
SE-RL:
x -
(x',r)
19

---

Published in Transactions on Machine Learning Research (03/2026)
Thus, the Markov processes resulting from policies и and xt with the same @ would have the same conditional probability densities,
(a x)) =p=" (a' |x); Pr
(35)
SB (r x:) - prr SP (r x)
(36)
Now, we can prove theorem
Proof. In SE-RL, for a certain policy π, the state value function (equation 13b given as
SE (a) = E Eu Xerope 1
Kt X0 = x
(37)
E Eu X X
7 T Xo = x
(38)
A=
/ In J. rpr SE [r] akk, us (us mxx)
k
Ps
SE
(1++1 [ (T xi) dr duu... dux dx dx.
(39)
1=0
(LTT)
TVr ( axx) JR
II
SE (#041 mi)) dr dx
(40)
k=0
where LTT refers to the law of total probability (Schervish 1995 theorem B.70). Here, (a) subsumes the lincarity of expectation as well as the chain rule of probability with the Markov property. similar derivation can be found in Bertsekas (2018 appendix A.2).
Similarly, for the corresponding safe policy n², we have
SP (x) Euf wes XP
g Xy x
(41)
Eul most X
*reek41 X0
(42)
1=
(a)
[ J3
rpr
SP (r Lkb 1 xxx)
k=
(uf
xx))s (#0+4 (0:+1 Bis
dr duf du. dar....drr
(43)
k-1
(LTT))
rp
SP (r xx
II
pe (w æ;)) dr do
(44)
k=
1i=0
Thus, with equation 35 and equation
36 for a policy II, there exists safe policy mt II+ such that,
SE U (x) = ( (x) Vx X.
(45)
Consider an optimal policy x* for SE-RL, then the corresponding value function is the optimal value function in SE-RL, i.e u (x) = SE' (æ), Yx e X. Using equation 45 there exists a safe policy म E II- such that 1 (x)) SP (x)), Va E X. However, it not guaranteod that this safe policy is optimal the MDP MSP, Next, we employ a proof by contradiction to establish that the safe policy and its associated state-value function SP constitute the optimal policy and optimal state-value function for SP-RL.
20

---

Published in Transactions on Machine Learning Research (03/2026)
180
033
330
035
020
1.00
7yye
Pauith
-130
Pouible
Prashla
Sule
Sude sel
Sude
-4.7
-2.0
ingodtryy
-1.5
injectory
Projection
Projeclsut
Projecias
Valce functut sotinate
Vahua fonation
Vahur Sunction
(a) Without safeguarding, the critic (b) SE-RL: Policy converges to an (c) SP-RL: Policy with differentiable
approximates the true objective fune- unsafe action that lies in the direction projection safeguard does not im- tion, showing that the optimal action normal to the safe optimal action and prove in the direction normal to the projection and eventually gets stuck vertex of the safe action set.
would be unsafe. Note that clip the safe set boundary. the policy to the feasible aclion set during policy improvement.
Figure 6: Effect of action aliasing on SE-RL and SP-RL algorithms using deterministic policies. We illustrate the policy improvement step for given state in the navigation task. The deterministic policy л (ax) is updated for 50 steps using a loss function based on the learned state-action value function ((æ, u) in the SF-RT, case and g(ar, u²) in the SP-RI case.
Suppose that there exists a safe stochastic policy म E achieving strictly better performance,
SP (a) > o (x), for some a E X
Since # x)) is supported on Uy C U, we can construct a corresponding SE-RL policy #(- x)) whose pushforward through equals a-(-1 m). Formally, for each w:
--) = #+
where 4y denotes the pushforward of the probability measure under
Intuition: for each safe action u sampled from (- Φ), select a preimage u E --'( according any probability distribution over that set. This is always possible because ob is surjective, ensuring every u e Th has al least one preimage in U.
Using equation SP-RL. Hence,
45 the
and reward distributions under म in SE-RL match those under # in
0x (x) = v (x)
o (x)
SE" (x))
contradicting the optimality of n²
Therefore, no such I exists, and # is also optimal in SP-RL. Consequently, the optimal value functions coincide,
SE' (x) = SP (x), Vx E X.
21

---

Published in Transactions on Machine Learning Research (03/2026)
033
830
035
020
1:
00
Naxiti
-130
Pouible
Prashle
Sale
Sado
Sude
-477
-2.0
ingodtryy
-12
injectory
Projection
Projeclson
Projecias
Valce funcout sotinate
Vahua fonation
Vahur Sanction
(a) SE-RL: Adding a penalty to the (b) SP-RL: An additional policy loss (c) SP-RL: Learning an additional reward addresses the flat-lining critic term that penalizes the distance be- penalty critic very similar to adding problem and improves convergence tween unsafe and safe action improves penalty to the reward in SE-R RL. toward the optimal safe action.
convergence the optimal safe Note that the penalty critic con- lion. However, if the optimal safe ditioned unsafe actions, display action would lie the interior the objective over the entire action the safe action set, it might not be space.
reached.
Figure Effect of improvement strategies when using a diffcrentiable safeguard during policy updates for given state æ in the navigation task. The deterministic policy ()( is updated for 50 steps using a loss function based the learned state-action value function q<x, u) in the SE-RL case and q(x, u²) in the SP-RL case
A.4 Proof of Lemma 1: Equivalence of SE-RL and SP-RL for Stochastic Policies and GAE
Proof. With ST (x, u²) (G w²) and USE (x, (xx estimates for SP-RL for stochastic policies as
a.AF u), rewrite the policy gradient
VoJ(TO)
EE,1
ST (xx u Vo log (u æ]] [a
E
(x, u
V ( x)]
E
(TA) 8i Vo log
E
(7A) - ( - SP (x+++)) Vo log Te( x))
and, similarly,
VOJ(TO) Ez Σ (rA) 70
SB (224444)) ST (F(+))))) log (o [))
Using the same initial parameters & = -h and 0 = gin, both policy gradient estimates are the same for the first policy update. Similarly, as the value functions SE and SP only depend on the state, the gradient L for the loss defined in equation 9 also equivalent. Consequently, if all other algorithm hyperparameters are the same for both SE-RL and SP-RL, all subsequent parameter updates will also be the same.
22

---

Published in Transactions on Machine Learning Research (03/2026)
Pendulum
Quadrotor
Seeker
EMS
Panally (Nommloud)
1.0
03
2m0
04
04
00
1000
Training
Traniing Steps
Training Steps
20C
Sabsaadd Inteventenn
130
162
14.
3X
208
.000
0000
2000
2000
2000
20000
20300
20000
2000
40000
5000
8000
100000
10000
2000
20000
1000
2000
Trarinng Stops
Training Stape
Training Steps
Training Staps
Figure Comparison of the penalty critic loss and the number of saleguard interventions during training for different scaling factors w.
A.5 Proof of Lemma 3: Flat-lining Critic SE-RL
Any action 4 (u"") projected u² and thus transitions to the same next x' and yields the same reward pb pr(r] ae, u²). Since the stale-action value function satisfies (Sutton & Barto 2018
Proof.
In (x, u) Er yo (x') x. u,
we obtain
Ir (x, u²) = qn (x, u²) Vu E U
and, consequently,
ag (x, u²) = ax (x, u²)
Vu US we receive
Furthermore, for the GAE according to equation
GAF (x. u²) (x) Vu E U.. 0g
A.6 Action Aliasing Visualization for Seeker Task
We provide second minimal example visualize the impact of action aliasing in deterministic policies. It is based the secker navigation task described in appendix A.. All other settings the same as listed in section 6.3 The main difference to the previously described example is that task performance and safety are not aligned in the navigation task, as one obstacle will always be situated between the initial and the goal position. Therefore, actions that are optimal with respect to the task performance are usually unsafe, and the optimal safe action lies on the boundary of the safe action set. This is visualized in figure D Due to these different task characteristics, the impact of the different improvement strategics becomes more similar shown in figure 7
A.7 Details of Penalty Critic Implementation
The penalty critic is a separate neural network with the same architecture as the task critic. In contrast the latter, the penalty critic is conditioned the unsafe action u, not the safe action. Its parameters ş are
23

---

Published in Transactions on Machine Learning Research (03/2026)
trained using the mean-squared error loss in equation
9 where the target y is computed
Y/ us u
+79 pen
(+++1,u(+1)
(46)
be
Consequently, the scaling factor has strong impact the learning stability of the penalty critic. Figure shows that while for the pendulum and the EMS all penalty factors lead to stable loss progression and reduced safeguard interventions, only higher values for w can achieve this for the quadrotor environment. For the seeker navigation task, where performance and safety are not aligned (see section 8.1 a. low penalty factor enables the best convergence behavior.
A.8
Benchmark Problems
A.8.1
Pendulum Stabilization Task
Our pendulum environment closely related to the OpenAl Gym Pendulum- vo environment with the difference that we limit the one-dimensional control input |u] S 8rads-!. The environment has the state 0., 8] and the dynamics
2 =
(47)
sin(0)
sin
+
me²
where g is gravity and m, l are the mass and the length of the pendulum, respectively. We discretize the dynamics using the explicit Euler method. The desired equilibrium state is xx [o.0]" The reward is -00 0.1 + 0.001=²).
A.8.2 Quadrotor Stabilization Task
We use the model of quadrotor operating in the r-z-plane that is proposed in Stolz et al. 2024). The system has two independent. thrusts u [u²,u2] u²] bounded by lower and upper limits u, u, respectively. The state of the system x = [e", [e",e", ,", 0, o)", where 6x,z are the positions along the x- and z-axis, respectively. The system dynamics are
E² (u* u² K sin(O)
w-
(48)
-9 + (u' + K cos(O)
2
i
dg0 - diú + no no + u²)
where do, d1, no, are constants and = [w] w²] are disturbances. The linearized dynamics are obtained using a first-order Taylor expansion around the equilibrium point x [0, 1, 0, 0) and are discretized in time using the explicit Euler method. The disturbances sampled uniformly from compact disturbance
set W c R². The reward is computed using r -1 - exp (-Ils -81/2 0.01
x²-x²"
ll). where
77 P-
[e e+].
A.8.3 Seeker Navigation Task
In this two-dimensional navigation task, a simple massless robot has to find the shortest path to the goal while avoiding a fixed number of obstacles. The environment. configured such that least one obstacle always lies between the initial position of the seeker and the goal position e The initial position, goal position, and the positions and radií of the obstacles are pseudo-randomly sampled at the beginning of each episode. The state [e",e" 200 consists of the positions and velocities in the x-y-plane. The
htttps://gymaaiiu http::///...aaa..........oo.oons http://sa...aaaaoeeaeeee.aeaaaaa
24

---

Published in Transactions on Machine Learning Research (03/2026)
actions 10 |0] are accelerations in the respective directions and bounded by |ē",, |291 S max with
nmax mm/8 8-. We use the simplified system dynamics
0
0
(49)
w
>
and discretize them using the explicit Euler method. The disturbances w = [w= w²]² are uniformly sampled from a compact disturbance set W C RR. The position of the secker at time is et, and the step reward is exp([le, egll). In experiments, we set the number of obstacles to three.
In each time step, safe action set (x) computed to avoid collisions with obstacles and meet the boundarics of the map. This simplifies the projection problem in equation 12 to
(a2 u)
2 arg min ú 2
1/ - u1
(50)
s.t. ū E x''
(51)
(52)
A.8.4 Energy Management System Optimization Task
The goal in this task (introduced in Walter et al. (2025) is to control battery and heat pump in building that has non-flexible electric base load and non-controllable PV generator. The state æ [soc, Oin, @retT comprises the state of charge of the battery, the indoor temperature, and the re- turn ue of the floor heating system. The policy of the agent is conditioned on the observation o= [x, Ouut Pitt PP @ta+H]: which is composed the current value and H forecasts the outdoor such that the
temperature, the PV power, the inflcxible load, and the electricity price. We use observation space has a dimensionality 23. The actions u j " are the continuous power points for the battery and the heat pump, respectively. The system follows the simplified dynamics
BB O + Area 000
Oout 0
(53)
cg "pp"P
where the outdoor temperature acts as a bounded noise. The computation of the coefficients cp-4 is detailed
in Bianchi (2006, equation 2.17). Performance is measured through the deviation of the indoor temperature from a desired set point Ostt as well as through the electricity costs such that r = -p HP + +p- -:ppppPyp p (@in @ The state constraints [0, 10] [18, 24] x [10, 100] are enforced using reachability analysis. During training, one episode corresponds to one day, and the initial state is randomized.
A.9 Wall Clock Time Results
Table 2: Comparison of wall clock time for TD3 (deterministic policy case) over 10,000 training steps, normalized to the vanilla SF-RI. approach.
States
SE-RL
SE-RL Penalty
SP-RL
SP-RL PSL
SP-RL PenC
Pendulum
1.0
0.99
6.04
5.59
5.49
Seeker
1.0
1.0
3.56
3.55
3.71
Quadrotor
1.0
1.0
12.01
10.51
9.02
EMS
1.0
0.98
12.25
12.38
12.51
25

---

Published in Transactions on Machine Learning Research (03/2026)
A.10
Comprehensive Results
Table 3: TD3, pendulum: mean and standard deviation of returns and safeguard interventions for different scaling factors
SE-RL
SE-RL Penalty
SP-RL
SP-RL PSL
SP-RL PenC
t = 0.1
Interventions
0.01 0.20
0.03 0.24
4.13 = 5.10
1.33 + 1.35
5.49 + 18.23
Returns
-11.52 + 7.24
113.23 + 8.19
-10.52 + 7.41
-8.06 + 6.16
-8.71 + 6.22
# == 0.5
Interventions
0.01 0.20
0.0 0.0
4.13 = 5.10
2.51 + 4.17
8.90 + 18.29
Returns
-11.52 + 7.24
29.34 19.43
10.52 + 7.41
-8.77 + 6.21
-7.60 = 5.99
== 1.0
Interventions
0.01 = 0.20
0.07 : 0.35
4.13 = 5.10
1.36 + 1.46
Returns
-1.52 7.24
23322 + 21.19
-00.52 + 7.41
-8.33 + 6.84
-9.57 + 7.08
w 2.0
Interventions
0.01 + 0.20
0.0 0.0
4.13 + 5.10
16.53 - 25.80 18.57 = 34.74
Returns
11.55 + 7.24
-5..88 - 23.71
-00.52 + 7.41
-8.50 + 6.30
-7.47 + 5.70
Table 4: TD3, quadrotor: mean and standard deviation of returns and safeguard interventions for different scaling factors
SE-RL
SE-RL Penalty
SP-RL
SP-RL PSL
SP-RL PenC
w = 0.1
Interventions
194.23 + 14.01
170.13 48.93
199.71 + 0.64
199.04 + 1.49
192.60 :: 23.79
Returns
-49.43 + 6.76
-49.75 12.35
66.34 = 16.76
-66.87 = 17.41
-62.52 + 13.00
w = 0.5
Interventions
194.23 + 14.01
64.16 + 46.66
199.7 069.71 + 0.64
170.44 51.05
145.31 + 49.21
Returns
-49.43 + 6.76
443.76 = 13.68
-66.34 + 16.76
-59.11 - 18.65
-53.46 + 17.47
w 1.0
Interventions
194.23 + 14.01
65.63 + 60.98
199.71 + 0.64
181.79 + 37.02
141.80 + 58.84
Returns
-49.43 + 6.76
- -4..33 = 16.68
-66.34 + 16.76
-49.97 + 7.68
-54.22 + 16.73
w 2.0
Interventions
194.23 + 14.01
4 + 40.76
199.71 + 0.64
192.51
21.70
134.93 52.98
Returns
-49.43 + 6.76
- 4
66.34 + 16.76
49.34 5.74
44.34 + 10.26
26

---

Published in Transactions on Machine Learning Research (03/2026)
Table 5: TD3, seeker: mean and standard deviation of returns and safeguard interventions for different scaling factors
SE-RL
SE-RL Penalty
SP-RL
SP-RL PSL
SP-RL PenC
V 0.1
Interventions
83.90 34.00
85.60 + 32.19
81.7 + 35.16
42.24 + 43.90
49.90 44.94
Returns
-34.20 = 20.05
34.16 + 20.14
-51.33 + 22.97
-6.31 + 20.13
-31.66 = 20.43
w 0.5
Interventions
83.90 = 34.00
80.00 = 37.10
81.7 = 35.16
44.01 + 39.53
42.63 - 44.65
Returns
-34.20 + 20.05
32.73 + 20.70
-1.33 + 22.97
-6.100 18.19
33.02 + 19.22
w
Interventions
83.90 34.00
93.96 + 22.04
81.7 + 35.16
43.19 + 38.87
46.03 45.75
Returns
-34.20 + 20.05
-99.12 + 16.59
-1.33 + 22.97
-28.17 + 17.49
-36.06 + 19.26
w 2.0
Interventions
83.90 34.00
97.47 + 14.86
81.7 + 35.16
40.40
39.20
39.31 + 2.97
Returns
-34.20 = 20.05
-48.09 15.86
-1.33 + 22.97
28.90 19.73
-42.04 = 17.69
Table 6: TD3, FMS: mean and standard deviation of returns and safeguard interventions for different. scaling factors
SE-RL
SE-RL Penalty
SP-RL
SP-RL PSL
SP-RL PenC
w 0.1
Interventions
17.19 7.42
3.1 + 4.36
22.99 + 0.12
16.23 + 4.46
14.11 + 7.75
Returns
-16.11 + 8.81
-18.71 + 8.77
26.45 + 19.25
-2..60 = 15.51
-14.99 + 6.46
w 0.5
Interventions
17.19 + 7.42
0.43 + 0.79 -20.02 + 8.65
22.99 + 0.12
13.9 = 8.78
15.13 + 4.55
Returns
-16.11 + 8.81
26.45 + 19.25
-15.63 + 7.11
-12.65 + 4.59
w
Interventions
17.19 + 7.42
0.0 0.0
22.99 + 0.12
15.81 + 4.94
11.07 7.03
Returns
-16.11 + 8.81
-23.06 = 12.00
26.45 + 19.25
-13.57 4 5.87
-14.41 + 6.23
2.0
Interventions
17.19 7.42
0.0 - 0.0
22.99 + 0.12
8.56 = 7.47
11.71 + 7.16
Returns
-16.11 + 8.81
20099 = 8.03
-26.45
19.25
-18.40 13.90
-2.59 + 5.13
27

---

Published in Transactions on Machine Learning Research (03/2026)
Table 7: A2C, pendulum: mean and standard deviation of returns and safeguard interventions for different scaling factors
SE-RL
SE-RL Penalty
SP-RL
SP-RL PSL
SP-RL PenC
W = 0.1
Interventions
170.03 + 53.11 1177.09 + 31.28
1.04 2.82
170.03 + 53.11
6.06 = 8.88
1.04 2.82 32.90 + 57.86
Returns
-32.90 = 57.86
117.009 + 31.28
-7.60 = 6.33
w 0.5
Interventions
170.03 = 53.11
0.48 = 2.15
170.03 + 53.11
7.84 11.95
0.48 + 2.15
Returns
1177.09 + 31.28
24..88 + 40.57
1177.09 + 31.28
-7.52 + 6.54
-4.558 + 40.57
w == 1.0
Interventions
170.03 + 53.11
50.79 = 80.19
170.03 + 53.11
27.50 + 36.68
50.79 + 80.19
Returns
1177.09 + 31.28
-114.15 + 77.99
-177.09 + 31.28
-8.02 + 7.40
-114.15 + 77.99
2.0
Interventions
170.03 + 53.11
38.66 + 68.81
170.03 + 53.11
11.66 + 20.81
38.66 68.81
Returns
117..09 31.28
156.37 + 64.00
117..09 + 31.28
-7.06 = 6.42
-15.377 64.00
Table 8: A2C, quadrotor: mean and standard deviation of returns and safeguard interventions for different scaling factors
SE-RL
SE-RL Penalty
SP-RL
SP-RL PSL
SP-RL PenC
w 0.1
Interventions
199.90 +: 0.46 -66.09 + 20.09
161.56 + 50.16
199.90 + 0.46
198.27 + 4.67
161.56 + 50.16 550.46 + 8.22
Returns
-50.46 + 8.22
66.09 = 20.09
-67.58 = 25.27
w 0.5
Interventions
199.90 +: 0.46 98.27 + 64.17
199.90 +: 0.46
198.46 + 2.19
98.27 + 64.17
Returns
-66.99 + 20.09
-2.75
12.63
-6.009 + 20.09
-58.60 + 9.41
-52.75 + 12.63
w 1.0
Interventions
199.90 + 0.46
63.96 + 63.01
199.90 + 0.46
196.94 + 4.97
63.96 + 63.01
Returns
-66.09 + 20.09
-50.29
13.41
-66.09 + 20.09
62.94 + 10.30
-50.29 + 13.41
2.0
Interventions
199.90 + 0.46
80.19 - 82.43
199.90 + 0.46
195.77 + 5.66
80.19 82.43
Returns
-66.09 + 20.09
-1.52 = 14.81
66.09 + 20.09
63.81 + 12.06
-51.52 + 14.81
28

---

Published in Transactions on Machine Learning Research (03/2026)
Table 9: A2C, seeker: mean and standard deviation of returns and safeguard interventions for different scaling factors w.
SE-RL
SE-RL Penalty
SP-RL
SP-RL PSL
SP-RL PenC
Interventions
56.93 44.75
66.42 + 44.02
56.93 44.75
43.71 + 42.01
66.42 44.02 -22.04 = 20.23
Returns
-99.46 + 19.92
22.04 + 20.23
19.466 + 19.92
-44.2 17.54
w 0.5
Interventions
56.93 = 44.75
97.43 + 15.16
56.93 + 44.75
61.33 + 40.95
97.43 + 15. 16
Returns
-19.46 + 19.92
-1..3 + 15.46
-9.44 + 19.92
-35.58 + 13.76
-41.33 = 15.46
w 1.0
Interventions
56.93 44.75
100.00 + 0.00
56.93 + 44.75
72.61 + 34.33
100.00 + 0.00
Returns
-19.46 + 19.92
-50.22 + 15.04
199.46 + 19.92
-38.61 + 13.28
-50.22 + 15.04
2.0
Interventions
56.93 44.75
100.00 + 0.00
56.93 - 44.75
64.86 - 36.08
100.00 + 0.00
Returns
-19.46 19.92
-56.06
14.32
19.46 + 19.92
40.67 + 13.33
-6.00 + 14.32
Table 10: A2C, EMS: mean and standard deviation of returns and safeguard interventions for different scaling factors w.
SE-RL
SE-RL Penalty
SP-RL
SP-RL PSL
SP-RL PenC
w 0.1
Interventions
17.49 + 1.67
12.6 + 4.02
17.49 + 1.67
16.23 + 4.46
10.87 + 5.51
Returns
144.14 7.92
-14.12 7.91
-14.14 + 7.92
-21.60 + 15.51
-13.2 7.70
0.5
Interventions
17.49 + 1.67
7.34 4.27
17.49 + 1.67
13.9 + 8.78
0.6 + 1.45
Returns
-14.14 7.92
-3.99 7.84
-14.14 7.92
-15663 7.11
-13.01 7.62
1.0
Interventions
17.49 1.67
4.7 +: 3.59
17.49 + 1.67
15.81 4.94
0.0 0.0
Returns
-14.14 7.92
13.69 7.79
-14.14 7.92
-13.77 5.87
-1323 + 7.64
w = 2.0
Interventions
17.49 1.67
2.16 2.59
17.49 + 1.67
8.56 + 7.47
0.0 0.0
Returns
-14.14 7.92
-13.38 7.82
- -14.14 + 7.92
-18.40 + 13.90
13.4 7.65
29


Reviews:
[{"review_id": "IFoA1BUNJu", "reviewer": "Reviewer_oNPV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers differences between two approaches to safe RL: one where a safeguard mapping unsafe to safe actions is put into the environement (SE-RL), and one where this safeguard is put into the policy (SP-RL) such that policy gradients are calculated through the safeguard.\n\nThe paper makes a number of theoretical contributions: it shows that in some ways SE-RL and SP-RL are equivalent approaches. Value functions optimal for SE-RL are also optimal for SP-RL. Further, for stochastic policies, both have identical policy and critic gradients, meaning that both will converge to the same local optima. For deterministic policies, it is shown that both SE-RL and SP-RL suffer from action aliasing, where several unsafe actions are mapped to the same safe action, but that both are affected differently by it.\n\nThe paper then considers strategies for mitigating the effects of aliasing on both SE-RL and SP-RL. In SE-RL settings, prior work has commonly introduced a penalty into the reward function that penalizes the difference between an unsafe action and its safe replacement. This penalty therefore has a non-myopic effect on the optimal policy. For SP-RL, prior work has proposed to introduce additional terms into the policy loss, also penalizing distances between unsafe and corresponding safe actions, but doing so in a myopic way. This paper therefore proposes for SP-RL to introduce an additional penalty critic. This yields policy gradients that push unsafe actions closer to safe actions, and analogues to the SE-RL approach, but does so in a non-myopic manner.\n\nExperimentally the paper shows that the proposed penalty critic for SP-RL improves return and safety for SP-RL, and outperforms the myopic loss penalty approaches from prior work on two out of three tasks considered.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Some of the core results in the paper are theoretical in nature and have accompanying proofs in the appendix. The proposed penalty critic for SP-RL has theoretical justification and is tested empirically. Empirical results could have been more extensive, with experiments carried out in only three relatively simple environments, but are sufficient in combination with the theoretical contributions.", "audience_yn": "Yes", "audience_explanation": "This is a relevant paper or people engaged in Safe-RL research. The paper does an fantastic job presenting safeguards in Safe RL and provides an excellent overview of related work, making it a valuable introduction to the topic for a broader audience of RL/ML researchers.", "requested_changes": "I have no issues that would prevent me from recommending acceptance.\n\nI do have some minor points:\n1. In section 5.2 when introducing the SP-RL setting, you explain that action adjustment is part of the policy. Yet when describing the MDP you introduce new transition and reward densities $p_x^{SP}$ and $p_r^{SP}$, despite these being exactly the same as in the original MDP. For SE-RL it made sense to introduce new densities as these now included the safeguard, but would it not be better to simple use the original $p_x$ and $p_r$ for SP-RL to emphasize the fact that the MDP does not change?\n2. On page 11, first paragraph, you write “policy actions that are inherently safe or closer to the feasible region are preferred” Is it not supposed to be “safe region” rather than “feasible region”?", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MIelrY3dHD", "reviewer": "Reviewer_t7Dp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary**\n\nThis paper conducts a systematic comparison between two projection-based safety-enforcing mechanisms in reinforcement learning, safe environment (SE) and safe policy (SP). The paper presents theoretical analyses that establish the equivalence and difference of the two approaches under certain settings. It also demonstrates the effect of action aliasing across both approaches and proposes a variant, PenC, to improve SP. Some experiments on RL benchmarks validate the theoretical claims.", "claims_and_evidence_yn": "Yes", "claims_explanation": "**Strength**\n\nI'm not an expert in RL, but I find this paper clearly motivated and well written. I also think a systematic comparison between two existing paradigms is important.", "audience_yn": "Yes", "audience_explanation": "**Weaknesses**\n\nI have two minor comments on the presentation and experiments.\n\n1. Limited theoretical explanations of action aliasing\n\n   The paper theoretically explains the connection between SE and SP in section 6 and presents the solutions to action aliasing. But I find that the formal definition and explanation of action aliasing itself is lacking and somewhat hidden in the discussion of section 6.3 on deterministic policies. Besides, I also feel that the motivation behind the newly proposed PenC approach should be more carefully explained.\n\n2. Scope of experiments\n\n   The experiments compare the performance of SE and SP in terms of return and safety intervention, which are two primary metrics for the task. I believe it would also be useful to include experiments on the practical computational efficiency of the two approaches (as mentioned in the discussion section), since this would provide guidance for practitioners when choosing between them.\n\n**Questions and Minor issues**\n\n1. Page 2\n\n   What does \"the sensitivity of the safe action with respect to the unsafe action\" mean?\n\n2. Page 5\n\n   $\\mathbf{g}_t$ => $g_t$.\n\n3. Page 6\n\n   The order of $u, x$ in $p_x^{SE}(x' | u, x)$ and $p_x^{SP}(x' | u, x)$ seems inconsistent with THE definition of $p_x$.\n\n4. Page 7\n\n   $\\Psi_{\\pi^\\perp}^{SP}$ is not clearly explained here.\n\n5. Page 7 and 11\n\n   Border and boundary are used interchangeably.\n\n6. Page 9\n\n   Please define normal cone formally.\n\n7. Page 10, 11\n\n   Additional fullstop after the equations for $q_\\pi^{aug}$ and the figure on page 11.\n\n8. Page 11\n\n   $\\mu$ seems not defined.\n\n9. Page 13\n\n   Figure 5 seems to be mentioned before Figure 4.", "requested_changes": "See weaknesses and questions above.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dPPQYpsfuS", "reviewer": "Reviewer_pag5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### 1. Summary\n\nThe paper provides a theoretical and empirical comparison between two common approaches for projection-based safe reinforcement learning: Safe Environment RL (SE-RL) and Safe Policy RL (SP-RL). The authors propose a unified framework to formalize these methods and prove that, theoretically, both approaches share the same optimal value functions. A key contribution is the analysis of \"action aliasing,\" which manifests differently in each setting (flat-lining critic in SE-RL vs. rank-deficient Jacobians in SP-RL). To address the limitations of SP-RL, the paper introduces a \"Penalty Critic\" aimed at mitigating the zero-gradient problem. Experiments on standard control tasks (Pendulum, Quadrotor) and a navigation task are conducted to validate the proposed methods against baselines.\n\n### 2. Strengths\n\n- **Clarification of Theoretical Equivalence:** The paper offers a rigorous proof showing that SE-RL and SP-RL target equivalent optimal value functions. This theoretical grounding clears up ambiguities in the existing literature regarding the fundamental relationship between safeguarding the policy versus the environment.\n\n- **Mechanism Analysis:** The analysis of how \"action aliasing\" negatively affects learning is insightful. Specifically, distinguishing between the \"flat-lining critic\" in SE-RL and the \"zero-gradient/rank-deficient Jacobian\" problem in SP-RL provides a useful taxonomy for understanding failure modes in safe RL.\n\n- **Sound Mitigation Strategy:** The proposed \"Penalty Critic\" for SP-RL  is mathematically well-motivated and serves as a valid strategy to re-introduce gradient information that is otherwise lost during projection.\n\n### 3. Weaknesses\n\n- **High Computational Cost:** As the authors acknowledge, the improved SP-RL methods incur a significant computational overhead ($2-10 \\times$) compared to SE-RL due to the need for differentiating through the optimization layer. This substantial cost somewhat diminishes the practical appeal of the proposed method, especially since the performance gains are not always drastic.\n\n- **Mixed Experimental Results:** The empirical advantage of the proposed SP-RL method is not consistent across all tasks. For instance, in the Seeker environment, the simpler SE-RL baseline (with penalties) performs comparably to or even better than the more complex SP-RL variants. This suggests that the \"Penalty Critic\" is not universally superior and may depend heavily on the alignment between safety constraints and task objectives.\n\n- **Limited Evaluation Scope:** The experiments are limited to relatively low-dimensional tasks (Pendulum, Quadrotor, Point-mass Seeker). It remains unclear how the \"rank-deficient Jacobian\" issue and the proposed remedy scale to high-dimensional state/action spaces where action aliasing might be more severe and the computational cost of projection layers would be even more prohibitive.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper's theoretical claims are supported by rigorous mathematical derivations. Specifically, the proof of Theorem 1, which establishes the equivalence of optimal value functions between SE-RL and SP-RL, is sound. The subsequent analysis identifying the distinct mechanisms of \"action aliasing\" in both frameworks — characterizing them as \"flat-lining critics\" in SE-RL and \"rank-deficient Jacobians\" in SP-RL — is logically convincing and well-supported by the formal definitions.\n\nRegarding the empirical claims, the authors provide clear evidence for the effectiveness of their proposed \"Penalty Critic\" strategy. The experiments on the Pendulum and Quadrotor tasks  convincingly demonstrate that the proposed method helps recover performance in SP-RL settings. Furthermore, the evidence is presented accurately and transparently: the authors candidly report that in the \"Seeker\" navigation task, the simpler SE-RL baseline remains competitive. This inclusion of mixed results enhances the credibility of the submission, as it avoids overclaiming the universality of the method and instead provides a nuanced view of performance across different task types.", "audience_yn": "Yes", "audience_explanation": "Safe Reinforcement Learning is a significant and active area of research within the TMLR community. This paper addresses a fundamental practical question: whether to integrate safety filters into the policy or the environment. The theoretical insights regarding \"action aliasing\" and the comparative analysis of failure modes (flat-lining critic vs. rank-deficient Jacobians) provide valuable guidance for researchers and practitioners working on constrained optimization and safety-critical control.", "requested_changes": "### \n\n- **Expanded Discussion on Computational Trade-offs (Critical):** While the paper acknowledges a $2-10\\times$ computational overhead for improved SP-RL methods due to sensitivity computations, the current discussion could be more prescriptive for practitioners. I request the authors add a paragraph to the Discussion or Conclusion section that explicitly guides readers on when the performance gains of SP-RL justify this cost. Specifically, under what task characteristics (e.g., safety-criticality vs. training speed, state-space complexity) should a practitioner prefer the computationally expensive SP-RL over the robust and cheaper SE-RL baseline?\n- **Wall-Clock Time Analysis (Critical):** The paper mentions that detailed wall-clock analysis was excluded due to software-specific variations. However, providing even a rough comparison (e.g., a simple table showing average seconds per training step for one environment) would significantly strengthen the evidence regarding the computational cost mentioned above. This would give readers a concrete sense of the \"price\" of using differentiable projection layers.\n- **Clarity on \"Penalty Critic\" Implementation (strengthen):** It would be beneficial to slightly expand the description of the \"Penalty Critic\" implementation details in the main text or appendix to ensure full reproducibility, specifically regarding how the penalty scale hyperparameter ($w$) interacts with the critic's loss function stability during the early phases of training.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)