Breaking Safety Paradox with Feasible Dual Policy Iteration
Authors: Yujie Yang
Jinglin Teh
Ziyu Lin
Kaicheng Yu
Tao Zhang
Shengbo Eben Li
Venue: ICLR 2026 Poster (2026)
Keywords: Safety paradox
safe reinforcement learning
feasible policy iteration
feasibility function
Primary area: reinforcement learning
TL;DR: Introduces an additional violation-seeking policy to overcome sparsity of unsafe samples.

Abstract: Achieving zero constraint violations in safe reinforcement learning poses a significant challenge. We discover a key obstacle called the safety paradox, where improving policy safety reduces the frequency of constraint-violating samples, thereby impairing feasibility function estimation and ultimately undermining policy safety. We theoretically prove that the estimation error bound of the feasibility function increases as the proportion of violating samples decreases. To overcome the safety paradox, we propose an algorithm called feasible dual policy iteration (FDPI), which employs an additional policy to strategically maximize constraint violations while staying close to the original policy. Samples from both policies are combined for training, with data distribution corrected by importance sampling. Extensive experiments show FDPI's state-of-the-art performance on the Safety-Gymnasium benchmark, achieving the lowest violation and competitive-to-best return simultaneously.

Published as a conference paper at ICLR 2026
BREAKING SAFETY PARADOX WITH FEASIBLE DUAL POLICY ITERATION
Yujie Yang"), , Jinglin Teh Ziyu Lin', Kaicheng Yu², Tao Zhang 1.3 Shengbo Eben Lit 'Tsinghua University, "Westlake University, *SunRising Lab (yangyj21, zhengi124] a Emails tsinghua edu. en, 1in ziyu@cau. edu en, kyu@westlake, edu. en, zhang. t 1998899m311111 com, 11 wwwssshottssigghhe edu. .n
ABSTRACT
Achieving zero constraint violations in safe reinforcement learning poses a sig- nificant challenge. We discover key obstacle called the safety paradox, where improving policy safety reduces the frequency of constraint-violating samples, thereby impairing feasibility function estimation and ultimately undermining pol- icy safety. We theoretically prove that the estimation error bound of the feasibility function increases the proportion of violating samples decreases. To overcome the safety paradox, we propose an algorithm called feasible dual policy iteration (FDPI), which employs an additional policy strategically maximize constraint violations while staying close to the original policy. Samples from both policies are combined for training, with data distribution corrected by importance sam- pling. Extensive experiments show FDPI's state-of-the-ee performance on the Lafety-Gymnasiun benchmark, achieving the lowest violation and competitive- to-best return simultaneously.
INTRODUCTION
Reinforcement learning (RL) has achieved promising performance in many challenging tasks such video games ( (Vinyals al. 2019), board games (Schrittwieser et al.) 2020), autonomous driv- ing (Wurman et al. 2022), and drone racing (Kaufmann et al. 2023). RL solves an optimal control problem by finding policy that maximizes the expected sumulative rewards. However, real-world control tasks often demand more than reward maximization- they require strict adherence to safety constraints, where violations in these tasks remains significant challenge.
rare violations lead to catastrophic outcomes. Achieving zero constraint
key element in safe iL the feasibility function, which evaluates whether state can satisfy safety constraints over an infinite horizon. This function not only defines the feasible region of a policy but also serves
safety-oriented learning target. Examples of feasibility functions include cost value function (CVF) (Altman 202T). Hamilton-Jacol (HJ) reachability function (Bansal et al.) (017), and constraint decay function (CDF) (Yang et al., 2023b). These functions are typically learned through fixed-point iteration based on their risky self-consistency conditions (Li 2023 Yang et al.) 2024)-analogou to the Bellman equation for value functions. These conditions establish recursive relationships between temporally adjacent states, allowing feasibility functions to capture long-term
safety at all states.
While learning- based feasibility functions are crucial in ensuring safety, discover that they in- herently prevent policies from achieving zero violations due to a phenomenon we term the safety paradox. Our analysis reveals that policy safety improves and violating samples become sparser, the estimation error of the feasibility function increases. This impairs the accuracy of the feasible region and introduces bias into the policy's learning target, ultimately undermining safety perfor- mance. This phenomenon differs fundamentally from the sparse reward problem in standard RL, where achieving higher rewards directly facilitates further reward improvement. In contrast, the safety paradox forms a self-defeating cycle where improving safety degrades the learning condi- tions for further safety optimization.
"Equal contribution
Corresponding author

---

Published as a conference paper at ICLR 2026
Existing methods for addressing sample sparsity, which categorize as passive and active, inadequate for resolving the safety paradox. Passive methods such as prioritized experience replay (PER) (Schaul et al. 2013) reweight samples in the replay buffer emphasize critical transitions. However, their efficacy is limited when critical samples are inherently rare, and they fail to break the safety paradox's s self-defeating cycle. Active methods such curiosity-driven exploration (Pathak) et al. (2017) modify the environment or agent behavior to generate critical samples. While poten- tially more effective, these methods induce behavioral shifts that can steer the policy away from optimality, and their implementation often requires intrusive task modifications, which may not be feasible practice.
In this paper, we propose algorithm called feasible dual policy iteration (FDPI), which breaks the safety paradox by incorporating an additional dual policy designed to maximize constraint vio- lations. This approach effectively increases the proportion of constraint-violatiin samples without increasing the total number of samples, thereby reducing feasibility function estimation error and pushing policy safety to higher level. A challenge of this approach is the distributional shift that occurs when combining data from both policies. We address this through an importance sampling (IS) scheme that approximates the marginal state distribution with truncated trajectory distribu- tion. We further introduce KL divergence constraints between the two policies to ensure numerical stability of IS. Extensive experiments on the Sefety-Gymmasium benchmark demonstrate FDP['s state-of-the-art performance.
2 RELATED WORK
Safe RL algorithms A prominent class safe RL algorithms called iterative unconstrained RL, which transforms the safe RL problem into series of unconstrained RL problems, typically via the method of Lagrange multipliers (Paternain et al. 2019). Under this framework, researchers explored different kinds of feasibility functions. including CVF (Chow al. 2018 Tessler al., 2018). HJ reachability function (Yu et al., 2022 2023), and control barrier function (Yang et al. 2023a b).
Another class is called constrained policy optimization, which incorporates safety constraints in each iteration of policy optimization. The most representative example is CPO (Achiam et al., 2017), which adopts a trust region update with linearized objective and constraints. Several improvements over CPO have been proposed, including projection methods (Yang et al. 2020; 2022) and first-order methods (Zhang al., 22200) 2022). For Enite-horizon problems, Zhhoo et al. (2023) and (Zhao et al.) (2024) convert state-wise constraints to cumulative constraints and bound the worst-case violation. A common practice of these algorithms is to estimate feasibility functions from sampled data.
Critical sample augmentation in RL There are two kinds of methods to increase critical samples in RI.: passive methods and active methods. Passive methods focus on biasing the replaying process to prioritize experiences that are likely to be more informative for learning. A representative exam- ple is PER (Schaul et al., 2015), which replays samples with larger temporal difference (TD) errors more frequently. Other methods include prioritizing similar experiences the current policy (No- vati & Koumoutsakos 2019) and modifying certain information in replayed samples (Andrychowicz al. (2017). Active methods involve modifying the environment or the agent's behavior deliber- ately generate critical samples. Some algorithms employ adversary policies to generate challenging scenarios (Pinto et al., 2017 Feng et al.) 2023), , while others use auxiliary rewards to guide explo- ration (Jaderberg et al. 2016, Pathak et al. 2017). Unlike these methods, our algorithm requires no environment modifications or reward shaping, maintaining the integrity of the original task. There are also methods that address the sample sparsity challenge through representation learning, which are discussed in Appendix D.I
PROBLEM STATEMENT
Safe RL addresses control problems which an agent aims to maximize long-term rewards while strictly adhering to safety constraints at every step. We consider Markov decision process (MDP) (X, 24, Punt P. r. r), where X' C R" is the state space, 24 C I is the action space, Pinit E AX is the initial state distribution, P X x 2 + AX is the transition probability,
X' x U + RR is the reward function, and 0 < 7 < 1 is the discount factor. We consider a stochastic policy π X - Au,
whose state-value function is defined V* (xx) E 1o (r( ut) Iz x/. Safety is specified
2

---

Published as a conference paper at ICLR 2026
through a state constraint expressed as inequality h(()) 0, where h X - R is the constraint function. The state constraint must be satisfied at every step:
h(x))) I O, 14 2 0.
The goal of safe RL is to find policy that maximizes the state-value function while satisfying the state constraints over an infinite horizon.
3.1 FEASIBILITY IN SAFE REINFORCEMENT LEARNING
Feasibility is a core concept in safe RL that describes the long-term safety of a state. To formally define feasibility, we first introduce the reachable set. Definition (Reachable set). The reachable set of policy ι from state x E X, denoted R (I), is the set of states that can be reached with non-zero probability under π in finite time:
(²)x) = fa's E X 2 O, s.t. P(z, = x' [x, σ) > 0}.
where P(x+ = 12, πT is the probability of reaching x' at time t starting from and following л.
The reachable set includes all states that will possibly be visited by π given an initial state. Feasibil- ity is defined based whether all states in the reachable set is constraint-sane -aatsiiin...tiiyiinng...
Definition 2 (Feasible region). The feasible region of a policy π. denoted X", is the set of states from which every reachable set under satisfies the safety constraint:
x== E X(v2'' E (R),)) 0}
In safe RI., we need to find a policy whose feasible region includes all possible initial states. This requirement can be expressed as single constraint by the feasibility function.
Definition 3 (Feasibility function). Function F + R is a feasibility function of a if its zero-
sublevel set equals the feasible region of i.e., {x X (x) S 0} X".
An example of a feasibility function is the CDF (Yang et al. Definition (Constraint decay function). The CDF of a policy n is defined as
2023b
F.t (x) EE [2))
(4)
where 7 E (0. 1) is the discount factor, ( U1 11 u1.... 3 a trajectory sampled by n, and N(T) E NN (+oo) is the time step of the first constraint violation in T.
In the rest of this paper, we use CDF as a concrete example of feasibility function. However, analysis also applies other feasibility functions with similar properties such CVE. The feasibility function is also called a constraint aggregation function (Yang et al., (2024) because we [I] with single expressed by the feasibility
replace the original infinitely many constraints function, leading to the following safe RL problem:
max Egaepan [V*(x) s.t. Ee [F" (=) 5.
(5)
4 SAFETY PARADOX
A core problem in safe RL is to estimate the feasibility function. We discover that as the policy becomes safer, the estimation error of the feasibility function tends increase. This makes the identified feasible region less accurate, which, in turn harms policy update and deteriorates policy safety. This phenomenon is called the safety paradox.
4.1 ESTIMATION ERROR BOUND OF CDF
In safe RL, the CDF is computed by solving its risky self-consistency condition with fixed- point iteration (Yang et al. 2023b):
F (x) (ww. (c(x) + ((x)))FF(3))
3

---

Published as a conference paper at ICLR 2026
where c(z) [[{(x)) 0] is an indicator function for constraint violation. In practice, the expec- tation above is estimated by sample average. Equation (6) can be viewed as a one-step TD estimate of the CDF. Since TD involves bootstrapping of the estimated CDF itself, the analysis of estimation error becomes complicated. Here, we consider a Monte Carlo (MC) estimate instead for theoret- ical simplicity: F" 1/K -iKE starting from x sampled by и. We discuss extension TD estimate at the end of Section 44.
-i=1 (++),, where 72.72.....77 are K independent trajectories
An inaccurate CDF leads to incorrect identification of the feasible region, i.e., feasible states misidentified as infeasible and vice versa. To minimize misidentification, we must bound the es- timation error. We show that the bound of the relative estimation error of CDF is related to the expectation and variance of the number of steps to the first violation. Before that, we assume that these two quantities are finite.
Assumption 1. For any infeasible state I E X under policy let 13)()) = [ = ON
and 2, (x) (Nr)=a x/. We have uz)(x) < +o0 and o 2.t
N (x:) < +oc.
Theorem 1. For any infeasible state x E X under policy x. let F (x) be the MC estimate of the CDF Under Assumption 7 the expected relative estimation error is bounded by:
F (x) - (x) (T(
N 2. (x) 1 [nnlo + (Inn)² VK
1
E Pamort
(7)
7/3((0)
Sketch. Sketch. Construct two auxiliary functions (² (0 and (xT (x) where (x) = 1/K N(r,). Use Taylor expansion to obtain the bounds of |FT (x) H (x)].
(x)()) (or)-Fro (x)), and (x) (x)). The result follows by the triangle inequality. See Appendix
A.I for the complete proof.
The number of samples K in the error bound 7 is related to the batch size and is constant through- out training. The only two variables relevant to the error bound is the expectation and variance of steps to violation. While is obvious that the expectation increases as the policy becomes safer, how the variance changes is not easily observed and requires further analysis.
4.2 RELATIONSHIP BETWEEN POLICY SAFETY AND ESTIMATION ERROR BOUND
In this section, we show that under mild assumptions, the variance of steps to violation increases as the policy becomes safer. To begin with, we introduce function measure the "distance" to constraint violation.
Assumption 2. There exists a continuous function D X + R. such that Vx E X, D(x) 2 0 and D(x) = 0 + h(r)) > O.
Examples of D include the Fuclidean distance to obstacles in collision avoidance tasks, and the margin to speed limit in velocity-constraine tasks. With distance function, we can define what "safer" policy is.
Definition 5. A policy n' is safer than a policy a state x if P(D(4') D(+)) s P(D(2)) D(x)), where x²= Ps (-Ix) is the next state under π.
A safer policy has lower probability of decreasing the distance to violation. This definition natu- rally connects to the proportion of violating samples: if the distance violation decreases slower, the number of steps to violation will be larger, resulting in fewer violating samples. The following assumption requires the continuity of D and the system dynamics. This holds in most systems, where the changing rate states and the control inputs are bounded.
Assumption 3. The difference of D between any two adjacent states is bounded by 0, i.e., for any x, x' such that p(æ' [r, u) > 0. it holds that D(z) - D(x')l < 8.
We split (xx) into a sequence of consecutive intervals: [D x 18, D(x) (i - 1)8),
1, 1,,,.... m(x) 11, where m(x)= DD= With Assumption 3 the agent must visit the (i 1)th interval before visiting the ith interval. We denote N (z) as the first-visiting time from the (i (i- 1)th interval to the ith interval. Note that ((x) is not necessarily monotonie, i.e., the agent may revisit
4

---

Published as a conference paper at ICLR 2026
stage where it has visited before. NT is counted until the first time the next stage is visited. The following assumption requires that for states far from violation, NN (z) is weakly correlated with the sum of the rest first-visiting times.
Assumption 4. There exists M > O, such that for all a with m(x) 2 M, it holds that Cov (NT Em)) NT S (x)(22
The intuition behind this assumption is that for a state far from violation, the initial step NT (x) primarily influenced by local dynamics. contrast, the subsequent trajectory ΣmEh Ng (x) is governed by future stochastic events. Their correlation is weak relative to the variability of the initial step itself. With the above assumptions, we have the following theorem.
Theorem 2. Let TT, x' be two policies. Consider set of states XM C X such that for all I E XM. (1) x is infeasible under both π and n', (2) m(x)= 2 M, (3) min(((((xz)) < D())), P(D(x")) <
D(x))} 2 0.5. Under Assumptions if is safer than π. in all states in X we have o 2,5 2.
N (z) 2
NN (1r), Vx E N--
Proof Sketch. First prove that for given policy, state further to violation has larger variance by stage decomposition. Then, prove that in a given state, a safer policy has a larger variance by law of total variance. Finally, extending this result to all states in XM proves the theorem. See Appendix A.2 for the complete proof.
Remark The condition (3) in Theorem follows from the requirement of finite mean and variance. To ensure the violation happens in finite steps, the probability of approaching the violation, i.e., d(x') d(z), must be larger than that of moving away from it in each step. Theorem 2 tells us that a safer policy (in the sense of lower probability of approaching violation) has larger variance of steps to violation in infeasible states far from violation under mild assumptions. Combined with Theorem
1 we conclude that safer policy leads to a larger CDF estimation error bound. To help better understand this result, we give an intuitive example of a one-dimensional random walk in Appendix A.3 It is worth mentioning that although the above analysis is based on MC estimate, it can be extended to TD estimate. The core reason is that the increased variance identified in Theorem 2 propagates to the TD target. The analysis can also be extended the CVF widely used in the constrained MDP. A detailed explanation of the extensions can be found in Appendices D.2
FEASIBLE DUAL POLICY ITERATION
The only way break the safety paradox is increase constraint-violating samples. We achieve this by training an additional policy, called the dual policy, that intentionally violates the constraint.
5.1 COLLECTING MORE VIOLATING SAMPLES WITH A DUAL POLICY
We denote the dual policy as va, and for distinguishing purposes, we call the original policy the primal policy and denote Tp. We train dual feasibility function Ga(ar, u) = (x-(r) Jm0 &, 14g u| to help optimize the dual policy. Compared with Fa, Ga further fixes the current action un and computes the feasibility value starting from the state-action pair (z, u). The dual policy is updated by maximizing the dual feasibility function:
max Ex,ux [Ga u)].
Both the primal policy and the dual policy are used to sample data. In practice, the sampling ratio is controlled by a hyperparameter called the dual threshold d, which is fixed at 0.95 in our experiments. If the rumning averaged proportion of feasible states is greater than d, the dual policy will be activated and collect half of the samples. We ensure that the total number of samples used by our algorithm, i..., collected by both the primal and dual policies, equals that of other algorithms.
5.2 CORRECTING SAMPLE DISTRIBUTION VIA IMPORTANCE SAMPLING
The problem of directly using data collected by the dual policy is that the data distribution for computing the expectation in loss functions is shifted. Take the loss function of the primal feasibility
5

---

Published as a conference paper at ICLR 2026
function an example:
G. ".ggp"",
(c(o) + C1-c(=))GG((t v())
Among the four random variables involved in the expectation, and have shifted distributions because some of their samples are collected by the dual policy. To solve this problem, we use IS correct their distribution. The IS ratio for u can be readily computed by the ratio of probabilities under two policies. The IS ratio for z involves the marginal state distribution under policy, which can be computed as: (x) = Σi= P( = z/T) Ei= E, I[2 = xT)" where (**(r) is the probability of trajectory under ma:
(T) = Pinit (zo) P(ri un ma) ... Pinit (ro)
II Ta( (zx) P(rtt1lla ut). t=
To correct the distribution of from n to ph, we need to insert an IS ratio pp)(x) (=)( into the loss function. However, directly computing the ratio intractable because we cannot sum over all possible trajectories. Instead, we approximate the ratio with single trajectory T that con- tains x:
(T) (T)
o Tp (u (ate) Ta uu (
t(x))
Tp mp(ualtrs = fpa())) Ta (a(uul1z) (x,)
(p)(x) RM
where the second approximation operation truncates the product up to t(x), which is the appearing step of x. This is because future actions beyond t(x) do not affect the probability reaching x, and truncating them reduces the variance of IS. To additionally account for the distribution shift of u, we define the approximated IS ratio for a state-action pair as (z, u) = (()l)))) The approximated IS ratios from the primal policy to the dual policy, (x)( and fap u), can be defined similarly.
The sequential multiplication in the IS ratio can easily cause numerical underflow since the proba- bility of an action under another policy is usually lower than that under the behavior policy. We find that this problem can be effectively alleviated by constraining the KL divergence between the two policies. Observe that
Ta (u|z) Tp (u|z)
Tp(u(zr) Taa [1(=)))
DKL
(Tal [p)](]] = E.
log
<
(
log
2 -.
The KL divergence constraint ensures that the expected logarithm of each term in the product (8) is not too small, thus preventing the IS ratio from collapsing to zero.
5.3 OVERALL ALGORITHM
Our algorithm, called feasible dual policy iteration (FDPI). follows the framework of feasible policy iteration (FPI) proposed by Yang et al. (2023c). On the basis of FPI, we combine the maximum entropy RL method from soft actor-critic (SAC) (Haarnoja et al. 2018 and name the resulting al- gorithm as SAC-FDPI. Our algorithm learns two action-feasibility networks Gp.,,= Ga. an action- value network Qw, and two policy networks p. Ta,8a, where Φ. w. and denote their parameters. We additionally introduce a hyperparameter e > and approximate feasibility by Gø(a, u) < c. This is because, in practice, approximation error causes the CDF to be positive almost everywhere since learning target non-negative. This approximation is valid under the assumption that the steps to violation is uniformly bounded (Thomas et al. (02T). In our experiments, we find that fixed value of = 0.1 works well for all environments.
The loss functions for the action-feasibility networks are
LG (x+) =E [ u) (G+ (2, u) - yau (2, u())
(9)
where "#" stands for "p" "d", fz, u) = i İ2)( u) for (æ, u) sampled by πa. and equals 1 for (I,, u) sampled by Tp. falæ, u) is defined similarly, and
yG. (,, u') c(x) + (1 c(x))) (4', w')

---

Published as a conference paper at ICLR 2026
where ø denote the parameters of the target networks. The loss functions for the action value net- works are
LQ(w) E (p(x, u) (Qw u) uQ u')) v'))
(10)
(2 x', u') 1 r(z, u) (Qw ud) a log Tp [=))),
where a is a learnable parameter for entropy temperature. The primal policy network is updated by maximizing the action-value network inside the feasible region and minimizing the primal action- feasibility network outside the feasible region. The loss function is as follows:
mm (8p) E (x) (Irfa, 2/ (a log p, (u(x) (w,
(1 If
(11)
where (z, u| = 1 if Gp.o, (I, u) < and 0 otherwise. The dual policy is updated by maximizing the dual featon-feasibility network under the constraints of KL divergence. The constraints addressed by the Lagrange multiplier method. The loss function for the dual policy is as follows:
Ling (0a) [a( (z, u)] + DKL (Ta, ou Tp.,,) Dxp (TT.,.))
(12)
The Lagrange multipliers Aap and Adp are updated by:
>ap + (>ap + (Fa - 4) Apd (Apa + (DKL (1 -6)) (13)
where is the learning rate and (-)+ denotes the projection to Rzo. The pseudocode of our algorithm
Appendix
EXPERIMENTS
We aim answer the following questions through our experiments:
Q1 How does SAC-FDPI perform in terms of safety and return compared to existing algorithms?
Q2 Does learning an additional dual policy help increase violating samples?
Q3 Does the estimation error of the feasibility function decrease with more violating samples?
6.1 EXPERIMENT SETUPS
Our experiments cover 14 environments in the Safety-Gymnasium benchmark [Ji et al.) 2023a). including navigation and locomotion. The navigation enviromments include two robots. i.c., Point and Car, and four tasks, i.c. Goal, Push, Button, and Circle, with all difficulty levels set as I and constraints set as default. The locomotion environments include six classic robots from Gymnasium's 's MuJoCo environments, i.e., HalfCheetah, Hopper, Swimmer, Walker2d, Ant, and Humanoid, with maximum velocity constraints.
Environments
Baselines We compare algorithm with wide variety of mainstream safe RL algorithms im- plemented in the Omnisafe toolbox (ji al.) 2023b), including iterative unconstrained RL algo- rithms RCPO (Tessler al. 2018). PPO-Lag (Ray et al. 2019). and SAC-Lag (Ha et al.] 2021). and constrained policy optimization algorithms
0.7
+
0.5
Nomelized
CPO
RCPO
CPO (Achiam et al., 2017), FOCOPS (Zhng et al.
FOCOPS
0.3
CUP
2020 and CUP (Yang et al. 2022). addition, we combine the state-of-the-an thet-rrt unconstrained RL algo- rithm DSAC-T (Duan et al. 2023) with the penalty method and name the resulting algorithm DSAC-T- Pen. We also include version of our algorithm without the dual policy, named SAC-FPI. Hyperpa- rameters for all algorithms are detailed in Appendix
PPO-Lag
0.2
SAC-Lag
DSAG-T-Pan
SAC-FPI
10
10" Nomalized
C.1
Figure 1: Normalized cost-return plot. Error bars represent 95% confidence intervals.

---

Published as a conference paper at ICLR 2026
6.2 EXPERIMENT RESULTS
Cost-return performance In safe RL, we evaluate algorithms by two metries: (1) episode cost, representing the average number of constraint-violatiin steps per episode, and (2) episode return, representing the average cumulative rewards per episode. To perform comprehensive evaluation, we place the scores of all algorithms in a cost-return plot in Figure The scores are first normalized by those of PPO and then averaged all 14 environments. SAC-FDPI simultaneously achieves the lowest cost and an almost tied highest return (with SAC-FPI) among all algorithms, demonstrating its superior performance. We further plot the training curves SAC-FDPI and six baselines across eight environments in Figure 2 with remaining results in Appendix
C.2 SAC-FDPI achieves near- zero constraint violations on enviromments while maintaining comparable or higher returns than the baselines. Notably, SAC-FPI exhibits persistent cost spikes even at final training stages, while SAC-FDPI maintains near-zero violations by continually feeding controlled trickle of unsafe tran- sitions through its dual policy. These results answer Q1.
025
675
1.0
0.5
6:5
0.5
675
125
1.5
0.5
615
6.5
815
1.5
180
Diviemment 190
Divivement
RCPO
POCOPS
DUP
SAG-Lag
DSAD-TTPPn
SAC-FFI
SAC-PORI
Figure 2: Training curves on eight environments in Safety- Sammay-Gymmnsium benchmark. The shaded areas represent 95% confidence intervals over 5 seeds.
Proportion of violating samples Figure 3 compares the average proportion of violating samples in the replay buffer during the final 10% of training iterations for SAC-FPI and SAC-FDPI. It shows that SAC-FDPI maintains a significantly higher proportion of violating samples than SAC-FPI- an order of magnitude greater in most environments. SAC-FPI's violation ratio falls below 1% in nearly all environments, undermining the accuracy of its feasibility function and leading to the cost spikes observed in its training curves. In contrast, SAC-FDPI's dual policy mechanism ensures persistent availability of proper number of violating samples, which answers Q2. It is worth mentioning that our method is designed for training simulator instead of the real world, and thus additional violation does not cause real damage. A detailed discussion can be found in Appendix D.3
8

---

Published as a conference paper at ICLR 2026
SAC-FORI SAC-FPI
Vicleton semple raño
0.125
0.100
0.075
0.050
0.025
0.000
CarGrdd
PainiClecle
Hopervabbciyy
Hett Veloody
Walker Eatabyyyy
CarPaali
CarGess
PointGoa
PaisPuat
Antiaboily
PoiiButton
SwinmerValieh
GarBation
Humanaistattch
Figure 3: Average proportion of violating samples in the replay buffer. The error bars represent 95% confidence intervals over 5 seeds.
MAOVER
127
12
G
Figure 4: Distribution of feasibility function estimation error after convergence.
Accuracy of fcasibility function Figure comparres the estimation error of the feasibility function learned by SAC-FPI (blue) and SAC-FDPT (orange) in four environments. The on 100k states collected by the two policies after convergence. The true values of the fcasibility function are computed by definition on collected trajectories. shows that SAC-FDPI produces sharp, symmetric peak tightly centered at zero, which is an evidence of low bias and estimation enror. In contrast, SAC-FPI exhibits flatter, more dispersed errors, reflecting the inflated estimation error bound caused by vanishing violating samples under the safety paradox. An empirical evidence of the relationship between violating samples and estimation error can be seen by combining Figures 3 and 4. Figure 3 shows that SAC-FDPI maintains about 10x more violating samples compared to SAC- FPI in most environments. This richer violation data directly leads to significantly lower estimation errors shown in Figure 4. These results support our theoretical analysis that richer violation data leads to better feasibility estimation, answering Q3.
are computed
Exploration patterns To understand how the dual policy specifically helps collect violating samples, we visualize the trajectories of the pri- mal policy and dual policy in Figure 5 The primal policy conservatively steers around the hazards that no violation incurred. In con- trast, the dual policy augments the samples col- lected by the primal policy by deliberately cut- ting through the hazards, injecting constraint- violating samples while staying close to the pri- mal policy. This richer mixture of safe and unsafe data keeps the feasibility function well- estimated, allowing the primal policy to verge to higher performance and with lower constraint violations.
(a) CarGoal
(b) PointGoal
Figure 5: Trajectories of primal policy (red) and dual policy (cyan). Blue disks mark hazards, and green cylinder denotes the goal.

---

Published as a conference paper at ICLR 2026
7 CONCLUSION
This paper discovers a fundamental obstacle in safe RL. called the safety paradox, where improved policy safety leads to increased estimation enror bound of the feasibility function, and ultimately harms policy safety. To address this paradox, we propose FDPI, which introduces a dual policy maximize constraint violations while staying close the primal policy through KL divergence constraints. We incorporate IS to correct distribution shifts between two policies. Extensive ex- periments on Seffety-Gymmasium show that FDPI significantly increases violating samples, reduces feasibility function estimation error, and achieves state-of- of-th---tt performance.
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning. pp. 22-31. PMLR, 2017.
Eitan Altman. Constrained Markov decision processes. Routledge, 2021.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. Advances in neural information processing systems, 30, 2017.
Somil Bansal, Mo Chen, Sylvia Herbert, and Claire J Tomlin. Hamilton-jacobi reachability: A brief overview and recent advances. In 2017 IEEE 56th Annual Conference on Decision and Control (CDC). pp. 2242-2253. IEEE, 2017.
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforeement learning with stability guarantees. Advances in neural information processing sys- tems, 30, 2017.
Zhepeng Cen, Yihang Yao, Zuxin Liu, and Ding Zhao. Feasibility consistent representation learning for safe reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning, volume 235 Proceedings of Muchine Learning Research, pp. 6002-6019. PMLR, 21-27 Jul 2024.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained rein- forcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(167); 1-51, 2018.
Jingliang Duan, Wenxuan Wang, Liming Xiao, Jiaxin Gao, and Shengho Eben Li. Dsac-t: Distribu- tional soft actor-critic with three refinements. arXiv preprint arXiv:2310.05858, 2023.
Shuo Feng, Haowei Sun, Xintao Yan, Haojie Zhu, Zhengxia Zou, Shengyin Shen, and Henry X Liu. Dense reinforcement learning for safety validation of autonomous vehicles. Nature, 615(7953): 620-627, 2023.
Sehoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, and Jie Tan. Learning to walk in the real world with minimal human effort. In Conference on Robot Leurning, pp. 1110-1120. PMLR, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning, pp. 1861-1870. Pmlr, 2018.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv 2016.
preprint ar 161
Jaminn Ji, Borong Zhang, Jiayi Zhou, Xuchai Pan. Weidong Huang, Ruiyang Sun, Yiran Geng, Yi- fan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforcement learn- ing benchmark. Advances in Neural Information Processing Systems, 1et11886449999 2023a.
Jaaming Ji. Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun. Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. arXiv preprint 200.::2300000004,,. 2023b.
10

---

Published as a conference paper at ICLR 2026
Elia Kaufmmann Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620(7976):922-9 9927987, 2023.
Shengbo Eben Li. Reinforcement learning for sequential decision and optimal control. Springer Verlag, Singapore, 2023.
Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior-point policy optimization under constraints. the AAAI conference on artificial intelligence, volume 34, pp. 4940-494449 2020.
In Proceedings the
Guido Novati and Petros Koumoutsakos. Remember and forget for experience replay. In Interna- tional Conference on Machine Learning, pp. 4851-4860. PMLR, 2019.
Santiago Paternain, Luiz Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained re- inforcement learning has zero duality gap. Advances in Neural Information Processing Systems, 32, 2019.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778 2787. PMLR, 2017.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In International conference on machine learning, pp. 2817-2826. PMLR, 2017.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708, 7(1):2, 2019.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay, arXiv preprint arXiv:1511.0000 1.05952, 2015.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-669 2020.
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv: 1rXhv:1805.11074 2018.
Garrett Thomas, Yuping Luo, and Tengyu Ma. Safe reinforcement learning by imagining the near future. Advances in Neural Information Processing Systems, 141138899-1866) 2021.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforeement learning. Nature, 575(7782)350--35, 2019.
Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Out- racing champion gran turismo drivers with deep reinforcement learning. Nature, 602(7896):223-- 228, 2022.
Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning, pp- 11490-11491. PMLR, 2021.
Long Yang. Jiaming Ji. Juntao Dai, Linrui Zhang, Binbin Zhou, Pengfei Li. Yaodong Yang, and Gang Pan. Constrained update projection approach to safe policy optimization. Advances in Neural Information Processing Systems, 35:9111-9124 2022.
Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-based constrained policy optimization. arXiv preprint arXiv:2010.03152. 2020.
Yujie Yang, Yuxuan Jiang, Yichen Liu, Jianyu Chen, and Shengbo Eben Li. Model-free safe rein- forcement learning through neural barrier certificate. IEEE Robotics and Automation Letters, 13)-1295-1302, 2023a.
11

---

Published as a conference paper at ICLR 2026
Yujie Yang, Yuhang Zhang, Wenjun Zou, Jianyu Chen, Yuming Yin, and Shengbo Eben Li. Syn- thesizing control barrier functions with feasible region iteration for safe reinforcement learning. IEEE Transactions Automatic Control, 69(4):2713-27200 2023b.
Yujie Yang, Zhilong Zheng, Shengbo Eben Li, Jingliang Duan, Jingjing Liu, Xianyuan Zhan, and Ya-Qin Zhang. Feasible policy iteration. arXiv preprint arXiv:2304. 08845, 2023c.
Yujie Yang, Zhilong Zheng, Shengbo Eben Li, Masayoshi Tomizuka, and Changliu Liu. The feasibility of constrained reinforcement learning algorithms: tutorial study. arXiv preprint 20l::2404.10044, 2024.
Dongjie Yu, Haitong Ma, Shengho Li, and Jianyu Chen. Reachability constrained reinforcement learning. International conference on machine learning, pp. 25636-25655. PMLR, 2022.
Dongjie Yu, Wenjun Zou, Yujie Yang, Haitong Ma, Shengbo Eben Li, Yuming Yin, Jianyu Chen, and Jingliang Duan. Safe model-based reinforcement learning with an uncertainty-awar reachability certificate. IEEE Transactions on Automation Science and Engineering, 21(3):412-4442 2023.
Linrui Zhang. Li Shen, Long Yang, Shixiang Chen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Penalized proximal policy optimization for safe reinforcement learning. arXiv preprint arXiv:2205. 11814, 2022.
Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 13:15338-15349 2020.
Weiye Zhao, Rui Chen, Yifan Sun, Tianhao Wei, and Changliu Liu. State-wise constrained policy optimization. arXiv preprint arXiv:2306. /2594, 2023.
Weiye Zhao, Feihan Li, Yifan Sun, Yujie Wang. Rui Chen, Tianhao Wei, and Changliu Liu. Absolute state-wise constrained policy optimization: High-probability state-wise constraints satisfaction. arXiv preprint 2200022410.0121,, 2024.
12

---

Published as a conference paper at ICLR 2026
A PROOFS
A.1 PROOF OF CDF ERROR BOUND
Theorem 1. For any infeasible state E X under policy let (x) be the MC estimate of the CDE Under Assumption
7 the expected relative estimation error is bounded by:
FT (()))FF
- (1)
1
In ll (2) + (In 7) 2x (x) K
E T2..
(7)
(T)
Proof. Construct two auxiliary functions
(x) g(3(())
11 f1*(x) = -P3(())
where
(M = Σ N(T).
Perform second-order Taylor expansion with the Lagrange remainder to (x):
(x) ( + (+))())) In (x) - (x(( + 2 (())
1 M
where M is a point between (x) and 23 (z). Thus,
(x) (=)) (
In (x)( (x(x))
2 3 (1n7) (z)()))
1
S ( Iny (AT xx)())) + 2 7
M (ln>) (1)((())))))))))))))))))()) - NN (x))
() ynllA (x) (x) ( (z) (o)
where the last inequality holds because TM < Since the squared function (-)2 is a convex function, by Jensen's 's inequality, have
[x) (x)) S (x)()(())))))))))))))))))))))))) -x)))))))))))))) (x))²= (x)x)
VK
Thus, (x] - (x)() S ( (x) (x)(0|| + 1 (x)))
1n (z) (In (1n (z)
2K
(14)
For any trajectory Tr perform Taylor expansion to y N(t)) (T)) = - (x) + - (z) Iny (N(r)) - px(z)) + 7 M: (
-
where M, is a point between NTT,)) and (x)x)). Then,
K
1 (z) - [x)) =
1
y N(x) - (x)
K
(In 72 1 K
(x(()) In
1
Σ - (M)))+
(Im
Σ
( -xx(2))3
K
K
i=1
i=1
1
k
2 (I 712
Mi
- -())))>))))))))))
K
k
< 2 7)2 7)²
Σ
- (()(m)))(((00)))
K
13

---

Published as a conference paper at ICLR 2026
Using the relationship between sample variance and variance:
K
E
Σ
(3))
= ON 2:² (x),
we have
K
(x) f1" (x)]l - 7
Σ (N(T)) -
2
K
(15)
(1n oN'" (z).
2K
For any trajectory 7, perform another Taylor expansion to 7 (tr):
(T.) = -xx(() (ar) + (0
(1)) Iny (N) (x)))) + Mi (
where Mj is point between (T:)) and (x). Then,
[F (x) H² (+)]=
M (m))
( Iny E[N(T)) xx)x)) (In (M -1x(()))
((m Mi (N()T) (x)))
2
1
$ (In (((())
- (
2
2 (2 (x)
Thus,
ElIP" - (x) < (In 7 (x).
(16)
Combining 14).
15
and
16
we have
(x) Fr (x)
E FT (x)
Ei FT Fm
In In (5(1) oN (x) + 1
2K (1n))3
K-1
+
2K (In (x) (z) + (In 7) 7)20 (x)
1
In (0) (x) (In 20 27 (z)
VK
Since N a convex function for y E (0, 1), by Jensen's 's inequality, we have
F Bl7 (T) > [[N(T)] (x)()))
Thus,
f (x) (x) FT) (x)
(x) (x) F (x) (z)
1 In + (In 22n))200 2, (x)
E
SE E
(x)
H (x)
Vk
713((0))
which proves the theorem.
A.2 PROOF OF VARIANCE RELATIONSHIP
Theorem 2. Let n. n' be two policies. Consider a set of states XM C X such that for all x E XM.
(1) x is infeasible under both π and n'. (2) m(x) 2 M, (3) (P
D(z)), P(D(x')) <
D(x))} 2 0.5. Under Assumptions 2 if n' is safer than x in all states in XMs, we have o (z) 2 2,5
2.
NN (I:), Vx XM-
14

---

Published as a conference paper at ICLR 2026
Proof. We prove the theorem in three steps.
Step 1: For a given policy, a state further to violation has a larger variance.
Let 2.+ be state with (z++) (D(x), D(x) + 6). To violate the constraint starting from x.,, the agent first needs move to some state x_ with D(x_) E [D(x) 6, D(x)). Thus, we have
m(x+)
(= +
N7(x+) NT(x+) -( (x
Take the variance on both sides:
NN (x+) = (ar(
+ 2, (x 2Cov (C Nz
m(x_)
(ar(NT + NN (x_) + 2Cov NT(x)),
Σ 2=2
NT (x+)
2, 2 ON (x-).
The last inequality follows from Assumption
Step 2: In a given state, safer policy has a larger variance.
Starting from x and taking one step under n. the agent will arrive state x_ with D(x_) (D(x) - 5, D(x)) with probability P(D < D(r)) and arrive at state I+ with D(x+) E
((()), with probability 1 P(D (c')) D(x)). Pe = (()(2)) < D(r)), we have
(x-
w.p. Pr
N
1 N
w.p. 1 Pr.
According to the law of total variance,
(x)( = P (r-) (1 - P 2. (x+) PP (=_ -
Consider policy मे that is safer than π in x and identical to . in all other states, i.e., गे only modifies the action in x and follows æ thereafter. We have
Pa S Px,
(*(x)) = N (z-),
N (x+) = No (x+).
Since 2 2,5 (x-) < ON
2,5 (I+),, we have
p+o N (x + (1 - (1-1=)o0 (x+) 2 P=O 2,5 (x- + (1 p=)o 2,7 (x++).
Since pr P 2 0.5, we have
Pe (1 p* (1x - uwv (x+ ))² Px (a) 13 (x+
Summing the above two inequalities, we have,
2,77 2. N (x) 0 N (z)
Step 3: For all states in XM, safer policy has larger variance.
A safer policy can be obtained by modifying the actions state by state. Each time we modify the action in single state, obtain safer policy in that state. The variance in that state increases, and the variance in other states remain unchanged. After modifying the actions in all states in XM,
obtain the safer policy n' with 2,a NN (x) 2 2
2.a (x), Vx E XM. which proves the theorem.
Note that modifying the actions state by state is just a technique to facilitate the proof. It is not required in the actual policy update. The policy can be updated "globally" at once, which is common practice for function approximated policies. The intermediate "virtual" policies, with each one safer in state, constructed only for the proof and do not need be found in practice. I1
15

---

Published as a conference paper at ICLR 2026
A.3 CASE STUDY: ONE-DIMENSIONAL RANDOM WALK
Consider a one-dimensional random walk where the state space X' = 2 and the action space / O, and the transition dynamics follows x' x + u. In
{-1,1}. The initial state is fixed each step, the policy chooses u = 1 with probability p. and u -1 with probability 1 - We require p > 0.5 to ensure finite expectation and variance steps to violation. The state x = is constraint-violatin
and every time the agent reaches this state, it is reset to x 0. Let N be the number of steps to the first violation, with its expectation and variance denoted as JuN and oj To measure policy safety, let r be the expected proportion of violating samples under continuous sampling. easily observed that r 1/pN- Our aim is to find the relationship between r and N.
Observe that can be decomposed as the sum of random variables: N LM, where M is the number of steps to the first time the agent visits the step its right. This is because going from x = 0 to x L can be decomposed as going to the step on the right times. We derive the expectation and variance of M, denoted as jaM and oh respectively. Consider the following two cases:
1. With probability p. the agent moves right and reaches x + in one step.
With probability - p. the agent moves left to x - 1. Now, the agent must first return to x and then proceed to x + 1. The expected number of steps of returning to from 1 is the same as that of moving from x to æ + 1, which equals μ.M Thus, the total expected number of steps in the case + 2pM.
Thus, the expectation PM satisfies
PM = p) (1+ 2pm)
which gives
AM 2p -
To derive the variance of M, first consider the second-order moment R[M²). Following the above analysis, we have
E[M² = p- 12 + (1 p) -[[[ + M' + +))
where M' is the number of steps to move from æ to æ for the first time. Since M' and M are ii.d., we have
[[(1 + M' + M) = 11 + 2(M' + M) + (M' + M)
1 + 4μm + B[(M' M
1 + 4/m E[M + 2M' M M²)
+1+ 4μm 2 24
Thus,
E[M = + (1 (1-p) p) (1 4p + 2E[M² 2 2p31)
Solving for EE [[M];
E = 1 4(1 p)jM + 2(1 p)
2p
Thus, the variance is
3 -EE - - 4p(1 - p) (2p 1)3"
Since N L.M, by the property of the summation of random variables, we have
1
NN LpM
L,
2p
2N = 1'02 = 4p(1 - p) L
(2p 1)3
16

---

Published as a conference paper at ICLR 2026
With 1/PN, have
2p- I. (2p- - 1)JNN
r
Thus,
4p p) NN
(2p 1)2
4p(1 p)
(2p 1)3
p2 2p
,
The above equation reveals that as the policy becomes safer, the variance of steps to violation in- creases. Combining with the error bound in Equation (7). we conclude that safer policy leads to higher relative estimation error bound.
B PSEUDOCODE
Algorithm 1: Soft actor-critic with feasible dual policy iteration (SAC-FDPI)
Initialize: Network parameters 6p. dh. Pp. Oa. IS ratios řap fpa 1. Replay buffer D 0. Feasibility threshold 0.95.
for each iteration do
// Sample with primal policy
Sample action Up ~ -
Get next state z', reward r, and cost from environment;
Store transition in replay buffer D < U u, c, ,,,, f.p));
Update IS ratio f'p <- F'p (a(ull)//T(((u
or Fap 1 if episode ends; Rumning averaged proportion of Gp( u) S greater than d then
// Sample with dual policy
Sample action Ta Te,
Get next state z', reward r, and cost from environment;
Store transition in replay buffer D +- D ((x,u,,', pp))
10
Update IS ratio fpa + fN (p(u(x)/ Ta (u|x)) or Ppa f- episode ends;
else 11
// Sample with primal policy Repeat Lines 2-5;
12
13
end
// Update network parameters
14
Update primal -easibliyyyy network + Lo, // Equation Update dual fetton-feasibility network da da (V
15
// Equation
16
Update action-value network w + 1/ (w))
// Equation
17
Update primal policy network 0p + 6p V LT (0p):
Equation
18
Update dual policy network 0u Aa V L (O)):
Equation
19
Update Lagrange multipliers by Equation 13
20
end
17

---

Published as a conference paper at ICLR 2026
C EXPERIMENTS
The Srfety-Gymmnsium benchmark (Ji et al. 2023a) and the Omnisafe toolbox (Ji et al. 2023b) are both released under the Apache License 2.0.
All experiments are conducted on a workstation equipped with Intel(R) Xeon(R) Gold 6246R CPUs (32 cores, 64 threads), NVIDIA GeForce RTX 3090 GPU, and 256GB of RAM. A single perimental trial- comprising one environment, one algorithm, and one random seed takes about hours execute. Executing all experiments with properly configured concurrent running scheme requires approximately 400 hours.
C.1 HYPERPARAMETERS
Table I: Hyperparameters
Category
Hyperparameter
Value
Shared
Number of vector environments
2
Number of samples per iteration
2
Number of updates per iteration
1
Replay buffer size
2e6
Batch size
256
Reward discount factor
0.99
Cost discount factor
0.97
Cost limit
0
Actor learning rate
le-4
Actor network hidden sizes
(256, 256)
Actor activation function
ReLU
Critic learning rate
le-4
Critic network hidden sizes
(256, 256)
Critic activation function
ReLU
Network weight initialization method
Truncated normal
Optimizer
Adam
Target network soft update weight
0.005
Maximum gradient norm
40
SAC
Initial entropy temperature
Target entropy
dim(2()
Entropy temperature learning rate
le-4
Penalty
Penalty coefficient
1.0
Lagrangian
Initial multiplier Multiplier learning rate
0.0
le-4
Multiplier update delay
10
FDPI
Primal policy step per iteration
Dual policy step per iteration Feasibility threshold e
0.1
Maximum KL divergence 6
5.0
C.2 ADDITIONAL RESULTS
18

---

Published as a conference paper at ICLR 2026
Diviooment Map
Divicment
Poe
Givic
025
535
025
Catennily
RCPO
FOCOPS
CUP
SAC-Lag
DBAC-T-Pe
SAC-FPI
SAC-FDM
Figure 6: Training curves on all 14 environments in Safety-Gymnasium benchmark. The shaded areas represent 95% confidence intervals over 5 seeds.
19

---

Published as a conference paper at ICLR 2026
Table Average cost and return the last 10% iterations. Mean Std over 5 seeds.
PointGoal
PointPush
VellChh
Algorithm
Cost
Return
Cost
Return
Cost
Return
CPO
0.90 + 0.45
1.45 + 0.99
0.60 = 0,66
0.19 0.75
0.03 = 0.02
853.39 + 239.61
4.08 1.18
13.14 1.92
1.96 = 0.81
7.24 1.10
1,29 : 0.38
2389.86
341.21
FOCOPS
3.16 0.83
1,85 0.56
+ 1.18
0.24
34.15 14.09
2606.12 = 214.08
CUP
3.01 3.224
2.91 0.98
0.91 = 0.51
1.37 0.52
0.91 = 0.90
1619.77 = 259.74
PPO-Lag
3.337
3.83
1.89 1:23
+
2234.27 499.64
SAC-Lag
0.100
11.06
-
2.27 + 3.29
14.18 3.33
0.00 = 0.00
2783.51 +: 26.26
DSAC-T-Pen
0.58 0.28
24.65
0.62
0.38 + 0:27
21.13
6.82
0.00 + 0.00
2760.54 + 14.76
SAC-FPI
2809.33
SAC-FDPI
0.09 4: 0.06
25.77
0.49
0.44 + 0.67
22.71
1.21
0.00 + 0.00
2831.97 + 9.13
Velocr2dVelocity
HlmanooddVelocity
SufetyHopper
Algorithm
Cost
Return
Cost
Retum
Cost
Return
CPO
0.02 0.01
188.35
::
250.84
18.05
519.61 :: 509.11
-4599.10
6207.6 66
RCPO
1.1 0.51
1913.68
765.27
0.28 0,10
3862.31
405.31
577.12 = 413.10
1481.77 = 639.77
FOCOPS
2.72 0.99
1629.19
1: 516.49
0.35 0.13
562.29
74.98
763.58
19.10
1332.01
= 297.91
CUP
0.50
088.166814..2
0.06 0.03
482.02
34.07
206.51 =: 213.09
-1266.47
1504.71
PPO-Lag
2.00 2.00 0.61
595.72 2166.22
0.22 0.07
1300 15 = 425.44
507.99
- 209.61
1337.05 : 204.90
SAC-Lag
0.00 0.00
126.54
0.10 0.09
5212.90
110.20
84.39 20.99
24 + 1252.72
DSAC-T-Pen
0.01 0.02
2752.25
106.80
0.05 + 0.05
5229.07
176.58
6.70 6: 4.83
3122,00 179.75
SAC-FPI
0.01 0.02
2787.28
126.27
0.09 + 0.08
5351.34 + 183.96
2,23 4 3.15
2947.83
: 227.60 ..6
0.00
2619.63
3118.77
Note: Bold values indicate top algorithms. Colored cells indicate top 2 in both cost and returm.
20

---

Published as a conference paper at ICLR 2026
We evaluated the feasibility threshold {0.05, 0.2) across 8 environments. The results indi- cate that a smaller leads to more conservative behavior, i.c., lower cost and lower return.
Table 3: Normalized cost and return under different feasibility threshold
Environment
0.05
0.1
0.2
Cost
Return
Cost
Return
Cost
Return
PointGoal
0.001
0.998
0.002
1.073
0.017
1.080
PointPush
0.000
0.822
0.002
0.770
0.001
0.816
PointCircle
0.000
0.703
0.004
0.797
0.040
0.862
CarGoal
0.000
1.087
0.001
1.115
0.000
1.085
CarPush
0.001
0.862
0.016
0.934
0.008
0.888
CarCircle
0.001
0.681
0.003
0.619
0.016
0.824
HalfCheetah Velocity 0.000
1,386
0.000
1.385
0.000
1.418
ClumanoidVVeloiioe
0.001
0,487
0.000
0.810
0.000
0.736
Average
0.000
0.878
0.003
0.938
0.010
0.964
We evaluated the dual threshold d e {0.5, 0.9, 0.95, 0.98). The results show that within reasonable range (i.e., for 2 0.9), smaller d, which corresponds to more frequent activation the dual policy, leads to lower costs without sacrificing return. However, excessively small d results in higher costs, possibly because severe distributional shift.
Table 4: Normalized cost and return under different dual threshold
Environment
0.50
d 0.90
0.95
0.98
Cost
Return
Cost
Return
Cost
Return
Cost
Return
PointGoal
0.001
0.988
0.000
1.091
0.002
1.073
0.008
1.076
PointPush
0,003
0.747
0.000
0.746
0.002
0.770
0.005
0.798
PointCircle
0.004
0.773
0.000
0.787
0.004
0.797
0.019
0.861
CarGoal
0.003
1.107
0.000
1.142
0.001
1.115
0.000
1.110
CarPush
0.150
0.943
0.005
1.028
0.016
0.934
0.021
0.789
CarCircle
0,000
0.625
0,001
0.739
0.003
0.619
0.001
0.703
HalfCheetah Velocity
0,000
1.374
0.000
1.382
0.000
1.385
0.000
1.374
Humanoid Velocity
0.000
0.825
0.000
0.682
0.000
0.810
0.000
0.763
Average
0.020
0.923
0.001
0.949
0.003
0.938
0.007
0.934
21

---

Published as a conference paper at ICLR 2026
We evaluated the KL divergence threshold 8 {2, (0). The results show that the overall per- formance stable within a reasonable range (ó - 5). with a smaller á slightly decreases both cost and return. However, an excessively large å significantly increases cost due excessive policy divergence.
Table 5: Normalized cost and return under different KI. divergence threshold
Environment
8 2
6 10
Cost
Return
Cost
Return
Cost
Return
PointGoal
0.002
1,000
0.002
1.073
0.015
1.036
PointPush
0.000
0.335
0.002
0.770
0.061
0.647
PointCircle
0.005
0.775
0.004
0.797
0.012
0.861
CarGoal
0.000
1.108
0.001
1.115
0.002
1.144
CarPush
0.010
0.857
0.016
0.934
0.387
0,021
CarCircle
0.002
0.766
0.003
0.619
0.005
0.824
HalfCheetah Velocity
0.000
1,433
0.000
1.385
0.000
1.425
Humanoid VemcnoidVeloccty
0.000
0.843
0.000
0.810
0.000
0.709
Average
0.002
0.890
0.003
0.938
0.060
0.833
We compared BAC-FDPI with IPO Liu et al.
2020) and CRPO (Xu et al. 2021 8 environments.
The results show that SAC-FDPI achieves a lower cost and higher return than these two algorithms.
Table 6: Normalized cost and return comparison with IPO and CRPO.
Environment
IPO
CRPO
SAC-FDPI
Cost
Return
Cost
Return
Cost
Return
PointGoal
0.060
0.086
0.142
0.607
0.002
1.073
PointPush
0.188
0.063
0.057
0.193
0.002
0.770
PointCircle
0.052
0.816
0.013
0.918
0.004
0.797
CarGoal
0.122
0.555
0.123
0.546
0.001
1.115
CarPush
0.551
0.131
0.831
0.175
0.016
0.934
CarCircle
0.045
0.624
0.025
0.677
0.003
0.619
HalfCheetah Velocity
0.073
1.435
0.004
1.146
0,000
1.385
Vommanoid ellciiiyy
0.001
0.162
0.002
0.635
0.000
0.810
Average
0.136
0.484
0.149
0.612
0.003
0.938
22

---

Published as a conference paper at ICLR 2026
D DISCUSSION
D.1 RELATED WORK ON REPRESENTATION LEARNING
Another class of methods address the challenge of sparse violation signals in feasibility function estimation through representation learning. Cen et al. (2024) propose Feasibility Consistent Safe Reinforeement Learning (FCSRL), which extracts safety-related information from the observation to improve feasibility function learning. This is achieved by learning a encoder that maps the obser- vation latent states, which serves as a better input for the feasibility function. FCSRL focuses on representation learning, which tries to better exploit safety-related information from given data. In contrast, our method addresses the safety paradox, where violation signals become increasingly sparse the policy becomes safer. Our solution is to actively collect more violating data using dual policy, targeting the root cause of the sparsity. Therefore, FCSRL and our method are orthogonal: one improves data utilization, while the other improves data collection.
D.2 EXTENSION TO TD ESTIMATR AND CVF
Our theoretical analysis can be extended to TD estimate. Specifically, Section 4.2 proves that the variance of steps to violation, (T)), increases with safer policies. Consider an imitial state x, its subsequent state z', and the sub-trajectory 7' starting from x'. Since N N(r') = [N(T) 1, the variance of N(r') also increases with policy safety. This implies a higher variance in the true feasibility value F" (x'). Consequently, the TD target, which is computed by F" (xx', inherits this increased variance, leading larger estimation error in fT(x))
Our analysis can also be extended to the CVF widely used the CMDP, defined as
F (I) ETAY
Σ \'((I))
The key insight is that the CVF can be decomposed into a discounted sum CDF-like terms. Specifically, we can break down any infinite-horizet trajectory into segments that end immediately after a constraint violation. The total CVF is the discounted sum of costs along the entire trajectory. We can now group the costs by the segment in which they occur. The cost incurred in each segment is, by construction, a discounted sum that starts from an initial state and ends with a violation. Crucially, the value of the CVF in each segment is precisely CDE The total CVF can be expressed as:
F
E Ediccuudte cost of Segment 1] + Efdiscounted cost of Segment 2] +...
CDF term
CDF temn
where T; is the time step of the ith violation. Our main theoretical result establishes that the esti- mation error bound of a single CDF term increases as the policy becomes safer (i.e., as violations become rarer, making each segment longer). This directly implies that the error bound for the total CVF must also increase.
D.3 ADDITIONAL VIOLATION DURING TRAINING
In the broader safe RL community, there are two training and implementation modes: (1) offline training and online deployment (OTOD), which first trains a policy in simulator and then deploys the real world, and (2) simultaneous online training and deployment (SOTD), which directly interacts with the real world to collect data for training. The OTOD mode only requires the final policy to be safe because intermediate policies will not be deployed in the real world. The SOTD model requires both the final policy and all intermediate policies be safe.
FDPI, along with the baselines we compare in the paper, belongs to the OTOD mode. We focus primarily on learning safe policy convergence, rather than guaranteeing safety during training. Therefore, FDPI does not involve unsafe exploration in the real world during training.
For the SOTD mode, must ensure safety throughout training. Achieving this goal would require integrating additional safe exploration techniques like those proposed by [Berkenkapp et al. (2017) and Yu al. 2022). These methods typically employ model of the environment, either known
23

---

Published as a conference paper at ICLR 2026
learned. Constraint violations allowed in the model but not allowed in the environment. They alternate between learning a safe policy within the current model and refining the model with newly collected data. These approaches are complementary to our contribution, which focuses solving the safe policy within a fixed model/envirrnment.
E LARGE LANGUAGE MODEL USAGE DISCLOSURE
We used Large Language Model (LLM) solely for the purpose of improving grammar and polishing writing. The LLM not used for any core research tasks such retrieval, discovery, ideation, or analysis.
24


Reviews:
[{"review_id": "q3fqRxlwgV", "reviewer": "Reviewer_EGeQ", "summary": "This paper studies safe RL. It first introduces a problem termed as safety paradox, which describes a phenomenon that as a safe RL agent's policy improves, it encounters constraint-violating states less frequently. The consequence of this paradox is that an inaccurate feasibility function estimate leads to an incorrect identification of the feasible region. This biases the policy update and undermines policy safety. To address it, the paper proposes an algorithm called Feasible Dual Policy Iteration. The algorithm maintains and trains two policies in parallel. The dual policy is trained to maximize constraint violations by maximizing the dual feasibility function. This mechanism ensures that even as the primal policy becomes too safe, the dual policy continually supplies the necessary critical samples to maintain an accurate feasibility function estimate.", "questions": "How are the hyperparameters chosen, such as the feasibility threshold (0.1) and the dual threshold (0.95)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It proposes several definitions to formulate an important problem in safe RL and also provides an algorithm to address it.\n- It has both theoretical works and empirical experiments.", "weaknesses": "- There is a gap between the paper's theoretical foundation and its algorithmic implementation. Theorem 1 is built upon a Monte Carlo estimate of the feasibility function. However, the proposed FDPI algorithm is based on SAC and uses a TD estimate. \n\n- The paper frames the \"safety paradox\" as a fundamental and general obstacle in safe RL. However, the entire analysis is built upon a specific problem formulation. It is unclear whether this paradox, as defined, holds for other significant branches of safe RL.\n\n- The proposed solution, FDPI, hinges on training a dual policy that is explicitly optimized to maximize constraint violations. This approach may be impractical and dangerous for many real-world, safety-critical applications.\n\n- The IS ratio is defined as a sequential product of policy probabilities, which is prone to high variance. To address it, the algorithm has a KL divergence constraint. However, the hyperparameter chosen for the KL divergence is high (5.0), suggesting the algorithm could be unstable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dgKjzKrq3u", "reviewer": "Reviewer_HJpx", "summary": "This paper identifies the “safety paradox” in safe reinforcement learning, where making a policy safer reduces constraint-violating samples and thus worsens safety estimation. It proposes Feasible Dual Policy Iteration (FDPI), which adds a dual policy that deliberately induces controlled violations to improve feasibility function accuracy using importance sampling and KL constraints. Empirical experiments show FDPI achieves strong safety and returns, breaking the paradox", "questions": "1. What is the impact of the sampling fraction from the dual policy? In the paper, it seems to be fixed at 0.5. In particular, if compute allows, an interesting ablation would be how altering both the dual threshold and sampling fraction would impact the final policy.  \n2. In Eq 5, the optimization problem is formulated with a constraint that the expectation of the CDF is less than 0, ie $\\\\mathbb{E}\\_{x\\\\sim d\\_{init}}\\\\left\\[F^{\\\\pi}(x)\\\\right\\]\\\\leq 0$ However in equation 4, the CDF is defined as the expectation of $\\\\gamma^{N(\\\\tau)}$ where $\\\\gamma \\\\in (0,1)$, so how is constraint possible?  \n3. Could the authors clarify the precise relationship with FPI? It appears the core primal policy update (Eq. 11\\) is adopted from FPI, and the primary novelty is the dual-policy data-augmentation mechanism designed to fix FPI's data-starvation failure mode (the paradox). Is this an accurate characterization of the contribution?  \n4. Are the proposed data centric solutions and existing representation centric solutions mutually exclusive? Could FDPI's dual-policy sampler be combined with a safety-aware embedding (like that from FCSRL) to potentially achieve even greater stability and performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* **Problem Significance:** The paper addresses the well-known and critical problem of learning from sparse cost signals in safe R. Accurately modeling safety boundaries, especially when violations are rare , is a key obstacle to deploying RL in the real world\n* **Writing:** This paper is in general well written, barring some minor issues, see weaknesses and questions for a detailed list.\nIntuitive Solution: The proposed mechanism of using a dedicated \"dual policy\" to actively generate the very data that is missing (i.e., constraint violations) is an intuitive and direct approach to solving the sparse data problem. \n* **Experimental Results:** The experimental results presented in Figure 1 and 2 show that FDPI (and its ablative variant, SAC-FPI) achieve a superior cost/return trade-off when compared against the chosen set of baselines.", "weaknesses": "1. **Missing Context of the \"Safety Paradox.”** At the crux, this paradox is due to the sparsity of the safety constraint, which becomes more sparse as the policy becomes more sparse. The authors address this for one specific policy iteration algorithm, FPI, by using a dual policy to sample unsafe data. There have been existing works that tackle this problem from a different angle. For example “Feasibility Consistent Representation Learning for Safe Reinforcement Learning”[1] proposes using representation learning to extract the safety-related information from the raw state. Perhaps the authors could include a discussion or comparisons to such methods.  \n2. **Baselines:** The current suite of baselines are limited and rather old, with the newest being from 2023\\. Perhaps some newer baselines could be added, such as ESPO [2] and PCRPO [3], and the FCSRL[1] method mentioned above?  \n3. **Ablations:** There are no ablations of the key design hyperparameters such as $\\\\epsilon$ and $d$.  \n4. **Over usage of $d$ (Minor)** $d$ is used several times in the paper for different notations. For example, d is defined as a continuous function that measures the distance to a constraint violation in Assumption 2\\. But then in lines 259-260, it is defined again as a dual threshold hyperparameter.   \n5. **Intuitive Justification of Assumption 4 (Minor)** The intuitive justification for Assumption 4 is not particularly clear.\n\n[1] Cen, Zhepeng, et al. \"Feasibility consistent representation learning for safe reinforcement learning.\" Proceedings of the 41st International Conference on Machine Learning. 2024.\n\n[2] Gu, Shangding, et al. \"Enhancing efficiency of safe reinforcement learning via sample manipulation.\" Advances in Neural Information Processing Systems 37 (2024): 17247-17285.\n\n[3] Gu, Shangding, et al. \"Balance reward and safety optimization for safe reinforcement learning: A perspective of gradient manipulation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 19. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ChmHFe6YLb", "reviewer": "Reviewer_2igd", "summary": "The paper identifies and formalizes a **safety paradox**: as a policy becomes safer, **violation samples become sparse**, which makes the **feasibility function** (e.g., a CDF) harder to estimate and actually increases its error bound—degrading feasible-set identification and, paradoxically, safety. To break this loop, the authors propose **FDPI (Feasible Dual Policy Iteration)**: alongside the primal policy, a **dual policy** deliberately generates violation samples; data from both policies are **importance-sampled (IS)** to correct distribution shift, and a **(symmetric) KL constraint** stabilizes the IS ratios. On 14 Safety-Gymnasium tasks, FDPI achieves **the lowest violation cost** while maintaining **best or near-best return**.", "questions": "1. Can you provide an **error-propagation analysis under TD learning**, or at least a joint training trace that links **violation coverage → feasibility-estimation error → final cost**, to empirically validate the paradox-breaking story?\n2. Please include **sensitivity analyses** for the **KL threshold**, **dual activation/sampling ratio**, and **IS truncation length**; also report the **distribution of IS ratios** (mean/variance/overflow/underflow rates) over training.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* **Clear, broadly relevant problem framing**: Elevates the “sparse violations → higher feasibility error → less safety” feedback loop into a unifying paradox for feasibility-function–based safe RL.\n* **Coherent method design**: The **dual policy** raises violation coverage without increasing total interactions; **truncated-trajectory IS** approximates the state-marginal ratio, and a **KL constraint** tames ratio underflow—practically implementable.\n* **Compatible with common policy-improvement frameworks** (e.g., FPI/SAC) with concrete loss forms and Lagrangian updates; reproducibility looks feasible from the described components.\n* **Empirical scope and metrics are solid**: 14 environments with cost–return trade-off plots showing **lowest cost** and **competitive returns**, plus targeted ablations that match the stated research questions.", "weaknesses": "* **Theory–practice gap**: Error bounds are developed under MC/CDF estimation, while training uses **TD learning with function approximation**; the bridge between them is under-substantiated.\n* **Assumption generality**: Key results rely on continuity/neighboring-state bounds and other conditions whose validity in contact-rich or discontinuous dynamics is not fully justified.\n* **IS approximation and robustness**: The state-marginal ratio is approximated along a single trajectory and truncated at first appearance, which can bias estimates; there’s no systematic **sensitivity study** for the **KL threshold, truncation length, or dual sampling fraction**.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ag2bRLu6D7", "reviewer": "Reviewer_3DTX", "summary": "The paper introduces a new theoretical problem in safe reinforcement learning (safe RL) called the safety paradox. This paradox arises because as an agent’s policy becomes safer—producing fewer constraint violations—it also generates fewer “unsafe” samples. However, those samples are crucial for accurately estimating the feasibility function, which measures whether future states will satisfy safety constraints. The result is a paradoxical loop: improving safety reduces the ability to learn safety accurately, which then harms the policy’s overall safety performance.\n\nTo address this issue, the authors propose a new algorithm called Feasible Dual Policy Iteration (FDPI). FDPI introduces an auxiliary “dual” policy that deliberately maximizes constraint violations while staying close to the original policy. This strategy increases the diversity and number of violating samples without requiring additional data collection. The algorithm then combines samples from both policies and corrects for the resulting distribution shift using importance sampling.\n\nThe authors conduct extensive experiments on the Safety-Gymnasium benchmark and show that FDPI achieves state-of-the-art results—producing the lowest constraint violation rates and competitive or best reward performance among all compared algorithms.\n\nIn essence, the paper both identifies a fundamental limitation in safe RL and provides a theoretically grounded, empirically validated solution to overcome it.", "questions": "1. Could you further explain the intuition behind the assumptions introduced in Section 4 (e.g., continuity of the distance-to-violation function, bounded state transitions, and weak covariance conditions)? It would be helpful to understand how these assumptions can still hold or approximately apply in practical scenarios, such as high-dimensional control or discontinuous dynamics.\n\n\n2. While the motivation for Feasible Dual Policy Iteration (FDPI) is strong and well-argued, the paper does not appear to include a theoretical bound or convergence analysis for the proposed method. Could you clarify whether there exists any theoretical guarantee—such as stability, safety improvement, or sample-efficiency bounds—for FDPI, and how it connects to the theoretical framework established earlier?\n\n\n3. In real-world applications where safety-critical systems cannot deliberately generate unsafe behaviors, how can the dual policy—designed to intentionally cause constraint violations—be implemented or approximated safely? Is the concept of a “dual policy” operationally feasible in physical or safety-limited environments, and what mechanisms could ensure it remains practical?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.Novel Theoretical Insight — the “Safety Paradox”\nThe paper uncovers and rigorously explains a previously overlooked phenomenon in safe reinforcement learning: improving a policy’s safety reduces constraint-violating samples, which paradoxically increases the estimation error of the feasibility function and degrades future safety. This insight is both conceptually fresh and theoretically significant.\n\n2.Principled and Effective Solution — Feasible Dual Policy Iteration (FDPI)\nThe proposed algorithm introduces a dual policy that deliberately generates controlled constraint violations to maintain balanced data for learning safety. With importance sampling and KL constraints ensuring stability, FDPI smartly breaks the paradox without altering the environment or reward structure.\n\n3.Strong Empirical Validation\nExtensive experiments on 14 Safety-Gymnasium environments demonstrate that FDPI achieves state-of-the-art safety performance (lowest violations) and competitive rewards. The empirical results clearly support the theoretical claims, showing reduced estimation error and consistent safety improvements across diverse tasks.", "weaknesses": "1.Too Many Theoretical Assumptions\nThe theoretical framework depends on several restrictive assumptions—such as smoothness of the distance-to-violation function, bounded state transitions, and weak covariance between trajectory segments. These simplify analysis but may not hold in high-dimensional or discontinuous environments, limiting the generality and real-world validity of the theory.\n\n2.Limited Analysis of Hyperparameter Sensitivity\nFDPI introduces new hyperparameters, including the dual activation threshold \nd, KL divergence bound \nδ, and feasibility tolerance \nϵ. However, the paper fixes these values without investigating their sensitivity or adaptive tuning. Since these parameters directly affect the balance between exploration and safety, a lack of analysis leaves uncertainty about robustness across different tasks.\n\n3.Computational and Implementation Overhead\nThe algorithm maintains two policy networks and two feasibility estimators while applying importance-sampling corrections, which significantly increase computational and implementation complexity. The paper does not report runtime or convergence cost, raising concerns about scalability to larger or real-time safety-critical systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iFKHtGas7I", "reviewer": "Reviewer_opKu", "summary": "This paper points out a safety paradox in safe RL: as a policy gets safer, it visits constraint-violating states less, making learning a feasibility function harder. The authors formalize this by showing the estimation error of the constraint decay function grows when violating samples become rare. To break this, the authors propose to run a primal policy that tries to be safe with a dual policy that seeks violations but is kept KL-close, and mix them with a truncated importance sampling.", "questions": "Please see weaknesses.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes a novel algorithm that uses a second KL-coupled policy to actively seek violating rollouts.\n2. It is useful in the context of safe RL to formalize the safety paradox.", "weaknesses": "1. The bound that \"error grows when violations are rarer\" depends on the chosen CDF function, which is not motivated enough, making the paradox narrower than claimed.\n2. FDPI creates unsafe data via the dual policy, and the paper does not quantify how many unsafe transitions it generates, or how to cap it in non-simulation settings.\n3. The main difficulty, the state-distribution shift, is handled with a single-trajectory, truncated IS ratio plus a KL constraint. However, there is no bias/variance analysis on truncation depth, despite this is the core technical contribution.\n4. The paper can be benefited from a broader scope of experiments. Currently all tasks are from Sarety-Gym with similar constraint types", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LVrLoKPPdz", "reviewer": "Reviewer_W7n2", "summary": "**DISCLAIMER: I was one of the reviewers of this paper at an earlier conference.**\n\nThe paper proposes \"safety paradox\" as a main obstacle towards zero constraint violation in safe RL, where the improvement in policy safety causes the estimation error of the feasibility function to increase and in turn harms policy safety. An algorithm is proposed to solve this issue. It works through introducing another policy that seeks to violate the constraint as much as possible, so as to provide more samples for the training of the feasibility function. Importance sampling and KL-divergence constraints are employed to mitigate the distributional shifts between the this newly introduced policy and the original learning policy. Empirical valuation shows the effectiveness of the proposed method.", "questions": "My questions were resolved during the previous rebuttal phase of this paper, at an earlier conference. The only thing left: It would be nice if the additional results and discussions from the previous rebuttal can be added to the paper.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The flow of this paper is clear and easy to follow.\n- Achieving zero constraint violations has great practical significance, and the estimation error of the feasibility function has been a key issue to this. In this regard, this work is well-motivated.\n- I did not go through the proof, but the theoretical analysis makes sense to me, and can serve as a valid justification of the proposed safety paradox.\n- The experiment results of the proposed method look quite promising.", "weaknesses": "- The theory (Theorem 1) only shows that the upper bound of the estimation error will become larger, not exactly the error itself. I know showing how the exact error relates to constraint violations may be too much to ask, so I recommend showing some empirical evidence about this, which would be a good supplement for the theory. (In the previous rebuttal phase, the authors gave empirical evidence demonstrating the relationship between constraint-violating samples and estimation error. I would appreciate it if the results can be added to the paper.)\n- The proposed method solves the safety paradox by introducing another policy that aims to intentionally violate the constraint, which causes extra cost during training and thus may not be practical in the real world.\n- There are some other existing baselines this paper is missing: IPO [1] and CRPO [2]. In the previous rebuttal phase, the authors showed their performance. I would appreciate it if the results can be added to the paper.\n- The metric used in the empirical evaluation may be a problem. See Questions.\n\n[1]: Liu et al., 2020, Ipo: Interior-point policy optimization under constraints.\n\n[2]: Xu et al., 2021, Crpo: A new approach for safe reinforcement learning with convergence guarantee.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)