Constrained Reinforcement Learning with Smoothed Log Barrier Function
Authors: Baohe Zhang
Yuan Zhang
Hao Zhu
Shengchao Yan
Thomas Brox
Joschka Boedecker
Venue: Accepted by TMLR ()

Abstract: Deploying reinforcement learning (RL) in real-world systems often requires satisfying strict safety constraints during both training and deployment, which simple reward shaping typically fails to enforce. Existing constrained RL algorithms frequently face several major challenges, including  instabilities during training and overly conservative policies.
To overcome these limitations, we propose CSAC-LB (Constrained Soft Actor-Critic with Log Barrier), a model-free, sample-efficient, off-policy algorithm that requires no pre-training. CSAC-LB integrates a linear smoothed log barrier function into the actor’s objective, providing a numerically stable, non-vanishing gradient that enables the agent to quickly recover from unsafe states while avoiding the instability of traditional interior-point methods. To further enhance safety and mitigate the underestimation of constraint violations, we employ a pessimistic double-critic architecture for the cost function, taking the maximum of two cost Q-networks to conservatively guide the policy.
Through extensive experiments on challenging constrained control tasks, we demonstrate that CSAC-LB significantly outperforms baselines by consistently achieving high returns while strictly adhering to safety constraints. Our results establish CSAC-LB as a robust and stable solution for applying RL to safety-critical domains.

Published in Transactions on Machine Learning Research (11/2025)
Constrained Reinforcement Learning with Smoothed Log Barrier Function
Baohe Zhang
thangbBes..uii-fribbbe www-freiburg.....
Deportment of Computer Science University of Freibury
Yuan Zhang
www.://swwss.nifbbrrrr.... de
Department Computer Science University of Freiburg
Hao Zhu
zhuh@css catiib/ccsuun-ffibbrr..... de
Department of Computer Science University of Freibury
Shengchao Yan
yan uni-
Department Computer Science Universily of Freibury
Thomas Brox
brom&es brozeess.unifreettee www.iesttiiifffrrrr..... de
Department of Computer Science University of Freiburg
Joschka Boedecker
jboedeck www.e:eesss.nnnuneeene wni-freibury......mm
Department of Computer Science University of Freihurg
Reviewed on OpenReview: https: openreview. net/ forum? id= Amh95oURa
Abstract
Deploying reinforcement learning (RL) in real-world systems often requires satisfying striet safety constraints during both training and deployment, which simple reward shaping typ- ically fails to enforce. Existing constrained RL algorithms frequently face several major challenges, including instabilities during training and overly conservative policies. To over- come these limitations, we propose CSAC-LB (Constrained Soft. Actor-Crita with Log Barrier), a model-free, sample-efficient, off-policy algorithm that requires no pre-training. CSAC-LB integrates linear smoothed log barrier function into the actor's objective, pro- viding numerically stable, non-vanishing gradient that enables the agent. to quickly recover from unsafe states while avoiding the instability of traditional interior-point methods. To further enhance safety and mitigate the underestimation of constraint violations, em- ploy pessimistic double-critic architecture for the cost function, taking the maximum of two cost Q-networks to conservatively guide the policy. Through extensive experiments on challenging constrained control tasks, we demonstrate that CSAC-LB significantly out- performs baselines by consistently achieving high returns while strietly adhering to safety constraints. Our results establish CSAC-LB as robust and stable solution for applying RL to safety-critical domains.

---

Published in Transactions on Machine Learning Research (11/2025)
1 Introduction
formulations, rewards are usually the only metric of performance. However, in many real-world applications, the performance of algorithm is not necessarily measured by single objective. For example, in the context of an autonomous driving task, the agent must drive as quickly as possible while still adhering to traffic rules and avoiding collisions. Designing a reward function that considers all individual cases can be challenging. Reward engineering, which involves tuning the weights of different reward components, may also lead to a suboptimal solution. In the context of deep reinforcement learning, where neural networks are used as function approximators, it is often not feasible to apply conventional constrained optimization methods due to the large number of parameters to be optimized.
In traditional Reinforcement Learning (RL) (Sutton Barto 1998)
If no prior knowledge is available, constraint violation is unavoidable during training as the RL agent must visit the unsafe states of the environment to gradually learn a safe policy. A fundamental challenge is to predict the boundary of the safe region using a neural network, since neural networks typically cannot extrapolate well to unseen data (Barnard & Wessels, 1992). To predict this safe margin, the agent must first see some samples from the unfeasible set. However, dramatically violating constraints is also generally undesirable, as this may cause severe damage to the system being optimized. This means an RL agent must avoid extensive constraint violations and learn efficiently from as few violations as possible.
Considering these challenges, existing approaches often require limiting assumptions, such as a known sub- optimal recovery policy (Thananjeyan et al. 2021 Banerjee et al. 2024) or prior knowledge of the system dynamics (Liu et al. 2025 Luo & Ma 2021. This restricts gencralization and requires domain expertise. The challenge is particularly critical in tasks where the optimal policy lies on the boundary of the safe region. Success these tasks hinges directly on the agent's ability to explore the safe margin efficiently. This requirement, however, reveals a common weakness in existing algorithms (Achiam) (021) as found in Marchesini et al. (2022), which often fail to strike the necessary balance, resulting in policies that are either too reckless and unsale or too conservative and suboptimal.
Motivated by the above, we propose CSAC-LB, which applies a linear smoothed log barrier function (Ker- vadec et al.) 2019) to the Soft Actor-Crite Critic (SAC) algorithm (Haarnoja et al.) 2018) augmented by a safety critic (Srinivasan et al. 2020). It serves as a general-purpose method that can handle various types of cost functions without requiring any additional information or pre-training. Being an off-policy algorithm, it can leverage data collected previously for elficient learning. The goal of this algorithm is to effectively explore the safe margin of given problem during training and to learn a well-performing policy for future deployment in real-world tasks. Crucially, the smoothed log barrier provides stable, non-vanishing gradient even in unsafe states, enabling the agent to recover quickly from constraint violations and to explore the boundary between safe and unsafe regions without suffering from numerical instability or vanishing gradients. This facilitates more effective and targeted exploration, allowing the agent to learn the true constraint boundary with fewer violations and greater sample efficiency than prior methods. This distinguishes it from other works that aim for zero constraint violations during training. Our contributions are the following:
We propose new actor objective that integrates a linear smoothed log barrier function. This function provides stable, non-vanishing gradient signal even when the policy is in an unsafe state, allowing for rapid recovery without the numerical issues of standard log barriers or the gradient saturation of clipping methods. This enables effective exploration along the constraint margin, which is critical for learning in tasks where optimal policies lie near the feasibility boundary.
To combat the critical problem of cost underestimation by the critic network, we employ pessimistic double-critic architecture for the cost function. By taking the maximun of the two cost Q-values, we ensure a conservative estimate of constraint violations, leading to more reliable and safer policies.
We provide a theoretical analysis of CSAC-LB, establishing a bound on its suboptimality and guaran- teeing bounded constraint. violation. Through extensive experiments ten challenging benchmark
2

---

Published in Transactions on Machine Learning Research (11/2025)
tasks, we demonstrate that CSAC-LB consistently outperforms state-of-the-art baselines, achieving a superior balance of high returns, strict constraint satisfaction, and stable convergence.
2 Related Work
Many prior works have been proposed for addressing the safety of reinforcement learning using various approaches. The reviews by Garcia & Fernández (2015), Arnold et al. (2021), and Ma et al.) (2022) provide a comprehensive overview.
One direction which has been explored for solving constrained RL problems is safe policy search. Constrained Policy Optimization (CPO) (Achiam et al. 2017 is the first general-purpose approach that solves constrained reinforcement learning using trust-region method and is able to provide certain theoretical guarantees. Polymenakos et al. (2019) extend safe policy search by augmenting it with a Gaussian Process model for risk estimation. Other approaches leverage cost or safety critics to guide the policy. For instance, Ha ct al. (2020) extend SAC (Haarnoja et al., 2018) with a cost function and optimize the constrained problem by introducing Lagrangeemulliplirr However, this method suffers in terms of training robustness when constraint violations occur only rarely during training on the task. Building on this idea, the method by Srinivasan et al. (2020) learns to predict the cumulative constraint violations and update the policy accordingly. Further improvements are presented by Yang el al. (2021) where a distributional safely critic is combined with the CVaR metric. By changing the percentile of this CVaR metric, the sensitivity to risk can also be changed accordingly. The work presented by Ying et al. (2022) also uses CVaR as the metric, but in conjunction with the on-policy method Proximal Policy Optimization (PPO) (Schulman et al.) 2017)) as the RL algorithm. In a separate line of inquiry, Lyapunov functions are also widely applied to constrained RL problems (Chow et al. linearized Lyapunov constraints during the policy update, these methods can be applied to any policy gradient method, such as DDPG (Lillicrap et al. 2016) or PPO.
2018 2019 By projecting policy parameters onto the feasible solutions from
Another line of work is close to Model Predictive Control (MPC) (Rawlings et al. 2017 J. Model-basod
approaches, such as (Berkenkamp et al.] 2017 Cheng et al. 2019 use a dynamics model to certify the safety of the system. These approaches usually assume that the dynamics model is calibrated and the constraints are known. This, however, limits the general applicability of the algorithms. A framework for learning barrier certificates (Ames et al., 2019) and the policy iteratively, achieving zero training-time constraint violations in an empirical analysis, is presented by Luo & Ma (202T). The work by Pereira et al. (2020) combines stochastic barrier functions with safe trajectory optimization and is able to recover the optimal policy under certain conditions.
As et al. (2022) improve data efficiency in constrained RL by using a Bayesian World model. Huang el al.] (002)) learns a world model for training and planning, leading to a higher data-efficiency.
Close to our work is the application of interior-point methods to PPO (Zeng & Zhang 2018 Liu et al. 2020 However, these methods are on-policy methods and have numerical stability issues due to the log barrier function. To the best of our knowledge, we are the first to propose a numerically stable, off-policy interior- point algorithm. By integrating a novel linear smoothed log barrier with the highly sample-efficient SAC framework, our work, CSAC-LB, addresses the critical challenges of both training instability and sample efficiency in constrained reinforcement learning.
3 Background
3.1 Constrained Markov Decision Processes
A Markov Decision Process (MDP) provides the mathematical foundation for sequential decision-making, It is formally defined as tuple (,, 7), where S is the state space, A is the action space, P: S x Axx + [0, 1] is the state transition probability function, R: x A + RR is the reward function and E (0, 1) is the discount factor.
An agent interacts with the environment by following a policy :: S + P(A), which maps states to a probability distribution over actions. The primary goal in reinforcement learning is to find optimal policy
3

---

Published in Transactions on Machine Learning Research (11/2025)
where the expected cumulative discounted reward, given by
Jn(#) =
I (ScoA))m
y R(st. at)
(3.1)
is maximized.
A Constrained Markov Decision Process (CMDP) (Eitan 1999 extends the MDP framework by augmenting the MDP tuple with set of cost functions, (Ci,..., Cm), where C:: S A + IR, i = ,,..., maps a given state-action pair to the constraint violation penalty. For each cost function, we define an expected cumulative discounted cost as
Jc. (x)
E (Ne,0))
Σ [((((,)
(3.2)
The objective in CMDP is find policy that maximizes the expected return 3.1 while ensuring that each expected cost (3.2) remains below predefined threshold d, Let II denote the set of all possible policies, then this leads to the following constrained optimization problem:
maximize well
Jn(#) Jc, (T) S di, i= I,...,m.
(3.3)
subject to
3.2 SAC-Lagrangian
The Lagrange Multiplier Method The Lagrange multiplier method is a standard technique for solving constrained optimization problems (Bertsekas) 1982). Let f.g: R + R be real valued functions on R", a general constrained optimization problem, ollen called the primal problem, can be written as:
maximize #ER"
f(z)
(3.4)
subject to g(x) S.
This problem can be transformed into an unconstrained optimization by introducing the Lagrangian function:
L(x, A) = f(x) Ag(x),
then transformed to solving the following saddle-point (or max-min) problem, known the Lagrangian dual problem:
where 2 0 is the Lagrange multiplier. The original problem 3.4
maximize zER"
inf L(x, A).
(3.5)
SAC-Lagrangian SAC-Lagrangian (SAC-Lag) Ha et al. 2020 applies the Lagrange multiplier frame- work to solve constrained Markov Decision Processes (CMDPs) within the Soft Actor-Critic (SAC) paradigm (Haarnoja et al. 2018). The goal to find a policy æ that maximizes the entropy-regularized return while ensuring the expected cumulative cost does not exceed predefined limit d. This forms the primal optimization problem for the policy:
maximize nell
E (st,ae)~xx )- t=
y (R(s az) + aH((T()))
f(x)
(3.6)
subject to
E (0t)0))
Σ
r' ACaa,, st)
d d d.0.
t=
g(T))
where H: P(A) R is the entropy function, and C: S A R is the cost function.
4

---

Published in Transactions on Machine Learning Research (11/2025)
Following the dual formulation in 3.5
construct the Lagrangian and solve the saddle-point problem
maximize infx2o ((T,
mell
Since this problem has analytical solution, SAC-Lag employs actor-critic architecture and solves it using dual gradient descent. The architeclure consists of an actor network T(;; d)), a reward Q-network Qr,s, a; 0y), and cost Q-network Qe(8, a; 0c). where the parameters 0, and Og are updated by minimizing the respective TD-errors. The policy parameters e and the Lagrange multiplier À are updated iteratively using samples from a replay buffer D. The multiplier is adjusted via gradient descent on the dual objective
Ja(A) 1 ( A(d - Qc(s),a)))
This update increases when the expected cost Qe exceeds the limit d, thereby increasing the penalty for constraint violations in the policy's objective. The actor Th then updated maximize the Lagrangian, which corresponds to minimizing the actor loss:
(Ø) = ES,ND a log (s: @) Q,(s,, Φ)) + AQc(st, ((6:
(3.7)
4 Approach
4.1 Log Barrier Method
The log barrier method provides an alternative to Lagrange multipliers for handling inequality constraints like those in (3.4.. It works by augmenting the objective with barrier function that is only defined on the feasible set, i.e., goes to infinity when approaching the feasible set boundary from the interior, given by
(x) log(-zz))
(4.1)
where μ > 0 is parameter that controls the steepness of the barrier. The constrained problem 3.4 is then approximated by the unconstrained problem
maximize RER"
f(x) - ()(x))) = f(x) + [oo((=((x))
The barrier term is negligible when the solution is far from the constraint boundary (1(x) < 0) but ap- proaches -c as the boundary is neared (g(z) + 07), effectively preventing the optimizer from leaving the feasible set. The key advantage is that it replaces the non-differentiable hard constraint with a smooth, differentiable penalty.
4.2 Linear Smoothed Log Barrier Function
A critical limitation of the standard log barrier 4.1 is that it undefined for points not in the interior of the feasible set, e.g. consider the problem (3.4). the barrier function v(((x)) is undefined for æ R" with g(x) 0. This poses significant challenge in deep reinforcement learning, where: 1. Gradient-based optimizers may temporarily step into the infcasible region during an update. 2. The policy network is often randomly initialized, resulting in unsafe policy at the start of training.
Directly clipping the cost (define g(z) = 0 if g(z) > 0) would cause gradient saturation and prevent the agent from learning to escape unsafe regions. To address this, we adopt the lincar smoothed log barrier function proposed by Kervadee et al. (2019), defined as:
1 log( (x)) 1E log +++++ otherwise.
if
((x)
(4.2)
As shown in Fig. the function (4.2) is continuous and differentiable everywhere. It preserves the logarithmic barrier within the fcasible region but transitions a linear penalty for constraint violations. This allows meaningful gradients to flow even from unsafe states, guiding the policy back towards feasibility without numerical instability.
5

---

Published in Transactions on Machine Learning Research (11/2025)
Function Values
Derivatives wrt Constraint Value
10.0
Standard Log Barrier
Penalty Value
Log Barrier with Clipping
7.5
Numerical Instability (Value + w))
Gradient
Smoothed Log Barrier
5.0
Stable Non-Zero Gradient
Constraint Boundary
(Enables recovery from unsafe states)
Smooth
2.5
Transition
0.0
Vanishing Gradient
-2.0
-1.5
-1.0
-0.5
0.0
0.5
-2.0
-1.5
-1.0
-0.5 0.0
0.5
Constraint Value (Ve - d)
Constraint Value Uc d)
Figure 1: Comparison of different barrier functions and their derivatives. The left figure shows the penalty value, while the corresponding gradient is shown in the right figure. The slandard log barrier is numerically unstable for non-negative constraint values (æ: 2 0). Clipping the input (or gradient) solves this but results in a vanishing gradient for unsafe states (r > 0), halting learning. Our proposed smoothed barrier remains well- defined and provides a stable, non-zero gradient, enabling the policy to recover from constraint violations. The barrier parameter set to = 2.0 for visualization.
4.3 CSAC-LB
Our method, CSAC-LB, integrates the linear smoothed log barrier (4.2 into the SAC framework solve the constrained RL problem (3.6). Although the smoothed log barrier function provides mechanism for learning safe policy, its effectiveness relies the cost critic accurately estimating the policy's expected constraint violations. If the cost critic underestinates the policy's 's cost, recovery from unsafe policy be slow and unreliable. van Hasselt et al. (2016) identified an overestimation bias in Q-learning and proposed taking the minimum across multiple Q-networks to mitigate it. similar issue arises in the constrained setting; during actor optimization, there is a bias that can lead to an underestimation constraint violations. This is critical for the policy update, as may hinder the agent's 's ability to recover from unsafe policies.
To address this problem, architecture employs twin Q-networks for both the reward and cost critics. For the reward critic, follow the standard practice from Double DQN of taking the minimun of the Q-values to reduce overestination. For the cost critic, however, underestimation is the primary concern, it could lead the agent to select unsafe actions. Therefore, to ensure a conservative and pessimistic estimate of the expected cumulative cost, we take the maximum of the two cost Q-values. While other methods (Ji et al. 2024) may use a single cost critic, our double-critic approach for costs is designed explicitly mitigate the underestimation of costs.
A further challenge in RL is that the cost critic become overly conservative due biases in replay data and the approximation errors. To add flexibility and prevent the non-zero gradients of the barrier function from causing suboptimal reward-seeking, we introduce tunable offsct 5 2 0 (where 8 = 1 is used for all the experiments in this paper). Our final barrier function, vi, is applied to the output of the cost critic when training the actor, i.e.,
((Qc(s,)) = (max Qc.(ss - d- -)-- 1=1,
(4.3)
The combination of the fast-acting log barrier penalty and the pessimistic cost estimation provides strong, immodiate corrective signal when approaching or excceding constraint boundarics. This allows CSAC-LB recover from unsafe states more rapidly and reliably than methods relying on slower-adapting Lagrange multipliers optimistic cost estimates. The complete procedure is detailed in Algorithm.

---

Published in Transactions on Machine Learning Research (11/2025)
The critic networks Qr and Qe are trained by minimizing their respective mean-squared TD errors, where the actor loss is given by replacing the last term in (3.7) for SAC-Tagrangian, i.e.,
J(,) Es a ((s))()))))
(4.4)
4.4 Theoretical Analysis: Safety and Performance Guarantees
Our theoretical analysis provides two key guarantoes for the CSAC-LB algorithm: a guarantce on the salislaction of the safety constraints and a bound on the suboptimality of the final policy's 's performance. detailed proof can be found in the Appendix. A.4
Theorem (Suboptimality Guarantee). Let p² be the optimal point of the reinforcement learning problem under CMDP given by 8.5 i.e.,
p' inf (-JR(TT) Jc, (T) d: 50,, i m.).
(4.5)
where JR(z) is the expected return given by 3.1 and Jc, (T) is the expected cummllative cost for the i-th construint, given by (3.2). We also assume that. the reward and cost functions R(s,a), C(s,a) a) are bounded. Let nT be the optimal policy found by CSAC-LB, i.e.,
x²
argmin tE
-JR((T) + Σ (Ja.(T)) - -d, -6.)
(4.6)
where the function db is the linear smoothed log barrier (4.2), and 8; 2 0 is the offsel for the i-th constroint. If the constrained problem (.3) is strictly feasible, (i.e., Ie E 11 such that Jc, (Λ) < de for all i = .....,m) and the functions -J((T), (T) are conver, then the suboptimality bound of the solution x' is given by
-JR(m²) -p' m p
where is the wamber of construints and 0 is the barrier porameter.
Theorem 2 (Bounded Constraint Violation). Let n² be the optimal policy found by CSAC-LB, given by (466), with some hyperparameter μ for the barrier function (4.2). Suppose the reward and cost functions are bounded and the original constrained reinforcement learning problem (.3) is strictly feasible, then have
lim Jc. (T*) ) S d, + + 1-000
for all i= ===1,... 1....,m, m, i..., the constraint violation Jc, (x*) d, is at most 8, as + oc.
Theorem shows that CSAC-LB is formal approximation of the true constrained problem, with subop- timality gap of m/μ that can be controlled. This reveals the fundamental trade-off of the barrier parameter μe larger l reduces the theoretical performance gap by creating a steeper penalty landscape, but such sharp penalties can destabilize the neural network's optimization process as shown in Fig. 7 When the barrier parameter l set to be too large, the agent may struggle with exploring the boundary area and consequently learn conservative policy. We provide an empirical analysis in our ablation study in Sec. A..2 It is important to note that the convexity assumption required by our formal guarantees is generally not met in deep reinforcement learning. However, still learns robustly safe and high-performing policies in practice. Theorem 2 explains the rolc of the offset when μ + oo. In practice, while adjusting 5 is functionally similar to changing the cost limit d, its true utility comes from its interaction with the shape of our smoothed barrier function. By shifting the barrier's steep logarithmic region from d d it creates a "soft" " lincar penalty zone for minor constraint violations. This is particularly beneficial when the cost limit is very strict (e.g., d 0), as prevents the immediate, harsh penalty at the boundary and thus smooths the training process.
empirical results demonstrate (see Fig, 4 CSAC-LB
7

---

Published in Transactions on Machine Learning Research (11/2025)
5 Experiments
5.1
Experiment Setup
Environment To evaluate the generalization of our algorithm CSAC-LB and the other basclines, we conduct experiments in 10 tasks introduced by Luo & Ma (2021) and Ji et al. (2023). These high-risk, high-reward tasks cover a wide spectrum ranging from 2D navigation task to continuous control tasks and from simple pendulum Tilt task to high-dimensional humanoid locomotion task with speed limits. All tasks
are depicted in the Fig. P Notably, there are four tasks derived from Pendulum and Inverted Pendulum. A detailed introduction can be found in the Appendix, A.1
Figure 2: Visualization of the experimental environments. Inverted Pendulum, with two distinct tasks for each defined in Luo & Ma (2021
The enviromments include Pendulum and
Baseline We evaluate CSAC-LB against fix prominent baseline methods, chosen to represent different approaches to constrained reinforcement learning: 1) SAC with Reward Shaping, a standard SAC imple- mentation (Haarnoja et al. 2018) where penalty term is subtracted from the environment reward upon constraint violation (i.c. the modified reward function R(s, a) R(s, a) penalty factor C(s, where R(s,a) is the true reward function from the environments). The setup follows the methodology from Luo & Ma (2021). 2) SAC-Lagrangian (SAC-Lag), an off-policy algorithm that extends SAC using the Ta- grangian method to handle constraints (Achiam 2021). as detailed in Section 3.2 3) Worst-Case SAC (WCSAC), an extension of SAC-Lagrangian that incorporates the Conditional Value-at- (CVaR) of the cost distribution as its risk measure (Yang et al. 2021). 4) CPO, a widely-cited on-policy algorithm that enforces constraints using trust region optimization, providing strong thcoretical guarantees (Achiam) eet al. 2017). 5) APPO, variation of PPO with Lagrangian method by augmenting the policy loss with quadratic deviation term (Dai et al. 2023). Recognizing that the performance of SAC with reward shaping and WCSAC is highly sensitive to their respective penalty factors and risk levels, we conducted a hyper- parameter search to ensure a fair and robust comparison. Based on this search (summarized in Fig. + we selected optimal values of for the SAC penalty factor and 0.5 for the WCSAC risk level. For clarity in our results, we refer to these tuned baselines as SAC-6 and WCSAC-0.5, respectively. All algorithms were ro- re-implemented and trained with the same hyperparameters, as detailed in Table. 2
5.2 Experiment Results
Comparison with Baselines Fig. 3 depicts the learning curves of the performance and constraint satis- faction of all algorithms across environments. We set a challenging cost limit of 0 for the simpler tasks (shown in the upper row) to test numerical stability, and d = 25 for the more complex locomotion tasks (in the lower row). The results show that CSAC-LB consistently achieves high returns while adhering to the specified safety constraints across all environments. For example, in the Walker2d and Ant. tasks, CSAC-
8

---

Published in Transactions on Machine Learning Research (11/2025)
Tafe
SafePenddulmmiirightedd
400
SefetyCarCircieL-g
Retuim
SAC
SMC Log
SAC
-3
SAC
SACrue
-10
SSC
30
Lau
380
APPO
A7PO
Steps
Steps (K)
Stops (K)
Steps (K)
Stops [M]
SaferyyHopprr/elcctty..
Safety/biller2dWWlllcttyyy
Safe4ntVetocityy-/
SoferyHumanoitteelooyyyyy
6000
CPR
ON
SAL
1000
CANCLa
7000
CANCLA
Epicede
2005
2000
1500
2006
0.
000
100
000
Epicade
Sre-
400
410
200
200
100
5.5 Steps (M4
012
25 120
25
022 03 22:
Steps (M)
Steps (M)
Steps (M)
Steps (M)
Figure 3: Training performance and safety over time. The top and bottom panels show the episodic return and cost of constraint violations, respectively, versus training steps. Solid lines represent the mean evaluation performance across 5 independent random seeds, while the shaded regions denote one standard deviation. The dashed horizontal line in the cost plot indicates the cost limit for the task.
Normalizen ized Epicade Return
1.32
Sight Irart
Viy Unsile J4,
Exiornly Limile II
CSAC-BB
1.00
SAC-Lag
0P0
0.8:
0.74
0.61
0.67
APPO
0.6
SAC-1
0.4
0.41
0.40
0.2
0.0
0.00
0.13
SAC-10
SAC-8
WCSAC00
CSAC-LB
WCSAC-0. 0.5
OPO
APPO
0.18
SAC-1
SAC33
SACB8
SAC.a
SAC-10
WCSAC-0.3
WCSAC-0.
SAC-Lag
WCSAC-0.7
WCSAC-03
10
Environments
Figure 4: Aggregate performance and safety analysis across all environments, (Left) Normalized average return. The performance of each algorithm is scaled such that method (CSAC-LBB) scores 1 and the conservative baseline (WCSAC-0.5) scores 0, highlighting relative improvement. (Right) Distribution of constraint satisfaction. Evaluation runs are categorized into discrete safety levels based on their final cumulative cost. The bars show the number of environments for each algorithm that fall into each category. Both plots show results averaged over the final 5 evaluation episodes across 5 independent random seods.
LB achieves approximately 24% and 150% higher final returns, respectively, than the next best-performing safe baseline, SAC. In contrast, other methods demonstrate clear limitations. SAC-Lag often achieves high returns but at the cost of significant constraint violations, with average costs reaching up to 700 in the locomotion lasks. WCSAC and SAC generally remain safe but yield substantially lower returns. Moreover, WCSAC fails to learn safe policies in the Ant and HalfCheetah environments. CPO consistently underper- formod, cxhibiting extremcly conservative policy in locomotion tasks. However, we do note a performance degradation for CSAC-LB in the final stages of the PendulumTilt task and CSAC-LB is no longer among the best-performing safe policies. We hypothesize this occurs because once the policy converges, the entropy
9

---

Published in Transactions on Machine Learning Research (11/2025)
bonus from SAC can encourage unnecessary exploration near the boundary, leading to minor violations and a subsequent over-correction by the barrier function. As we discuss in Appendix A this can be mitigated by adjusting the offset parameter ò to prevent the convergence. We further conduct a statistics of the over- all performance of different algorithms as shown in Fig. The left panel presents the normalized average return, which is scaled to highlight the performance improvement over a conservative but suboptimal base- line (WCSAC-0.5). The results show that despite scoring at 1.32, SAC-Lag severely violates constraints. Whereas, other methods, such as the SAC variants, achieve significantly lower relative scores of 0.75 at their highest, 25% less than our methods. All APPO, WCSAC and CPO runs performed poorly. The right panel provides the corresponding safety analysis, showing the distribution of constraint satisfaction across the ten environments. CSAC-LB demonstrates a high degree of reliability, successfully solving 9 out of 10 tasks with safe policies. In only one task (Inverted PendulumMove) did it produce a slightly unsale policy, with a minor constraint violation of 0.6. In contrast, the high performance of SAC-Lag is shown to be at a cost, as it results unsafe policies in 7 out of the 10 environments. While SAC variants are predominantly safe, their low scores in the left panel confirm that this salety comes al significant performance cost.
CSAC-LB
SAC-Lag
SAC-6
WCSAC-0.7
1000
150
2.5
Episode Cost
20-
2.0
100
600
15
1.5
400
10
100
50-
1.0
200
0.
0
0f*
0
-1000
0
1000 2000
3000
2000
4000
-1000
1000
2000
-2000
0
2000
Episode Retum
Episode Return
Episode Return
Episode Return
Figure 5: Evolution of learned policies for different algorithms on the SafeAnt V task. Each point represents the return and cost of a single evaluation episode, with color indicating training progress (from blue/carly to red/late). The black dashed line indicates the cost limit d, while the crosses mark the Pareto front. The plot illustrates that our method, CSAC-LB, learns to safely explore the regime around the cost limit while efficiently converging towards high-return and safe solutions. The baseline methods showcase clear failure modes: SAC-Lag and WCSAC are unsafe, frequently breaching the cost limit, while SAC-6 learns an overly safe yet conscrvative policy.
Analysis of Exploration Behavior To provide a qualitative understanding of the advantage of CSAC-LB in terms of exploration, Fig. 5 visualizes the training behaviors of different algorithms on the SatoontVelooc ity task. Each point represents the return and cost of a policy during an evaluation episode, with its color indicating the training step at which the evaluation took place. Initially, CSAC-L.B adopts a conservative policy, operating well within the safe margin achieve a return of approximately 1000. As training pro- gresses, it briefty explores beyond the cost limit of d 25, behavior enabled by the stable linear penalty of our smoothed barrier, which allows the agent to find higher-reward states without destabilixing. In the final training stages, its policies converge to the constraint boundary, with most later-stage policies (red points) clustering near the cost limit. By precisely exploring this boundary, CSAC-LB successfully pushes the Pareto-optimal frontier, achieving a high final return of approximately 3000 while satisfying the safety constraint. The baseline methods, contrast, exhibit less effective strategies. SAC-Lag and WCSAC display high variance in their exploration; their evaluation points are scattered across the plot and different colored points all mix up together, indicating failure to converge to a stable policy. Critically, numerous later-stage policies (red points) significantly exceed the safety limit, demonstrating unreliable constraint satisfaction. Conversely, SAC-6 represents the opposite extreme: it is overly conservative. While it quickly learns a safe policy, it fails to explore the space near the boundary, causing it to prematurely converge to a suboptimal return of around 2000. Overall, CSAC-LB achieves a good balance between safety and performance. This success is attributed to our smoothed log barrier design, which permits the agent to effectively explore the crucial region near the constraint boundary. Compared to the bascline algorithms, CSAC-LB allows for gathering more information from the crucial region near the constraint boundary.
10

---

Published in Transactions on Machine Learning Research (11/2025)
Approximation Errors of Cost Critic To validate design choice of using twin cost critic network, we performed an ablation study the number of cost crities (N) used in our architecture. The results, presented in Fig. 6 highlight the critical issue of the critic network's value estimation accuracy and its effects. We use SafoCarCircle--v task with its cost limit d 0. This creates extra challenges for the cost critics. The single critic network (N=1) fails to maintain safely, consistently showing high episodic costs. This can attributed to the value underestimation. The left panel shows the single critic predicting large negative Q-values, a clear approximation error given that episodic costs are always non-negative. This leads the policy to incorrectly evaluate dangerous states as safe, resulting in the observed constraint violations. The introduction of second critic (N = 2) and taking the maximum of their Q-values rectifies this issue. This approach introduces a pessimistic bias that effectively counteracts the underestimation tendency, stabilizing the cost estimate at a value greater than O, mostly, and leading to safe behaviour in the end. While using more critics (N = 3, 4) also ensures safety, these configurations become overly pessimistic and prevent the agent from discovering higher-reward policies, causing their final returns be significantly lower than the N 2 case. Therefore, we conclude that using two cost critics provides the best. balance: it introduces sufficient pessimism to ensure stable and safe learning while avoiding the overly conservative behaviour with more critics.
Num of 0-network=1
17.5
O-netwerk-2
Minimum Value O.
Num 3-netwark=3
15.0
Episode Cost
Num of Q-network=4
12.5
10.0
-2
Episode
30-
7.5
5.0
10-
2.5
0.0
0.0
1.5 Steps (M)
2.0
0.5
1.0
2.5
2.0
Steps (M)
Steps (M)
Figure Training curves of different number of cost critic networks Qe in SafeCarCirelel-vd task. The left panel shows the minimum Q-value predicted by the larget cost critic(s) during batch updates, while the middle and right panels show the corresponding episodic returns and costs. Notably, the single critic configuration demonstrates significant underestination. Despite only non-negative costs occurring in the environment, it predicts negative Q-values. This flawed estimation leads the agent to perceive unsafe actions as safe, resulting in poor constraint iatistaciion.
6 Limitations
While CSAC-LB demonstrates robust performance, it important to acknowledge its limitations. The scope of our evaluation was conducted entirely in simulation; deploying CSAC-LB in real-world systems would require addressing additional challenges, such as the sim-to-real gap. Our formal guarantees on suboptimality and constraint satisfaction rely on idealized assumptions, including convexity, which do not hold in the general deep RL setting. Thus, the theory serves as valuable guidance for the algorithm's design, while its practical effectiveness is confirmed by our empirical results.
7 Conclusion
In this work, we introduced Constrained Soft Actor-Critic with Log Barrier (CSAC-LB), novel off-policy algorithm designed to overcome the eritical challenges of training instability and sample inefficiency in safe reinforcement learning. Our approach successfully integrates a linear smoothed log barrier function into the actor's objective. This core innovation provides a numerically stable gradient signal that allows the agent to learn effectively from constraint violations, enabling rapid and reliable recovery from unsafe states without the oscillations in Lagrangian methods or the gradient issues of traditional barriers. To further
11

---

Aaron D. Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control barrier functions: Theory and applications. In 2019 18th European Control Conference (ECC), pp. 3420-344 3431, 2019. doi: 20 Published in Transactions on Machine Learning Research (11/2025)
improve its safety, CSAC-LB employs a pessimistic double-critic architecture for the cost function, which provides conservative estimate of expected costs and robustly prevents the policy from exploiting critic underestimation errors. These design choices are supported by a theoretical analysis that provides guarantees on the policy's suboptimality and constraint violation.
Our extensive empirical evaluation across ten diverse and challenging benchmark tasks demonstrated the clear superiority of CSAC-LB. The algorithm consistently achieved a superior balance of high returns and strict constraint satisfaction, succeeding where other prominent methods failed, all while avoiding the need learned dynamics model. While baselines either violated constraints
for pre-training, reward shaping, in pursuit of high rewards or settled for overly conservative, suboptimal policies, CSAC-LB learned to proficiently and safely explore the crucial boundary of the feasible region. This a allowed it to discover policies that are both performing and safe, highlighting the effectiveness of its design.
The robust performance and stable learning dynamics of CSAC-LB establish as a powerful and practical solution for CMDPs. Future work could proceed along several avenues. One direction is to enhance the core algorithm by exploring adaptive mechanisms for the barrier's parameters (μ, 6) by integrating more advanced critic representations to tackle complex, high-dimensional problems. A second, broader direction is to extend our approach to more complex problem domains, such as real-world applications, multi-agent systems, enviromments with time-varying constraints, or problems involving stochastic safety specifications. Overall, CSAC-LB provides a reliable framework for developing intelligent agents that can operate effectively and safely in safety-critical domains.
8 Acknowledgement
This work funded by the Deutsche Forschunggemmcnscchattt (DFG, German Research Foundation) Project-ID 499552394 SFB 1597 and the German Federal Ministry for the Environment, Nature Con- servation and Nuclear Safety (BMU) on the basis of resolution of the German Bundestag as part of the 'Kl-Lecchtturm' project 'Intelligence for Cities' (I4C). This project is also funded by the German Research Foundation (DFG) under grants 417962828, 428605208 and 539134284.
References
Joshua Achiam. Exploration and Safety in Deep Reinforcement Learning. PhD thesis, EECS Department, ber.///eel.e.c...bb...llll......
University of California, Berkeley, May 2021. URL http://www2.eet htt.eee...reeeeeeeeeeebbbee////e www//ubss/ecchppteec R021/EECS-2021-8 httl
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 22 31. PMLR, 2017. URL http:://proceedings mlr, ppwpss//70/acchaaa///aa/
Yarden As, Iluura Usmanova, Sebastian Curi, and Andreas Krause. Constrained policy optimization via bayesian world models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, OpenReview.net, 2022. URL https:///penrrvvier. http:///opmmeeeieee.effffrrrrraa net/fforum?id PRZoSmCinhf
Arko Banerjee, Kia Rahmani, Joydeep Biswas, and Isil Dillig. Dynamic model predictive shielding for provably safe reinforcement learning, 2024. URL https://arxiv..ot www.aaawwiiioigrrrrrrr/aaaa org/abs//400s 13863
E. Barnard and L.F.A. Wessels. Extrapolation and interpolation in neural network classifiers. IEEE Control Systems Magazine, 12(5):50-53, -5,, 1992. doi: 10.1 10.1109/37.158800
12

---

Published in Transactions on Machine Learning Research (11/2025)
Felix Berkenkamp, Matteo Turchetta, Angela P. Schoellig, and Andreas Krause. Safe model-based reinforce- ment. learning with stability guarantees. In Isabelle Guyon, Ulrike von Tuxburg, Sany Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 908 918, 2017. URL htttp:://proceedings neurips. cwppsscccppoe cctppprr///ppp/////////pppp 2017 ht57h//26666bba02222200000111123 absssaaatt...e-tttaaatt.t......a
paper/
Dimitri P. Bertsekas. Chapter 4 exact penalty methods and lagrangian methods. In Dimitri P. Bertsekas (ed.); Construined Optimization and Lagrange Multiplier Methods, pp. 179-301. Academic Press, 1982.
ISBN 978-0-12-099340055. doi:
URL https:/tp:://wx
Stephen Boyd and Lieven Vandenberghe. Conver Optimization. Cambridge University Press, 2004. doi: 10.1017/CR0978001180444
Richard Cheng, Gábor Orosz, Richard M. Murray, and Joel W. Burdick. End-to-end safe reinforcement learn- ing through barrier functions for safety-critical continuous control tasks. In The Thirty-Thinn A Con- ference on Artificial Intelligence, AAAI 2019, The Thirty-Fire First Innovative Applications of Artificial Intel- ligence Conference, TAAT 2019, The Ninth AAAT Symposium on Educational Advances in Artificial Intel- ligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 Februory 1, 2019, pp. 3387 3395. AAAI Press, 2019. doi: 10.1609/ v33i0.33
URL atttps//doo org/10. (669//aaaa) v33101.33013381
Vinlam Chow, Ofir Nachum, Edgar A. Duéjez-Guzmále and Mohammad Ghavamzadeh. A lyapunov-based approach safe reinforcement learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchh, and Roman Garnett (eds.), Advances in Newral Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. aat.aaaaaeettteteaaaabata....... Abstract. html 2018 00119/2a//E850400033776666888880
2018. URL https://proceeddig// mertiiinnnnneeeiis...ee.. neurips.cc/ppper//
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Mohammad Ghavanzadeh, and Edgar A. Auéiez-Guzzáán Lyapunov-based safe policy optimization for continuous control. CoRR, abs/ Abs/1991.10031,, 2019. URL
http:://rxx: http::///xxvvooo//aaaa/a abs
www. 19911, wrtgabb//191..000..
Juntao Dai, Jiaming Ji, Long Yang, Qian Zheng, and Gang Pan. Augmented proximal policy optimization for safe reinforcement learning. In Brian Williams, Yiling Chen, and Jennifer Neville (eds.), Thirty- Seventh AAAT Conference on Artificial Intelligence, AAAT 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advonces in Arlificial Intelligence, FAAT 2023, Washinglon, DC, USA, February 7-14, 2023, pp. .3716.25888 Press, 2023. doi: 10.1609/ V3716.3.....1 25888, URL httph: //doi org/ 1609/668888.
AAAT
Gabriel Dulac-Arnold, Nir Levine, Danicl J. Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: defiiitions, benchmarks and analysis. Mach. Learn., 110(9):2419- 2468, 2021. doi: 00.0007//109944-02--00961 URL http///....r//0.110000////
810994-021-059911-
Altman Eitan (ed.). Constrained Markov Decision Processes. CRC Press, 1999.
Javier García and Fernando Fernández. A comprehensive survey on safe reinforcement learning. J. Mach. Learn. Res., 16:1437 1480, 2015. doi: 2155555/77992722888779. URL httpp://dl. acm. org/doi/10.
5555/2789272.u 2886795
Sehoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, and Jie Tan. Learning to walk in the real world with minimal human effort. In Jens Kober, Fabio Ramos, and Claire J. Tomlin (eds.), 4th Conference on Robot Learning, CoRL 2020, 16-18 November 2020, Virtual Event / Cambridge, MA, USA, volume 155 of Proceedings of Machine Learning Research, pp. 1110-1120. PMLR, 2020. URL http:s::/prooceedinnna mlr. wwt.:rrrssss11e111 hwwpe/5//hha...... html
13

---

Published in Transactions on Machine Learning Research (11/2025)
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause (eds.); Proceedings of the 35th International Conference on Machine Learning, 1CML 2018, Stockholmsmássan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1856- 1865. PMLR, 2018. URL http://proceediing mll. wtt.:iiws.......rrreereereeeeeee htrees///8//aaarojjaBbbbbb
Weidong Huang, Jiaming Ji, Chunhe Xia, Borong Zhang, and Yaodong Yang. Safodreamer: Safe re- inforcement learning with world models. In The Twelfth International Conference on Learning Rep- resentations, ICI.R 2024, Vienma, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: wttp://wwww..ettttoommmitt--oeed
Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang. Safety gymnasium: A unified safe reinforcement learning benchmark. In Thirty-seventh Conference on Newral Information Processing Systems Dalasels and Benchmarks Track, 2023. URL
Jiaming Ji, Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. Journal of Machine Learning Research, 25 25(285):116 2024. URL http://jmlr. http:////mm..oo/pppprr....
v25/23-0681 html
Hoel Kervadec, Jose Dolz, Jing Yuan, Christian Desrosiers, Eric Granger, and Ismail Ben Ayed. Constrained deep networks: Lagrangian optimization via log-barrier extensions, 2019. URL RtL.https: https://arxiv //arxiv. org/ abs/
1904. 04200]
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Contimuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, Son Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arxiv betexxivv..r//aee. htt//bbs/1500 02971
Puze Liu, Haitham Bou-Amman Jan Peters, and Davide Tateo. Safe reinforcement learning on the constraint manifold: Theory and applications. IEEE Transactions on Robotics, 41:3442-3400 34611 2025. doi: 10.1109/ TTRO.2025.3667411
Yongshuai Liu, Jiaxin Ding, and Xin Liu. TPO: interior-point policy optimization under constraints. Tn The Thirty-Forr Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AA Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7- 12, 2020, pp. 4940-4947. AAAT Press, 2020. URL https://ojs aaai wwwoor///n//.e. phtp:/aatt/arrtcce///ee////////p
Yuping Luo and Tengyu Ma. Learning barrier certificates: Towards safe reinforcement learning with zero training-time violations. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (ods.), Advances in Newral Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6- 14, 2021, virtual, pp. 25621-25632, 2021. URL https://proceedings.noo ctppppptppeppppppppppppe//e// httpp///ppppe////aaaa////// hashh/ 67718380444000221111110022266610 Awst44-bbtrrtt.ttcc
Haitong Ma, Changliu Liu, Shengbo Eben Li, Sifa Zheng, and Jianyu Chen. Joint synthesis of safety certificate and safe control policy using constrained reinforcement learning. In Roya Firoozi, Negar Mehr, Esen Yel, Rika Antonova, Jeannette Bohg, Mac Schwager, and Mykel J. Kochenderfer (eds.), Learning for Dynamics and Control Conference, L.4C 2022, 23-24 June 2022, Slonford Universily, Stanford, CA, USA, volume 168 of Proceedings of Mochine Learning Research, pp. 97-109. PMLR, 2022. URL https://proceedii mlr. pwwpss/vv6/mm222aa html
Enrico Marchesini, Davide Corsi, and Alessandro Farinelli. Exploring safer behaviors for deep rein- forcement learning. In Siitt AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty- Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Sym- posium on Educational Advances in Artificial Intelligence, FAAT 2022 Virtual Event, February 22
14

---

Published in Transactions on Machine Learning Research (11/2025)
March 1, 2022, pp. 7701-7709. AAAI Press, 2022. doi:
URL
https:
//doi.org/10 1609/aaai: v3617. 20737
Yurii Nesterov and Arkadii Nemirovskii. Interior-poine Polynomial Algorithms in Conver Programming STAM, 1994.
Marcus Pereira, Ziyi Wang, Toannis Exarchos, and Evangelos A. Theodorou. Safe optimal control using stochastic barrier functions and deep forward-backe backwadd sdes. In Jens Kober, Fabio Ramos, and Claire J. Tomlin (eds.), 4th Conference on Robot Learning, CoRL 2020, 16-18 November 2020, Virtual Event Cambridge, MA, USA, volume 155 of Proceedings of Machine Learning Research, pp. 1783 1801. PMLR, 2020. URL httpps://proceeeddnggg mlr. wtt.:pes/v1//ppeeerraa222
Kyriakos Polymenakos, Alessandro Abate, and Stephen Roberts. Safe policy search using gaussian process models. In Proceedings of the 18th international conference on autonomous agents and multiagent systems, pp. 1565 1573, 2019.
J. Rawlings, D.Q. Mayne, and Moritz Diehl. Model Predictive Control: Theory, Compulation, and Design. Nob Hill Publishing, 01 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. CoRR, abs/ 0os/1707.06347 2017. URL http: http://xxivooogg/baa alg07
Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. Learning be safe:
Deep RL with safety critic. CoRR, abs/2010.14603, 2020. URL http::://arx...orgg////////.... 14603.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning - an introduction. Adaptive computation and machine learning. MIT Press, 1998. ISBN 978-0-262- 19398-9.. URL
org/
0c1c/37293240
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery RL: safe reinforcement learning with learned recovery zones. TEEE Roboties Autom. Lett., 6(3):4915-4922, 2021. doi: 10.1109/
LRA. 2021.00225... URL htttps://oo.. org/ 10. 1109/LRA 2021.3070257 3070252
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. Dale Schuurmans and Michael P. Wellman (ods.), Proceedings of the Thirtieth AAAI Conference on Arlificial Intelligence, February 12-17, 2016, Phoenir, Arizona, USA, pp. 2094 2100. AAAI Press, 2016.
URL http://www. aaai. org/ ocs/ index. php/ /A//AAAAAAAAAAAA /ttp///AAAA////e//e/e///////////
Qisong Yang, Thiago D. Simão, Simon H. Tindemans, and Matthijs T. J. Spaan. WCSAC: worst-case soft actor critic for safoty-constraincl reinforcement learning. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirly- Third Conference on Innovative Applications of Arlificial Intelligence, TAAT 2021, The Fleventh Symposium on Educational Advances in Artificial Intelligence, FAAT 2021, Virtual Event, February 2-9, 2021, pp. 10639 10646. AAAI Press, 2021. URL httph://tpo/. aaai...aaaa....... org/ index. php/ hhtp/AAA//aticcee//eeaae 1 17272
Chengyang Ying, Xinning Zhou, Hang Su, Dong Yan, Ning Chen, and Jun Zhu. Towards safe reinforcement learning via constraining conditional value-at-risk. In Luc De Raedt (ed.), Proceedings of the Thirty- First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pp. 3673-3680 ijcai.org, 2022. doi: 10.24963 11196/3/jcaai2222//5/1 2022/510
URL https://dol. org 10. 24963/1jcai
Cheng Zeng and Hongming Zhang. A logarithmic barrier method for proximal policy optimization. CoRR, abs/1812.06502, 2018. URL http:// wttpr//..rr//b//11...00
15

---

Published in Transactions on Machine Learning Research (11/2025)
A Appendix
A.1 Environment Descriptions
This evaluation uses a diverse set of reinforcement learning environments designed to test different aspects of algorithm performance and safety.
low-dimensional Tasks
These lour tasks are based on classic control problems (Luo Ma 2021), but are modified to create high-risk, high-reward scenarios where the agent must operate near a safety boundary succeed.
Upright Based on Pendulum, the agent must keep the pole perfectly upright. The reward is maximized by minimizing the tilt angle, while safety constraint prevents the pole from falling.
Tilt Also based on Pendulum, this task requires the agent to balance the pole at a large, specific tilt angle, making exploration near the safety boundary necessary for high rewards.
Move Based CartPole, the agent is rewarded for moving the cart far from its starting point while keeping the pole nearly vertical and within light safety bounds.
Swing Also based on CartPole, the agent is rewarded for swinging the pole to a large angle, requiring it to manage its position while avoiding falling.
Gymnnsimmmm Benchmark
These tasks are from the standard Safety-Gymmnsium benchmark (Ji et al. 2023). testing navigation and locomotion under explicit safety constraints.
CarCircle-v1 A car-like robot must navigate circular path. The reward encourages the agent drive as fast as possible and be as close possible to the circle, while cost is incurred for crossing the track boundaries.
Velocity-Constraine Locomotion Tasks This suite environments (Hopper, Walker2D, HalfCheetah, Ant, Humanoid) features common MuJoCo agents with a shared objective. In each task, the agent is rewarded for maximizing its forward velocity while being constrained by safety limit set to 50% of the velocity achieved by vanilla PPO in the original task.
A.2 Ablation Study
To evaluate the sensitivity of CSAC-LB to its key hyperparameters, conduct ablation study the barrier parameter l and the offset 8. The results, presented in Fig. 7 show the training performance across different hyperparameter settings in two representative environments: the numerically challenging SafeCarCirclel-v00 and the complex locomotion task Safety
In CarCircle task, CSAC-LB demonstrates high robustness. While larger values of μ (e.g. 10) introduce minor training fluctuations, the final performance remains consistent across all tested values. This suggests that for tasks with simpler dynamics, the exact steepness of the barrier is less critical. The offset d shows more significant effect. Although most values yield similar outcomes, setting 3.0 leads to notable improvement in the final return (from approxinately 10 with our default settings to 17) while maintaining zero cost. This highlights the utility of 8 in creating "soft' linear penalty zone around the constraint boundary, ineventing the policy from becoming overly conservative, especially when the cost. limit strict (d=00 0 in this case).
Conversely, the Hopper task is more sensitive to the choice of μ.. As shown, lower values (μ=33 3.4) achieve high return of approximately 1500. However, increasing μ 6 or 10 causes significant performance
16

---

Published in Transactions on Machine Learning Research (11/2025)
SaretyCarCirdeelvv
SefetyCarCincle--v0
SadetyHlopprWWlccclyy--
Sefetyhoper/eiicillyy.
2000
1750
Jum3
6-0.5
1750-
6=100
Pponne beborn
12.5
10.0-
pnd
p-10
1250
1i=10
1250
0-2.0
0-6.0
1000
Eptsod
1000
7.5
5.0
790-
5.0-
2.5
500
500
0.0
250
23
1.5 Steps (90)
Stepa
Steps
50-
50
50-
40-
40-
40-
Ephocce Core
Ejhoo0e Cast
Eplegon Cast
30
20-
-04
50.
10-
0-
0.0 0.5
3.5
2.0
2.5
1.0
2.0
0.0
0.5
3.0
1.5
3.0
Figure 7: Ablation study on the barrier parameter μ and offset 8. Training curves for episodic return and cost are shown for SafeCarCirclel-v) and SafetyHopper Velopptrr.... The dashod horizontal line in the cost plots indicates the cost limit d. The results highlight the fundamental trade-off of the barrier parameter je: while the algorithm is broadly robust, excessively large values can destabilize training in complex tasks like Hopper. The offset d provides a mechanism for fine-tuning, improving performance in the strict CarCircle task without compromising safety.
drop around 600. This aligns with the trade-off identified in our theoretical analysis (Thoorem 1 large I creates steep penalty landscape that, while theoretically optimal, can destabilize the neural network's optimization process in complex control tasks, leading to suboptimal policy. Notably, all tested values successfully keep the cost below or very near the limit (d - 25). The algorithm is largely robust to the choice of 8 in this environment, with all settings converging to a similar high-performanne safe policy.
Overall, this study demonstrates the general robustness of CSAC-LB. It suggests that μ 4 provides good balance between theoretical optimality and practical stability, while ō can be used as an effective parameter for fine-tuning performance, particularly in tasks with very strict constraints.
A.3 Failure Case Analysis
In our main results (Fig. 3 we identified a late-stage performance degradation for CSAC-LB in the Pendu- TumTilt task. Figure g provides a deeper analysis of this behavior and demonstrates how the offset parameter serves as an effective remedy.
The issue stems from subtle interaction between SAC's automatic entropy tuning and the sharp constraint boundary enforced by our log barrier. With the default 5 1, the policy quickly converges to near- optimal, low-entropy state. As shown in the left panel of Fig. g this causes the policy's entropy to drop below SAC's fixed target entropy. Consequently, the automatic tuning mechanism dramatically increases the temperature parameter a to encourage more exploration. This forced, stochastic behavior near a strict constraint boundary inevitably lcads to constraint violations (right panel), which are then heavily penalized by the barrier function. This penalty, in turn, causes the policy to become overly conservative, leading to the observed collapse in performance (middle panel).
This analysis provides a clear guide for practitioners on managing this behavior. The most direct solution is to increase the ofīset parameter J (e.g, to or 3), which creates a wider "buffer zone" around the true constraint limit d. This allows the policy to maintain a slightly higher entropy without triggering the harshest region of the logarithmic penalty, causing the temperature a to remain stable and averting the performance collapse. An alternative, more advanced approach is to manually lower the target entropy for SAC, which directly reduces the pressure for forced exploration that causes the issue. While these tuning strategies are
17

---

Published in Transactions on Machine Learning Research (11/2025)
effective, exploring fully automated methods for managing this trade-off is promising direction for future work.
0
6=1.0
5
1.5
-10
Entropy Coefficient
6=2.0
6=3.0
Episode Return
-20
4-
Episode Cost
1.0
-30
3
-40
2
-50
-60
0.0
0
-70
60
100
20
60
100
40
80
Steps (K)
Steps (K)
Steps (K)
Figure 8: Analysis of the performance degradation in SafePondulummTilt-wwe and its mitigation using the offset 5. The figure displays the evolution of the SAC entropy coefficient (left), episodic return (middle), and cost (right) for different values of the offset parameter 5. With the default 5 = 1, the policy converges successfully but then degrades as the entropy coefficient rises to enforce exploration, leading to increased costs. Increasing 8 to 2 or 3 stabilizes the training by creating a larger buffer around the constraint boundary, preventing this late-stage collapse and maintaining high-performing, safe policy.
A.4 Proof of Theorem 1
Proof.
The Lagrangian for the constrained problem 3.3
given by
- +
(J)
(A.1)
where Ả (A1...., dm) ) RT is the Lagrange multiplier. The corresponding dual function is then
g(A) inf ((x,)). =ell
(A.2)
Since the policy n* given by (.6 is the optimal policy found by CSAC-LB via solving an unconstrained problem, the first-order optimality condition requires the gradient of the objective to be zero, i.c.,
VJn(m") Σ (J,,, d, V.,, = 0.
(A.3)
We can then construct a Lagrange multiplier X that is dual feasible by setting each of its component as:
x = (Jc, (TT) - de 6,) i= I...,,
(A.4)
(We have X 2 O since the derivative & is always nonnegative.) Substituting A.4 into A.1 and then taking the first derivative with respect to a for L(TT X*), we have
m -
By the optimality condition A.3 follows immediately that V=L(T", X²) 0. Since the Lagrangian is convex by our assumption, this implies that n² minimizes the Lagrangian L( X²) for this specific choice of
18

---

Published in Transactions on Machine Learning Research (11/2025)
X given by A.4 We can therefore evaluate the dual function A.2 at A
g(x²) inf I(T,,x))= I(x",*))
-JJR(T") + Σ (x")
= =Jn((T)) Σ - - di).
(A.5)
Notice that by weak duality, the primal optimal value p" bounded by the dual function, i.c., p" 2 g(²)) Together with (A.5), we have the following suboptinality gap:
-JR(T") - p" Σ (Ja((( - de (6)))(Jc. - d,).
(A.6)
Now we need find a uniform upper bound for the summation on the right-hand side of (A.6). Let 24 Jc, (x*) di, i == m...,,.. We then need bound the term
M(x)) - (z: 6z)x...
Case 1: Logarithmic Region. Suppose a; - 8, S -1/p², by 4.2 we have
V) 6.) - 1/ (1(1s 5.)).
Then h(z;)) can be expressed as:
h(x,) -r, m(zz 64)
-1
32
1
1+
8;
p(z; 6.) 1
Ij
6
Since 6, 2 0 and the denominator H, ò; negative and non-zero, we have conclude that
5
S 0. Hence, we
2.-5,
h(x;) - -
Case 2: Linear Region. Suppose xa 8, > - 1/11² by
4.2 we have
w (z; 5,) = μ.
Then h(xz)) can be expressed as:
M(x))) = -UT
Recall that xa > - 1/1²2 by our assumption, and hence, we have
1
1
h(xx)) -pr < 11 0;
dzj < 1
1
1
In both cases, each individual term in the summation of A.6 is bounded above by 1/p. We can therefore bound the entire sum
Σ
(Ja((x - d, (.), - d,) < m
(A.7)
i=
Substituting
A.7 back into A.6 we conclude that
-Jn(m²) p² 5 p
(A.8)
which ends the proof.
This result shows that the suboptimality of the solution is proportional the number of constraints m and inversely proportional the barrier parameter μ. The gap can therefore be made arbitrarily small by choosing sufficiently large μ.
19

---

Published in Transactions on Machine Learning Research (11/2025)
A.5 Proof of Theorem 2
Proof. According to the expression of the linear smoothed log barrier function 4.2
have vp s/
- 0o (where v is the original log barrier function given by (4.1) Let I_: IR - R U {00) be the indicator function for the nonpositive reals, given by
(-x)= 0
x > 0,
we have z -> I_ as ja >> oo Nesterov & Nemirovskij 1994 Boyd & Vandenberghe 2004).
Then follows
that v I_ as p + oo, and hence, the optimization problem corresponding to 4.6 given by
minimize aell
-Jn(m) + Ei (J).(T)) d; -:))
(A.9)
converges
minimize #ell
-JR(T)
(A.10)
subject to (T) S d, + 6,, i= i=
μ - Recall that x² is defined as the optimal point of the problem A.9 for some μ, and let n be the optimal point of the problem (A.10). The optimal point T. exists since the problem 4A.10 is relaxation of the original constrained reinforcement learning problem 3.3) whose optimal point exists by our assumption.
Then it follows from the discussion above that we have + To as + 20. Since the policy T satisfies
Jc, (T S d 6
for all i == I...., m. and Jc, (TT) + Jc. (TT))) as π- - To we conclude that
lim Ja, - d, 8, 4+0
for all I,..., m. which ends the proof.
20

---

Published in Transactions on Machine Learning Research (11/2025)
Table 1: Environments Overview
Environment
Basc
Agent
Observations
Actions
Upright
Pendulum
3
1
Tilt.
Pendulum
Move
CartPole
2
Swing
CartPole
2
CarCircle-vl
Car
24
2
Hopper
11
3
Wulker2D-Velocity--
Walker2D
MalfChecctb--elloctyy..v
HalfCheetah
17
6
Ant- Vnl-Velocity-vl
Ant
105
8
Humanoid- --ennniideeloottyyvll
Humanoid
348
17
Table 2: Hyperparameter Configuration
Hyperparameter
Value
Common Parameters
Batch Size
256
Network Architecture
[256, 256]
Discount Factor (7)
0.99
Random Steps
100
Learning Rate
x 10-3
Actor Update Frequeney
Critic Update Frequency
Polyak Update Factor
0.005
Initial Temperature
1.0
Normalize Reward
Yes
CSAC-LB Parameters
Offset
1.0
Log Barrier Factor
4.0
WCSAC Parameter
Damp Scale
10
CPO Parumelers
GAE Lambda (A)
0.95
Line Search Max Iterations
15
CG Max Steps
15
Normalize Advantage
Yes
21

---

Published in Transactions on Machine Learning Research (11/2025)
Algorithm Constrained Soft Actor-Critic with Log Barriers (CSAC-LB)
Initialize: Actor parameters æ.
Initialize: Twin reward critic parameters ,,,1,
Initialize: Twin cost critic parameters 0c,1,0e.
4: Initialize: Target networks with f²,; t Or,s and f + dc, for i = 1,2.
5: Initialize: Empty replay buffer D.
Hyperparameters: Cost limit d, offset ó, barrier factor μi, discount 7, entropy coef. a, target update rale T, learning rales λz,A@. Aq-
for each episode do for cach environment step do
Select action from policy: Execute a,, observe reward rt, cost Ct. and next state 81+1- Store transition (8t, at, Te, Ct St+1) in D.
(-(3))
10:
11:
12:
Sample a minibatch of transitions = ((,,,,,,,,s)) from D.
{-
Critic Updates }
With target policy actions a' Tx (-Is)):
Compute the reward target (using clipped doublc-Q):
13:
yr r [min;=1,2 Q6 (s',a') log n. (a' (a'ls")
Compute the cost target (using pessimistic double-Q):
Ye + c+ y (max;=1,2 Vo (s',a)).
14:
Update reward critics by one step gradient descent on the MSE loss: [(Or.)) (,,,)= [(Qe,, (s, a) ), for i 1,2.
15:
Update cost critics by step of gradient descent the MSE loss:
16:
[(Or.s) = (s.00= (8, y for i 1,2.
{- Actor Update }
For actions from the current policy Opolicy Tx (-18):
17:
Compute pessimistic cost Q-value: Qmaa (s, Apoliey) = max Q8 (s, apolicy).
18:
Compute the log barrier penalty: P(s, Apolicy) ((Q (s, Apolicy) d - 8).
Update actor by one step of gradient descent on the objective:
19:
J(T) = EB-BBann
min;=1,2 Q0.. (s, + a log nH (als) P(s,a))
Target Network Updates
Update all target networks using Polyak averaging:
20:
0' 7 (1 -)0,, for 1,2.
21:
0' 70 + T)0' for = 1,2.
22:
end for
23: end for
22


Reviews:
[{"review_id": "SHLlq2VLtG", "reviewer": "Reviewer_uWry", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes CSAC-LB, a variant of Soft Actor-Critic (SAC) that incorporates a linear smoothed log barrier function and a pessimistic double-critic design for handling constraints. The goal is to improve training stability and safety in constrained reinforcement learning (RL). The paper provides theoretical analysis and experimental evaluation across multiple benchmark tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper addresses an important and timely problem in constrained RL, where balancing performance with safety is critical.\n\nThe idea of integrating a smoothed log barrier function into the SAC framework is novel and provides a stable alternative to standard methods.\n\nThe use of pessimistic double critics for costs is well-motivated and supported by ablation experiments.\nThe paper is clearly written, and the proposed method is presented in a structured manner.", "audience_yn": "Yes", "audience_explanation": "The paper proposes an interesting and practically relevant method with clear theoretical and empirical contributions.", "requested_changes": "1.\tInsufficient references in Introduction:\n\nThe introduction makes several strong claims about the limitations of existing constrained RL methods (e.g., instability, conservatism, difficulty in handling constraints), but it lacks sufficient recent references to back these points. The authors should expand the literature coverage, particularly to include recent advances.\n\n2.\tBaselines are outdated:\n\nThe experimental section primarily compares CSAC-LB against classical baselines such as SAC, SAC-Lag, WCSAC, and CPO before 2021. You can compare with recent safe RL methods to strengthen the empirical evaluation.\n\n3.\tPerformance vs. SAC-Lag:\n\nFrom the reported results, it appears that SAC-Lag sometimes achieves higher returns. Since SAC-Lag is still a widely used baseline, a deeper analysis of why CSAC-LB underperforms in return compared to SAC-Lag in certain cases would be useful. For example, is the trade-off between safety and return inherent to the barrier design, or is it a matter of hyperparameter tuning?", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XEiBMmOHFn", "reviewer": "Reviewer_MQEj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes The paper proposes an off-policy safe RL algorithm in constrained settings where its primary aim is to learn a policy that is recoverable from unsafe state transitions while being averse to unsafe states. The authors provide a comprehensive strategy to implement their intended goal by two mechanisms mainly (i) through the use of a smoothed log barrier function for policy recovery (ii) correcting underestimation / overestimation bias of the cost and reward  Q-values respectively so as to make the final policy to converge towards the true optimal policy. They demonstrate through varied experiments the effectiveness of their approach in terms of boosting both the performance and reducing incursions to unsafe states. They also provide certain desirable theoretical properties that emerge from employing the policy updates they propose.\n\nStrengths:\n1. I think the paper is written very well providing a good flow of thought while also providing the motivation pretty clearly.\n2. Safety-constrained RL is a pressing challenge for real-world applications. The authors articulate well why existing methods (Lagrangian, CPO, clipping, etc.) are unstable or conservative.\n3. I found the experimental setup convincing for the most part and found it to address the claims made in the paper.\n\nWeaknesses:\n1. My main concern is in the theoretical results. While understand and appreciate the sub-optimality guarantee,  I am concerned about this guarantee holding due to the restrictive convexity assumption on the $-J_r(\\pi), J_c(\\pi)$. Under the more general case of non-convexity, there exists no guarantee that the $-J_r(\\pi*)$ closely matches the true intended objective $p*$. In such a case, can $\\pi*$ indicate a policy that is minimally safe (much like SAC-Lag in the experiments). I want to emphasize that this is a theoretical nuance, but should be stated clearly in the paper. Can you provide your thoughts on this?\n2. The paper suggests CSAC-LB is widely applicable without pretraining or prior knowledge, but all experiments are in simpler simulated benchmarks. I believe this is minor and may need to be discussed in a limitations section.\n\nQuestions:\n1. When you mean, SAC penalty factor , I want to make sure you mena the following:|\nReward  = true_reward - penalty_factor * cost\n\n2. Please add intuition for the suboptimality gap vs. barrier parameter trade-off in the main body, not just the appendix.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s two core ideas are well supported: (i) the linear smoothed log barrier is precisely defined (Eq. 4.2) and contrasted against standard barriers with value/gradient plots showing stability and non-vanishing gradients in unsafe regions (Fig. 1), directly supporting the “stable recovery from violations” claim.\n\n(ii) The pessimistic double cost-critic is motivated and ablated: using the max of two Qc’s counteracts cost underestimation; the ablation (varying the number of cost critics) shows a single critic predicts impossible negative costs and leads to violations, while two critics strike the best safety/return balance (Fig. 6).", "audience_yn": "Yes", "audience_explanation": "I believe the current work shows promise in the current safe RL landscape.", "requested_changes": "Please try to address the weakness and questions in the summary of the review above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "r6FGiys3dD", "reviewer": "Reviewer_zicq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes CSAC-LB, a model-free, sample-efficient, and off-policy reinforcement learning algorithm designed for safe RL in real-world systems where strict safety constraints must be satisfied during both training and deployment. The key innovation is integrating a linear smoothed log barrier function into the actor's objective, which enforces constraint satisfaction without requiring pre-training or suffering from instability issues common in traditional constrained RL methods. To further mitigate underestimation of constraint violations, the authors employ a pessimistic double-critic architecture for the cost function. Theoretical analysis provides suboptimality and constraint violation bounds, and comprehensive experiments on ten benchmark tasks validate the method’s robustness, safety, and performance advantages over existing baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "They provided comprehensive experimental results.", "audience_yn": "Yes", "audience_explanation": "Safe RL would interest some researchers.", "requested_changes": "Explicitly state the assumptions required for the theoretical guarantees, and clarify whether the results generalize to all CMDPs.\n\nDiscuss possible extensions, such as to multi-agent settings, time-varying constraints, or stochastic safety specifications.", "broader_impact_concerns": "No", "additional_comments": "### Strengths\n\n1. **Well-Motivated and Practical Problem**\n    \n    The paper targets a critical challenge in safe RL, **balancing safety and training stability**, which has direct implications for real-world deployment in safety-critical domains.\n    \n2. **Key Innovation: Linear Smoothed Log Barrier**\n    - Overcomes the limitations of standard log barrier methods that are undefined outside the feasible set.\n    - Provides **non-vanishing gradients**, enabling meaningful updates even from unsafe states and avoiding instability from abrupt gradient saturation.\n    - Naturally **penalizes constraint violations** while remaining negligible far from the constraint boundary, enabling **efficient exploration**.\n\n3. **Comprehensive Experiments**\n    - Conducted across **ten challenging tasks**, demonstrating the **consistency** of CSAC-LB in achieving **high returns** with **strict constraint satisfaction**.\n    - Qualitative analysis of **exploration behavior** (e.g., SafeAntVelocity) shows that CSAC-LB can explore near constraint boundaries more effectively than baselines.\n    - Visualization and analysis of **cost critic approximation errors** further support the method’s robustness and provide insight into architectural decisions.\n---\n\n### Weaknesses\n\n1. **Lack of Explicit Theoretical Assumptions**\n    - While the theoretical results are promising (e.g., suboptimality guarantee), **the assumptions are not clearly specified**. Does the guarantee hold for **arbitrary CMDPs**, or are there specific conditions required (e.g., bounded cost/reward, Lipschitz dynamics, compact state space)?\n2. **Performance Trade-offs**\n    - CSAC-LB is **not always the highest-performing method** in return, though it maintains safety. Discussion about **when and why CSAC-LB underperforms** in return (e.g., due to over-correction from the barrier function) could provide valuable insights and guide users.\n\n---\n\nOverall, this is an interesting and solid paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)