Handling Long-Term Safety and Uncertainty in Safe Reinforcement Learning
Authors: Jonas Günster
Puze Liu
Jan Peters
Davide Tateo
Venue: CoRL 2024 (2024)
Keywords: Safe Reinforcement Learning
Chance Constraint
Distributional RL

Abstract: Safety is one of the key issues preventing the deployment of reinforcement learning techniques in real-world robots. While most approaches in the Safe Reinforcement Learning area do not require prior knowledge of constraints and robot kinematics and rely solely on data, it is often difficult to deploy them in complex real-world settings. Instead, model-based approaches that incorporate prior knowledge of the constraints and dynamics into the learning framework have proven capable of deploying the learning algorithm directly on the real robot.
Unfortunately, while an approximated model of the robot dynamics is often available, the safety constraints are task-specific and hard to obtain: they may be too complicated to encode analytically, too expensive to compute, or it may be difficult to envision a priori the long-term safety requirements. In this paper, we bridge this gap by extending the safe exploration method, ATACOM, with learnable constraints, with a particular focus on ensuring long-term safety and handling of uncertainty. Our approach is competitive or superior to state-of-the-art methods in final performance while maintaining safer behavior during training.

Handling Long-Term Safety and Uncertainty in Safe Reinforcement Learning
Jonas Günster!
Puze Liu
Jan Peters
Davide Tateo?
1 TU Darmstadt, Germany German Research Center for AI, Germany
Abstract: Safety is one of the key issues preventing the deployment of reinforce- ment learning techniques in real-world robots. While most approaches the Safe Reinforcement Learning area do not require prior knowledge of constraints and robot kinematics and rely solely data, it is often difficult deploy them in complex real-world settings. Instead, model-based approaches that incorporate prior knowledge of the constraints and dynamics into the learning framework have proven capable of deploying the learning algorithm directly on the real robot. Un- fortunately, while an approximated model of the robot dynamics often available, the safety constraints are task-specific and hard to obtain: they may be too com- plicated encode analytically, too expensive to compute, or it may be difficult envision a priori the long-term safety requirements. In this paper, we bridge this gap by extending the safe exploration method, ATACOM, with learnable con- straints, with particular focus on ensuring long-term safety and handling of un- certainty. Our approach is competitive or superior to state-of-the-art methods in final performance while maintaining safer behavior during training."
Keywords: safe reinforcement learning, chance constraints, distributional RL
Introduction
Safety one of the most important problems when applying Reinforcement Learning (RL) to real- world applications, as it is fundamental to prevent RL agents from harming people or causing dam- age. To deal with this issue, RI. researchers developed Safe Reinforcement Learning (SafeRI.) techniques, to learn policies maximizing the task performance while satisfying the safety require- ments. Safe Exploration (SafeExp) aims to ensure the agent's safety during the exploration phase and formulates the safety problem as stepwise constraint that the agent should not violate.
Solving the SafeExp problem requires additional prior knowledge, such as constraints, robot dynam- ies, or previously collected datasets. While the SafeExp algorithms have been deployed successfully in complex real-world tasks [1, 2, 3, 4], most of these approaches suffer from many drawbacks pre- venting their application to complex out-of-the-lab tasks. First, safety specifications defined as constraints entail an in-depth understanding of the environment and dynamics. Designing and validating safety constraints requires extensive expertise and experience. Second, real-world appli- cations contain various uncertainty sources, such as sensor noise, model error, environmental distur- bance, and partial observability, which are often neglected in the design of constraints. Third, the op- timization techniques for constrained optimization problems are limited and often require a specific problem structure, such as quadratic programming with linear complementarity constraints. Lastly, the learning algorithms should also guarantee Long- Term Safety, which not only ensures the safety of the current step but also considers the safety of future trajectories. The term Long- Long-Term Safety
Code publicly available at httpss///ithinn wtt.cccccccaeeeeeeec wtt.m/uuue324//daaoeeee 324/d-44///0
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

unifies the Forward Invariance property for the static environment with known dynamics [5, 6] and the predictive safety in a dynamic environment without full knowledge of dynamics.
We argue that incorporating prior knowledge for safety-critical robotics applications can be benefi- cial, as prior knowledge, such kinematics and dynamics, often well-studied and readily avail- able. Furthermore, in robotics, can assume that a sufficiently good approximation of the dynam- ies model is available, while the complete model of the environment is not. Under this assumption, we combine techniques from model-free SafeRL methods and SafeExp approaches, showing how to exploit the knowledge of the dynamics while learning of unknown, long-term constraints. To achieve this goal, we extend the Acting on the TAngent Space of the COnstraint Manifold (ATA- COM) approach by dropping key assumptions that the constraints are predefined, allowing learning of long-term safety constraints directly from data. Furthermore, we explicitly model the constraints' uncertainty in distributional RL perspective, which provides with way estimate the to- tal uncertainty of the model. Consequently, we introduce Distributional ATACOM (D-ATACOM), allowing us to derive a awsk- policy by restricting the level of accepted risk.
Our experiments demonstrate that D-ATACOM achieves a safer performance during the training phase and reaches a similar or better performance at the end of training. For tasks where the optimal policy stays within constraints, D-ATACOM explores more cautiously the cost of slower learning speed. Instead, D-ATACOM show faster and safer behaviors during training whenever there is conflict between the policy optimization problem and the constraint satisfaction one.
Related Work In the last decades, SafeRL, is field of increasing interest for deploying learned safe agents to the real world. Constrained Markov Decision Processes (CMDP) [7, 8] framework as a first attempt from RL researchers has gained significant progress in recent research in solving constrained control problems. One important formulation of constraint is the expected cumulative cost. The RL agent aims to maximize the expected return while maintaining the expected cost below threshold [9, 10, 11, 12, 13, 14, 15, 16, 17]. This type of constraint has been extended different variants, such as the risk-sensitive constraint [18, 19, 20, 21] and the probabilistic constraint [22, 23, 24]. Different types of constrained optimization techniques are applied in the policy update process, such as the trust-region method [9, 20], the interior point method [12], and the Lagrangian relaxation method [7, 11, 13, 14, 18, 19]. Furthermore, the Lyapunov function is also used to derive a policy improvement procedure [10, 25, 26]. Notably, learning the value function of the cumulative cost has gained incremental attraction as it addresses long-term safety. Recent works have shared a common view that the value function of constraint provides a predictive estimation of safety, such as the feasibility value [27, 28), control barrier function [29, 30], and safety critic [31, 21]. In this paper, we leverage the idea of the safety value function. Instead of penalizing the unsafe policy in the objective, we combine the safety value function with a model- exploration method ATACOM.
2 Preliminaries
We formulate the safety problem in the framework of CMDP. A CMDP is defined tuple (S, A, P,r, k, k, 7) with state space S, action space A, stochastic function P S x A x S - I that represents the transition probability from a state to another state by an action, reward function r(s, a) E [Tmin m. constraint function k(s) E [kmin, kmax], and a discount factor E [0, 1). We first define the safety and feasibility following [28] as:
Definition 1. Consider constraint function k S - R and policy π S - A. i. A state is Safe if k(s) & O. ii). The Safe Set is defined as Sg = {sE S: k(s)) & 0). ii. The Unsafe Set is the complementary set S5 = St,S. iv. A state s is Feasible under a policy π if k(ss) & for all t ee {0, I,.., 20), 50 = 8, 0g = (st).
The SafeRL problem formulated as constrained optimization problem
max
E
rr(s oc)
s.t. F(s) & 0,
(1)

---

where F, depending on the perspective on the safety problem [32], can take different forms, such as
k(s)) & Vt
(2a)
P(k(st) > 0) nex Vi
(2b)
E
& ne (2c)
The first constraint in (2a) describes the hard constraint, to be satisfied at each time step. However, we cannot enforce these constraints in the setting of stochastic environments. To address this issue, can use chance constraints, as shown in (2b), restrict the probability of the violations be smaller than threshold Vee- Both (2a) and (2b) are stepwise constraints that focus on safety at the current time step. The last type of constraint, as shown in (2c), forces the cumulative cost of the trajectory to be smaller than a threshold T/- In this paper, we will focus on the long-term safety constraint in stochastic formulation, a combination of (2b) and (2c).
Distributional Reinforcement Learning Unlike the typical RL setting that considers the expected value of the reward (the cumulative cost in our case), distributional RL treats the reward (cost) as a random variable and, therefore. the value function describes the distributions of the random cumulative return (cumulative cost). Distributional RI. has shown superior performance in many benchmarking tasks, such as Atari Games [33, 34, 35] and Mujuco tasks [36] since the distributional value function contains more information beyond the first moment. The value function is represented as a random variable nd instead of a scalar of the expected value Q", the random variable Bellman equation has a similar form Zd (*) 2
R(s) + 7Z" where the distribution of the random variable S" depends on policy π and dynamics P. The distribution Z=(s) can be represented by different types of models, such as the network with fixed support [33, 34], Gaussian Network [37] and Implicit Quantile Networks (IQN) [35]. We demonstrate our method using Gaussian Networks. However, our method is not limited to the model type as shown in Appendix using IQN.
Safe Learning on the Constraint Manifold We briefly introduce the ATACOM approach [38, 3, 4], which forms the basis of our approach. ATACOM addresses stepwise hard constraint, as defined in Equation (2a). Furthermore, ATACOM assumes that the dynamic system of the robot is given nonlinear affine system, á ʃ(s) + G(s)a. ATACOM constructs the Constraint Manifold by introducing a slack variable as M = {(s, μ1) E D c(s, μ1) 0} with c(s, μt) k(s) + μ.. We assume that is equipped with a dynamic system مز ( and a is a class IC function Using the concept Constraint Manifold, a safe controller can be obtained by setting de e u)) = 0. The resulting controller has the following form
d
= ((s = -S. - -JSC + B.u,
(3)
U
(s(s) _ k()ss) and the Constraint Drift with Ju(s. = [Ja(s) Ac(s).Jc(s) =c(s))
(s)) {s)) induced by the system drift f(s). A(μx) (p))() (1,..., K) is K-dimensional diagonal matrix for the slack variable. B, is set of basis vectors tangent to the manifold. The first and the second terms on the right-hand side compensate the drift &b and retract the system to the manifold; the last term is the tangential term that drives the system along the constraint manifold. An RL agent only needs to learn a policy for the tangential action u ~ T(s), while the safety is guaranteed by the controller structure. Alg. 3 in Appendix A showed a detailed process of action mapping. In this work, we extend ATACOM for the chance constraint and propose new method simultaneously learn the policy and the long-term constraint online.
3 Long-term Safety under Uncertainty
In this section, we will introduce method to estimate the constraint for long-term safety, and then we show how to integrate the time-varying constraint into the ATACOM learning framework. The overall algorithm Alg. I can be found in Appendix A.
A 2 B denotes that two random variable and are equal distributions. "lass IC function: (1) continuous; (2) strictly increasing: (3) a(0) 0.
3

---

3.1 Feasibility Value Function for Long-Term Safety
To ensure long-term constraint satisfaction, we introduce the concept of Feasibility Value Function (FVF), which describes the expected cumulative constraint violation under policy л with an infinity horizon. Formally, we define the feasibility value function under a policy x as
V
E
max((((s))))))) so
(4)
We assume the constraint k(s) E [Rminn max is bounded and consequently V [0, fman When is an indicator function of the constraint violation, the FVF is analogous to the Constraint Decay Function (CoDF) introduced in [28], defined as (*))) NN((e), with N=(s) the number of steps to the first constraint violation. Unlike the CoDF, which assumes the unsafe state to be absorbing,
I-7
the FVF does not assume episode termination at the unsafe state, allowing the agent to retract back to a safe state in the future. Therefore, V (8) > (s). The feasibility value function estimates the expected discounted cumulative cost of max (ss)..) under policy æ. We can use the standard Bellman operator (B" (s) = max(k(s), 0) + >VF(s) to update the estimate. For continuous state-action space, a common choice to use a neural network approximate the value function and update the value function using TD learning. When V O, have max(k(s).
indicating ksss & Thus, the Feasibile Set Sp {s S VF (s) 0} is subset of the Safe Set. Ensuring the feasibility value function to be zero is sufficient to guarantee stepwise safety.
3.2 Distributional Feasibility Value Iteration
The original FVF definition in Eq. (4) evaluates the expected value over the future cost. This timation does not capture the distribution of the future cost and could fail to ensure safety when the distribution is multi-modal or heavy-tailed. Instead, we can exploit the theory of Distributional RL [39] that learns the parametric model to approximate the distribution of the future expected cost. Then, we can construct Value-at-Risk (VaR) /Conditiia Value-at-Rinn (CVaR) constraints that con- sider both the safety and the uncertainty of the prediction when drawing actions.
Different parametric models have been used to approximate the distribution of the random value function. In this paper, we approximate the target distribution max(k(s), VV with Gaussian support to the 2nd-order moment [37, 31], i.e. we assume (s) ~ - (I* (s), (s)). We can compute the mean of the target distribution f (s) = 1 k((s) + tpp (s') and the variance
(+) - (s)) + 2 (s) E E. (x + J-pp
E (<)))) + (N (<)) (n²
Here, k'(s) max(k(s);)) 0) and PT (s'1s) is the transition probability under policy æ. The distri- bution of FVF is parameterized by Gausian ((u5(s), (s), (s)). The TD error between the target distribution N ((n((s) of (s)) and the parameterized distribution N N( (s), (s)) with respect to the Wasserstein distance can be computed as
CF CF = (1= N5 (2) Tr (x) (x) (s))() 1/2
(5)
Since the FVF is one-dimensional, Eq. (5) becomes LF - 1| We use a Softplus activation for the mean and a Exponetial parameterization for the standard de- viation ensure both values are positive. We the Gaussian parameterization for illustration purposes. However, our method is not restricted to such parameterization. Experiments using IQN [35, 21] can be found in Appendix C.
([oF(()) (s)
During training, we keep a replay buffer of limited size. As training progresses, the agent will behave safer and encounter fewer constraint violations, flushing away unsafe transitions when using single replay buffer. Instead, we would like the agent remember the failures and avoid being overly optimistic, using separate, smaller, Failure Buffer Dy to store the unsafe transitions. In each data batch, we sample proportional number of data coming from this buffer.

---

3.3 Uncertainty-Awann Constraint using (Conditional) Value-at-Risk
VaR and CVaR quantify the risk of a random variable. VaR is the smallest value where the prob- ability of Z is bigger than a a and CVaR measures the mean of the a-tail of the distribution.
(Z)., inf(2 RIF(2) ah
(6a)
CVaRa = 10 [2|2 > [ (6b)
where F()) the Cumulative Distribution Function (CDF) of the random variable Z. When F continuous and strictly increasing, the VaR. is uniquely defined as VaRa(Z) (a), i.e., the quantile function. The VaR and CVaR offer risk- aware constraint formulation by restricting their values to be smaller than threshold 6. The CVaR constraint for the Gaussian distribution [40] of FVF
CV) n - (()) E
(7)
VaR,
where ip and & the Probability Density Function (PDF) and the CDF of the standard normal distribution, respectively, determines the probability of constraint satisfaction; thus, the risk is
1/2
CVAR,
Using the CVaR constraints (7), the constraint in problem (1) is defined as F(s) = CVaR (aaR_(a0 (s)
Figure 1: Distribution of V and illustration of mean, VaR (red), and CVaR (green). The shaded area shows the cumu- lative probability a.
During exploration, the TTuss), which is then converted to safe action (ss, u) using ATACOM. Detailed algorithm of ATACOM is shown in Alg. 3 in Appendix A.
agent draws an action
Adaptive constraint threshold estimate While ideally, we would like to have the FVF to be al- ways equal to zero, setting 8 = 0 is neither a practical choice since the network's mean is always bigger than O, nor beneficial for the training of FVF as it restricts the exploration. The threshold trades off the constraint violation and the exploration and requires further engineering. The experi- ment comparing different thresholds is illustrated in Appendix E.
To alleviate the engineering effort, propose an adaptive scheme that updates the å based on the current episodic cost and the esti- mation of the FVF. We use Softplus parametrization to keep the positive. The 5 parameter is updated during the learning process after each episode of horizon H using the following loss
Statc Space
H Cs
Σ 1=0
Litubrr (de(si). CVaRa
(8)
H
Feasible Sct
where the term (s():) DH bf C computes the differ- ence between empirical discounted cost starting from 8; and the accepted discounted cost budget Ō and CVaR, 8 computes the distance of the estimated FVF its threshold. We use Huber Loss [41] obtain a more robust update against outliers. As shown in Figure 2, 8 is tuned based on the empirical cost to its cost budget. If do(s;) is bigger than 0 (the actual cost is bigger than the budget), we reduce 8 for a more conservative policy. Conversely, if (s;) is smaller than 0, we increase å to loose the constraints.
O-level CVaR
Illustration of the feasible (light blue), the learned FVF at 0-level red and threshold 8. The thresh- old ó provides a small feasi- ble region (white) explore within a small cost budget.
Figure/ 2:
3.4 Policy Iteration with Learnable Constraint using ATACOM
As introduced in Section 2, ATACOM constructs a safe action space by determining the basis vectors of the tangent space of the constraint manifold. However, previous work assumes that the constraint given, fixed, and deterministic. This assumption is longer valid when the constraint trained during the learning process. Since the constraint function changes during training, the safe action space changes accordingly, leading to non-stationary Markov Decision Process (MDP), which
5

---

causes the failure of the training. Specifically, let a E A be the action applied to the environment and the u E I be the control input obtained from the policy u ~ T(s). ATACOM constructs an affine mapping W U - A. defined in (3), that maps an action in safe space into the original one. Combining ATACOM with the actor-critic framework, a value function estimator Qw( u) is trained approximate the expected return. Then, the policy Te(s) (s) updated by maximizing the (s, u). However, when the constraint is updated during the training process, the action mapping W will also change. The same action will result in different a at different steps. This variation of the action space leads to an unstable update of (s, u) and To()))
In the following, we will demonstrate how to address this problem for the Soft Actor Critic (SAC) algorithm [42]. However, it possible to use the same methodology to extend most Deep RL algorithms, e.g., DDPG [43] TD3 [44], PPO [45]. To solve this issue, we learn the value function of the original action space Qw (s, a), which is invariant to the constraints. Since Q is represented by neural network and can be differentiated, we can use the reparameterization trick to obtain the gradient for 0 (similar to TD3 and SAC [42]), the objective and policy gradient can be obtained as
max J No
E
(s, (u))},
Vol. = (sQu(s, a) W
(w)) (u) VeTa(s)
Note that in SAC, the soft Q-function includes the entropy term (((((()))) to encourage explo- ration. The entropy term is indeed constraint-dependent. Thus, updating the constraints may change the entropy of the policy, consequently, the estimated value function is not anymore proper. We gue that practically, the variation of the entropy bonus has negligible effect on the training stability because the entropy term is scaled down by coefficient. Furthermore, online training in the value function allows us to quickly adapt to the new policy entropy. The overall algorithm of D-ATACOM and the modified SAC algorithm are illustrated in Alg. 1 and Alg. in Appendix A.
Experiments
In this section, we compare the performance of our approach in three different environments with different characteristics. Details of the environment description can be found in Appendix B. We compare with SafeRL baselines such as the LagSAC [46] and the WCSAC [31]. All environments and algorithms based on SAC are implemented using the MushroomRL framework [47]. We use the implementations provided by OmniSafe [48] for PPO-based algorithms. We conducted hyper- parameter search on the learning rates, cost budget, and accepted risk with 10 random seeds. We present the results with 25 seeds whose hyperparameters perform the best tradeoff between perfor- mance and safety. Further hyperparameter search experiments are in Appendix D.
Cartpole This environment extends the classical Cartpole benchmark. The pendulum is initialized in upright position, and the goal is move the pendulum tip desired point while keeping the pendulum upright. The constraint enforces angle smaller than 90 degrees from the upright position. We further added position limit to the cart. Despite the simplicity of the environment, designing a feasible long-term constraint is very challenging due to the actuator limitation and cart position limits. As we can see from the results in Figure 3, our approach is the best bert-performing one among the SafeRL baselines terms of learning speed, while achieving small constraint vio- lations. SAC achieves a policy with higher performance, but this policy heavily violates the safety constraints it completely disregards the pole angle constraint while reaching the goal. The RCPO algorithm [11] achieves better performance, but violations are comparable with SAC. Notice that D-ATACOM requires feasible cost budget to generate feasible actions since single constraint vi- olation will result in a high sum cost, reaching the maximum violation. An unachievable low budget will lead to conservative performance due to a lack of exploration, as shown in Appendix E.2.
In this task, the goal to control a differential-driven TIAGo+ robot and learn a navigation policy leading the goal position while avoiding collision with moving Fetch Robot. The Fetch robot moves to its randomly assigned target with moving arm. In this task, agents do not observe the Fetch robot's s joint positions while the end-effector's position is available. The safety
Navigation

---

Discounted Return
Maximum Violation per Episode
Episodic Sum of Cost
1.0
400-
20
0.5-
200-
0.01
0-
0.00
0.25
0.50 Steps
0.75
1.00
0.50
0.25
0:50
0.75
1.00
0.b0
0.25
0.50
0.75
1.b0
165
Stops
165
Steps
165
D-ATACOM@O0
WC
LagSAC
PPOLag
RCPO
SAC
Figure 3: Learning Curves for the Cartpole Environment
Discounted Return
Maximum Violation per Episode
Episodic Sum of Cost
0.2-
300
-400-
0.1-
2-
1-
500
0.00
0.50 Steps
0.75
1.00
0.b0
0.25
0.50
0.75
1.b0
0.00
0.25
0.50
0.75
1.00
166
Steps
Steps
D-ATACOM@0.5
WCSAC@0.5
LagSAC
PPOTag
RCPO
SAC
Figure 4: Learning Curves for the Navigation Environment
constraint considers the smallest distance between the two robots. In this setting, the impact of the arm's 's motion on the constraint is not explicit, which generates stochasticity on the constraint. From the result in Figure 4, D-ATACOM clearly outperforms all the state-of-the-art methods both in terms of safety and final task performance. Inspecting the learned policies, D-ATACOM is the only approach that performs active collision avoidance behaviors. These collision avoidance behaviors are mostly achieved by the model-based treatment of the constraint function. In this setting, using the constraint gradient and the dynamic model is a strong inductive bias for the method. On top of that, the control system can exploit the physical meaning of the variables, such as other obstacle velocity, allowing it to compensate in advance for the other robot movements. In this task, most approaches behave the same. We argue that the conflict between the task objective and the constraint function, forcing the robot to cause a detour, problematic for the Lagrangian approaches. Indeed, Lagrangian optimization is trying to balance constraint satisfaction and policy improvement in the update step, possibly causing the algorithm to get stuck in local minima.
3dof Robot Air Hockey The objective of this task is to control 3-DoF arm to score goal in the robot air hockey task. The arm is controlled by providing acceleration setpoints, and the constraints include the joint position/velocity limits and collision avoidance with the table. Since the optimal strategy for hitting the puck toward the goal is achievable within the tables's ' boundary, high reward performance and low constraint violations are achievable at the same time.
In addition to common baselines, we also compare D-ATACOM with the original ATACOM with pre-defined forward invariant constraint (ATACOM+FI) and non-forward invariant constraint (ATACOM nFI). As illustrated in Figure 5, D-ATACOM approaches the performance of the AT- ACOM + FI while maintaining low constraint violations. ATACOM + nonFI performs similarly to unconstrained algorithms SAC, both in return and cost, as a poorly defined constraint can not ensure safety. In this task, D-ATACOM safer than the other baselines at the cost of slower learning per- formance. The reason for this performance drop that our approach learns to expand the safe region progressively and improve the performance. The final performance is lower the robot hits more cautiously at the boundary regions to ensure safety. while other approaches allow the robot to go outside for stronger hitting, as shown in Appendix E.4. A characteristic of this environment is that the constraints do not majorly affect an optimal policy. Therefore, constraint satisfaction is more difficult in the initial phases of learning than in the final one. This allows classical lagrangian meth- ods to be particularly competitive in the task. Figure illustrates the learned constraint at different training steps. We can clearly observe that the feasible region expands progressively and reaches good coverage at the final epoch. Since FVF is policy-dependent value function, the prediction at the corner regions is poor, as the policy does not achieve higher performance.
7

---

Discounted Return
Maximum Violation per Episode
Episodic Sum of Cost
100-
0.1
2.5
50-
2-
0.0
100+
0.0
0.00
0.25
0.50
0.75
1.00
0.50
0.25
0.50
0.75
1.00
0.b0
0.25
0.50 Steps
0.75
1.00
lo5
Stepe
166
165
D-ALACOM@0
WCSAC@09
LagSAC
PPOLag
RCPO
ATACOM non-HI
ATACOM +FI
SAC
Figure 5: Learning Curves for the Air Hockey Environment
Epoch 2
Epoch 10
Epoch 20
Epoch 50
Epoch
Table Boundary
Goal
Center Line
CVaR 6
Figure 6: Learned FVF in the Air Hockey task. The color demonstrates the value of the learned constraints k(s) CVaR (s) 8 in the half of the air hockey table. After short training with an initial dataset, the feasible region shrinks to a small region at Epoch 2 and then increases progres- sively, reaching considerable coverage at the end of training.
Limitations While this paper is first step towards safer and more efficient learning without pre- defined constraints, the above-mentioned methodologies cannot solve complex control tasks, such the 7-DoF Robot Air Hockey task with equality constraints. Furthermore, training directly real robots remains challenging with the current approach, as robots need to explore unsafe states to obtain the FVF. Another limitation of this approach is that it requires knowledge of the robot dynamics. However, even in the presence model mismatch, the FVF can still impose robust and safe behaviors, as the unsafe transitions caused by the incorrect nominal model are included during training. We show preliminary study on the robustness of D-ATACOM Appendix E.5. For this reason, fine-grained domain randomization is crucial step in successful sim-to-real transfer. Finally, this paper does not explore the possibility of combining constraint learning with known constraints. This can be easily implemented in the D-ATACOM framework. However, the resolution of confficts between constraints remains to be explored.
5 Conclusion
In this paper, we started to bridge the gap between SafeExp methods and model-free SafeRL ap- proaches. We extended the ATACOM framework to work with learned constraints, ensuring long- term safety and properly dealing with constraint uncertainty. Our results show that our method competitive with state-of-the-an approaches, outperforming them in terms of safety, keeping an on-par learning speed, and achieving similar better performance for environments with differ- ent characteristics. Also, the method does not require excessive parameter tuning, as it includes automatic tuning rules for the most important hyperparameters. Therefore, our work proves that including prior knowledge in data-driven methods can actually be beneficial for scaling SafeRL approaches. Although all of the experiments are trained from scratch, we believe starting with an offline dataset and pre-training will significantly reduce the initial violations, leading safe per- formance. In future work, we will further investigate how to integrate known local constraints with long-term safety. This integration will allow scaling the ATACOM approach real-world robotics tasks involving complex long-term constraints and human-robot interaction.

---

Acknowledgments
Research presented in this paper has been supported by the China Scholarship Council (No. 201908080039) and partially supported by the German Federal Ministry of Education and Research (BMBF) within the subproject "Modeling and exploration of the operational area, design of the Al assistance well as legal aspects of the of technology" of the collaborative KIARA project (grant no. 13N16274).
References
[1] A. Taylor, A. Singletary, Y. Yue, and A. Ames. Learning for safety-critical control with control barrier functions. In Learning for Dynamics and Control, pages 708-717. PMLR, 2020.
[2] K. Stachowicz and S. Levine. Racer: Epistemic risk-sensitive rl enables fast driving with fewer crashes. arXiv preprint arXiv:2405.04714 2024.
[3] Liu, K. Zhang. D. Tateo, S. Jauhri, Z. Hu, J. Peters, and G. Chalvatzaki. Safe Reinforcement Learning Dynamic High-Dinenssionl Robotic Tasks: Navigation, Manipulation, Interac- tion. Proceedings of the IEEE International Conference on Robotics and Automation. IEEE, 2023.
[4] P. Liu, H. Bou-Ammar, J. Peters, and D. Tateo. Safe reinforcement learning on the constraint manifold: Theory and applications. arXiv preprint arXiv:2404. 2024.
[5] K. Wabersich, A. Taylor, Choi, K. Sreenath, J. Tomlin, A. D. Ames, and M. N. Zeilinger. Data-driven safety filters: Hamilton-jacohi reachability, control barrier functions, and predictive methods for uncertain systems. IEEE Control Systems Magazine, 43(5):137- 177, 2023.
[6] A. D. Ames, S. Coogan, M. Egerstedt, G. Notomista, K. Sreenath, and P. Tabuada. Control barrier functions: Theory and applications. In 2019 18th European control conference (ECC), pages 3420-3431. IEEE, 2019.
[7] E. Altman. Constrained Markov Decision Processes with Total Cost Criteria: Lagrangian Approach and Dual Linear Program. Mathematical methods of operations research, 48(3); 387-417, 1998.
[8] E. Altman. Constrained Markov decision processes: stochastic modeling. Routledge, 1999.
[9] J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained Policy Optimization. In Interna- tional Conference on Machine Learning (ICML), 2017.
[10] Y. Chow, O. Nachum, F. Duenez-Guzman and M. Ghavamzadeh. Lyapunov-based Ap- proach Safe Reinforcement Learning. Conference on Neural Information Processing Systems (NIPS), 2018.
[11] C. Tessler, D. J. Mankowitz, and S. Mannor. Reward Constrained Policy Optimization. In International Conference on Learning Representations (ICLR), 2019.
[12] Y. Liu, J. Ding, and X. Liu. Ipo: Interior-point policy optimization under constraints. In AAAI Conference Artificial Intelligence (AAAI), volume 34(04), pages 4940 4947, 2020.
[13] A. Stooke, J. Achiam, and P. Abbeel. Responsive Safety in Reinforcement Learning by PID Lagrangian Methods. International Conference on Machine Learning (ICML), 2020.
[14] D. Ding, X. Wei, Z. Yang, Z. Wang, and M. Jovanovie. Provably Efficient Safe Exploration via Primal-Dual Policy Optimization. In International Conference on Artificial Intelligence and Statistics (AISTATS), volume 130, 2021.

---

[15] H. B. Ammar, R. Tutunov, and E. Eaton. Safe policy search for lifelong reinforcement learning with sublinear regret. In International Conference on Machine Learning. PMLR, 2015.
[16] A. I. Cowen-Rivers, D. Palenicek, V. Moens, M. Abdullah, A. Sootla, J. Wang, and H. Bou- Ammar. Samba: Safe model- & active reinforcement learning. Machine Learning, pages 1-31, 2022.
[17] M. Yu, Z. Yang, M. Kolar, and Z. Wang. Convergent policy optimization for safe reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
[18] V. Borkar and R. Jain. Risk-constrained markov decision processes. IEEE Transactions Automatic Control, 59())2574-2277) 2014.
[19] C. Ying, X. Zhou, H. Su, D. Yan, N. Chen, and J. Zhu. Towards safe reinforcement learn- ing via constraining conditional value-at-risk. International Joint Conference on Artificial Intelligence, 2022.
[20] D. Kim and S. Oh. Efficient off-policy safe reinforcement learning using trust region condi- tional value at risk. IEEE Robotics and Automation Letters, 7(3):7444-751, 2022.
[21] Q. Yang, T. D. Simão, S. H. Tindemans, and M. Spaan. Safety-constrained reinforcement learning with a distributional safety critic. Machine Learning. 11((3):55-887, 2023.
[22] N. C. Wagener, B. Boots, and C.-A. Cheng. Safe reinforcement learning using advantage-bar based intervention. In International Conference on Machine Learning, pages 10630-10640, PMLR, 2021.
[23] B. Peng, Y. Mu, Duan, Y. Guan, S. E. Li, and Chen. Separated proportional-integnn lagrangian for chance constrained reinforcement learning. In 2021 IEEE Intelligent Vehicles Symposium (IV), pages 193-199. IEEE, 2021.
[24] S. Pfrommer, T. Gautam, A. Zhou, and S. Sojoudi. Safe reinforcement learning with chance- constrained model predictive control. In Learning for Dynamics and Control Conference, pages 291-303. PMLR, 2022.
[25] Y. Chow, O. Nachum, A. Faust, E. Duenez-Guzmne and M. Ghavamzadeh. Lyapunov-based Safe Policy Optimization for Continuous Control. In Reinforcement Learning for Real Life (RLARealLife) Workshop in the 36 International Conference on Machine Learning. 2019.
[26] H. Sikchi, W. Zhou, and D. Held. Lyapunov barrier policy optimization. arXiv preprint 27X:v:2103.09230, 2021.
[27] Y. Z. Li, D. Yu, Y. Yang, S. E. Li, X. Zhan, and Liu. Safe offline reinforcement learn- ing with fcasibility- diffusion model. In The Twelfth International Conference on Learn- ing Representations, 2024. URL
wwwparvveeeeeeeeeeeeeeeeeeeeeeee nett/foruaAid=j3JJcaaaD
[28] Y. Yang, Z. Zheng, and S. E. Li. Feasible policy iteration. arXiv preprint arXiv:2304.08845, 2023.
[29] D. C. Tan, F. Acero, R. McCarthy, D. Kanoulas, and Z. A. Li. Your value function a control barrier function: Verification learned policies using control theory. arXiv preprint WxXiv:336.04022 2023.
[30] S. Liu, C. Liu, and J. Dolan. Safe control under input limits with neural control barrier func- tions. In Conference on Robot Learning, pages 1970-1980. PMLR, 2023.
[31] Q Yang, T. D. Simão, S. H. Tindemans, and M. T. Spaan. Wesae: Worst-case soft actor critic for safety- cenettccostt reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35 (12), pages 10639-10646, 2021.
10

---

[32] L. Brunke, M. Greeff, A. W. Hall, Z. Yuan, Shou, J. Panerati, and A. P. Schoellig, Safe learn- ing robotics: From learning- based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5:41 1-44,, 2022.
[33] M. G. Bellemare, W. Dabney, and R. Munos. A distributional perspective reinforcement learning. In International conference machine learning, pages 449-458. PMLR, 2017.
[34] W. Dabney, M. Rowland, M. Bellemare, and R. Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI conference on artificial intelligence, volume 32(1), 2018.
[35] W. Dabney, G. Ostrovski, D. Silver, and R. Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning, pages 1096-1105. 1105. PMLR, 2018
[36] J. Duan, Y. Guan, S. E. Li, Y. Ren, Q. Sun, and B. Cheng. Distributional soft actor-critic: Off-policy reinforcement learning for addressing value estimation error.. IEEE Transactions on Neural Networks and Learning Systems, 33(11)658466558 2022. d0i: 109//S 2021.3082568.
[37] Y. C. Tang, J. Zhang, and R. Salakhutdinov. Worst Cases Policy Gradients. In Proceedings of the Conference on Robot Learning, volume 100, pages 1078-1093. PMLR, 2020.
[38] Liu, D. Tateo, H. B. Ammar, and J. Peters. Robot Reinforcement Learning on the Constraint Manifold. In Conference on Robot Learning, pages 1357-1366. PMIR, 2022.
[39] M. G. Bellemare, W. Dabney, and M. Rowland. Distributional Reinforcement Learning. MIT Press, 2023, http://wwxz
[40] M. Norton, Khokhlov, and Uryasev. Calculating evar and bpoe for common probabil- ity distributions with application to portfolio optimization and density estimation. Annals of Operations Research, 299: 1099:12811311 2021.
[41] P. Huber. Robust estimation of a location parameter. In Breakthroughs in statistics: Method- ology and distribution, pages 492-518. Springer, 1992.
[42] T. Haarojaa, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[43] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wier- stra. Continuous Control with Deep Reinforcement Learning. In International Conference Learning Representations (ICLR), 2016.
[44] Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. International conference on machine learning, pages 1587-1596. PMLR, 2018.
[45] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347 2017.
[46] S. Ha, P. Xu, Z. Tan, S. Levine, and J. Tan. Learning to walk in the real world with minimal hu- man effort. In J. Kober, F. Ramos, and C. Tomlin, editors, Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 1110- 1120. PMLR, 16-18 Nov 2021. URL https: htttp::s//rrceedinga mll. crress/155/ha21c.. html.
[47] C. D'Eramo, D. Tateo, A. Bonarini, M. Restelli, and J. Peters. Mushroomrl: Simplifying reinforcement learning research. Journal of Machine Learning Research, 22(131):1-5, 2021. URL attp:://jll. www.oo/g/pp...o.oo.oo httpp/pppp////////////////// caterr//v22118000 106116..1.1111111111111111111 html.
11

---

[48] Jiaming Ji, Jiayi Zhou. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. arXiv preprint arXiv:2305.099344 2023.
[49] H. Sikchi, W. Zhou, and D. Held. Learning off-policy with online planning. In Conference Robot Learning, pages 1622-1633, PMLR, 2022.
[50] C. Dawson, Z. Qin, S. Gao, and C. Fan. Safe nonlinear control using robust neural Iyapunov- barrier functions. In 5th Annual Conference Robot Learning, 2021. URL https: // //opnrevieww
[51] G. Dalal, K. Dvijotham, M. Vecerik, T. Hester, C. Paduraru, and Y. Tassa. Safe Exploration in Continuous Action Spaces, arXiv preprint arXiv: 1801, 08757, 2018.
12

---

A Algorithms
Algorithm demonstrates the D-ATACOM, and Algorithm 2 shows the adapted SAC update used line
Algorithm D-ATTCOM with constraint learning
Initialize: FVF network é, numher of steps N, threshold 6, cost budget C, policy no, value function Q
1: for
I...N do
2: Construct CVaRR (s:) using p (st). Σg (sx) from Eq (7).
Draw action uy ze E and obtain safe action a, < ((s,, us) using Alg. 3.
4: Observe 85+1; Tt, kt from the environment.
5:
Save replay buffer (8,, a2, Te, ke, St+1] - D and (st, ke sc41) Dy kr > 0.
6:
If the episode terminates, update 8 using Eq. (8).
Sample batch transitions (s, r, k. from D u Dy. Update <- Φ - ag F using Eq. (5),
9: Update the value function Qw and policy т using SAC in Algorithm
10: end for
Algorithm SAC implementation for D-ATACOM
Initialize: policy parameters 0 and value function parameters learning rate 7)
Input: Batch of transitions = as a, s').
1: Draw action
obtain safe action a', B ((', using Alg. 3.
2: Compute log prohability log p'a'''" = log (1
log [B using the change of variahle rule. 1Qu (s', a²) + log ?')²
3: Update Q_ with the TD loss C. 1/ (Qu a)
4: Draw action and obtain safe a, B,, (u,, s) using Alg. 3.
Compute log probability, log pa log mo((s) log B.l. B/BBVa Va (Q, + a log pe). 6: Update policy 0 0 tH where VoQu(s,o) a) = Qa Vv)
Algorithm 3 Construct safe action with ATACOM
Input: state s, action u
1: Compute slack variable and constraint c(s, H1) = k(s) + μ.
2: Compute the Jacobian (s) (=s)))(s)) A(n)]. drift (iss) {s(s)((s).
3: Truncate the drift that have positive impact safety (h(s) max
Construct the tangent space basis B, by computing the kernel Ja using QR/SVD Decomposition.
5: Compute the safe action a using Eq. 3.
Output: safe action affine mapping matrix B.,
13

---

B Experiment Environments
In this Section, we provide the full description of the environments used for the experiments. In all environments, the cost value is continuous variable. A value greater than zero indicates how much the constraints are violated.
B.1 Cartpole
The cartpole environment, depicted in Figure 7a a classic control problem with the goal of moving the pole tip desired position (green point) by controlling cart. The pole one unit in length and is initialized in upright position cart is initialized on the left side of the rail, and the goal is move the cart towards the goal position of the pole tip on the right rail's side while keeping the pole upright.
the cart. The cart move rail 10 units long. The
The state space the environment s = [ar, sin 8 cos 0, 2 017 where r the position of the cart, t is the velocity of the cart, 0 is the angle of the pole with the vertical axis, and is the angular velocity of the pole. The action space a [- 1] where the action the force applied to the cart.
The reward function given goal position DG and pole tip position æT is defined r(s) clip(1 Is0-rrl 0, 1). The constraint function prevents the pole from deviating more than from the vertical axis. Thus we define the cost function as c(s) max (0.55 1,0).
B.2 Navigation
The Navigation task consists of two robots, one differentiall-riven TIAGo+ (white) that moves in a room while avoiding the Fetch robot (blue), as shown in Figure 7b. The Fetch robot constantly moves its robotic arm in a periodic motion, such that the end-effector draws lemniscate into the air in front of the robot. Additionally, the Fetch robot constantly moves to a randomly assigned target position using a hand-crafted policy that ignores the TIAGo. The agent controls the TIAGo robot to reach the target position while avoiding the Fetch robot, which serves as dynamic obstacle.
The state space consists of the cartesian position and velocity of the two robots, the target position of the TIAGo, the previous action, and the cartesian position and velocity of Fetch's end-effector. The action space the linear velocity in the x-direction and angular velocity around the z-axis of the TIAGo robot. These are converted into the left and right wheel velocities.
Given the distance to the goal de, the current orientation 0 and the goal orientation @G the reward is defined as:
r(s) dell sigmoid (30(Idel) 0.2)) 0 - 0.1|/a||
The constraint is the smallest 2d cartesian distance between the TIAGo base and every joint of the Fetch Robot. Additionally, the constraint also prevents the TIAGo from hitting the surrounding walls. Given the TIAGos position and cartesian position of the ith Fetch joint pip the Fetch cost (s(s) = max,(-(11prr Pfr w)) where a constant that accounts for the width of the robots. The wall cost is defined as ew (s) =
- w)) where &'an the distance to the wall.
The step cost c(s) max(cr( ow (3)).
(a) Cartpole Environment
(b) Navigation Environment
(c) Hockey Environment
Figure 7: The three Environments used for evaluation of all algorithms
14

---

B.3 Planar Air Hockey
In the Planar Air Hockey environment, the agent controls a 3-DoF robot arm with a mallet attached to the end-effector. The goal is to puck into the opponent's goal, located on the opposite side of the table, shown in Figure 7c. The episode terminates when the puck enters the goal or hits one of the table's walls.
The state space consists of the robots' joint positions, velocities, puck position, and velocity. The action space is the acceleration setpoint for each robot joint.
The reward for non-absorbing states is the change of distance between the puck and the goal. In absorbing states the reward depends the distance of the puck to the goal. Given the puck position [xt, v²1" at timestep t and the distance between puck and goal as d', we define the reward as:
50(d 1 )²)
not absorbing puck in goal
(1.55 5 clip(1='1,,,00.1))
r(s:)=
A(1 - 2 clip(ly') 0.1,0.0.33))
if puck on backboard next to goal if puck sidebars
(00.3 0.3 clip(1.43 0,1),
otherwise
where p constant that scales the reward. The constraint prevents the mallet from touching the sides of the table and the robot from violating its joint position and joint velocity limits. The mallet cost is defined (s) max;(-dip + w) where dip is the distance to the ith wall and is a constant that accounts for the width of the mallet. Given the joint positions and the joint
velocities d, the position cost is cp(s) = max; lu,i. -4 2i + Q1,i) and the velocity cost cv (s) =
max (9: - Ga.i: - + q..)). The total cost is c(s) = max(cr( (s), cv (v(s),))))))))))))))). eas (s).
15

---

C Implicit Quantile Network
IQN parametric model representing the quantile function of the distribution, which takes a quantile value as input and outputs threshold value so that the probability of being less or equal т. Let (s) be the quantile function at r E [0. 1] for the random feasibility value at state s. The TD error between samples T, T' (L0, 1J) for the transition (s, a, s , r,k)
(s) + >m (s') (3))
The IQN model can be optimized via the Huber quantile regression loss
[d2/2k, { [d] - k/2, otherwise
[d] <
(d) - where (x
In Figure we compare the Gaussian and IQN approaches for the navigation task. In this experi- ment, both algorithms use the same hyperparameters. The Gaussian approach slightly outperforms IQN in terms of performance and safety. We theorize that the souree of the performance difference the hyperparameters, which are tuned for the Gaussian assumption. The main difference in the constraint estimation is that the Gaussian approach predicts higher uncertainty leading higher per- formance and safety in this environment. To achieve the same similar with IQN, the cost budget or accepted risk has to be decreased. We plan to further investigate the performance of IQN-ATACOM in future work, especially in environments providing only sparse cost feedback.
Discounted Return
Maximum Violation per Episode
Episodic Sum Cost
0.2
330
3
2-
0.1-
500
0.00
0.25
0.50
0.75
1,00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.95
100
Steps
leb
leb
Steps
le6
D-ATACOM805
IQN-ATACOM@0.
Figure 8: Comparison between the Gaussian distribution assumption and the direct CDF estimation for the navigation task. Both experiments use the same hyperparameters.
16

---

D Hyperparameter tuning
In this section, we report the parameter tuning for the baselines in all tasks. general, we test the methods with different learning rates, cost budgets and safety parameters to ensure performance of the baseline is optimal. We report all the hyperparameter configurations we tried and indicate which configuration used for the main evaluation.
Every algorithm is first evaluated with the learning rates of le 5e and 1em3. To keep the computation reasonable, we use the same learning rate for the actor, the critic, the constraints, and the learning rates for the Lagrangian multiplier that are updated every step. We report the results of these experiments for each task in the following sections.
As second step, experimented with different cost budgets to get the best trade-off between safety and performance. Our goal is to get the least constraint violations possible while maintaining reasonable behavior. As we show in Section E.2, setting the cost budget too low can have an impact on the performance with no safety benefit.
Lastly, we tuned the cost-dampening parameters of LagSAC and WCSAC using the same principle used for the cost budget.
D.1 CartPole
Figure shows the results of the learning rate tuning for the CartPole task. We can see that RCPO and LagSAC have a learning rate that achieves the best performance. For PPOLag and WCSAC, the differences are more nuanced. Table shows all the parameters we tried for the Cartpole task. The resulting best parameters used for the main evaluation can found Table
RCPO
PPOLag
LagSAC WCSAC D-ATACOM
Sweeping parameter
learning rate actor/criticcconnstrant
(le-3 5e-1 1e-1)
cost budget
(0.1, 5, 25, 40) {1. 10)
cost dampening
learning rate lagrangian multipliers
0.035
0.035
{1e 5e -4 1.1e
accepted risk
(0.1. 0.5, 0.9}
Default parameter
epochs
100
100
100
100
100
steps per epoch steps per fit episodes per test network size batch size
20000
20000
10000
10000
10000
20000
20000
25
25
25
[128 128]
128
64
64
64
64
initial replay size max replay size
2000
2000
2000
200000
200000
200000
200000
200000
soft update coefficient
le-3
1e-3
le-3
warm-up transitions
2000
2000
2000
target kl update iterations
0.01
0.02
10
40
Table 1: Training Parameters for the CartPole task
17

---

RCPO
PPOLag
LagSAC
WCSAC
D-ATACOM
Sweeping parameter
learning rate actor/criticcccostraiit
5e 5e
le
4
5e
5
5e
cost budget
5
5
5
5
40
cost dampening
learning rate lagrangian multipliers
0.035
0.035
5e-4
5e-
5e-1
accepted risk
0.9
0.9
Table Result of hyperparameter tuning for the CartPole task
Discounted Return
Maximum Violation per Episode
Episodic Sum of Cost
RCPO
30
W
m
300
20
0.8-
200
10
0.6
100
0-
0.0
0.5
1.0
2.0
0.0
0.5
1.0 Steps
1'3
2.0
0.0
0.5
1.0 Steps
1.5
2'0
Steps
1e6
1e6
leh
20-
300
PPOLag
1.0
10
0.8-
200-
0-
0.3
1.0
2.01
0.0
0.5
2.0
1.5
2.0
Steps
1e6
Steps
le6
Steps
lef
1.0-
LagSAC
20
10-
0.5-
200-
0-
0.0-
0.50 Steps
0.00
0.25
5.000
0.75
Steps
Steps
WCSAC@0.5
20
1.0
10
2010
0.0-
0.00
0.50
0.75
1000)
0.5
0.50
0.75
1.b0
0.50 Steps
0.75
1.00
Steps
Steps
166
0.000
0.0005
1r00001
Figure 9: Learning rate ablation study for the Cartpole task. For each experiment we run 10 seeds with all learning rates the algorithm set to the respective value.
18

---

D.2 Navigation
Figure 10 shows the results of the learning rate tuning for the navigation task. We can see WCSAC the only algorithm where the learning rate has significant impact on the performance. Table shows all the parameters we tested for the navigation task. The resulting best parameters used for the main evaluation be found in Table 4.
RCPO
PPOLag
LagSAC
WCSAC D-ATACOM
Sweeping parameter
learning rate actor/critic/coonstaant
14 9,10
cost budget
{0. 1}
cost dampening
{1,10) 10) {le-
learning rate lagrangian multipliers
0.035
0.035
1e-1
accepted risk
{0.1, 0.5, 0.9}
Default parameter
epochs
100
100
100
100
100
steps per epoch steps per fit episodes per test network size batch size
20000
20000
10000
10000
10000
20000
20000
25
25
25
[128 128]
128
64
64
64
64
initial replay size max replay size
2000
2000
2000
200000
200000
200000
200000
200000
soft update coefficient
1e-3
1e
1e-3 3
warm-up transitions
2000
2000
2000
target kl update iterations
0.01
0.02
10
40
Table 3: Training Parameters for the navigation task
RCPO
PPOLag
LagSAC
WCSAC
D-ATACOM
Sweeping parameter
learning rate actor/critic/constrrint
1e-4
le-
1e-4
le
1e-4
cost budget
0
0
cost dampening
1
1
learning rate lagrangian multipliers
0.035
0.035
le-4
le-4
le-4
accepted risk
0.5
0.5
Table 4: Result of hyperparameter tuning for the navigation task
19

---

Discounted Return
Maximum Violation per Episode
Episodic Sum Cost
0.16
-350-
RCPO
0.14
-400-
0.12-
2-
0.20-
0.0
0.3
1.0 Steps
1.5
2.0
0.0
0.5
1.0 Steps
1.5
2.0 1cb
0.0
0.5
1.0 Steps
1.5
2.0
1c6
1c6
-350
PPOLag
0.16-
400
0.14
0.12
450
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
2.0
Steps
1c6
Steps
le6
Steps
le6
LagSAC
0.20
-00
0.15-
500
0.00
0.50
0.75
0.00
0.25
0.30
0.60
0.25
0.95
Sceps
Lef
Sceps
les
Steps
1e6
WCSAC@0.5
0.20-
3-
0.15-
2
-500-
0.10
0.25
0.50
0.00
0.25
0.50
hts
1.h0
0.25
0.50 Steps
0.75
1.60
Steps
Steps
0.0001
0.0005
1r00001
Figure 10: Learning ablation study for the Navigation task. For each experiment, 10 seeds with all learning rates of the algorithm set to the respective value.
20

---

D.3 Air Hockey
Figure 11 shows the results of the learning rate tuning for the air hockey task. We can see that RCPO and PPOLag learn safer behaviors compared to LagSAC and WCSAC, However, their discounted return is lower, and they need twice as many steps. Table 5 shows all the parameters we tested for the air hockey task. The resulting parameters used for the main evaluation be found in Table 6.
RCPO
PPOLag
LagSAC
WCSAC D-ATACOM
Sweeping parameter
learning rate actor/criticc/constrant
fle-3, 5e 14, 1,1e
cost budget
0
{0.
cost dampening
(1, 10)
learning rate lagrangian multipliers
0.035
0.035
(1e -5-- 16-1)
accepted risk
{0.1, 0.5, 0.9}
Default parameter
epochs
100
100
100
100
100
steps per epoch steps per fit episodes per test network size batch size
20000
20000
10000
10000
10000
20000
20000
25
25
[128 128]
128
64
64
initial replay size max replay size
2000
2000
2000
200000
200000
200000
200000
200000
soft update coefficient
le-3
1e-3
le-3
warm-up transitions
2000
2000
2000
target kl update iterations
0.01
0.02
10
40
Table 5: Training Parameters for the air hockey task
RCPO
PPOLag
LagSAC
WCSAC
D-ATACOM
Sweeping parameter
learning rate actor/criticcconsttrant
5e
le-3
5e
5e-4
5e-4
cost budget
0
0
0
0
1
cost dampening
learning rate lagrangian multipliers
0.035
0.035
5e
5e-4
5e
5e
accepted risk
0.9
0.9
Table 6: Result of hyperparameter tuning for the air hockey task
21

---

Discounted Return
Maximum Violation per Episode
Episodic Sum Cost
100
1,5
200
RCPO
100-
0.3-
0.0
0.5
1.0
1.5
2,0
0.0
0.5
1.0 Stops
1.5
2.0
0.0
0.5
1.0
1.5
2.0
1c6
1cb
Steps
1ch
200
PPOLag
1.0-
50
100-
0.5
0.0
0.5
1.0
2.0
0.0
0.5
1.0
1's
2.0
0.0
1.0
1.5
2.0
Steps
166
Steps
le6
Steps
leb
100-
1.5-
LagSAC
1.0-
50-
100-
0.5
0.04
0-
0.5s
0.00
0.2s
0.50
0.95
1.00
0.25
0.30
0.15
1.60
Steps
le6
Steps
le6
Steps
1e6
WCSAC@0.5
100-
1.5-
200
50
0-
0.25
0.75
0.25
0.75
..0
0.75
1.60
Steps
Steps
Steps
0.0001
0.0005
1r00001
Figure 11: Learning ablation study for the Air Hockey task. For each experiment, seeds with all learning rates of the algorithm set to the respective value.
10
22

---

E
Additional Experiments
E.1 Air-Hockey with Fixed Delta
The safety threshold 8 is important parameter controlling the trade-off between safety and explo- ration. A too-small threshold ensures safety at the cost of limiting exploration. Thus, the agent will increase the performance slowly. A higher threshold results in less restrictive exploration, but the constraint is then not effective to ensure safety.
Therefore, we propose an adaptive threshold that updates its value based on the empirical costs and its prediction of the FVF. In this experiment, compare D-ATACOM with multiple fixed values for ó in the planar air hockey task. Figure 12 compares these experiments with the introduced automatic tuning method. We can observe that fixed 8 has a detrimental impact on learning performance because exploration is restrictive by too strict constraints. Interestingly, this lack of exploration also hinders constraint estimation, which leads to slower convergence towards safe behaviors. With higher fixed å there are no exploration issues. However, a higher 8 consequently leads to higher constraint violations. With our adaptive threshold, we get a good trade-off between the learning performance and constraint satisfaction by having initial high 5 to encourage exploration, which then converges towards smaller sensible value that reflects given cost budget.
Discounted Return
Episodic Sum of Cost
100
50-
0.25
3.30 Stops
0.75
1.500
0.00
0.25
0.75
0.00
0.25
0.50 Steps
0.75
1.00
Stops
8Flearned
8=0.1
8=0.3
8=0.5
8=1
8=3
Figure 12: Performance of ATACOM with different fixed å values
23

---

E.2 CartPole with different Cost Budget
In this experiment, we will compare the impact of the cost budget parameter on D-ATACOM and WCSAC. We chose the CartPole task for this comparison because both algorithms do not learn completely safe policy. Figure 13 shows the performance D-ATACOM and WCSAC with differ- cost budgets. We observe that the performance of D-ATACOM is more sensitive the cost budget parameter compared to WCSAC. When the policy cannot achieve the given cost budget the performance of D-ATACOM degrades significantly. This performance drop occurs because the delta eventually will converge towards zero, which results in very conservative policy. The behavior for D-ATACOM with the cost budgets of 0.1 and 5 balancing to the pole in its initial position because the policy is too conservative to move towards the goal, as this will lead to constraint violations.
On the other hand, WCSAC is robust budget will increase the Lagrange multiplier, giving more weight the constraint. The difference is that the Lagrange multiplier does not set an explicit limit to the constraint like the delta does in D-ATACOM. Instead, WCSAC gives more weight to the constraint violations in the optimization problem, which has less impact on policy performance. Is worth noting that, depending on the application, one of the two behaviors would be preferable. In safety-critical applications, having an algorithm that strongly enforces the constraint violation, independently of the performance, preferable. Instead, when partial constraint satisfaction enough, may be better choose lagrangian-based algorithm.
the budget parameter. An unreasonable cost
Discounted Return
Maximum Violation per Episode
Episodic Cost
WCSAC@0.9 D-ATACOM@09
20-
1.0-
400-
0.5
200-
0
0.00
0.25
0.50 Steps
0.75
1.00
0.50
0.25
0.5g
0.75
1.00
0.00
0.25
0.50 Stops
0.75
1.00
1c6
Steps
166
166
1.0
400
0.5
200
0.0-
0.50
0.75
1.00
0.00
0.25
0.5u
0.00
0.25
0.50
0.75
1.00
le6
le5
Steps
106
cost budget=0.1
cost budget=5
cost budgot=25
cost. budget=40
Figure 13: Impact of the cost budget parameter on D-ATACOM and WCSAC performance in the CartPole task
24

---

E.3 Experiment with different Accepted Risk
In the distributional setting, the parameter accepted risk determines how much of the tail of the distribution we are willing to violate, i..,, how much risk we want to take. However, this not the only parameter that influences the safety of policy. Usually, there is another parameter that tuned with given cost budget that also influences how safe the behavior is. For WCSAC this parameter is the Lagrange multiplier beta, and for D-ATACOM it is the learned 6. To show the complete impact of the accepted risk, we fix to constant value such that it cannot compensate for the difference in the accepted risk. Figure 14 shows the performance of D-ATACOM with a fixed delta and different levels of accepted risk in the Navigation task. Clearly, lower accepted risk leads to safer behavior.
The impact of the accepted risk on the safety shrinks for D-ATACOM when the delta is learned. Delta compensate for high accepted risk, resulting in the safe policy lower accepted risk would produce. The accepted risk has an impact in this setting toward the beginning of the training when delta is not yet converged. Thus accepted risk determines how risky the exploration at the beginning of the training will be. Figure 15 shows the impact of different accepted risk settings on the air hockey task. Setting a lower accepted risk results in slower exploration, thus achieving slower convergence of the discounted return. The maximum violation and sum of cost are comparable for all accepted risk settings because, in the air hockey task, the constraint does not majorly affect the optimal policy.
Discounted Return
Maximum Violation per Episode
Episodic Sum of Cost
300
3-
0.15
2-
400
0.10
1-
0.05
0.00
0.25
0.50
0.75
1,00
0.00
0.25
0.50
0.75
1.b0
0.00
0.25
0.50
0.75
1.00
Steps
le6
le6
Steps
le6
accepted risk-0.1
accepted risk-0.5
accepted risk-0.9
Figure 14: Impact of accepted risk on performance the Navigation task with fixed delta. The plots are smoothed via the exponential moving average with 0.9 weight
Discounted Return
Maximum Violation per Episode
Episodic Sum of Cost
100-
0.050
025
100
0.000
0-
0.25
0.50 Steps
0.75
1.00
0.25
0.50 Steps
0.75
1.00
0.00
0.25
0.50 Steps
1.00
accepted risk-0.1
accepted risk-0.S
accopted risk-0.9
Figure 15: Impact of accepted risk on performance the air hockey task
25

---

E.4 Analysis of Air Hockey
In the air hockey task, D-ATACOM cannot reach the same discounted return as LagSAC and WC- SAC. We investigate the final performance of the policies to understand the differences that lead to the performance gap. As D-ATACOM results in a safer policy, we theorize that performance is lost when the puck initialized too close the edge of the table. To test this hypothesis, we evaluate the performance of the final policies with an adjusted region for the initial puck position, that omits these critical positions. Figure 16 shows the performance for the original and adjusted regions and the difference between them.
For the original region D-ATACOM has significant outliers in the discounted return compared to WCSAC and LagSAC. However, LagSAC and WCSAC have more outliers in the maximum vio- lation and sum of cost. Thus, WCSAC and LagSAC sacrifice safety gain a stable performance. The safe exploration of D-ATACOM results in the opposite behavior, where the policy will sacrifice performance to ensure safety.
When we evaluate the performance with the adjusted region, we can observe that the discounted return of D-ATACOM increases more compared to WCSAC and LagSAC. Additionally, the decrease in maximum violation and sum of cost is more significant for LagSAC and WCSAC. This result confirms our hypothesis that D-ATACOM does not properly hit the puck when it close the edge of the table because it not possible do safely. WCSAC and LagSAC learn hit the puck in these critical positions, but this comes at the cost of safety.
Discounted Return
Maximum Violation
Sum of Cost
100
2.0
0.08
1.5
Original
80
0.06
1.0
40
0.02
0.5
0.00
0.0
1.5
0.08
100
0.06
Adjusted
1.0
0.04
0.5
0.02
40
0.00
0.0
12
0.00
0.0
10
Difference
-0.02
0.5
-0.04
-1.0
2
0.06
Original Spawn Region
Adjusted Spawn Region
D-ATACOM
WCSAC
LagSAC
Figure 16: Performance of the final policy from D-ATACOM, WCSAC, and LagSAC in the air hockey task. The performance evaluated with the original and an adjusted region for the initial puck position.
26

---

E,5 Experiment with Dynamic Model Mismatch and Disturbances
The D-ATACOM exploits the knowledge of the dynamics model to derive the safe policy. The model mismatch could potentially lead to unsafe behaviors. However, D-ATACOM able to ensure safety under model-mismatch, as the FVF is learned from the dataset collected from the mismatching environment. In this section, we empirically show how dynamic model mismatch will affect the performance of the algorithm.
We use the 3-DoF Air Hockey task as the study example. The robot is controlled by acceleration, we use a simple dynamics model
To simulate the model mismatch and disturbances, the dynamics model used in the simulator is
i = a o9 +
where c <~ (N o) is the Gaussian disturbance with standard deviation a, and -oq is an unmod- elled damping term. The experiment comparing the effect of different 5 is shown in Figure 17.
The magnitude of the noise heavily affects the performance of the discounted return, the RL agent converges to more robust policy that is more conservative. However, the safety performance remains consistent. This is because the FVF is learned directly from the noisy simulator, and the adaptive threshold balances the exploration and empirical safety during the learning process.
Training directly on real robots remains challenging with the current approach, as robots need to explore unsafe states to estimate the FVF. However, we believe our approach can achieve zero-shot transfer when trained in realistic simulator. Moreover, since FVF is independent of the dynamic model and the adaptive threshold estimation, are able exploit domain randomization during the training process, leading to safe policies that are robust to dynamic mismatches.
Discounted Return
Maximum Violation per Episode
Episodic Sum of Cost
100-
0.050
2
0.025
100-
50-
000
0-
0.00
0.25
0.50
0.75
1.50
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
1.00
Steps
lef
Steps
leh
Steps
0-0.2
0.4
0.8
Figure 17: Impact of model mismatch and disturbances of D-ATACOMs in the air hockey task. We add zero mean Gaussian disturbances with standard deviation and an unmodelled damping term of the form -aq to the environment.
27

---

E.6 Model-Based Baselines
In this experiment, we compared D-ATACOM with other model-based approaches the Planar AirHockey task. We use the OmniSafe [48] implementation of SafcLOOP [49], which learns the dynamic model and the policy jointly the same time. Additionally, we implement CBF-SAC, which learns the Control Barrier Function and the policy jointly and tries ensure the forward invariant property via the agents dynamics. This approach is an adaptation of the work presented in [50] to the online reinforcement learning setting. Finally, use SafeLayerTD3, which is an exploration method that learns the constraints that encode the dynamics [51].
As shown in Figure 18, CBF-SAC and SafelayerTTD3 achieve on-par performance to D-ATACOM in discounted return. CBF-SAC manages reduce constraint violations throughout training and converges reasonable safety level. However, it converges slower and towards less safe solution compared to D-ATACOM. SafeLayerTD3 struggles to ensure safety properly because it learns the constraint using supervised learning. This approach fails when the target constraint does not induce long-term safety. SafeLOOP uses a learned dynamics model and look-ahead planning to check safety. Due to the large computational demand, we run the experiment for only 500.000 steps, which is enough for SafeLOOP to converge to a safe behaviour. However, SafeLOOP achieves poor performance in terms of discounted return as learns very conservative policy that barely moves the agent. In summary D-ATACOM achieves the same better performance with stricter adherence to constraints compared to the model-based baselines.
Discounted Return
Maximum Violation per Episode
Episodic Sum of Cost
100-
0.2
200-
50-
0.0
100
0.00
0.55
0.50 Steps
0.75
1.00 lc6
0.b0
0.25
0.50 Steps
0.75
1.0 lo6
0.b0
0.25
0.50 Steps
0.75
he le5
D-ATACOM@09
SnfeLOOP
CBF-SAC
SAC
Figure 18: Learning Curves for Model-Based algorithms in the Air Hockey Environment
28


Reviews:
[{"review_id": "dtMqppKxbC", "reviewer": "Reviewer_ydKm", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper focuses on safe reinforcement learning, building on ATACOM and proposing distributional-ATACOM to integrate dynamics knowledge with constraint learning. Specifically, it addresses long-term safety by estimating a distributional value function for constraint violation. Experiments in Cartpole and navigation demonstrate that the proposed method achieves a lower constraint violation rate, albeit at the cost of a reduced learning rate.", "summary_of_recommendation": "The paper addresses an important problem, but the method requires further clarification. Additionally, the discussion on how the method can be deployed in real-world scenarios needs to be expanded.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 2, "robotics_focus": 2, "quality_of_limitations": 3, "questions_for_rebuttal": "- Could you provide a general algorithm to show how the different parts of the method are connected?\n\n - Is it possible to deploy this method in a real robot setting? If not, what are the potential ways to explore deployment?\n\n - How does D-ATACOM compare to ATACOM?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XLwm5le3lk", "reviewer": "Reviewer_PBPX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors address a problem of safety with unknown constraints. To address this, the authors extend the existing framework of ATACOM to a setting where the constraint has to be learned. The ATACOM is combined with learning of the feasibility value function, using the distributional method concepts like VaR and CVaR.", "summary_of_recommendation": "The paper needs to improve clarity of the main algorithm, with a clear justification how the proposed scheme solves the problem the paper is addressing. Moreover, the results have to be more convincing that the proposed method can solve nonstationary and long-term safety problems effectively.", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 2, "quality_of_limitations": 3, "questions_for_rebuttal": "- Can ATACOM deal with control bounds? Also, if the B_u term in equation (3) vanishes, what happens? (This looks similar to the high relative degree problem of Control Barrier Functions.)\n- It is not clear to me why the concept of CVaR is necessary for addressing the nonstationary or long-term safety, the main theme of the paper.\n- It would be good to have a pseudo-code of the algorithm in the rebuttal so that I can understand the learning method in more detail.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BDJUaj9pOD", "reviewer": "Reviewer_2U5V", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents D-ATACOM, an algorithm combining the SafeExp and CMDP approaches towards Safe RL with the goal of minimizing safety violations during training while still converging towards a constraint-satisfying policy efficiently. To do so, the authors extend the ATACOM for SafeExp by jointly learning the safety value function during training and accounting for uncertainty and stochasticity in the safety constraint when defining the Constrained Manifold.", "summary_of_recommendation": "I believe the community would find this paper interesting, as it presents a novel angle to the problem of Safe RL which could be relevant to robotics.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 2, "quality_of_limitations": 3, "questions_for_rebuttal": "Comments:\n- What is $A(\\mu)$ defined after equation (3)?\n- How is the FVF used to construct the constraint manifold (and ultimately, the action mapping $W$)?\n- In Figure 6, what does the color represent? I'm assuming this is $CVaR - \\delta$  but this should be stated more clearly.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Post-rebuttal metareview**:

This paper extends the ATACOM framework by learning a feasibility value function for solving safe RL problems with known dynamics but unknown constraints. The setting is quite challenging and novel, and the proposed method is convincing. Some reviewers' concerns were addressed in the rebuttal phase. Adding real-world experiments and comparisons with model-based safe RL approaches will strengthen the paper.

------------
**Pre-rebuttal metareview**:

Strengths:
1. Safe RL with unknown constraints is an important and interesting setting.

Weakness:
1. The proposed method requires knowing true dynamics and cannot easily scale up.
2. Need better presentation, especially for algorithms.
3. Need more baselines.
4. The experiment design doesn’t fully show the main point of the proposed method.
5. Other issues mentioned by reviewers.
Decision: Accept