Handling Long-Term Safety and Uncertainty in Safe Reinforcement Learning

Authors: jonas g ünster
puze  liu
jan  peters
davide  tateo
t u darmstadt
  germany
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 01:08:36

Abstract: Safety is one of the key issues preventing the deployment of reinforcement learning techniques in real-world robots. While most approaches in the Safe Reinforcement Learning area do not require prior knowledge of constraints and robot kinematics and rely solely on data, it is often difficult to deploy them in complex real-world settings. Instead, model-based approaches that incorporate prior knowledge of the constraints and dynamics into the learning framework have proven capable of deploying the learning algorithm directly on the real robot.
Unfortunately, while an approximated model of the robot dynamics is often available, the safety constraints are task-specific and hard to obtain: they may be too complicated to encode analytically, too expensive to compute, or it may be difficult to envision a priori the long-term safety requirements. In this paper, we bridge this gap by extending the safe exploration method, ATACOM, with learnable constraints, with a particular focus on ensuring long-term safety and handling of uncertainty. Our approach is competitive or superior to state-of-the-art methods in final performance while maintaining safer behavior during training.

Main paper: 
 introduction: 
 Safety is one of the most important problems when applying Reinforcement Learning (RL) to realworld applications, as it is fundamental to prevent RL agents from harming people or causing damage. To deal with this issue, RL researchers developed Safe Reinforcement Learning (SafeRL) techniques, to learn policies maximizing the task performance while satisfying the safety requirements. Safe Exploration (SafeExp) aims to ensure the agent's safety during the exploration phase and formulates the safety problem as a stepwise constraint that the agent should not violate.Solving the SafeExp problem requires additional prior knowledge, such as constraints, robot dynamics, or previously collected datasets. While the SafeExp algorithms have been deployed successfully in complex real-world tasks [1, 2, 3, 4] , most of these approaches suffer from many drawbacks preventing their application to complex or out-of-the-lab tasks. First, safety specifications defined as constraints entail an in-depth understanding of the environment and dynamics. Designing and validating safety constraints requires extensive expertise and experience. Second, real-world applications contain various uncertainty sources, such as sensor noise, model error, environmental disturbance, and partial observability, which are often neglected in the design of constraints. Third, the optimization techniques for constrained optimization problems are limited and often require a specific problem structure, such as quadratic programming with linear complementarity constraints. Lastly, the learning algorithms should also guarantee Long-Term Safety, which not only ensures the safety of the current step but also considers the safety of future trajectories. The term Long-Term Safety unifies the Forward Invariance property for the static environment with known dynamics [5, 6] and the predictive safety in a dynamic environment without full knowledge of dynamics.We argue that incorporating prior knowledge for safety-critical robotics applications can be beneficial, as prior knowledge, such as kinematics and dynamics, is often well-studied and readily available. Furthermore, in robotics, we can assume that a sufficiently good approximation of the dynamics model is available, while the complete model of the environment is not. Under this assumption, we combine techniques from model-free SafeRL methods and SafeExp approaches, showing how to exploit the knowledge of the dynamics while learning of unknown, long-term constraints. To achieve this goal, we extend the Acting on the TAngent Space of the COnstraint Manifold (ATA-COM) approach by dropping key assumptions that the constraints are predefined, allowing learning of long-term safety constraints directly from data. Furthermore, we explicitly model the constraints' uncertainty in a distributional RL perspective, which provides us with a way to estimate the total uncertainty of the model. Consequently, we introduce Distributional ATACOM (D-ATACOM), allowing us to derive a risk-aware policy by restricting the level of accepted risk.Our experiments demonstrate that D-ATACOM achieves a safer performance during the training phase and reaches a similar or better performance at the end of training. For tasks where the optimal policy stays within constraints, D-ATACOM explores more cautiously at the cost of slower learning speed. Instead, D-ATACOM show faster and safer behaviors during training whenever there is a conflict between the policy optimization problem and the constraint satisfaction one.In the last decades, SafeRL is a field of increasing interest for deploying learned safe agents to the real world. Constrained Markov Decision Processes (CMDP) [7, 8] framework as a first attempt from RL researchers has gained significant progress in recent research in solving constrained control problems. One important formulation of constraint is the expected cumulative cost. The RL agent aims to maximize the expected return while maintaining the expected cost below a threshold [9, 10, 11, 12, 13, 14, 15, 16, 17] . This type of constraint has been extended to different variants, such as the risk-sensitive constraint [18, 19, 20, 21] and the probabilistic constraint [22, 23, 24] . Different types of constrained optimization techniques are applied in the policy update process, such as the trust-region method [9, 20] , the interior point method [12] , and the Lagrangian relaxation method [7, 11, 13, 14, 18, 19] . Furthermore, the Lyapunov function is also used to derive a policy improvement procedure [10, 25, 26] . Notably, learning the value function of the cumulative cost has gained incremental attraction as it addresses long-term safety. Recent works have shared a common view that the value function of constraint provides a predictive estimation of safety, such as the feasibility value [27, 28] , control barrier function [29, 30] , and safety critic [31, 21] . In this paper, we leverage the idea of the safety value function. Instead of penalizing the unsafe policy in the objective, we combine the safety value function with a model-based exploration method ATACOM. 
preliminaries: 
 We formulate the safety problem in the framework of CMDP. A CMDP is defined as a tuple pS, A, P, r, k, γq with a state space S, an action space A, a stochastic function P : S ˆA ˆS Ñ R that represents the transition probability from a state to another state by an action, a reward function rps, aq P rr min , r max s, a constraint function kpsq P rk min , k max s, and a discount factor γ P r0, 1q. We first define the safety and feasibility following [28] as: Definition 1. Consider a constraint function k : S Ñ R and a policy π : S Ñ A. i. A state s is Safe if kpsq ď 0. ii). The Safe Set is defined as S S " ts P S : kpsq ď 0u. iii. The Unsafe Set is the complementary set s S S " SzS S . iv. A state s is Feasible under a policy π if kps t q ď 0 for all t P t0, 1, . . . , 8u, s 0 " s, a t " πps t q.The SafeRL problem is formulated as a constrained optimization problemEQUATIONwhere F, depending on the perspective on the safety problem [32] , can take different forms, such as kps t q ď 0, @t (2a) Ppkps t q ą 0q ď η c , @t (2b)E π « ÿ t γ t kps t q|s t ff ď η e (2c)The first constraint in (2a) describes the hard constraint, to be satisfied at each time step. However, we cannot enforce these constraints in the setting of stochastic environments. To address this issue, we can use chance constraints, as shown in (2b), restrict the probability of the violations to be smaller than a threshold η c . Both (2a) and (2b) are stepwise constraints that focus on safety at the current time step. The last type of constraint, as shown in (2c), forces the cumulative cost of the trajectory to be smaller than a threshold η e . In this paper, we will focus on the long-term safety constraint in a stochastic formulation, a combination of (2b) and (2c).Distributional Reinforcement Learning Unlike the typical RL setting that considers the expected value of the reward (the cumulative cost in our case), distributional RL treats the reward (cost) as a random variable and, therefore, the value function describes the distributions of the random cumulative return (cumulative cost). Distributional RL has shown superior performance in many benchmarking tasks, such as Atari Games [33, 34, 35] and Mujuco tasks [36] since the distributional value function contains more information beyond the first moment. The value function is represented as a random variable Z π instead of a scalar of the expected value Q π , the random variable Bellman equation has a similar form 2 Z π psq D " Rpsq `γZ π pS 1 q where the distribution of the random variable S 1 depends on policy π and dynamics P. The distribution Z π psq can be represented by different types of models, such as the network with fixed support [33, 34] , Gaussian Network [37] and Implicit Quantile Networks (IQN) [35] . We demonstrate our method using Gaussian Networks. However, our method is not limited to the model type as shown in Appendix C using IQN.We briefly introduce the ATACOM approach [38, 3, 4] , which forms the basis of our approach. ATACOM addresses stepwise hard constraint, as defined in Equation (2a). Furthermore, ATACOM assumes that the dynamic system of the robot is a given nonlinear affine system, 9s " f psq `Gpsqa. ATACOM constructs the Constraint Manifold by introducing a slack variable µ as M -tps, µq P D : cps, µq " 0u with cps, µq " kpsq `µ. We assume that µ is equipped with a dynamic system 9 µ " αpµqu µ and α is a class K function 3 . Using the concept of Constraint Manifold, a safe controller can be obtained by setting d dt cps, µq " 0. The resulting controller has the following formEQUATIONwith J u ps, µq " rJ G psq Apµqs, J G psq " J k psqGpsq, J k psq " d ds kpsq and the Constraint Drift ψpsq " J k psqf psq induced by the system drift f psq. Apµq " diagpα i pµ i qq, i P t1, . . . , Ku is a K-dimensional diagonal matrix for the slack variable. B u is a set of basis vectors tangent to the manifold. The first and the second terms on the right-hand side compensate the drift ψ and retract the system to the manifold; the last term is the tangential term that drives the system along the constraint manifold. An RL agent only needs to learn a policy for the tangential action u " πpsq, while the safety is guaranteed by the controller structure. Alg. 3 in Appendix A showed a detailed process of action mapping. In this work, we extend ATACOM for the chance constraint and propose a new method to simultaneously learn the policy and the long-term constraint online. 
long-term safety under uncertainty: 
 In this section, we will introduce a method to estimate the constraint for long-term safety, and then we show how to integrate the time-varying constraint into the ATACOM learning framework. The overall algorithm Alg. 1 can be found in Appendix A. 2 A D " B denotes that two random variable A and B are equal in distributions. 3 class K function: (1) continuous; (2) strictly increasing; (3) αp0q " 0.To ensure long-term constraint satisfaction, we introduce the concept of Feasibility Value Function (FVF), which describes the expected cumulative constraint violation under a policy π with an infinity horizon. Formally, we define the feasibility value function under a policy π asEQUATIONWe assume the constraint kpsq P rk min , k max s is bounded and consequently V π F P r0, kmax 1´γ s. When k is an indicator function of the constraint violation, the FVF is analogous to the Constraint Decay Function (CoDF) introduced in [28] , defined as F π psq " γ Nπpsq , with N π psq the number of steps to the first constraint violation. Unlike the CoDF, which assumes the unsafe state to be absorbing, the FVF does not assume episode termination at the unsafe state, allowing the agent to retract back to a safe state in the future. Therefore, V π F psq ě F π psq. The feasibility value function estimates the expected discounted cumulative cost of maxpkpsq, 0q under policy π. We can use the standard Bellman operator pB π V F qpsq " maxpkpsq, 0q `γV F psq to update the estimate. For continuous state-action space, a common choice is to use a neural network to approximate the value function and update the value function using TD learning. When V π F psq " 0, we have maxpkpsq, 0q " 0 indicating kpsq ď 0. Thus, the Feasibile Set S F " ts P S : V π F psq " 0u is a subset of the Safe Set. Ensuring the feasibility value function to be zero is sufficient to guarantee stepwise safety.The original FVF definition in Eq. ( 4) evaluates the expected value over the future cost. This estimation does not capture the distribution of the future cost and could fail to ensure safety when the distribution is multi-modal or heavy-tailed. Instead, we can exploit the theory of Distributional RL [39] that learns the parametric model to approximate the distribution of the future expected cost. Then, we can construct Value-at-Risk (VaR)/Conditional Value-at-Risk (CVaR) constraints that consider both the safety and the uncertainty of the prediction when drawing actions. Different parametric models have been used to approximate the distribution of the random value function. In this paper, we approximate the target distribution maxpkpsq, 0q`γV π F psq with Gaussian support up to the 2nd-order moment [37, 31] , i.e. we assume V π F psq " N `µF psq, Σ F psq ˘. We can compute the mean of the target distribution µ F psq " k 1 psq `γµ F ps 1 q and the varianceΣ F psq " k 1 psq 2 `2γk 1 psq E s 1 "P π " Σ F ps 1 q ‰ `γ2 E s 1 "P π " Σ F ps 1 q ``µ F ps 1 q ˘2ı´`µ F psq ˘2Here, k 1 psq " maxpkpsq, 0q and P π ps 1 |sq is the transition probability under policy π. The distribution of FVF is parameterized by Gausian N pµ F ϕ psq, Σ F ϕ psqq. The TD error between the target distribution N pµ F psq, Σ F psqq and the parameterized distribution N pµ F ϕ psq, Σ F ϕ psqq with respect to the 2-Wasserstein distance can be computed asL F " }µ F psq ´µF ϕ psq} 2 `Tr ˆΣF psq `ΣF ϕ psq ´2 ´ΣF psq 1{2 Σ F ϕ psqΣ F psq 1{2 ¯1{2 ˙.(EQUATIONSince the FVF is one-dimensional, Eq. ( 5) becomes L F " }µ F psq ´µF ϕ psq} 2 `}σ F psq ´σF ϕ psq} 2 . We use a Softplus activation for the mean and a Exponetial parameterization for the standard deviation to ensure both values are positive. We use the Gaussian parameterization for illustration purposes. However, our method is not restricted to such parameterization. Experiments using IQN [35, 21] can be found in Appendix C.During training, we keep a replay buffer of limited size. As training progresses, the agent will behave safer and encounter fewer constraint violations, flushing away unsafe transitions when using a single replay buffer. Instead, we would like the agent to remember the failures and avoid being overly optimistic, using a separate, smaller, Failure Buffer D f to store the unsafe transitions. In each data batch, we sample a proportional number of data coming from this buffer.VaR and CVaR quantify the risk of a random variable. VaR is the smallest value where the probability of Z is bigger than a α and CVaR measures the mean of the α-tail of the distribution.VaR α pZq " inftz P R|F pzq ě αu, (6a)CVaR α pZq " E rz|z ě VaR α pZqs. (6b)where F pzq is the Cumulative Distribution Function (CDF) of the random variable Z.When F is continuous and strictly increasing, the VaR is uniquely defined as VaR α pZq " F ´1pαq, i.e., the quantile function.The VaR and CVaR offer a riskaware constraint formulation by restricting their values to be smaller than a threshold δ. The CVaR constraint for the Gaussian distribution [40] of FVF isEQUATIONwhere φ and Φ are the Probability Density Function (PDF) and the CDF of the standard normal distribution, respectively. α determines the probability of constraint satisfaction; thus, the risk is 1 ´α.Using the CVaR constraints ( 7), the constraint in problem ( 1) is defined as Fpsq -CVaR F α psq ´δ ď 0. During exploration, the agent draws an action u " πpu|sq, which is then converted to a safe action a " W ps, uq using ATACOM. Detailed algorithm of ATACOM is shown in Alg. 3 in Appendix A.Adaptive constraint threshold estimate While ideally, we would like to have the FVF to be always equal to zero, setting δ " 0 is neither a practical choice since the network's mean is always bigger than 0, nor beneficial for the training of FVF as it restricts the exploration. The threshold δ trades off the constraint violation and the exploration and requires further engineering. The experiment comparing different thresholds is illustrated in Appendix E.1.Feasible Set 0-level CVaR To alleviate the engineering effort, we propose an adaptive scheme that updates the δ based on the current episodic cost and the estimation of the FVF. We use a Softplus parametrization to keep the δ positive. The δ parameter is updated during the learning process after each episode of horizon H using the following lossL δ " 1 H H ÿ i"0L Huber pd c ps i q, CVaR α ps t q ´δq (8) where the term d c ps i q " ř H t"i γ t´i k 1 ps t q ´C computes the difference between the empirical discounted cost starting from s i and the accepted discounted cost budget C and CVaR α ps t q ´δ computes the distance of the estimated FVF to its threshold. We use Huber Loss [41] to obtain a more robust update against outliers. As shown in Figure 2 , δ is tuned based on the empirical cost to its cost budget. If d c ps i q is bigger than 0 (the actual cost is bigger than the budget), we reduce δ for a more conservative policy. Conversely, if d c ps i q is smaller than 0, we increase δ to loose the constraints.As introduced in Section 2, ATACOM constructs a safe action space by determining the basis vectors of the tangent space of the constraint manifold. However, previous work assumes that the constraint is given, fixed, and deterministic. This assumption is no longer valid when the constraint is trained during the learning process. Since the constraint function changes during training, the safe action space changes accordingly, leading to a non-stationary Markov Decision Process (MDP), which causes the failure of the training. Specifically, let a P A be the action applied to the environment and the u P U be the control input obtained from the policy u " πpsq. ATACOM constructs an affine mapping W : U Ñ A, defined in (3) , that maps an action in safe space into the original one. Combining ATACOM with the actor-critic framework, a value function estimator Q ω ps, uq is trained to approximate the expected return. Then, the policy π θ psq is updated by maximizing the Q ω ps, uq. However, when the constraint is updated during the training process, the action mapping W will also change. The same action u will result in different a at different steps. This variation of the action space leads to an unstable update of Q ω ps, uq and π θ psq.In the following, we will demonstrate how to address this problem for the Soft Actor Critic (SAC) algorithm [42] . However, it is possible to use the same methodology to extend most Deep RL algorithms, e.g., DDPG [43] TD3 [44] , PPO [45] . To solve this issue, we learn the value function of the original action space Q ω ps, aq, which is invariant to the constraints. Since Q ω is represented by a neural network and can be differentiated, we can use the reparameterization trick to obtain the gradient for θ (similar to TD3 and SAC [42] ), the objective and policy gradient can be obtained asmax πθ J π " E s,u"πθ psq rQ ω ps, W puqqs, ∇ θ J π " ∇ a Q ω ps, aq∇ u W puq∇ θ π θ psq.Note that in SAC, the soft Q-function includes the entropy term HpW pπ θ psqqq to encourage exploration. The entropy term is indeed constraint-dependent. Thus, updating the constraints may change the entropy of the policy, consequently, the estimated value function is not anymore proper. We argue that practically, the variation of the entropy bonus has a negligible effect on the training stability because the entropy term is scaled down by a coefficient. Furthermore, online training in the value function allows us to quickly adapt to the new policy entropy. The overall algorithm of D-ATACOM and the modified SAC algorithm are illustrated in Alg. 1 and Alg. 2 in Appendix A. 
experiments: 
 In this section, we compare the performance of our approach in three different environments with different characteristics. Details of the environment description can be found in Appendix B. We compare with SafeRL baselines such as the LagSAC [46] and the WCSAC [31] . All environments and algorithms based on SAC are implemented using the MushroomRL framework [47] . We use the implementations provided by OmniSafe [48] for PPO-based algorithms. We conducted a hyperparameter search on the learning rates, cost budget, and accepted risk with 10 random seeds. We present the results with 25 seeds whose hyperparameters perform the best tradeoff between performance and safety. Further hyperparameter search experiments are in Appendix D.Cartpole This environment extends the classical Cartpole benchmark. The pendulum is initialized in an upright position, and the goal is to move the pendulum tip to a desired point while keeping the pendulum upright. The constraint enforces an angle smaller than 90 degrees from the upright position. We further added a position limit to the cart. Despite the simplicity of the environment, designing a feasible long-term constraint is very challenging due to the actuator limitation and cart position limits. As we can see from the results in Figure 3 , our approach is the best-performing one among the SafeRL baselines in terms of learning speed, while achieving small constraint violations. SAC achieves a policy with higher performance, but this policy heavily violates the safety constraints as it completely disregards the pole angle constraint while reaching the goal. The RCPO algorithm [11] achieves better performance, but violations are comparable with SAC. Notice that D-ATACOM requires a feasible cost budget to generate feasible actions since a single constraint violation will result in a high sum cost, reaching the maximum violation. An unachievable low budget will lead to conservative performance due to a lack of exploration, as shown in Appendix E.2.Navigation In this task, the goal is to control a differential-driven TIAGo++ robot and learn a navigation policy leading to the goal position while avoiding collision with a moving Fetch Robot. The Fetch robot moves to its randomly assigned target with a moving arm. In this task, agents do not observe the Fetch robot's joint positions while the end-effector's position is available. The safety 4 : Learning Curves for the Navigation Environment constraint considers the smallest distance between the two robots. In this setting, the impact of the arm's motion on the constraint is not explicit, which generates a stochasticity on the constraint. From the result in Figure 4 , D-ATACOM clearly outperforms all the state-of-the-art methods both in terms of safety and final task performance. Inspecting the learned policies, D-ATACOM is the only approach that performs active collision avoidance behaviors. These collision avoidance behaviors are mostly achieved by the model-based treatment of the constraint function. In this setting, using the constraint gradient and the dynamic model is a strong inductive bias for the method. On top of that, the control system can exploit the physical meaning of the variables, such as other obstacle velocity, allowing it to compensate in advance for the other robot movements. In this task, most approaches behave the same. We argue that the conflict between the task objective and the constraint function, forcing the robot to cause a detour, is problematic for the Lagrangian approaches. Indeed, Lagrangian optimization is trying to balance constraint satisfaction and policy improvement in the update step, possibly causing the algorithm to get stuck in local minima.3dof Robot Air Hockey The objective of this task is to control a 3-DoF arm to score a goal in the robot air hockey task. The arm is controlled by providing acceleration setpoints, and the constraints include the joint position/velocity limits and collision avoidance with the table. Since the optimal strategy for hitting the puck toward the goal is achievable within the tables's boundary, high reward performance and low constraint violations are achievable at the same time.In addition to common baselines, we also compare D-ATACOM with the original ATACOM with a pre-defined forward invariant constraint (ATACOM+FI) and a non-forward invariant constraint (ATACOM + nFI). As illustrated in Figure 5 , D-ATACOM approaches the performance of the AT-ACOM + FI while maintaining low constraint violations. ATACOM + nonFI performs similarly to unconstrained algorithms SAC, both in return and cost, as a poorly defined constraint can not ensure safety. In this task, D-ATACOM is safer than the other baselines at the cost of slower learning performance. The reason for this performance drop is that our approach learns to expand the safe region progressively and improve the performance. The final performance is lower as the robot hits more cautiously at the boundary regions to ensure safety, while other approaches allow the robot to go outside for stronger hitting, as shown in Appendix E.4. A characteristic of this environment is that the constraints do not majorly affect an optimal policy. Therefore, constraint satisfaction is more difficult in the initial phases of learning than in the final one. This allows classical lagrangian methods to be particularly competitive in the task. Figure 6 illustrates the learned constraint at different training steps. We can clearly observe that the feasible region expands progressively and reaches a good coverage at the final epoch. Since FVF is a policy-dependent value function, the prediction at the corner regions is poor, as the policy does not achieve higher performance. Limitations While this paper is a first step towards safer and more efficient learning without predefined constraints, the above-mentioned methodologies cannot solve complex control tasks, such as the 7-DoF Robot Air Hockey task with equality constraints. Furthermore, training directly on real robots remains challenging with the current approach, as robots need to explore unsafe states to obtain the FVF. Another limitation of this approach is that it requires knowledge of the robot dynamics. However, even in the presence of model mismatch, the FVF can still impose robust and safe behaviors, as the unsafe transitions caused by the incorrect nominal model are included during training. We show a preliminary study on the robustness of D-ATACOM in Appendix E.5. For this reason, fine-grained domain randomization is a crucial step in successful sim-to-real transfer. Finally, this paper does not explore the possibility of combining constraint learning with known constraints. This can be easily implemented in the D-ATACOM framework. However, the resolution of conflicts between constraints remains to be explored.'LVFRXQWHG5HWXUQ 6WHSV H 0D[LPXP9LRODWLRQSHU(SLVRGH 6WHSV H (SLVRGLF6XPRI&RVW 6WHSV H 0D[LPXP9LRODWLRQSHU(SLVRGH '$7$&20# 5&32 :&6$&# $7$&20QRQ), /DJ6$& $7$&20), 332/DJ 6$& 
conclusion: 
 In this paper, we started to bridge the gap between SafeExp methods and model-free SafeRL approaches. We extended the ATACOM framework to work with learned constraints, ensuring longterm safety and properly dealing with constraint uncertainty. Our results show that our method is competitive with state-of-the-art approaches, outperforming them in terms of safety, keeping an on-par learning speed, and achieving similar or better performance for environments with different characteristics. Also, the method does not require excessive parameter tuning, as it includes automatic tuning rules for the most important hyperparameters. Therefore, our work proves that including prior knowledge in data-driven methods can actually be beneficial for scaling SafeRL approaches. Although all of the experiments are trained from scratch, we believe starting with an offline dataset and pre-training will significantly reduce the initial violations, leading to safe performance. In future work, we will further investigate how to integrate known local constraints with long-term safety. This integration will allow scaling the ATACOM approach to real-world robotics tasks involving complex long-term constraints and human-robot interaction. 
 
 Appendix: Algorithm 1 demonstrates the D-ATACOM, and Algorithm 2 shows the adapted SAC update used in line 9.Initialize: FVF network ϕ, number of steps N , threshold δ, cost budget C, policy π θ , value function Qω 1: for 1 ¨¨¨N do 2:Construct CVaR F α pstq using µ F ϕ pstq, Σ F ϕ pstq from Eq (7).Draw action ut " π θ P U and obtain safe action at Ð W pst, utq using Alg. 3.Observe st`1, rt, kt from the environment. 5:Save replay buffer pst, at, rt, kt, st`1q Ñ D and pst, kt, st`1q Ñ D f if kt ą 0.If the episode terminates, update δ using Eq. ( 8). 7:Sample a batch of transitions ps, a, r, k, s 1 q from D Y D f . 8:Update ϕ Ð ϕ ´αϕ ∇ ϕ LF using Eq. ( 5), 9:Update the value function Qω and policy π using SAC in Algorithm 2. 10: end for Algorithm 2 SAC implementation for D-ATACOM Initialize: policy parameters θ and value function parameters ω, learning rate η Input: Batch of transitions B " ps, a, r, k, s 1 q. 1: Draw action u 1 and obtain safe action a 1 , B 1 u Ð W pu 1 , s 1 q using Alg. 3. 2: Compute log probability log p 1 pa 1 |s 1 q " log π θ pu 1 |s 1 q ´log |B 1 u | using the change of variable rule. 3: Update Qω with the TD loss Lω " 1{|B|pQωps, aq ´r ´γpQωps 1 , a 1 q `α log p 1 qq 2 . 4: Draw action u and obtain safe a, Bu Ð W pu, sq using Alg. 3. 5: Compute log probability, log p θ " log π θ pu|sq ´log |Bu|. 6: Update policy θ Ð θ ´η{|B|∇ θ pQωps, aq `α log p θ q, where ∇ θ Qωps, aq " ∇aQωps, aq∇uW puq∇ θ π θ psq.Input: state s, action u 1: Compute slack variable µ and constraint cps, µq " kpsq `µ. 2: Compute the Jacobian Jupsq " " J k psqGpsq Apµq ‰ , drift ψpsq " J k psqf psq. 3: Truncate the drift that have a positive impact on safety ψpsq " maxpψpsq, 0q. 4: Construct the tangent space basis Bu by computing the kernel of Ju using QR/SVD Decomposition. 5: Compute the safe action a using Eq. 3.Output: safe action a, affine mapping matrix BuIn this Section, we provide the full description of the environments used for the experiments. In all environments, the cost value is a continuous variable. A value greater than zero indicates how much the constraints are violated.The cartpole environment, depicted in Figure 7a is a classic control problem with the goal of moving the pole tip to a desired position (green point) by controlling a cart. The pole is one unit in length and is initialized in an upright position on the cart. The cart can move on a rail 10 units long. The cart is initialized on the left side of the rail, and the goal is to move the cart towards the goal position of the pole tip on the right rail's side while keeping the pole upright.The state space of the environment is s " rx, sin θ, cos θ, 9x, 9 θs T where x is the position of the cart, 9x is the velocity of the cart, θ is the angle of the pole with the vertical axis, and 9 θ is the angular velocity of the pole. The action space is a P r´1, 1s where the action is the force applied to the cart.The reward function given a goal position x G and pole tip position x T is defined as rpsq " clipp1 }xG ´xT } 4, 0, 1q. The constraint function prevents the pole from deviating more than π from the vertical axis. Thus we define the cost function as cpsq " max `θ 0.5π ´1, 0 ˘.The Navigation task consists of two robots, one differential-driven TIAGo++ (white) that moves in a room while avoiding the Fetch robot (blue), as shown in Figure 7b . The Fetch robot constantly moves its robotic arm in a periodic motion, such that the end-effector draws a lemniscate into the air in front of the robot. Additionally, the Fetch robot constantly moves to a randomly assigned target position using a hand-crafted policy that ignores the TIAGo. The agent controls the TIAGo robot to reach the target position while avoiding the Fetch robot, which serves as a dynamic obstacle.The state space consists of the cartesian position and velocity of the two robots, the target position of the TIAGo, the previous action, and the cartesian position and velocity of Fetch's end-effector. The action space is the linear velocity in the x-direction and angular velocity around the z-axis of the TIAGo robot. These are converted into the left and right wheel velocities.Given the distance to the goal d G , the current orientation θ and the goal orientation θ G the reward is defined as:rpsq " ´}d G } ´sigmoid p30p}d G } ´0.2qq θ G ´θ π ´0.1}a}The constraint is the smallest 2d cartesian distance between the TIAGo base and every joint of the Fetch Robot. Additionally, the constraint also prevents the TIAGo from hitting the surrounding walls. Given the TIAGos' position p T and cartesian position of the ith Fetch joint p i F the Fetch cost is c F psq " max i p´p}p T ´pi F } ´ωqq where ω is a constant that accounts for the width of the robots. The wall cost is defined as c W psq " max i p´pd i wall ´ωqq where d i wall is the distance to the ith wall. The step cost is cpsq " maxpc F psq, c W psqq. IQN is a parametric model representing the quantile function of the distribution, which takes a quantile value τ as input and outputs a threshold value z so that the probability of Z being less or equal to z is τ . Let η τ ϕ psq be the quantile function at τ P r0, 1s for the random feasibility value at state s. The TD error between two samples τ, τ 1 " U pr0, 1sq for the transition ps, a, s 1 , r, kq isd τ,τ 1 ϕ " k 1 psq `γη τ 1ps 1 q ´ητ ϕ psq The IQN model can be optimized via the Huber quantile regression loss L τ pdq " |τ ´Itdu|L k pdq, where L k pdq "EQUATIONIn Figure 8 we compare the Gaussian and IQN approaches for the navigation task. In this experiment, both algorithms use the same hyperparameters. The Gaussian approach slightly outperforms IQN in terms of performance and safety. We theorize that the source of the performance difference is the hyperparameters, which are tuned for the Gaussian assumption. The main difference in the constraint estimation is that the Gaussian approach predicts higher uncertainty leading to higher performance and safety in this environment. To achieve the same similar with IQN, the cost budget or accepted risk has to be decreased. We plan to further investigate the performance of IQN-ATACOM in future work, especially in environments providing only sparse cost feedback. 6WHSV H 'LVFRXQWHG5HWXUQ 6WHSV H 0D[LPXP9LRODWLRQSHU(SLVRGH 6WHSV H (SLVRGLF6XPRI&RVW 6WHSV H 0D[LPXP9LRODWLRQSHU(SLVRGH '$7$&20# ,41$7$&20#In this section, we report the parameter tuning for all the baselines in all tasks. In general, we test all the methods with different learning rates, cost budgets and safety parameters to ensure the performance of the baseline is optimal. We report all the hyperparameter configurations we tried and indicate which configuration is used for the main evaluation.Every algorithm is first evaluated with the learning rates of 1e ´4, 5e ´4 and 1e ´3. To keep the computation reasonable, we use the same learning rate for the actor, the critic, the constraints, and the learning rates for the Lagrangian multiplier that are updated every step. We report the results of these experiments for each task in the following sections.As a second step, we experimented with different cost budgets to get the best trade-off between safety and performance. Our goal is to get the least constraint violations possible while maintaining reasonable behavior. As we show in Section E.2, setting the cost budget too low can have an impact on the performance with no safety benefit.Lastly, we tuned the cost-dampening parameters of LagSAC and WCSAC using the same principle we used for the cost budget.Figure 9 shows the results of the learning rate tuning for the CartPole task. We can see that RCPO and LagSAC have a learning rate that achieves the best performance. For PPOLag and WCSAC, the differences are more nuanced. Table 1 shows all the parameters we tried for the Cartpole task. The resulting best parameters used for the main evaluation can be found in Table 2 . For each experiment we run 10 seeds with all learning rates of the algorithm set to the respective value.Figure 10 shows the results of the learning rate tuning for the navigation task. We can see WCSAC is the only algorithm where the learning rate has a significant impact on the performance. Table 3 shows all the parameters we tested for the navigation task. The resulting best parameters used for the main evaluation can be found in Table 4 . For each experiment, we run 10 seeds with all learning rates of the algorithm set to the respective value.Figure 11 shows the results of the learning rate tuning for the air hockey task. We can see that RCPO and PPOLag learn safer behaviors compared to LagSAC and WCSAC. However, their discounted return is lower, and they need twice as many steps. Table 5 shows all the parameters we tested for the air hockey task. The resulting parameters used for the main evaluation can be found in Table 6 . The safety threshold δ is an important parameter controlling the trade-off between safety and exploration. A too-small threshold ensures safety at the cost of limiting exploration. Thus, the agent will increase the performance slowly. A higher threshold results in a less restrictive exploration, but the constraint is then not effective to ensure safety.Therefore, we propose an adaptive threshold that updates its value based on the empirical costs and its prediction of the FVF. In this experiment, we compare D-ATACOM with multiple fixed values for δ in the planar air hockey task. Figure 12 compares these experiments with the introduced automatic tuning method. We can observe that fixed δ has a detrimental impact on learning performance because exploration is restrictive by too strict constraints. Interestingly, this lack of exploration also hinders constraint estimation, which leads to slower convergence towards safe behaviors. With a higher fixed δ there are no exploration issues. However, a higher δ consequently leads to higher constraint violations. With our adaptive threshold, we get a good trade-off between the learning performance and constraint satisfaction by having an initial high δ to encourage exploration, which then converges towards a smaller sensible value that reflects a given cost budget. In this experiment, we will compare the impact of the cost budget parameter on D-ATACOM and WCSAC. We chose the CartPole task for this comparison because both algorithms do not learn a completely safe policy. Figure 13 shows the performance of D-ATACOM and WCSAC with different cost budgets. We can observe that the performance of D-ATACOM is more sensitive to the cost budget parameter compared to WCSAC. When the policy cannot achieve the given cost budget the performance of D-ATACOM degrades significantly. This performance drop occurs because the delta eventually will converge towards zero, which results in a very conservative policy. The behavior for D-ATACOM with the cost budgets of 0.1 and 5 is balancing to the pole in its initial position because the policy is too conservative to move towards the goal, as this will lead to constraint violations.6WHSV H 'LVFRXQWHG5HWXUQ 6WHSV H δ 6WHSV H (SLVRGLF6XPRI&RVW 6WHSV H δ δ OHDUQHG δ δ δ δ δOn the other hand, WCSAC is more robust w.r.t. the cost budget parameter. An unreasonable cost budget will increase the Lagrange multiplier, giving more weight to the constraint. The difference is that the Lagrange multiplier does not set an explicit limit to the constraint like the delta does in D-ATACOM. Instead, WCSAC gives more weight to the constraint violations in the optimization problem, which has less impact on policy performance. Is worth noting that, depending on the application, one of the two behaviors would be preferable. In safety-critical applications, having an algorithm that strongly enforces the constraint violation, independently of the performance, is preferable. Instead, when partial constraint satisfaction is enough, it may be better to choose a lagrangian-based algorithm. In the distributional setting, the parameter accepted risk determines how much of the tail of the distribution we are willing to violate, i.e., how much risk we want to take. However, this is not the only parameter that influences the safety of a policy. Usually, there is another parameter that is tuned with a given cost budget that also influences how safe the behavior is. For WCSAC this parameter is the Lagrange multiplier beta, and for D-ATACOM it is the learned δ. To show the complete impact of the accepted risk, we fix δ to a constant value such that it cannot compensate for the difference in the accepted risk. Figure 14 shows the performance of D-ATACOM with a fixed delta and different levels of accepted risk in the Navigation task. Clearly, a lower accepted risk leads to safer behavior.The impact of the accepted risk on the safety shrinks for D-ATACOM when the delta is learned. Delta can compensate for a high accepted risk, resulting in the same safe policy as a lower accepted risk would produce. The accepted risk has an impact in this setting toward the beginning of the training when delta is not yet converged. Thus accepted risk determines how risky the exploration at the beginning of the training will be. Figure 15 shows the impact of different accepted risk settings on the air hockey task. Setting a lower accepted risk results in slower exploration, thus achieving a slower convergence of the discounted return. The maximum violation and sum of cost are comparable for all accepted risk settings because, in the air hockey task, the constraint does not majorly affect the optimal policy. 15 : Impact of accepted risk on performance the air hockey taskThe D-ATACOM exploits the knowledge of the dynamics model to derive the safe policy. The model mismatch could potentially lead to unsafe behaviors. However, D-ATACOM is able to ensure safety under model-mismatch, as the FVF is learned from the dataset collected from the mismatching environment. In this section, we empirically show how dynamic model mismatch will affect the performance of the algorithm.We use the 3-DoF Air Hockey task as the study example. The robot is controlled by acceleration, we use a simple dynamics model : q " a To simulate the model mismatch and disturbances, the dynamics model used in the simulator is : q " a ´σ 9 q `ϵ where ϵ " N p0, σq is the Gaussian disturbance with standard deviation σ, and ´σ 9 q is an unmodelled damping term. The experiment comparing the effect of different δ is shown in Figure 17 .The magnitude of the noise heavily affects the performance of the discounted return, as the RL agent converges to a more robust policy that is more conservative. However, the safety performance remains consistent. This is because the FVF is learned directly from the noisy simulator, and the adaptive threshold balances the exploration and empirical safety during the learning process.Training directly on real robots remains challenging with the current approach, as robots need to explore unsafe states to estimate the FVF. However, we believe our approach can achieve zero-shot transfer when trained in a realistic simulator. Moreover, since FVF is independent of the dynamic model and the adaptive threshold estimation, we are able to exploit domain randomization during the training process, leading to safe policies that are robust to dynamic mismatches. We add a zero mean Gaussian disturbances with standard deviation σ and an unmodelled damping term of the form ´σ 9 q to the environment.In this experiment, we compared D-ATACOM with other model-based approaches in the Planar AirHockey task. We use the OmniSafe [48] implementation of SafeLOOP [49] , which learns the dynamic model and the policy jointly at the same time. Additionally, we implement CBF-SAC, which learns the Control Barrier Function and the policy jointly and tries to ensure the forward invariant property via the agents dynamics. This approach is an adaptation of the work presented in [50] to the online reinforcement learning setting. Finally, we use SafeLayerTD3, which is an exploration method that learns the constraints that encode the dynamics [51] .As shown in Figure 18 , CBF-SAC and SafeLayerTD3 achieve on-par performance to D-ATACOM in discounted return. CBF-SAC manages to reduce constraint violations throughout training and converges to a reasonable safety level. However, it converges slower and towards a less safe solution compared to D-ATACOM. SafeLayerTD3 struggles to ensure safety properly because it learns the constraint using supervised learning. This approach fails when the target constraint does not induce long-term safety. SafeLOOP uses a learned dynamics model and look-ahead planning to check safety. Due to the large computational demand, we run the experiment for only 500.000 steps, which is enough for SafeLOOP to converge to a safe behaviour. However, SafeLOOP achieves poor performance in terms of discounted return as it learns a very conservative policy that barely moves the agent. In summary D-ATACOM achieves the same or better performance with stricter adherence to constraints compared to the model-based baselines. Code is publicly available at https://github.com/cube1324/d-atacom 8th Conference on Robot Learning (CoRL2024), Munich, Germany. 


Decision: **Post-rebuttal metareview**:

This paper extends the ATACOM framework by learning a feasibility value function for solving safe RL problems with known dynamics but unknown constraints. The setting is quite challenging and novel, and the proposed method is convincing. Some reviewers' concerns were addressed in the rebuttal phase. Adding real-world experiments and comparisons with model-based safe RL approaches will strengthen the paper.

------------
**Pre-rebuttal metareview**:

Strengths:
1. Safe RL with unknown constraints is an important and interesting setting.

Weakness:
1. The proposed method requires knowing true dynamics and cannot easily scale up.
2. Need better presentation, especially for algorithms.
3. Need more baselines.
4. The experiment design doesn’t fully show the main point of the proposed method.
5. Other issues mentioned by reviewers.

Reviews: review: title: title: Review
paper_summary: paper_summary: The paper focuses on safe reinforcement learning, building on ATACOM and proposing distributional-ATACOM to integrate dynamics knowledge with constraint learning. Specifically, it addresses long-term safety by estimating a distributional value function for constraint violation. Experiments in Cartpole and navigation demonstrate that the proposed method achieves a lower constraint violation rate, albeit at the cost of a reduced learning rate.
main_review: main_review: Strength:
 - The paper addresses an important problem: safe reinforcement learning. The setting is particularly challenging, given that the constraint is partially unknown.

Weakness:
 - Although the paper eliminates the assumption of known constraints, which is very important, it requires known dynamics and struggles to scale up to real-world robotics tasks. While assuming some level of knowledge about the dynamics is reasonable, the limited scalability to real robotics applications limits the significance of the contribution. Nevertheless, I appreciate that the authors acknowledged this limitation.
 - The overall algorithm is unclear. A significant portion of the paper is dedicated to problem formulation and preliminaries, but how each part is learned and linked together is not well-explained. Including an overall pseudo-algorithm for clarification would be beneficial.
 - The experiments did not use ATACOM as one of the baselines. Comparing the proposed method with ATACOM is important because ATACOM is the foundational method upon which this paper builds. Including this comparison would help readers better understand the contribution of the proposed approach.
questions: questions: - Could you provide a general algorithm to show how the different parts of the method are connected?

 - Is it possible to deploy this method in a real robot setting? If not, what are the potential ways to explore deployment?

 - How does D-ATACOM compare to ATACOM?
limitations: limitations: 3
review_summary: review_summary: The paper addresses an important problem, but the method requires further clarification. Additionally, the discussion on how the method can be deployed in real-world scenarios needs to be expanded.
title: title: Important problem and promising approach. The algorithm not clearly written, and the results are not clearly connected to the main title of the paper.
paper_summary: paper_summary: The authors address a problem of safety with unknown constraints. To address this, the authors extend the existing framework of ATACOM to a setting where the constraint has to be learned. The ATACOM is combined with learning of the feasibility value function, using the distributional method concepts like VaR and CVaR.
main_review: main_review: The setting of the problem where the model of the robot is assumed to be known and the constraints are assumed to be unknown is realistic, and in this sense, I believe that the authors are tackling an important problem.

The summary of ATACOM is appropriate and clear. Also, Section 3.1 and 3.2 where the feasible value function and its learning scheme using the CVaR concept is overall written well. 

However, I could not clearly understand Section 3.3 where the ATACOM is combined with the value function learning. In particular, I think it would be good if the authors include the algorithm table that incorporates every proposed components. 

Also, it is not clear to me why the concept of CVaR is necessary for addressing the nonstationary or long-term safety, the main theme of the paper. Relevant to this point, another limitation of the manuscript is that as opposed to the title of the paper, the experiments are not well designed to tackle “long-term” safety. The second experiment does tackle the non-stationary safety constraint, however, in the other two experiments, the safety constraints are just static, which does not differentiate this work much from existing works on learning safety. 

Finally, the experiments are compared to model-free methods, whereas the proposed method assumes known dynamics of the robot. Thus, it only makes sense that the proposed method results in better training curves. It will be more informative if the method is compared to existing model-based methods (if there is any).

Minor comments:

- p2 - “we will focus on the constraints in the form of (2c), but in the same stochastic formulation of (2b).”: What this means is not clear.
- Line 153 - reason / reference to why p-Wasserstein distance is used is not clear.
- eq (7) - reference?
- Line 197-200 - It will be better if the description of d_c appears first.
- Line 200 - Huber loss - reference?
questions: questions: - Can ATACOM deal with control bounds? Also, if the B_u term in equation (3) vanishes, what happens? (This looks similar to the high relative degree problem of Control Barrier Functions.)
- It is not clear to me why the concept of CVaR is necessary for addressing the nonstationary or long-term safety, the main theme of the paper.
- It would be good to have a pseudo-code of the algorithm in the rebuttal so that I can understand the learning method in more detail.
limitations: limitations: 3
review_summary: review_summary: The paper needs to improve clarity of the main algorithm, with a clear justification how the proposed scheme solves the problem the paper is addressing. Moreover, the results have to be more convincing that the proposed method can solve nonstationary and long-term safety problems effectively.
title: title: Interesting idea and compelling results, could improve on clarity and strength of ablations.
paper_summary: paper_summary: This paper presents D-ATACOM, an algorithm combining the SafeExp and CMDP approaches towards Safe RL with the goal of minimizing safety violations during training while still converging towards a constraint-satisfying policy efficiently. To do so, the authors extend the ATACOM for SafeExp by jointly learning the safety value function during training and accounting for uncertainty and stochasticity in the safety constraint when defining the Constrained Manifold.
main_review: main_review: Strengths:
- The approach is well motivated, and strikes an interesting balance in the field of Safe RL by using system dynamics knowledge to constrain the policy during exploration to actions whose CVaR on the safety value is estimated to be below a threshold. In this way, the approach achieves lower costs during training than penalty-based baselines. In addition, the approach addresses the realistic setting where agent-dynamics are known but the true safety value function must be learned through interaction (e.g., due to stochasticity in environmental dynamics). 
- The results are compelling, showing that D-ATACOM can indeed achieve faster learning than SafeExp methods while having fewer violations per episode compared to Lagrangian penalty based constrained RL methods.

Weaknesses:
- I found the discussion of how the learned feasibility value function is used to define the constraint manifold unclear. Laying out this connection more explicitly in the document would make the paper significantly easier to follow. 
- While the approach does strike a balance between dynamics-informed constrained safe exploration and penalty-based constrained RL formulations, the specific balance that is struck seems to be difficult to control. It seems that choosing a fixed $\delta$ risk level is not recommended, as the authors propose the Huber-loss based method to choose $\delta$ as a function of a cost-budget parameter. However, the ablation results varying the cost budget in the appendix seem to suggest that the cost-budget parameter can have a significant impact on discounted return, but does not have a significant impact in the violations per episode or the episodic sum of costs. This suggests that the interpretation of this parameter as defining a budget on the cost does not necessarily hold true in practice? What value does the Huber-loss based adaptive selection of $\delta$ provide over a fixed $\delta$? How does the scale of this loss impact the algorithm's performance?
questions: questions: Comments:
- What is $A(\mu)$ defined after equation (3)?
- How is the FVF used to construct the constraint manifold (and ultimately, the action mapping $W$)?
- In Figure 6, what does the color represent? I'm assuming this is $CVaR - \delta$  but this should be stated more clearly.
limitations: limitations: 3
review_summary: review_summary: I believe the community would find this paper interesting, as it presents a novel angle to the problem of Safe RL which could be relevant to robotics.