Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations

1 Introduction . Researchers have demonstrated that reinforcement learning ( RL ) can solve complex tasks such as Atari games [ Mnih et al. , 2015 ] , Go [ Silver et al. , 2017 ] , dexterous manipulation tasks [ Akkaya et al. , 2019 ] , and many more robotics tasks in simulated environments [ Haarnoja et al. , 2018 ] . However , deploying RL algorithms to real-world problems still faces the hurdle that they require many unsafe environment interactions . For example , a robot ’ s unsafe environment interactions include falling and hitting other objects , which incur physical damage costly to repair . Many recent deep RL works reduce the number of environment interactions significantly ( e.g. , see Haarnoja et al . [ 2018 ] , Fujimoto et al . [ 2018 ] , Janner et al . [ 2019 ] , Dong et al . [ 2020 ] , Luo et al . [ 2019 ] , Chua et al . [ 2018 ] and reference therein ) , but the number of unsafe interactions is still prohibitive for safety-critical applications such as robotics , medicine , or autonomous vehicles [ Berkenkamp et al. , 2017 ] . Reducing the number of safety violations may not be sufficient for these safety-critical applications— we may have to eliminate them . This paper explores the possibility of safe RL algorithms with zero safety violations in both training time and test time . We also consider the challenging setting where we are only given a safe but trivial-reward initial policy . A recent line of works on safe RL design novel actor-critic based algorithms under the constrained policy optimization formulation [ Thananjeyan et al. , 2021 , Srinivasan et al. , 2020 , Bharadhwaj et al. , 2020 , Yang et al. , 2020 , Stooke et al. , 2020 ] . They significantly reduce the number of training-time safety violations . However , these algorithms fundamentally learn the safety constraints by contrasting the safe and unsafe trajectories . In other words , because the safety set is only specified through the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . safety costs that are observed postmortem , the algorithms only learn the concept of safety through seeing unsafe trajectories . Therefore , these algorithms can not achieve zero training-time violations . For example , even for the simple 2D inverted pendulum environment , these methods still require at least 80 unsafe trajectories ( see Figure 2 in Section 6 ) . Another line of work utilizes ideas from control theory and model-based approach [ Cheng et al. , 2019 , Berkenkamp et al. , 2017 , Taylor et al. , 2019 , Zeng et al. , 2020 ] . These works propose sufficient conditions involving certain Lyapunov functions or control barrier functions that can certify the safety of a subset of states or policies [ Cheng et al. , 2019 ] . These conditions assume access to calibrated dynamical models . They can , in principle , permit safety guarantees without visiting any unsafe states because , with the calibrated dynamics , we can foresee future danger . However , control barrier functions are often non-trivially handcrafted with prior knowledge of the environments [ Ames et al. , 2019 , Nguyen and Sreenath , 2016 ] . This work aims to design model-based safe RL algorithms that achieve zero training-time safety violations by learning the barrier certificates iteratively . We present the algorithm Co-trained Barrier Certificate for Safe RL ( CRABS ) , which alternates between learning barrier certificates that certify the safety of larger regions of states , optimizing the policy , collecting more data within the certified states , and refining the learned dynamics with data . The work of Richards et al . [ 2018 ] is a closely related prior result , which learns a Lyapunov function given a fixed dynamics model via discretization of the state space . Our work significantly extends it with three algorithmic innovations . First , we use adversarial training to learn the certificates , which avoids discretizing state space and can potentially work with higher dimensional state space than the two-dimensional problems in Richards et al . [ 2018 ] . Second , we do not assume a given , globally accurate dynamics ; instead , we learn the dynamics from safe explorations . We achieve this by co-learning the certificates , dynamics , and policy to iteratively grow the certified region and improve the dynamics and still maintain zero violations . Thirdly , the work Richards et al . [ 2018 ] only certifies the safety of some states and does not involve learning a policy . In contrast , our work learns a policy and tailors the certificates to the learned policies . In particular , our certificates aim to certify only states near the trajectories of the current and past policies—this allows us to not waste the expressive power of the certificate parameterization on irrelevant low-reward states . We evaluate our algorithms on a suite of tasks , including a few where achieving high rewards requires careful exploration near the safety boundary . For example , in the Swing environment , the goal is to swing a rod with the largest possible angle under the safety constraints that the angle is less than 90◦ . We show that our method reduces the number of safety violations from several hundred to zero on these tasks . 2 Setup and Preliminaries . 2.1 Problem Setup . We consider the standard RL setup with an infinite-horizon deterministic Markov decision process ( MDP ) . An MDP is specified by a tuple ( S , A , γ , r , µ , T ) , where S is the state space , A is the action space , r : S ×A → R is the reward function , 0 ≤ γ < 1 is the discount factor , µ is the distribution of the initial state , and T : S ×A → S is the deterministic dynamics model . Let ∆ ( X ) denote the family of distributions over a set X . The expected discounted total reward of a policy π : S → ∆ ( A ) is defined as J ( π ) = E [ ∞∑ i=0 γir ( si , ai ) ] , where s0 ∼ µ , ai ∼ π ( si ) , si+1 = T ( si , ai ) for i ≥ 0 . The goal is to find a policy π which maximizes J ( π ) . Let Sunsafe ⊂ S be the set of unsafe states specified by the user . The user-specified safe set Ssafe is defined as S\Sunsafe . A state s is ( user-specified ) safe if s ∈ Ssafe . A trajectory is safe if and only if all the states in the trajectory are safe . An initial state drawn from µ is assumed to safe with probability 1 . We say a deterministic policy π is safe starting from state s , if the infinite-horizon trajectory obtained by executing π starting from s is safe . We also say a policy π is safe if it is safe starting from an initial state drawn from µ with probability 1 . A major challenge toward safe RL is the existence of irrecoverable states which are currently safe but will eventually lead to unsafe states regardless of future actions . We define the notion formally as follows . Definition 1 . A state s is viable iff there exists a policy π such that π is safe starting from s , that is , executing π starting from s for infinite steps never leads to an unsafe state . A user-specified safe state that is not viable is called an irrecoverable state . We remark that unlike Srinivasan et al . [ 2020 ] , Roderick et al . [ 2020 ] , we do not assume all safe states are viable . We rely on the extrapolation and calibration of the dynamics to foresee risks . A calibrated dynamics model T̂ predicts a confidence region of states T̂ ( s , a ) ⊆ S , such that for any state s and action a , we have T ( s , a ) ∈ T̂ ( s , a ) . 2.2 Preliminaries on Barrier Certificate . Barrier certificates are powerful tools to certify the stability of a dynamical system . Barrier certificates are often applied to a continuous-time dynamical system , but here we describe its discrete-time version where our work is based upon . We refer the readers to Prajna and Jadbabaie [ 2004 ] , Prajna and Rantzer [ 2005 ] for more information about continuous-time barrier certificates . Given a discrete-time dynamical system st+1 = f ( st ) without control starting from s0 , a function h : S → R is a barrier certifcate if for any s ∈ S such that h ( s ) ≥ 0 , h ( f ( s ) ) ≥ 0 . Zeng et al . [ 2020 ] considers a more restrictive requirement : For any state s ∈ S , h ( f ( s ) ) ≥ αh ( s ) for a constant 0 ≤ α < 1. it is easy to use a barrier certificate h to show the stability of the dynamical system . Let Ch = { s : h ( s ) ≥ 0 } be the superlevel set of h. The requirement of barrier certificates directly translates to the requirement that if s ∈ Ch , then f ( s ) ∈ Ch . This property of Ch , which is known as the forward-invariant property , is especially useful in safety-critical settings : suppose a barrier certificate h such that Ch does not contain unsafe states and contains the initial state s0 , then it is guaranteed that Ch contains the entire trajectory of states { st } t≥0 which are safe . Finding barrier certificates requires a known dynamics f , which often can only be approximated in practice . This issue can be resolved by using a well-calibrated dynamics model f̂ , which predicts a confidence interval containing the true output . When a calibrated dynamics model f̂ is used , we require that for any s ∈ S , mins′∈f̂ ( s ) h ( s ′ ) ≥ 0 . Control barrier functions [ Ames et al. , 2019 ] are extensions to barrier certificates in the control setting . That is , control barrier functions are often used to find an action to meet the safety requirement instead of certifying the stability of a closed dynamical system . In this work , we simply use barrier certificates because in Section 3 , we view the policy and the calibrated dynamics model as a whole closed dynamical system whose stability we are going to certify . 3 Learning Barrier Certificates via Adversarial Training . This section describes an algorithm that learns a barrier certificate for a fixed policy π under a calibrated dynamics model T̂ . Concretely , to certify a policy π is safe , we aim to learn a ( discretetime ) barrier certificate h that satisfies the following three requirements . R.1 . For s0 ∼ µ , h ( s0 ) ≥ 0 with probability 1 . R.2 . For every s ∈ Sunsafe , h ( s ) < 0 . R.3 . For any s such that h ( s ) ≥ 0 , mins′∈T̂ ( s , π ( s ) ) h ( s ) ≥ 0 . Requirement R.1 and R.3 guarantee that the policy π will never leave the set Ch = { s ∈ S : h ( s ) ≥ 0 } by simple induction . Moreover , R.2 guarantees that Ch only contains safe states and therefore the policy never visits unsafe states . In the rest of the section , we aim to design and train such a barrier certificate h = hφ parametrized by neural network φ. hφ parametrization . The three requirements for a barrier certificate are challenging to simultaneously enforce with constrained optimization involving neural network parameterization . Instead , we will parametrize hφ with R.1 and R.2 built-in such that for any φ , hφ always satisfies R.1 and R.2 . We assume the initial state s0 is deterministic ( the parameterization can be extended to multiple initial states . ) To capture the known user-specified safety set , we first handcraft a continuous function Bsafe : S → R≥0 satisfying Bsafe ( s ) ≈ 0 for typical s ∈ Ssafe and Bsafe ( s ) > 1 for any s ∈ Sunsafe.1 The construction of Bsafe does not need prior knowledge of irrecoverable states , but only the userspecified safety set Ssafe . To further encode the user-specified safety set into hφ , we choose hφ to be of form hφ ( s ) = 1 − Softplus ( fφ ( s ) − fφ ( s0 ) ) − Bsafe ( s ) , where fφ is a neural network , and Softplus ( x ) = log ( 1 + ex ) . Because s0 is safe and Bsafe ( s0 ) ≈ 0 , hφ ( s0 ) ≈ 1 − Softplus ( 0 ) > 0 . Therefore hh satisfies R.1 . Moreover , for any s ∈ Sunsafe , we have hφ ( s ) < 1 − Bsafe ( s ) < 0 , so hφ in our parametrization satisfies R.2 by design . Training barrier certificates . We now move on to training φ to satisfy R.3 . Let U ( s , a , h ) : = max s′∈T̂ ( s , a ) −h ( s′ ) . ( 1 ) Then , R.3 requires U ( s , π ( s ) , hφ ) ≤ 0 for any s ∈ Chφ , The constraint in R.3 naturally leads up to formulate the problem as a min-max problem . Define our objective function to be C∗ ( hφ , U , π ) : = max s∈Chφ U ( s , π ( s ) , hφ ) = max s∈Chφ , s ′∈T̂ ( s , π ( s ) ) −h ( s′ ) , ( 2 ) and we want to minimize C∗ w.r.t . φ : min φ C∗ ( hφ , U , π ) = min φ max s∈Chφ , s ′∈T̂ ( s , π ( s ) ) −h ( s′ ) , ( 3 ) Our goal is to ensure the minimum value is less than 0 . We use gradient descent to solve the optimization problem . We also derive the gradient of C∗ ( Lφ , U , π ) w.r.t . φ : ∇φC∗ ( hφ , U , π ) = ∇φU ( s∗ , π ( s∗ ) , hφ ) + ‖∇φU ( s∗ , π ( s∗ ) , hφ ) ‖2 ‖∇φhφ ( s∗ ) ‖2 ∇φhφ ( s∗ ) , ( 4 ) where s∗ : = arg maxs : hφ ( s ) ≤1 U ( s , π ( s ) , hφ ) and we defer the derivation to Appendix A. Computing the adversarial s∗ . Equation ( 4 ) requires us to compute s∗ efficiently . Because the maximization problem with respect to s is nonconcave , there could be multiple local maxima . In practice , we find that it is more efficient and reliable to use multiple local maxima to compute∇φC∗ and then average the gradient . Solving s∗ is highly non-trivial , as it is a non-concave optimization problem with a constraint s ∈ Chφ . To deal with the constraint , we introduce a Lagrangian multiplier λ and optimize U ( s , π ( s ) , hφ ) − λIs∈Chφ w.r.t . s without any constraints . However , it is still very time-consuming to solve an optimization problem independently at each time . Based on the observation that the parameters of h do not change too much by one step of gradient step , we can use the optimal solution from the last optimization problem as the initial solution for the next one , which naturally leads to the idea of maintaining a set of candidates of s∗ ’ s during the computation of∇φC∗ . We use Metropolis-adjusted Langevin algorithm ( MALA ) to maintain a set of candidates { s1 , . . . , sm } which are supposed to sample from exp ( τ ( U ( s , π ( s ) , hφ ) − λIs∈Chφ ) ) . Here τ is the temperature indicating we want to focus on the samples with large U ( s , π ( s ) , hφ ) . Although the indicator function always have zero gradient , it is still useful in the sense that MALA will reject si 6∈ Chφ . A detailed description of MALA is given in Appendix D. We choose MALA over gradient descent because the maintained candidates are more diverse , approximate local maxima . If we use gradient descent to find s∗ , then multiple runs of GD likely arrive at the same s∗ , so that we lost the parallelism from simultaneously working with multiple 1The function Bsafe ( s ) is called a barrier function for the user-specified safe set in the optimization literature . Here we do not use this term to avoid confusion with the barrier certificate . Algorithm 1 Learning barrier certificate hφ for a policy π w.r.t . a calibrated dynamics model T̂ . Require : Temperature τ , Lagrangian multiplier λ , and optionally a regularization function Reg . 1 : Let U be defined as in Equation ( 1 ) . 2 : Initialize m candidates of s1 , . . . , sm ∈ S randomly . 3 : for n iterations do 4 : for every candidate si do 5 : sample si ∼ exp ( τU ( s , π ( s ) , hφ ) − λIs∈Ch ) by MALA ( Algorithm 5 ) . 6 : W ← { si : hφ ( si ) ≥ 0 , i ∈ [ m ] } . 7 : Train φ to minimize C∗ ( hφ , U , π ) + Reg ( φ ) using all candidates in W . Algorithm 2 CRABS : Co-trained Barrier Certificate for Safe RL ( Details in Section 4 ) Require : An initial safe policy πinit . 1 : Collected trajectories buffer D̂ ← ∅ ; π ← πinit . 2 : for T epochs do 3 : Invoke Algorithm 3 to safely collect trajectories ( using π as the safeguard policy and a noisy version of π as the πexpl ) . Add the trajectories to D̂ . 4 : Learn a calibrated dynamics T̂ with D̂ . 5 : Learn a barrier certificate h that certifies π w.r.t . T̂ using Algorithm 1 with regularization . 6 : Optimize policy π ( according to the reward ) , using data in D̂ , with the constraint that π is certified by h. local maxima . MALA avoids this issue by its intrinsic stochasticity , which can also be controlled by adjusting the hyperparameter τ . We summarize our algorithm of training barrier certificates in Algorithm 1 ( which contains optional regularization that will be discussed in Section 4.2 ) . At Line 2 , the initialization of si ’ s is arbitrary , as long as they have a sort of stochasticity .

Summary: The paper proposes an RL algorithm that starts from an initial safe but low return policy that is iteratively improved while ensuring no safety violation occurs during the rollouts required by the training. The paper achieves this by learning barrier functions as safety certificates.  The two "simpler" barrier conditions are directly enforced in the structure of the learned function. In contrast, the "closed under a policy+environment step" condition R3 is checked heuristically on a probabilistic model of the environment with a Metropolis-adjusted Langevin algorithm.  Finally, the authors evaluate the achieved return and number of safety violations of their RL algorithm and other RL algorithms on variations of the Pendulum and CartPole gym environments.