Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations
Authors: Yuping Luo
Tengyu Ma
Venue: NeurIPS 2021 Poster (2021)
Keywords: safe reinforcement learning
barrier certificate
training-time safety

Abstract: Training-time safety violations have been a major concern when we deploy reinforcement learning algorithms in the real world.
This paper explores the possibility of safe RL algorithms with zero training-time safety violations in the challenging setting where we are only given a safe but trivial-reward initial policy without any prior knowledge of the dynamics and additional offline data.
We propose an algorithm, Co-trained Barrier Certificate for Safe RL (CRABS), which iteratively learns barrier certificates, dynamics models, and policies. The barrier certificates are learned via adversarial training and ensure the policy's safety assuming calibrated learned dynamics. We also add a regularization term to encourage larger certified regions to enable better exploration. 
Empirical simulations show that zero safety violations are already challenging for a suite of simple environments with only 2-4 dimensional state space, especially if high-reward policies have to visit regions near the safety boundary.  Prior methods require hundreds of violations to achieve decent rewards on these tasks,  whereas our proposed algorithms incur zero violations.

Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violations
Yuping Luo Princeton University yupinglocs princeton. wat.nngless.riinetoon.c.m
Tengyu Ma Stanford University tengyuma@stanf ord. edu
edu
Abstract
Training-time safety violations have been a major concern when we deploy rein- forcement learning algorithms in the real world. This paper explores the possibility of safe RL algorithms with zero training-time safety violations in the challenging setting where we are only given safe but trivial-reward initial policy without any prior knowledge of the dynamics and additional offline data. We propose an algorithm, Co-trained Barrier Certificate for Safe RL (CRABS),whidd iteratively learns barrier certificates, dynamics models, and policies. The barrier certificates are learned via adversarial training and ensure the policy's 's safety assuming cali- brated learned dynamics. We also add regularization term to encourage larger certified regions to enable better exploration. Empirical simulations show that zero safety violations
already challenging for suite of simple environments with only 2-4 dimensional state space, especially high-reward policies have to visit regions near the safery boundary. Prior methods require hundreds of violations achieve decent rewards on these tasks, whereas our proposed algorithms incur zero violations.
Introduction
Researchers have demonstrated that reinforcement learning (RL) solve complex tasks such as Atari games [Mnih et al. 2015). Go [Silver et al., 2017]. dexterous manipulation tasks [Akkaya et al. 2019), and many deploying RL algorithms to real-world problems still faces the hurdle that they require many unsafe environment interactions. For example, robot's unsafe environment interactions include falling and hitting other objects, which incur physical damage costly to repair. Many recent deep RL works reduce the number of environment interactions significantly (e.g.. see FFaarnoja al. 2018)
robotics tasks in simulated environments [Haarnoja et al. 2018). However,
Fujimoto et al.) [2018), Janner et al. [2019), Dong et al. [2020], Luo et al. [2019]. and reference therein). but the number of unsafe interactions still prohibitive for safety-critical applications such as robotics, medicine, or autonomous vehicles [Berkenkamp et al.) 2017].
(20199,/Chhi CCua et al. 2018
Reducing the number of safety violations may not be sufficient for these criticll applications we may have to eliminate them. This paper explores the possibility of safe RL algorithms with zero safety violations in both training time and test time. We also consider the challenging setting where we are only given a safe but trivial-reward initial policy.
A recent line of works on safe RI. design novel actor-critic based algorithms under the constrained policy optimization formulation [Thnnanjeyan et al.) [202]] Srinivasan et al.) 22200] Bharadhwaj et al. 2020 Yang et al.) 2020, Stooke et al. [2200]. They significantly reduce the number of training time safety violations. However, these algorithms fundamentally learn the safety constraints by contrasting the safe and unsafe trajectories. In other words, because the safety set is only specified through the
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

safety costs that are observed postmortem, the algorithms only learn the concept of safety through seeing unsafe trajectories. Therefore, these algorithms cannot achieve zero training-time violations. For example, even for the simple 2D inverted pendulum environment, these methods still require at least 80 unsafe trajectories (see Figure
2 in Section
Another line of work utilizes ideas from control theory and model-based approach [Cheng et al. 2019 Berkenkamp
2017 Taylor et al. 2019 Zeng al. 2020 These works propose sufficient conditions involving certain Lyapunov functions or control barrier functions that can certify the safety of a subset of states or policies [Cheng et al. 2019]. These conditions assume access to calibrated dynamical models. They can, in principle, permit safety guarantees without visiting any unsafe states because, with the calibrated dynamics, we can foresee future danger. However, control barrier functions are often non- -riviili handcrafted with prior knowledge of the environments [Ames et al. 2019 Nguyen and Sreenath 2016].
This work aims to design model-based safe RI algorithms that achieve zero training-time safety violations by learning the barrier certificates iteratively. We present the algorithm Co-trained Barrier Certificate for Safe RL (CRABS), which alternates between learning barrier certificates that certify the safety of larger regions of states, optimizing the policy, collecting more data within the certified states, and refining the learned dynamics with data.
The work of Richards et al. 2018 is closely related prior result, which learns a Lyapunov function given a fixed dynamics model via discretization of the state space. Our work significantly extends with three algorithmic innovations. First, we use adversarial training learn the certificates, which avoids discretizing state space and can potentially work with higher dimensional state space than the two-dimensional problems in |Richards al.. 2018). Second, we do not assume given, globally accurate dynamics; instead, we learn the dynamics from safe explorations. We achieve this by co-learning the certificates, dynamics, and policy to iteratively grow the certified region and improve the dynamics and still maintain zero violations. Thirdly, the work Richards et al. (2018) only certifies the safety of some states and does not involve learning a policy. In contrast, our work learns policy and tailors the certificates to the learned policies. In particular, our certificates aim to certify only states near the trajectories of the current and past policies thics allows us to not waste the expressive power of the certificate parameterization on irrelevant low-reward states.
We evaluate our algorithms on suite of tasks, including a few where achieving high rewards requires careful exploration near the safety boundary. For example, in the Swing environment, the goal is to swing rod with the largest possible angle under the safety constraints that the angle is less than 90%. We show that our method reduces the number of safety violations from several hundred to zero on these tasks.
2 Setup and Preliminaries
2.1 Problem Setup
We consider the standard RL setup with an infinite-horizon deterministic Markov decision process (MDP). An MDP is specified by a tuple (S,A, T, (u, T),, where S is the state space, AA is the action space, r : S x A + R. is the reward function, 0 S 7 < 1 is the discount factor, μ is the distribution of the initial state, and : A > S is the deterministic dynamics model. Let ((X) denote the family of distributions over set X. The expected discounted total reward of a policy π S - ((A) is defined as
(TT) E Σ
rrr(s)na))
where 50 ~ p.a, ag ~ T(S)). 8:+1 = T(s;, a.) for i 2 The goal is to find a policy a which maximizes J (
Let Sunsate C S be the set of unsafe states specified by the user. The user- user-specified safe set Satee is defined as S\ Sumatee A state s is (user-specified) safe if s E State. A trajectory is safe if and only if all the states in the trajectory are safe. An initial state drawn from ji is assumed to safe with probability 1. We say a deterministic policy is safe starting from state 8, if the infinite-horizon trajectory obtained by executing π starting from 8 safe. We also say policy π is safe it is safe starting from an initial state drawn from μ with probability 1. A major challenge toward safe RL is the existence of
2

---

irrecoverable states which are currently safe but will eventually lead to unsafe states regardless of future actions. We define the notion formally follows.
Definition 1. A state is viable iff there exists a policy a such that a safe starting from 8, that is, executing x starting from for infinite steps never leads to unsafe state, A user-specified safe state that is not viable is called an irrecoverable state.
We remark that unlike Srinivasan et al. 2020 Roderick et al. 2020 we do not assume safe states are viable. We rely on the extrapolation and calibration of the dynamics to foresee risks. A calibrated dynamics model T predicts confidence region of states T(s, a) S, such that for any state s and action a, we have T(s, a) (s.
2.2 Preliminaries on Barrier Certificate
Barrier certificates are powerful tools to certify the stability of a dynamical system. Barrier certificates are often applied to a continuous-time dynamical system, but here we describe its discrete-time version where our work is based upon. We refer the readers to Prajja and Jadbabaie [2004). Prajna and Rantzer 2005) for more information about continuous-time barrier certificates.
Given a discrete-time dynamical system St-1 = f(sr) without control starting from a function h : S R is a barrier certifcate if for any s E S such that h(s) O, h(f(s)) 2 O. Zeng al. 2020 considers a more restrictive requirement: For any state E S, s(f()) ah(s) for constant
1.
is easy to use a barrier certificate h to show the stability of the dynamical system. Let C₁ = {s h(s) 2 0} be the superlevel set of h. The requirement of barrier certificates directly translates to the requirement that if 8 E Cr. then [(s) E C.. This property of C.. which known as the forward-invariant property, is especially useful in safety-critical settings: suppose barrier certificate such that C₁ does not contain unsafe states and contains the initial state so, then it is guaranteed that Cn contains the entire trajectory of states {s:)e20 which are safe.
Finding barrier certificates requires known dynamics f. which often can only be approximated in practice. This issue can be resolved by using a well-calibrated dynamics model f. which predicts a confidence interval containing the true output. When calibrated dynamics model f is used, we require that for any S, min mim h(s') 0.
Control barrier functions Ames al. 2019 extensions to barrier certificates in the control setting. That is, control barrier functions are often used to find an action to meet the safety requirement instead of certifying the stability of closed dynamical system. In this work, simply use barrier certificates because in Section we view the policy and the calibrated dynamics model as a whole closed dynamical system whose stability we are going certify.
3 Learning Barrier Certificates via Adversarial Training
This section describes an algorithm that learns a barrier certificate for fixed policy æ under a calibrated dynamics model T. Concretely, to certify a policy is safe, we aim to learn a (discrete- time) barrier certificate h that satisfies the following three requirements.
R.1. For Sp ~ p. h(s)) 2 O with probability
R.2. For every E Sunsafe- h(s)) < 0.
R.3. For any such that 1(()) 2 0, min (eix M(s) 2 0.
Requirement R.1 and R.3 guarantee that the policy will never leave the set C. {s S: h(s) 2 0} by simple induction. Moreover, R.2 guarantees that C₁ only contains safe states and therefore the policy never visits unsafe states.
In the rest of the section, we aim design and train such a barrier certificate his parametrized by neural network ø.

---

he parametrization. The three requirements for a barrier certificate are challenging to simultane- ously enforce with constrained optimization involving neural network parameterization. Instead. we will parametrize he with R.T and built-in such that for any é, lig always satisfies R.1 and R.2
We assume the initial state so is deterministic (the parameterization initial states.) To capture the known user-specified safety set, we first handcraft a continuous function Bafe : Rgo satistying (sa R 0 for typical s E Saate and Baate > 1 for any s Sunsafe The construction of Bute does not need prior knowledge of irrecoverable states, but only the user- specified safety set State. To further encode the user-specified safety set into hg. we choose hg to be of form h& (s) = 1 Softplus(foss fa(so)) (s), where fo is neural network, and Softplus(z) log( E*).
be extended multiple
Because 8p is safe and (s0) R O, (80) R 1 - Softplus(O) > 0. Therefore h4 satisfies R.1 Moreover, for any Sunsafe: have (s) 1 Baaie (8) 0, so hg in our parametrization satisfies R.2 by design.
Training barrier certificates. We now move on to training ø to satisfy R.3
Let
h)) == max s'ET(s.0)
-h(s').
(1)
Then, R.3 requires (s, (s),, ha) S for any 8 E Chaz The constraint in R.3 naturally leads up to formulate the problem min-max problem. Define our objective function be
C² (he U, U) == max (ss (()) sECa,
max
-h(s'),
(2)
aECn s' (e,((()))
and we want to minimize C* w.r.t. Φ:
min C* (ho, U, "== min max 1 aECh 8' (s..)e))
-h(s'),
(3)
Our goal is to ensure the minimum value is less than O. We use gradient descent to solve the optimization problem. We also derive the gradient of (I.g, U, x) w.r.t. Φ:
x(s)), hz) + V n T(>)((())))))) ho)ll2 Voho(s"), V VoC' = (s (0=s""""ss"""_)))""""")))))"")))
(4)
[Vgha(s*)]l
where s" := arg max (1) (s, ग (s), ho) and we defer the derivation to Appendix A
Computing the adversarial s*
Equation 44 requires us to compute s* efficiently. Because the maximization problem with respect to is nonconcave, there could be multiple local maxima. In practice, we find that it is more efficient and reliable to use multiple local maxima to compute VOC" and then average the gradient.
Solving s* is highly non-trivial, as it is non-concave optimization problem with a constraint E Cr.- To deal with the constraint, we introduce a Lagrangian multiplier À and optimize U(s, π (s), ht) Alsecn, w.r.t. without any constraints. However, it is still very time-consuming to solve an optimization problem independently at each time. Based on the observation that the parameters of h do not change too much by one step of gradient step, we can use the optimal solution from the last optimization problem as the initial solution for the next one, which naturally leads to the idea of maintaining set of candidates of s during the computation of V VOC*
We netropolis-adjuster Langevin algorithm (MALA) maintain {81,..., 8m which are supposed to sample from exp(T(U(s, (s))),), h&) - Alssecng) ))- Here 7 is the temperature indicating want to focus on the samples with large (s), hz). Although the indicator function always have zero gradient, it is still useful in the sense that MALA will reject 8c E Ch.- A detailed description of MALA is given in Appendix D
of candidates
We choose MALA over gradient descent because the maintained candidates are more diverse, approximate local maxima. If use gradient descent to find s* then multiple runs of GD likely arrive at the same s", so that we lost the parallelism from simultancously working with multiple
'The function Bute (s) is called barrier function for the user-specified safe set in the optimization literature. Here do not use this term avoid confusion with the barrier certificate.

---

Algorithm Learning barrier certificate he for a policy л w.r.t. a calibrated dynamics model T.
Require: Temperature т. Lagrangian multiplier X, and optionally a regularization function Reg.
1: Let U be defined as in Equation [
2: Initialize m candidates of $1...., 5m 8 randomly.
3: for iterations do
4: for every candidate s; do
5: sample ~ (xp(rU(s, ((s), ha) - AIsEc,) ) by MALA (Algorithm 5p.
6: W + {s; (s,), 2 0, 2 m.].
7: Train e to minimize C* (ha, T, + Reg(ø) using all candidates in W.
Algorithm 2 CRABS: Co-trained Barrier Certificate for Safe RL (Details in Section 4)
Require: An initial safe policy Tinit-
1: Collected trajectories buffer D + 0; π +- Tinit-
2: for T epochs do
3: Invoke Algorithm to safely collect trajectories (using и as the safeguard policy and a noisy version of π as the > Add the trajectories to D.
4: Learn calibrated dynamics T with D.
5: Learn a barrier certificate that certifies π w.r.t. 7 using Algorithm 1 with regularization.
6: Optimize policy иT (according the reward), using data in D, with the constraint that x is
certified by h.
local maxima. MALA avoids this issue by its intrinsic stochasticity, which can also be controlled by adjusting the hyperparameter т.
We summarize our algorithm of training barrier certificates in Algorithm | (which contains optional regularization that will be discussed in Section 4.2). At Line 2, the initialization of s, 's is arbitrary, as long as they have sort of stochasticity.
4 CRABS: Co-trained Barrier Certificate for Safe RL
In this section, we present main algorithm, Co-trained Barrier Certificate for Safe RL (CRABS), shown in Algorithm 2, to iteratively co-train barrier certificates, policy and dynamics, using the
algorithm in Section gl In addition to parametrizing h by é, further parametrize the policy by 6, and parametrize calibrated dynamics model T by w. CRABS alternates between training a barrier certificate that certifies the policy अe w.r.t a calibrated dynamics model T.. (Line 5), collecting data safely using the certified policy (Line 3, details in Section (.1)) learning calibrated dynamics model (Line 4, details in Section 4.3). 3). and training a policy with the constraint of staying the superlevel
set of the barrier function (Line 6, details in Section 4.4 In the following subsections, discuss
how we implement each line in detail.
4.1 Safe Exploration with Certified Safeguard Policy
Safe exploration challenging because is difficult to detect irrecoverable states. The barrier certificate is designed to address this a policy certified by some h guarantees to stay within C₁ and therefore can be used for collecting data. However, we may need more diversity in the collected data beyond what can be offered by the deterministic certified policy slllegurd Thanks to the contraction property R.3 we in fact know that any exploration policy π expl within the superlevel set C, can be made safe with raleguurd being safeguard policy-we can first try actions from n "xxl and see they stay within the viable subset Cn and none does, invoke the safeguard policy x safeguardd Algorithm 3 describes formally this simple procedure that makes any exploration policy expl safe. By simple induction, one can see that the policy defined in Algorithm 3 maintains that all the visited states lie in Ca.
The safeguard policy x safeguard is supposed to safeguard the exploration. However, activating the safeguard too often is undesirable, as it only collects data from n. safoguurd there will be little
5

---

Algorithm 3 Safe exploration with safeguard policy safoguuad
Require: (1) A policy x zafeguard certified by barrier certificate h. (2) any proposal exploration policy π TTxpl
Require: A state Cha-
1: Sample actions 01,... On from (s).
2: if there exists an ag such that (s, ag, h) I then
3: refurn: aq
4: else
5:
return: x Safeguaard (s).
exploration. To mitigate this issue, often choose n to be noisy version of safoguuard that expl will be roughly safe by itself. Moreover, the safeguard policy Safeguard will be trained via optimizing the reward function as shown the next subsections. Therefore, a noisy version of π saleguard will explore the high-reward region and avoid unnecessary exploration.
Following Haarnoja et al. 2018). the policy To is parametrized as tanh(po(s)), and the proposal expl is parametrized as tanh(uu(s) (s(s)() for Ç ~ (0, I), where μs and exploration policy mg 0g are two neural networks. Here the tanh is applied to squash the outputs to the action set [-1, 1].
4.2 Regularizing Barrrier Certificates
The quality of exploration is directly related the quality of policy optimization. In our case, the exploration only within the learned viable set Cn,, and it will be hindered if Ch. is too small or does not grow during training. To ensure a large and growing viable subset Ch.- we encourage the volume of Ch. to be large by adding regularization term
④:: h) E..es - (s))))
Here h is the barrier certificate obtained in the previous epoch. In the ideal case when Reg(ó; h) O. we have Ch.p - Cj. that is, the new viable subset C.,, least bigger than the reference set (which is the viable subset in the previous epoch.) We compute the expectation over S approximately by using the set of candidate s's maintained by MALA.
In summary, to learn he in CRABS, we minimize the following objective (for a small positive constant X) over @ as shown in Algorithm [
[(;; U, ne, h) = (Lg, + AReg(á;h) h).
(5)
We remark that the regularization is not the only reason why the viable set Ch. grow. When the dynamics becomes more accurate as we collect more data, the Ch, will also grow. This is because an inaccurate dynamics will typically make the Cr2, smaller-it is harder to satisfy R.3 when the
confidence region f(s, T (s)) in the constraint contains many possible states. Vice versa, shrinking the size of the confidence region will make easier certify states.
4.3 Learning Calibrated Dynamics Model
It is a challenging open question obtain dynamics model T (or any supervised learning model) that is theoretically well-calibrated especially with domain shift [Zhao et al.) 2002)] In practice, heuristically approximate calibrated dynamics model by learning ensemble of probabilistic
dynamics models, following common practice in RL (Yu et al. 2020 Janner et al. 2019 Chua et al.,
2018). We learn K probabilistic dynamics models fay..... fax using the data in the replay buffer D. (Interestingly, prior work shows that an ensemble of probabilistic models can still capture the error of estimating a deterministic ground-truth dynamics [Janner et al. 2019 Chua et al. 2018 Each probabilistic dynamics model fa; outputs a Gaussian distribution (Mas (8, a), diag (a (s, a)) with diagonal covariances, where Mw: and ow, are parameterized by neural networks. Given replay buffer
D. the objective for probabilistic dynamics model fw; is to minimize the negative log-likelihood:
Lr(ux) -IR (9.. D |- log Sw, (8'18, a))
6

---

Figure I: Illustration of environments. The left figure illustrates the Pendulum environment, which is used by Upright and Tilt tasks. The right figer illustrates the CartPole environment, which is used by Move and Swing tasks.
(a) Pendulum
(b) CartPole
The only difference in the training procedure these probabilistic models is the randomness in the initialization and mini-batches. We simply aggregate the means of all learn dynamics models coarse approximation the confidence region, i.c., T(s, a) {mw (s. a))e(k-
4.4 Policy Optimization
We describe our policy optimization algorithm in Algorithm 4 The desiderata here are (1) the policy needs certified by the current barrier certificate h and (2) the policy has high reward possible. We break down our policy optimization algorithm into two components: First, we optimize the total rewards (TTg) of the policy πe: Second, we use adversarial training to guarantee the optimized policy can be certified by hg. The modification of SAC is to some extent non-essential and mostly for technical convenience of making SAC somewhat compatible with the constraint set. Instead, it the adversarial step that fundamentally guarantees that the policy is certified by the current has
Adversarial training. We use adversarial training to guarantee Te can certified by hg- Sim- ilar to what we've done in training he adversarially, the objective for training Ty is to min- imize C* Ch U, U,) Unlike the case of o, the gradient of C* (ha mo) w.r.t. 0 is simply (sU(s" To(s") ), ha), as the constraint hs(s) is unrelated to wo. We also use MALA to solve s* and plug it into the gradient term VoU(s", ,,)(s),))))
Optimizing (TT)). We use modified SAC (Haarnoja et al. 2018 to optimize J(To). As the modification for safety concerns and is minor, we defer Appendix B As side note, although expl expll here, Ty is also optimized implicitly because Tp" simply outputs the mean
we only optimize n. of deterministically.
5 High-risk, High-reward Environments
We design four tasks, three of which are high-risk, high-reward tasks, to check the efficacy of our algorithm. Even though they all based on inverted pendulum cart pole, choose the reward function to be somewhat conflicted with the safety constraints. That is, the optimal policy needs to take trajectory that is near the safety boundary. This makes the tasks particularly challenging and suitable for stress testing our algorithm's 's capability of avoiding irrecoverable states.
These tasks have state dimension dimensions between 2 to 4. We focus the relatively low dimensional environments to avoid conflating the failure to learn accurate dynamics models from data and the failure to provide safety given learned approximate dynamics. Indeed, identify that the major difficulty to scale up to high-dimensional environments that it requires significantly more data to learn decent high-dimensional dynamics that can predict long-horizon trajectories. We remark that we aim to have zero violations. This is very difficult to achieve, even if the environment low dimensional. As shown by Section many existing algorithms fail do
(a) Upright. The task is based on Pendulum-v0 Open AT Gym [Brockman et al.) 2016). as shown in Figure Tal The agent the tilt angle 9 which is defined to be the angle between the pole and a vertical line. The safety requirement is that the pole does not fall below the horizontal line. Technically, the user-specified safety set {0 |0] < mmmx = 1.5) (note that the threshold is very close to which corresponds to
apply torque to control a pole. The environment involves the crucial quantity:
90%.) The reward function r is r(s, a) = , so the optimal policy minimizes the angle and angular speed by keeping the pole upright. The horizon is 200 and the initial state so (0.3, -0.9).
(b) Tilt. This action set, dynamics, and horizon, and safety set are the same as in Upright. The reward function is different: r(s, a) -(0 - 0)2. The optimal policy is supposed to stay tilting near the
7

---

Upright
Tilt
Move
Swing
total rewards
1000
-1-
-100
500
25
-150
500
1000
urnsafe apisodes
300
700
1.50
460
200
200
100
500
1000
1000 # episodes total
100
500
1000 # episodes in total
100
500
500
1000
episodes in total
# episodes in total
CRABS
SAC
RecoveryAL
CPO
SQRL
Figure 2: Comparision between CRABS and baselines. CRABS can learn a policy without any safety violations, while other baselines have lot of safety violations. We each algorithm four times with independent randomness. The solid curves indicate the mean of four runs and the shaded areas indicate standard deviation around the mean.
angle 0 uiiit where Miit 0. 44.411 1111516844 is the largest angle the pendulum can stay balanced. The challenge is during exploration, it is easy for the pole to overshoot and violate the safety constraints.
(c) Move. The task is based cart pole and the goal
cart (the yellow block) control the pole (with color teal), as shown in Figure 1b The cart has an æ position between -1 and 1, and the pole also has an angle
1 2 2] with the same meaning as Upright and Tik.. The starting O. We design the reward function to be r(s,a) x² The user-specified safety set is {(x, 0) : |0] < mmx 0.2, |a|| < 0.9) where 0.2 corresponds to roughly 11 Therefore, the optimal policy needs to move the cart and the pole slowly in one direction, preventing the pole from falling down and the cart from going too far. The horizon is set to 1000.
position is
(d) Swing. This task is similar to More, except for few differences: The reward function is r(s, a) = 82; The user-specified safety set is {(x, 0) |0] < m = 1.5, |x| S 0.9}. So the optimal policy will swing back and forth some degree and needs control the angles well so that it does not violate the safety requirement.
For all the tasks, once the safety constraint is violated, the episode will terminate immediately and the agent will receive reward of -30 as a penalty. The number -30 is tuned by running SAC and choosing the one that SAC performs best with.
Experimental Results
In this section, we conduct experiments to answer the following question: Can CRABS learn reasonable policy without safety violations in the designed tasks?
Baselines. We compare algorithm CRABS against four baselines: (a) Soft Actor-Critic (SAC)
[Haarnoja et al., 2 one of the state-of-the-art RL algorithms, (b) Constrained Policy Optimiza- tion (CPO) Achhamm al. 2017], a safe RL algorithm which builds trust-region around the current policy and optimizes the policy in the trust-region,
(c) Re. (Thananjeyan et al. 2021] which leverages offline data to pretrain risk-sensitive Q function and also utilize two policies to achieving two goals (being safe and obtaining high rewards), and (d) SQRL (Srinivasan et al. 2020 which leverages offline data in an easier environment and fine-tunes the policy in a more difficult environment. SAC and CPO are given an initial safe policy for safe exploration, while RecoveryRL RL and SQRL are given offline data containing 40K steps from both mixed safe and unsafe trajectories which are free and are not counted. CRABS collects more data at each iteration in Swing than in
other tasks to learn better dynamics model 7. For SAC, we use the default hyperparameters because

---

0.2
1,0
02
1,0
0.2
1.0
0.2
1.0
Irajedory
0.3
0.0
040
0.0
0.0
0.00
0.0
0.0
002
-0.5
0.5
-10 -02
-0.5
0.0
-10
-02
-0.5 0.6
-1.
-02
-6.5 5.0
0.5
-00
posation
posttoo
position
postiton
(a) Cr.p after O epochs (b) Cr. after 5 epochs
(c) Ch. after 10 epochs (d) Cr.p after 15 epochs
Figure 3: Visualization of the growing viable subsets learned by CRABS in Move. To illustrate the 4-dimensional state space, we project state from [xr 0, ž; 0J to [x, 0J. The red curve encloses superlevel set Ch.. while the green points indicate the projected trajectory of the current safe policy. We can also observe that policy learns to move left as required by the task. We note that shown states in the trajectory sometimes seemingly are not be enclosed by the red curve due the projection.
we found they are not sensitive. For RecoveryRL and SQRL, the hyperparameters are tuned in the same way as in (Thananjeyan al.) [22T]] For CPO, we tune the step size and batch size. More
details of experiment setup and the implementation of baselines can be found in Appendix C
Results. Our main results shown in Figure From the perspective of total rewards, SAC achieves the best total rewards among all of the algorithms in Move and Swing. In all tasks, CRABS can achieve reasonable total rewards and learns faster at the beginning of training, and we hypothesize that this is directly due to its strong safety enforcement. RecoveryRL and SQRL learn faster than SAC in Move, but they suffer in Swing. RecoveryRL and SQRL are not capable of learning in Swing, although we observed the average return during exploration at the late stages of training can be as high as 15. CPO quite sample-inefficien and does not achieve reasonable total rewards well.
From the perspective of safety violations, CRABS surpasses all baselines without a single safety violation. The baseline algorithms always suffer from many safety violations. SAC, SQRI, and RecoveryRI. have a similar number of unsafe trajectories in Upright, Tilt, Move, while in Swing, SAC has the fewest violations and RecoveryRL. has the most violations. CPO has lot of safety violations. We observe that for some random seeds, CPO does find safe policy and once the policy trained well, the safety violations become much less frequent, but for other random seeds, CPO keeps visiting unsafe trajectories before it reaches its computation budget.
Visualization of learned viable subset Cr.- We visualized the viable set Ch. in Figure As shown in the figure, our algorithm CRABS succeeds in certifying more and more viable states and does not get stuck locally, which demonstrates the efficacy of the regularization at Section 4.2
Handcrafted barrier function. To demonstrate the advantage of learning a barrier function, we also conduct experiments on a variant of CRABS, which uses handcrafted barrier certificate by ourselves and does not train it, that is, Algorithm 2 without Line 5. Our goal for the handcrafted barrier function to only show that it's non-trivial to find a subset viable states by handcrafting a region and therefore to demonstrate the necessity to learn the barrier certificates. In the experiments, we handcrafted a function g. whose superlevel set is a subset safe states, and restricted exploration in this region (the superlevel set of g) in similar manner (A (Algorithm 3). Here only satisfies R.1 and R.2 but not necessarily R.3 Therefore, strictly speaking, the designed function g is not a barrier certificate, but only merely attempts to characterize subset of safe states and restrict the exploration to it. More details the handcrafted function are available at Appendix C
current exploration scheme
The results show that this variant does not perform well: It does not achieve high rewards, and has many safety violations. We hypothesize that the policy optimization is often burdened by adversarial training, and the safeguard policy sometimes cannot find an action to stay within the superlevel set Ch.

---

7
Related Work
Prior works about Safe RL take very different approaches. Dalal et al. 2018 adds an additional layer, which corrects the output of the policy locally. Some of them Lagrangian methods to solve CMDP, while the Lagrangian multiplier is controlled adaptively (Tessler et al. 2018) or by a PID (Stooke et al. 2020 AAchiam al. 2017]. Yang et al.] [2020] build trust-region around the current policy. Eysenbach et al. 2017] learns a reset policy so that the policy only explores the states that can go back the initial state. Turchetta et al.) 2020] introduces learnable teacher, which keeps the student safe and helps the student learn faster in a curriculum manner. [Srinivasan et al. [2200] pre-trains policy in a simpler environment and fine-tunes it in more difficult environment. Bharadhwaj et al. [2020) learns conservative safety critics which underestimate how safe the policy is, and uses the conservative safety crities for safe exploration and policy optimization. Thananjeyan ct al. [202T] makes use of existing offline data and co-trains a recovery policy.
Another line of work involves Lyapunov functions and barrier functions. Donti et al. 2020 constructs sets of stabilizing actions using a Lyapunov function, and project the action to the set, while Chow et al.
2019 projects action parameters to ensure the decrease of Lyapunov function after step. Ohnishi et al. 2019) is similar to ours but it constructs barrier function manually instead of learning such one. Ames et al. 2019 gives an excellent overview of control barrier functions and how to design them. Perhaps the most related work to ours is Cheng et al. (2019). which also uses a barrier function to safeguard exploration and uses reinforcement learning algorithm learn policy, However, the key difference is that we learn barrier function, while Cheng et al. [2019] handcrafts one. The works on Lyapunov functions [Berkenkamp al. 2017 Richards al. 2018 require the
discretizating the state space and thus only work for ow-diimessional space.
8 Conclusion
In this paper, we propose novel algorithm CRABS for training- taining-timme safe RL. The key idea is that we co-train a barrier certificate together with the policy to certify viable states, and only explore in the Icarned viable subset. The empirical rscults show that CRABS can learn some tasks without single safety violation. We consider using model-based policy optimization techniques to improve the total rewards and sample efficiency promising future work. Another fascinating direction is how to deal with less accurate learned dynamics model in higher dimension environments.
Acknowledgement
We thank Changliu Liu, Kefan Dong, and Garrett Thomas for their insightful comments. YL supported by NSF, ONR, Simons Foundation, Schmidt Foundation, DARPA and SRC. TM acknowledges support of Google Faculty Award, NSF IIS 2045685, and JD.com.
References
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference Machine Learning, pages 22-31. PMLR, 2017.
lige Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint arXiv:1910.0711: 2019.
Aaron D Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control barrier functions: Theory and applications. In 2019 18th European Control Conference (ECC), pages 3420-3431. IEEE, 2019.
Felix Berkenkamp, Matteo Turchetta, Angela P Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. arXiv preprint arXiv: 1705008575 2017.
Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety critics for exploration. arXiv preprint arXiv:2010.14497 2020.
10

---

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Richard Cheng, Gábor Orosz, Richard Murray, and Joel W Burdick. End-to-end safe reinforcement learning through barrier functions for safety cafety-critical continuous control tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3387-3395, 2019.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. arXiv preprint arXiv: 2019.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. arXiv preprint arXiv: 1805./2/14, 2018.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint 101.v01801.08757 2018.
Kefan Dong, Yuping Luo, Tianhe Yu, Chelsea Finn, and Tengyu Ma. On the expressivity of neural networks for deep reinforcement learning. In International Conference on Machine Learning, pages 2627-2637. PMLR, 2020.
Priya Donti, Melrose Roderick, Mahyar Fazlyab, and Zico Kolter. Enforcing robust control guarantees within neural network policies. arXiv preprint arXiv:2011.08105.. 2020.
Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave trace: Learning to reset for safe and autonomous reinforcement learning. arXiv preprint 1rX1v21711.06782, 2017.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference on Machine Learning, pages 1587-1596. PMLR, 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, al. Soft actor-critic algorithms and applications. arXiv preprint 1812.098,, 2018.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. arXiv preprint arXiv: 08255, 2019.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 2014.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In Interna- tional Conference Learning Representations, 2019. httllattps///ppeerriee...ee
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 51 518(7540):529-533 2015.
Quan Nguyen and Koushil Sreenath, Exponential control barrier functions for enforcing high relative- degree safety-critical constraints. In 2016 American Control Conference (ACC), pages 322-328. IEEE, 2016.
Motoya Ohnishi, Li Wang, Gennaro Notomista, and Magnus Egerstedt. Barrier-certifee adaptive reinforcement learning with applications brushbot navigation. IEEE Transactions on robotics, 2019. 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVoto, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, rechpperformance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché
11

---

Buc, E. Fox, and R. Gamett, editors, Advances in Neural Information Processing Systems 32. pages 8024 8035. Curran Associates, Inc., 2019. URL http:////w/pp////ppppp/ooaaaaaaa 9015 pdf
Stephen Prajna and Ali Jadbabaie. Safety verification of hybrid systems using barrier certificates. In International Workshop on Hybrid Systems: Computation and Control, pages 477-492. Springer, 2004.
Stephen Prajna and Anders Rantzer. On the necessity of barrier certificates. IFAC Proceedings Volumes, 38(1):526 531, 2005.
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Searching for activation functions. arXiv preprint arXiv:1770.055441 2017.
Spencer M. Richards, Felix Berkenkamp, and Andreas Krause. The дyарunov neural network: Adaptive stability certification for safe learning of dynamical systems, 2018.
Melrose Roderick, Vaishnavh Nagarajan, and J Zico Kolter. Provably safe pac-mdp exploration using analogies. arXiv preprint www.iv:2007.0577222 2020.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou. Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):3543333 2017.
Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. Learning to be safe: Deep with a safety critic. arXiv preprint arXiv:2010. Ttfi:20000.460033 14603, 2020.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pages PMLR, 2020.
Andrew Taylor, Victor D Dorobantu, Hoang M Le, Yisong Yue, and Aaron Ames. Episodic learning with control іyарunov functions for uncertain robotic systems. In 2019 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 6878 77888-684..88 IEEE, 2019.
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805.11077 2018
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 6(3): 4915-4922, 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012.
Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe rein- forcement learning via curriculum induction. arXiv preprint arXiv:2006. 12136, 2020.
Tsung- Teu Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Accelerating safe reinforcement learning with constraint-mismatched policies. arXiv preprint arXiv:2006./ 1645, 2020.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-base offline policy optimization. arXiv preprint arXiv:2005.132.99 2020.
Jun Zeng, Bike Zhang, and Koushil Sreenath. Safety-critical model predictive control with discrete- time control barrier function. arXiv preprint arXiv:2007.11718 2020.
Shengjia Zhao, Tengyu Ma, and Stefano Ermon. Individual calibration with randomized forecasting. arXiv preprint arXiv:2006. 10288, 2020.
12


Reviews:
[{"review_id": "hGBQ8vMP7bN", "reviewer": "Reviewer_DXuC", "summary": "This work proposes an algorithm of iteratively learning dynamical system model, barrier certificate and policies, for safe RL.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The overall idea of co-learning of dynamics, barrier certificate and policy are interesting and important.\nAnd the paper is clearly written.\nHowever the reviewer has the following concerns.\n\nMajor concerns:\n1) The author(s) mention that the proposed method potentially works for higher dimensional systems; however the experiment is only considering rather simple models.\nAlso, the work stresses that the proposed method aims at zero training-time violations and compared it against existing safe RL work; however, as the author(s) also mention in the experiment section that more data are necessary to achieve zero violations, safety violations ultimately depend on data, representational capacity, disturbance, convergence of algorithm, approximation of confidence interval etc etc. in practice.  Note theoretically it is easy to say zero-training violation if one makes certain assumptions.  Therefore, it is very important to clearly discuss why the proposed method outperforms others in terms of safety violations from several perspectives.\n\n2) There is already an existing work that does co-learning of dynamics, Lyapunov function (through value function learning), (and policy). (e.g. Berkenkamp)  This work basically replaced Lyapunov function by barrier certificate, and generalized the problem setting.\n(please consider the nominal known dynamical system model as that can be obtained by initial safe policies assumed in this work)\nTherefore, it is again very important to discuss why the use of barrier certificate with the proposed adversarial training may be better than that. \n\nIf the above concerns are addressed, the reviewer is happy to raise the score.\n\n----------\nreview updated after the initial response", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are well discussed.  However, as long as the author(s) stress zero training-time violaton, the limitation is crucial.", "review_text": "", "extra_scores": {"ethical_concerns": "The reviewer believes there is no significant ethical concern in this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "XakN8QT3x83", "reviewer": "Reviewer_78sq", "summary": "The paper proposes an RL algorithm that starts from an initial safe but low return policy that is iteratively improved while ensuring no safety violation occurs during the rollouts required by the training. The paper achieves this by learning barrier functions as safety certificates.  The two \"simpler\" barrier conditions are directly enforced in the structure of the learned function. In contrast, the \"closed under a policy+environment step\" condition R3 is checked heuristically on a probabilistic model of the environment with a Metropolis-adjusted Langevin algorithm. \nFinally, the authors evaluate the achieved return and number of safety violations of their RL algorithm and other RL algorithms on variations of the Pendulum and CartPole gym environments.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper is well written.\nNonetheless, I noticed a few unclear parts of the paper that I would like to authors to clarify.\n\nTo check R3 requires solving the non-convex/concave global optimization problem in Eq. (1) and its derived forms. As MALA works only with a set of local maxima, no guarantees about the global optimality can be made (it seems to be only a heuristic). I think this design choice is okay but needs to be communicated more clearly, i.e., as it is very well done in section 4.3 that there is a tradeoff between feasibility and correctness of the transition model (in the sense that a flawed dynamics model can lead to incorrect safety guarantees of the barrier function).\n \n\nThe section about the handcrafted barrier function is a bit imprecise. As a barrier function is policy dependent, I suppose the \"handcrafted\" function is a barrier function of the initial safe policy. Therefore, I would then expect that the achievable return when training the initial policy very much depends on the specific barrier certificate, i.e., if by the initial policy non-reachable but actually by a better policy reachable safe states are considered safe or unsafe.\nMoreover, how does the scenario with a handcrafted barrier function lead to safety violations? This shouldn't be the case? Please elaborate.\n\nIn Figure 2 top row, it seems that CRABS learns faster than unconstraint SAC. This is counter-intuitive and suggests that the hyperparameters are inadequately chosen, i.e., either SAC hyperparameters are non-optimal or CRABS hyperparameters received a much larger tuning budget.\n\nI like that the authors discuss the limitations of their approach with respect to higher dimensional problems, i.e., that learning a dynamics model becomes challenging and leads to safety violations.\n\nI positively acknowledge that the authors provide the code for their experiments.\n\n\nI am looking forward to reading the author's response.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think most of the limitations of the paper are sufficiently communicated (except my first concern in the main review).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "9ACVLiTDJ-", "reviewer": "Reviewer_SUah", "summary": "This paper proposes a safe reinforcement learning algorithm, CRABS, which can achieve zero training-time safety violations. The key idea is to iteratively learn the dynamic model, the policy and the barrier certificates. The barrier certificates are learned via adversarial training and can constrain the exploration of the policy in the safe set. The proposed algorithm is evaluated in four simple classical control problems (variants of inverted pendulum and cartpole). The evaluations demonstrate zero training-time violations and comparable learning performance compared to other safe learning baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Safe learning with zero training-time violations is critical for applying reinforcement learning to real-world systems, such as robotics. This paper proposes a method that nicely combines reinforcement learning (SAC) with control theory (barrier certificates) to tackle this exact challenge. The paper is well written. The method is novel. The results are promising.\n\nAlthough I like the proposed method and I believe that it has good technical contributions, I did not give the paper a high score mainly due to the evaluations. I am disappointed that the method is only evaluated on simple control problems based on inverted pendulum and cartpole. Although it seems that CRABS can beat baselines (CPO, recovery RL) in these environments, it is not evaluated in more complicated robotic environments, such as locomotion, navigation, and manipulation, where the baselines work well. To make CRABS useful and impactful, experiments that show CRABS can work equally well or better in complex environments are essential.\n\nHere are a few detailed suggestions how the paper can be improved:\n1) It is important to include more evaluations on challenging environments and show the advantages of CRABS. I understand that the model-learning part is the bottleneck to scale up the method to high-dimensional environments. Since for many control problems, the model is well known (e.g. the unicycle model for wheeled robots), it is still valuable to show that if the ground-truth or an approximate model is given, CRABS can guarantee safety during the training process. A navigation task with wheeled robots (safety gym) seems to be a good example to add.\n\n2) It appears to me that CRABS learns a model and uses that model in barrier certificate, but does not leverage the model in policy learning. The policy learning is a modified SAC, a model-free method. As we all know, model-based RL is more data efficient than its model-free counterparts. I wonder whether the policy update of CRABS can use the learned model, and then becomes more data efficient. This is just a suggestion for the long-term research direction, not a criticism for the current paper.\n\n3) The second paragraph of 2.2 seems incorrect to me. According to the text, If h(s)>=0 and h(s')>=0, and then h is a barrier certificate. If this is the case, finding a barrier certificate would be easy: h = |s|^2. Did I miss anything?\n\n4) Throughout of the paper, there are a few equations that have argmax_{s:h_\\phi(s)<=1} U. I am not sure where \"h_\\phi(s)<=1\" comes from. Shouldn't it be h_\\phi(s)<=0?\n\n5) Since the effectiveness of CRABS heavily depends on the model accuracy, it would be nice to have an ablation study to understand the relation between the number of training-time violations and the model accuracy.\n\n6) It would be great to give more details about \"Handcrafted barrier function\" paragraph in Section 6. Since how the barrier certificate is handcrafted would significantly impact the performance of the algorithm, the conclusion that \"this variant does not perform well\" is less convincing without details of handcrafting.\n\n7) Typo: Line 352: rseults->results", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Both the limitation and the potential societal impact are discussed.\n\nA suggestion: I like Appendix E, which is a comprehensive summary of the limitations. It would be great to make limitation more explicit in the main text, instead of \"hidden\" in the Appendix.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: I agree with the reviewers that this paper deserves to be accepted; it has a novel foundational contribution and is well written. 

The paper presents a new approach that iteratively generates a barrier certificate, an environment model and a policy, in a way that ensures that there are no safety violations even during training. The algorithm requires an initial safe (but possibly suboptimal) policy, and optimizes it while ensuring that there will not be any safety violations during training. 

As pointed out by one of the reviewers, the main downside of the paper is that the experiments are not very ambitious; the approach is evaluated on only a few problems with a very low-dimensional state space. However, the task of training with no safety violations is already challenging even for these relatively simple environments. There are also a few places where the explanations could be a little clearer, for example, the way algorithm 2 is presented, it is not entirely clear where the initial barrier certificate comes from. But overall, this is a very strong paper with a solid contribution to the state of the art in safe reinforcement learning.


Decision: Accept (Poster)