Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments
Authors: Yixuan Wang
Simon Sinong Zhan
Ruochen Jiao
Zhilu Wang
Wanxin Jin
Zhuoran Yang
Zhaoran Wang
Chao Huang
Qi Zhu
Venue: Submitted to ICLR 2023 (2023)

Abstract: It is quite challenging to ensure the safety of reinforcement learning (RL) agents in an unknown and stochastic environment under hard constraints that require the system state not to reach certain specified unsafe regions. Many popular safe RL methods such as those based on the Constrained Markov Decision Process (CMDP) paradigm formulate safety violations in a cost function and try to constrain the expectation of cumulative cost under a threshold. However, it is often difficult to effectively capture and enforce hard reachability-based safety constraints indirectly with such constraints on safety violation cost. In this work, we leverage the notion of barrier function to explicitly encode the hard safety constraints, and given that the environment is unknown, relax them to our design of \emph{generative-model-based soft barrier functions}.  Based on such soft barriers, we propose a safe RL approach that can jointly learn the environment and optimize the control policy, while effectively avoiding the unsafe regions with safety probability optimization. 
Experiments on a set of examples demonstrate that our approach can effectively enforce hard safety constraints and significantly outperform CMDP-based baseline methods in system safe rate measured via simulations. 

Under review as a conference paper at ICLR 2023
ENFORCING HARD CONSTRAINTS WITH SOFT BARRI- ERS: SAFE REINFORCEMENT LEARNING IN UNKNOWN STOCHASTIC ENVIRONMENTS
Anonymous authors Paper under double-bling brind review
ABSTRACT
It is quite challenging to ensure the safety of reinforcement learning (RL) agents in an unknown and stochastic environment under hard constraints that require the system state not to reach certain specified unsafe regions. Many popular safe RL methods such as those based on the Constrained Markov Decision Process (CMDP) paradigm formulate safety violations in cost function and try to con- strain the expectation of cumulative cost under a threshold. However, is of- ten difficult to effectively capture and enforce hard reachability- based safety con- straints indirectly with such constraints on safety violation cost. In this work, we leverage the notion of barrier function explicitly encode the hard safety con- straints, and given that the environment unknown, relax them to our design of serrrative-moee---eee
soft barrier functions. Based on such soft barriers, we propose a safe RL approach that can jointly learn the environment and optimize the control policy, while effectively avoiding the unsafe regions with safety prob- ability optimization. Experiments on a set of examples demonstrate that our ap- proach can effectively enforce hard safety constraints and significantly outperform CMDP-based baseline methods in system safe rate measured via simulations.
INTRODUCTION
Reinforcement learning (RL) (Sutton & Barto 2018) has shown promising successes in learning complex policies for games (Silver et al.) 2018), robots (Zhao et al., 2020), and recommender sys- tems (Afsar et al. 202T), by maximizing cumulative reward objective as the optimization goal. However, real- weal-world safety-critical applications, such as autonomous cars and unmanned aerial ve- hicles (UAVs), still hesitate to adopt RL policies due to safety concerns. In particular, these appli- cations often have hard safety constraints that require the system state not reach certain specified unsafe regions, e.g., autonomous cars not deviating into adjacent lanes UAVs not colliding with trees. And it is very challenging to learn a policy via RL that can meet such hard safety constraints, especially when the environment is stochastic and unknown.
In the literature, the Constrained Markov Decision Process (CMDP) (Altman lar paradigm for addressing RL safety. Common CMDP-based methods encode safety constraints through a cost function of safety violations, and reduce the policy search space to where the ex- pectation of cumulative discounted cost is less than threshold. And various RL algorithms proposed to adaptively solve CMDP through the primal-dual approach for the Lagrangian problem of CMDP. However, it is often hard for CMDP- methods to enforce reachability- safety constraints (i.c., system state not reaching unsafe regions) by setting indirect constraints on the expectation of cumulative cost. In particular, while reachability-based safety constraints are de- fined on the system state at each time point (i.e., each point on the trajectory). the CMDP constraints only enforce the cumulative behavior. In other words, the cost penalty on the system visiting the unsafe regions at certain time point may be offset by the low cost at other times. There is a recent CMDP approach addressing hard safety constraints by using the indicator function for encoding fail- probability /Wagener et al. 2021), but it requires safe back-up up policy for intervention, which difficult to achieve in unknown environments. Safe exploration with hard safety constraint has been studied in (Wachi al. 2018 Turchetta al. 2016 Moldovan & Abbeel 2012). However,
1999 is a popu-
hard

---

Under review as a conference paper at ICLR 2023
they focus discrete state and action spaces where the hard safety constraints defined set of unsafe state-action pairs that cannot be visited, different from our continuous control setting.
On the other hand, current control-thcorettial approaches for model-based safe RL often try to leverage formal methods to handle hard safety constraints, e.g., by establishing safety guarantees through barrier functions control barrier functions (Luo & Ma 2021), or by shielding mechanisms based on reachability analysis to check whether the system may enter the unsafe regions within a time horizon (Bastani et 2021). However, these approaches either require explicit known system models for barrier or shielding construction, or an initial safe policy to generate safe trajectory data deterministic environment. They cannot be applied to our unknown stochastic environments.
To overcome the above challenges, we pro- pose safe RL framework by encoding the hard safety constraints via the learning of generative-model-bae
Ours: You are bounded by soft barrier function along each paint on the trajectory, and is highly likely to be safe.
soft barrier function. Specifically, we formulate and solve novel bi- level optimization problem to learn the policy with joint soft barrier function learning, gener- ative modeling, and reward optimization. The soft barrier function provides guidance for avoiding unsafe regions based on safety prob- ability analysis and optimization. The genera- tive model accesses the trajectory data from the environment-policc closed-loop system with stochastic differential equation (SDE) represen- tation learn the dynamics and stochasticity of the environment. And we further optimize the policy by maximizing the total discounted re- ward of the sampled synthetic trajectories from the generative model. This joint training framework is fully differentiable and can be efficiently solved via the gradients. Compared to CMDP- methods, our approach more directly encodes the hard safety constraints along each point the agent trajectory through the soft barrier function, as shown in Figure| [ While given the unknown stochastic environment, approach cannot pro- vide a hard barrier and hence no deterministic safety guarantee, experimental results demonstrate that in simulations, ours can significantly outperform the CMDP-based baselines in system safe rate.
CMDP:
The expectation of cumulative
looks good.
should fine?
Figure An RL-based robot navigation exam- plc that shows the difference between our ap- proach and CMDP-based ones encoding the hard safety constraints.
The paper is organized as follows. Section introduces related works, Section presents our ap- proach, including the bi-level optimization formulation, safe RL algorithm with generative mod- cling, soft barrier function learning, and policy optimization to solve the formulation, and theoretical analysis of safety probability. Section shows the experiments and Section concludes the paper.
2 RELATED WORK
Safe RL by CMDP: CMDP- methods encode the safety violation cost function and set constraints on the expectation of cumulative discounted total cost. The primal-dual approaches have been widely adopted
solve the Lagrangian problem of constrained policy optimization, such as
PDO CCow et al. 2017). OPDOP DDing et al. 2021), CPPO (Stooke et al. 2020). FOCOPS (Zhang al. 2020), and CRPO (u al.
2021). l). Other works leverage world model learning (As al.
2021 or the Lyapunov function to solve the CMDP (Cow et al., 2018), or add safety layer for the safety constraint (Dalal et al. 2018). However, the constraints in CMDP cannot directly encode the hard reachability- systems. recent CMDP-based work uses the indicator function for encoding failure probability as hard safety constriants, but it requires safe backup policy for intervention (Wagener et al., 2021).
safety properties, which hinders its application to many safety-critical
Model-based Safe RL by Formal Methods: Formal analysis and verification techniques have been proposed in model-based safe RL enforce the system not reach unsafe regions. Some works develop shielding mechanisms with backup policy based on reachability analysis
(Shao et al.
2021
Li & Bastani 2020 Bastani et al. 202 Other works adopt (control) barrier functions or
(control) Lyapunov functions for provable safety (Emam et al. 2021 Choi et al.
2020
Cheng et al,
2019
Wang et al. 2022
Ma al. 2021 Luo & Ma 202
Berkenkamp et al.
2017
Taylor al.
2

---

Under review as a conference paper at ICLR 2023
Generetive Modeling
Pulicy
(f 83) Ey a))
R(Se) B(s)) + D())
Optimtration
Barrier Bg
Barrerr
(s.ar.5)
a, P,5)) Generative Model Ma.
SDE
M (S. P. (.)
Environment
G3. E, ())(())
Pollcyy Wy
(3)
2f (8)
Figure 2:
The overview of our safe RL framework based on wwwediiee-oool---eee soft barrier function. The real environment and generative model share the learning policy and the generative model is abstracted as discrete-time stochastic differential equation.
2020).
Moreover, recent work (Yu ct al. 2022) adopts reachability analysis with CMDP to compute safe feasible sets. However, these methods either require known dynamics, safe initial/backup policy, or human intervention, and thus do not apply to our setting.
Barrier Function for Safety: Barrier function is introduced as a safety certificate afflicted to the control policy for deterministic and stochastic systems (Prajna & Jadbabaie/ 2004 Prajna et al. 2004). In classical control, finding a barrier function is time-consuming and requires a lot of manual effort, where common idea is relax the conditions of barrier function into optimization for- mulations such as linear programming (Yang et al.] (0166), quadratic programming (Ames et al.) 2016), and aum-of-square programming (Wang et al. 2022). However, these optimization- approaches can hardly scale to high-dimensional systems. To this end, recent works have shown great promise jointly training barrier function and safe policy by neural network representation for better scalability (Qin et al. 202I). Our approach leverages the paradigm of barrier function, but develops the concept soft barrier to address unknown stochastic environments.
RI. with Generative Model: Previous works of generative-model-besse RL. mainly focus sam- ple efficiency and policy optimization for the total expected return (Agarwal et al.] 20206) Li al.) 2020a Tirinzoni et al.) 2020). Some works (HasanzadeZonuzy dress safe RL by CMDP with generative model, but only solve the tabular discrete state and action space. Besides policy optimization, the generative model in framework also plays important role in building a soft barrier function to facilitate the probabilistic safety analysis and optimization.
al.) 2021 Maeda et al.) [02 l) ad-
3 OUR APPROACH
In this section, we present our framework for safe RL in unknown stochastic environment that en- forces hard safety constraints with soft barrier functions. In Section 3.1 we present bi-level optimization formulation for the problem. which maximizes a total expected return while trying to avoid unsafe regions. Specifically, model-based soft barrier function the lower problem and maximize the performance with gen- erative model learning in the upper problem. We then present our safe RL algorithm solve the bi-level optimization formulation, by jointly learning the generative model (Section 3.2), soft bar- rier function (Section 3.3), and policy optimization (Section via first-order gradient, as shown in We conduct theoretical analysis for safety probability of the learned policy in Section 3.5
encode the hard safety constraints with novel generative-
Figure 2
3.1 BILEVEL OPTIMIZATION PROBLEM FORMULATION FOR SAFE RL
We assume that the environment can be abstracted as a finite-horizon continuous MDP M ~ A, r, T), where S E R" represents the continuous state space, A E TT indicates the continuous action space, and the function class P S x A [0, 1) denotes the unknown con- tinuous and smooth stochastic environment dynamics without jump condition. The rewards func- tion r(s,a) a) S x A + E is known and the discount factor & E [0. A deterministic con- tinuous NN-based policy no S - A maps the states st)) S to action a(t) E A at
3

---

Under review as a conference paper at ICLR 2023
Algorithm Safe RL with the Generative-model-oee Soft Barrier Function
Input: Unknown environment M, initial policy ne, generative model Ma,as barrier network B3. Parameter: [0, a, f].
Output: Policy xe with Ma,a based soft barrier function B3-
1: For k in 0, ... .N
2: For in 0,. -0,0........ .M
3: Sample processes T&K by policy ve with M and synthetic processes fg. by no with Ma
4: Compute generative loss function Lgg with rá and f0.a in Equation
aCg
5: Compute barrier function loss Ly by sampling synthetic op.a as in Equation
6: Compute total discount reward (TT) by sampling synthetic f0,a as Equation
7:
8
dCn aJ 09 86
B B OCn.
08
time t as a(t) = (t(s())), where s(t) is a random variable at timestep t. The environment has several known spaces, i.e., the state space S C S, the initial space So S, and the un- safe space Su C S. The RL objective is to maximize the total discounted expected return as max J == P('')es >'ro"r(sst),,... a(t)) with P E P. To encode the hard safety constraint S,, we formulate bi-level optimization problem for our framework as the following. We use to denote the elements related to the generative model.
Definition (Bi-level Optimization Problem for Safe RI.)
max ( (0) (ogTTo.fo,al 8,0
where 1f (0, or) is the optimal objective to lower-level problem of the generativ--madel--uuel soft barrier function with s as the state in the generative model:
BB($) 2 O, BB( 2 1 S.,, BB(3) S n e So, E B 1))[((0) & (s(8 (( Ma,a(((t)), At E To Su,
min y s.t. B
(1)
where 9 is the parameter of policy π. is the parameter of the generative model Me, = (Ga, Ea), which stochastic differential equation (SDE) with G. as the drift function and E. as the dif- fusion function for the stochasticity, as shown later in Equation 2 A 2 O is penalty multiplier. Ta == {s(0), s(1), ... s(T}} and ε.... : {s(0), 5(1), ... { are the sampled realizations of stochastic processes (trajectories) from the environment and from the generative model by the soft barrier function
policy ne. respectively. 3 is the parameter of the generative- weeeeeeoooelaaaaaeeeeee Bg R + R We encode the hard safety constraint by the generative-modee model--aaeee soft bar- rier function Bg in the lower problem, which minimizes (8,, a) as the upper bound of the unsafe probability for Me,a in Section 3.3 The upper problem aims to optimize the policy's 's expected return J(Te) and learn the generative model by the maximum likelihood loss Lg (TH, fa.a) between the processes and ia.0 shown later in Equation 3 Moreover, the upper problem penalizes "7" (0, a), which can back propagate the gradient information through Me.. to тe for pushing the agent to avoid S, in the environment MDP M as much as possible if Me,a behaves similar to M.
We can compute the gradient from (f, a:) for Ty through Ma,a with current auto auto-differential tools. This cannot be done in M differentiable and can be solved efficiently. Figure 2 shows how the components in our framework interact with each other. The overall algorithm solve the bi-level problem is shown in Algorithm I Next, we are going to introduce the details of each module.
unknown. Therefore, the overall bi-level problem end-to-end
3.2 GENERATIVE MODELING
The role of the generative model in our framework is two folds: (1) Because the barrier function requires environment model encode the hard safety constraints, the generative model serves
4

---

Under review as a conference paper at ICLR 2023
surrogate model to build this barrier function, where 17" (0,aa) propagates the gradient to ma through MM for improving system safety. (2) The generative model can generate synthetic process (trajectory) Ta to optimize the performance of the policy efficiently by gradient propagation.
We learn the generative model Me,a as a discrete-time SDE to capture the dynamics and stochas- ticity of the environment and serve as a base for the construction of the soft barrier function:
Ma,a 8(t + 1) = (8(t), (o(((t))) (((()))
E.. (s()))
where G. R Rm R is an unknown drift function, Eng R Rn is an unknown n matrix based on 8, and (t) E R4 is the Brownian motion (also known as Wiener Process) with dimension d, encoding the stochasticity. When the environment is deterministic, we can simply set the (s) as O. We design the generative model to share the learning control policy with the real environment, shown in Figure For the inference, the generative model starts from sample s(0) So and rolls by drift function G.. diffusion function Σ., and policy sy- Therefore, the computation graph contains the learning policy; thus, the auto-differential tools can obtain the gradient for the learning policy by propapropgg through the generative model.
Remark We use the fully-comnected neural networks encode such an SDE. Due to the continuity of the neural net, such SDE specification requires the environment dynamics to be continuous and smooth. Therefore our approach cannot handle hybrid dynamics with jump conditions such as the contact dynamics in Mujoco and Safety Gym. Such an assumption is not uncommon, as it remains an open problem to learn the discontinuous dynamics (Parmar et al. 2027 Pfrommer et al. 2021
The generative model training is to reduce the following loss function.
min (T fo.a) = min Σ log P (s(tt (.V(()(), E(((()
(3)
where Lg is the maximum likelihood loss, P (()) ( (.. is the likelihood probabil- ity of the observed st)) under the normal distribution of the SDE representation. We use torchsde (Li al. (220b)) fit the data Ta = {s(0), s(1, ... s{T) to the generative model by updating its pa-
rameter which is shown in Lines 2 to 4 in the Algorithm T
3.3 SOFT BARRIER FUNCTION LEARNING
To encode the hard constraints, we introduce novel generative- baaeeiibbeddd soft barrier function.
Definition (Safety Probability Lower Bound) safety probability lower bound (s), (T)) is defined as
1 of the entire trajectory (process) ((((1) & (a)s(0) E So, YL [0, 2 1-1
Definition 3 (Barrier Function for SDE) Given policy ng, B3 is a generative-model-buur -assed soft barrier function for the discrete-time SDE Ma as in Equation if twice differentiable and satisfies the constraints of the lower problem in Equation
Lemma Prajna et al. 2004 Let B(s(()) be a supermartingale of the process 8(1) and B(s) 2 0, V& E S. Then any 8(0) E So, c > 0, P(supizu B(((t) 2 el 8(0) E So) < B(((0))
Theorem With barrier function as in Definition 3 the generative-model SDE with policy (Equation 2 has safety probability lower bound 1 - nf" where 17" is the optimal value in the lower problem of Equation as Vt E [0, T), P(s(t) & S. E So) 2 - 17" s(t + 1) (o()(((t)).
Proof: With the last condition of the constraints in the lower problem of Equation 1 we have
E ((B(((())))(44) B(S(1)), VT 2 2 2
5

---

Under review as a conference paper at ICLR 2023
which indicates that the barrier function B()) is supermartingale. Then by leveraging the Lemma above from Prajna et al. (2004), we have
P ((((t) Su, for some 1 E [0, 8(0) E So) (BB((tt) 1, for some E [0, ((0)) So))
S P sup ((s(t)) 2 1 (() So (E[O.T]
s B(s()) S nn
Therefore, safety probability lower bound is - 11" and Theorem Tacoormmooolss holdds
We further translate the constraints of the lower problem in Equation | with their sampling mean:
(0)) 1 n, (0) E So. E (B() 2 1, 3 E Su. BB( 2 0, E - (++ S (B),
min 17 s.t.
N
Mo,a( v [o, e S S...
The third non non-negative condition is easy to satisfy by setting the output activation function as Sig- moid for the barrier neural network. The last condition is to make B as supermartingale, which is
the key to deriving the lower bound of safety probability. In practice, we use supervised-learning- based method to optimize this problem by minimizing the following loss function:
N
N
N
M
min LB 0.3
=
N
BB(S (0))
+
N
(1 Ba(
N
M
Σ j=1
(t+ - BB((+(1))
1=1
1=1
+ (1)+1)==A Ma. (8(())))))))))))))) (4), L E (.T),,3(0 ES
(4) where st + 1) the next state of (t) sampled from the generative model Ma.a with policy Te. LB essentially reduces the barrier mapping value on So (the maximum is 17" (0, a)) and projects the unsafe space S, to with Sigmoid output, and decreases the expectation along with trajectory. It is worthy to note that Lg cannot be approximated by the real environment M, as we cannot sample from any intermediate time point s(t) to s(t + 1) in the space 5 \ S. to compute the third sample mean in Equation 4 which is relatively feasible and simple to do with Me. as in Equation 2 The barrier training can be terminated if the second and third sample mean in Equation 4 are non-positive. The soft barrier training is shown as Line 5 in Algorithm
the barrier function
3.4 POLICY OFTIMIZATION
As stated before, we use the generative model to generate synthetic data +/.a {8 (0), (....."88)))))))) (i)) [1, NJ) with policy mo maximize the total expected return (TTa as:
max ( E B(0
(5(1) s.t. 5(t+ ((++)) 1) (eet(((()))) Vt [0,2].
We use the sample mean from the synthetic trajectories as an estimate for the expectation:
N T
max ((To)
Σ
s.t. st + 1) = Me.a Vt E [O, (5)
N 2= 1=0
With policy me in the forward computation graph of Ma,a. we can directly obtain the backwards
gradient for Tg from Equation The policy optimization is shown as Line 6 in the Algorithm
3.5 THEORETICAL ANALYSIS OF SAFETY PROBABILITY BY SOFT BARRIER
For the final learned policy, we conduct a theoretical analysis its safety probability (as defined in Definition 2) derived from the peeerative-mode--beeee soft barrier function in framework.
Lemma 2 (Theorem 21 in (Agarwal et al. 2020a)) Given & (0, 1), a learned deterministic policy (s(s) and assume the environment enviromment-policy transition dynamics as pa (s'lss) E P with the function class (P] 0o (s' represents the next state of s), let the environment and policy generate dataset

---

Under review as a conference paper at ICLR 2023
of n traeectoiiss D == ((s) s' = .n.. s(t) D (s ( :::tt11) 1)). Note that D' is a martingale depending the previous examples. Let the generative model M9,a maximize the likelihood of the dataset by its transition dynamics P via Equation Then with at least probability 1 the expectation of total variation distance between P and P is bounded as:
T Σ
T
2log(1P1/6)
EgnDe ddr(P', p,) t=0
Σ 2=
E Egn
P
(*
(6)
Lemma 3 (proof provided in the Appendix A Given a random variable X,, 0 on probability space S2, if En[Xn] - 0 as n + 0a, then P(Xn = 00) + 1.
Proposition 1 (Asymptotic Lower Bound of Safety Probability) Given the learned policy no. let the generative model fit sample trajectories Ti = -,.... n)) from environment M with me by Equation 3 learn the generaive-model-bassee assume that it formally satisfies the constraints in Equation 7 then the real environment M with policy is safe with at least probability (1 77") when n 0C
soft barrier function B3 by Equation f with vy" and
Proof of Proposition 1 Given (S, B) as the measure spaces with S as the state space and B {B + I. || S 1f. where B is a cewpattiee-mdee-bbaaaaeee soft barrier function with Sigmoid output, then according the definition of total variation distance and Lemma 2 we have
Σ BinDi (drv(P", P)]
T
sup (s1ea)) - -BPG)) llog( BEB
E
E
D'
2
n
When n + 00, set 8 = n, let X, = 2 sup Ept - Ep [B(s'), (B(r))
and therefore E|X,] + O. We know X, 2 O, since X, = 0 when P = P. Therefore, according to Lemma 3
P(X, + 0) +1 1. We then assume (t [o, 7T) can uniformly cover the space S as n - oc, thus
the soft barrier becomes a true barrier function for the real environment and Proposition I hoold.. O
Remark 2 (Practical Safety Probability Lower Bound) In addition to the asymptotic safety prob- ability, we propose a maite- practical safety probability lower bound. We first sample the generative model and the environment with the final learned policy quantify their maximum dis- 1s (t) - (t)), and then enlarge the unsafe region with A by Minkowski sum as S S., ① A. Next, retrain another generative-model-bon soft bar- rier function B with Sy Finally, we conservatively report (1 max B(a)) as the final lower bound of safety probability by the soft barrier function.
tance per state as A = max (O.T)e
Remark (During-learning Safety) The above asymptotic and practical safety bounds are derived for the final learned policy. It is possible that 1 rf" is not valid safety probability bound during learning, as there exist a modeling gap between the generative model and the real environment. However, we optimize 1 1-v" during learning to increase the chance of finding safer learned policies at the end, as demonstrated in our experiments below.
EXPERIMENTAL RESULTS
Experiment Settings and Examples: We compare our approach with two state-of-the-art open- source CMDP-based methods, PPO-L (Ray et al. 2019) and FOCOPS (Zhang al. 2020). For these two basclines, we design the cost function such that the state is safe if its cost is less than O. It is worth noting that PPO-L has stronger safety constraint than FOCOPS the PPO-L with the expectation of cost per state as E[c(s, a)] S 0, rather than the cumulative cost FOCOPS as E"' ((x, a) S D'
implemented
FOCOPS, We conservatively set D' = -60 for the and cartpole examples below, and -200 for the Rocket and UAV examples, to improve its safety. We mark this -riieted version #OCOPS* We mainly compare the converged final policy of each method in system safe rate measured via simulations we call it empirical safe rate. We

---

Under review as a conference paper at ICLR 2023
Table 1: Comparison of our approach with CMDP-based baselines PPO-L and FOCOPS*. se the
safe rate by simulating 500 random initial states from Sp. 1 1-1) is the practical lower bound of safety probability in our approach as (1 - (ef), B(s)), derived by Remark 2 Our approach achieves significantly higher se than the baselines. It is observed that 1 17 is a lower bound of se.
Metric
Methods
2D
Cartpole
Rocket
UAV
Ours
99.9(0.09) %
100%
100%
99.6(0.3)%
Se. empirical safe rate
PPO-L FOCOPS*
98.9(0.08)
89.3(5.5)%
96.4(6.3)%
100%
98.7(0.18))
84.2(4)%
100%
91(4.2)%
1-M n) lower bound
Ours
97.6(1.3)%
86.6(2.9)%
89.9(1.))%
93.2(2.2)%%
PPO-L, FOCOPS*
Ours
-67.(44.9
-24.4(4.7)
-143.2(1.6)
-6.7.166.5)
(T), performance
PPO-L
-66.3(5.3)
-3..1(6.7)
-895.5(4.3) -249.1(1.4)
-151,4(3.6)
FOCOPS
-69.8(3.2)
-15.2(2.3)
-734.1(3.3)
also perform safety probability analysis for our method (CMDP cannot provide one), and compare different methods in total reward return.
Note that learning safe control policy for high-dimensional systems is quite challenging. Current state-of-the-aat works of certificate-based policy learning mainly focus on low-dimensional systems with fewer than 6 dimensional states (Luo Ma [021 Lindemann et al. 202T](Chang et al. 2019 Berkenkamp et al. 2017p. In this paper, we test approach on 13D UAV and Rocket examples.
2- 2-Dimensional SDE (Prajna et al. 2004) has the unknown dynamics M as si = 0.882, dag = (a Wiener process.) Initial space So = {({ + 2) + 82 S 0.01]. and unsafe space S. {s} E [-1,0], $2 [1.2,1.7]3 The goal is stabilize the system near (0, 0).
ddt + 0. 0.2dW(t) (t) (W(1)
Cartpole Balancing Brockman et al.) 2016) has a 4-dimensional vector s [ax, 0, 6) as the sys- tem state, where x is the position and 0 is the angular enror the upright. The initial space So {xx, 0, Φ; f] space S (1x O, it 0) x < 0.753. The goal is to keep the cartpole balanced upright.
0-..667, 0.033), 0 [-0.6, 0.5), -0.35, A 0.53), and unsafe
Powered Rocket Landing (Jin et al. 2027) has 6 DoF (degrees of freedom) with 13 system states and 3 action variables. The goal to land the rocket close the original point while avoiding an unsafe region. Its state vector is s = [p qw] E IR 13 where = (z, 2) E R and v = (Uz, By. v2)= E 3 represent the position and velocity of the rocket, respectively. q E R the unit quaternion for attitude and w E R3 is the angular velocity with respect the inertial frame. There are three trust forces for the rocket as the control input = [Tg, Ty. 22] E R². The initial space So p = (x. y. 2x 10)² + (y + 8)2 + (( 5)2 < 0.01, v = =0.q (0.73, 0, O, 0.68), ~ = 0, and unsafe space S., p - (xx, y, (x 5)2 + y2 S 1, < - 5, |,|l,|,, S 10, ||l| < 10.
UAV Maneuvering (Jin et af.) 2027) is to maneuver an UAV close to the original point while avoid- ing an obstacle. The 6-DoF UAV has 13 system states and 4 action variables. Its state vector is s dp q (w e IR 13 same with above Rocket example. The control input u = [T, T3, Ta
E R2 in- + ((- 0.01, = 0, q 0,, 0), = O,S. p = (z,, y, z) + 4.5) 1, 5.
cludes the four rotating propellers of the quadrotor. So p (z. (z.y, y. (2)8)
Comparison and Effectiveness of Our Approach: Table shows the comparison results in simulation-based system safe rate (based on 500 simulations for each example, with random ini- tial states), safety probability, and performance. We can see that by directly enforcing hard safety constraints via soft barrier functions, our approach can achieve significantly higher system safe rate than the CMDP-based baselines. Our approach also provides a practical lower bound of safety probability, which the CMDP- methods cannot provide. CMDP achieves better perfor- mance (total reward return) in some cases, but we view safety as the first priority for these systems and the focus of this work.
Figure shows the control trajectories by the learned policies from our approach and the baselines. The agent is always safe with our learned policy, while there exist unsafe cases by both PPO-L and FOCOPS. Moreover, our generative model behaves very similarly to the real environment, which shows the usefulness of the generative modeling for constructing the soft barrier function and opti-
8

---

Under review as a conference paper at ICLR 2023
2D Sys
Cartpele
Racket Powered Landing
Ovm UAV Manesvering
1984 TOCOYS
RCL
Dut Gos HOL ictrs
0.2
-05
0-
-10
-15
-1.0
(a) 2D system
(b) Cartpole balancing
(c) Rocket landing
(d) UAV maneuvering
Figure 3: Control trajectories by the learned policies from our approaches and baselines. "Gene" in- dicates the synthetic trajectory from the final learned generative model, which behaves very similarly the real environment with "Ours" policy, showing its effectiveness for barrier function construc- tion. We can see that our approach learns safer policies than the baselines.
Barrier Loss
Barrier Value A4&M
Barrier Loss A
Barrier Value M&&
863t Ai
MiR) M
5:00
Lie
Lie
222 Blxul maxelxxl
L. Bix()
max
5:00
0:00
Iteration
1/5
Iteration
U/S
Figure 4: Barrier function training and testing in the 2D SDE example.
Figure 5: Barrier function training and testing in the UAV example.
mizing the control policy. We also show the learning process of the soft barrier function the gen- erative model and its testing in the real environment in Figures 4 and 5 for the 2D system and UAV examples are in the Appendix Aj. The learned barrier function
examples (results of the other maps the initial space to near 0 and the unsafe space to 1 with the third sample mean in Equation 4 to (marked as Lie in Figures). The barrier function has similar value along with the trajectories in the real environment and the generative model. Again, this indicates that the generative model behaves very similarly the environment, as shown in Figure 3 Although the barrier function decreases or stays constant most of the time, it can increase at some point. This is due to that 1) the possible modeling error between the generative model SDE and environment 2) the supervised learning cannot cover all possible cases for barrier training.
I Limitations: As stated earlier, one key assumption of this work is the smoothness and continuity of the system behavior, which prevents its application to hybrid dynamics with jump conditions such as the contact dynamics in Mojuco and Safety Gym. One possible solution is to learn an ensemble generative model hybrid system to deal with those discontinuous contact dynamics, and plan to explore it in future work. Another limitation of our framework is on the computation complexity of the generative model (e.g., it takes around hours learn policy for the Cartpole example and I day for the UAV and Rocket examples). In future work, we plan to improve the efficiency of this part by exploring techniques such as Continuous Latent Process Flows (CLPF) Deng al. (202T)
CONCLUSION
We present safe RI. approach for unknown stochastic environment that enforces hard reachability- based safety constraints through senerative-moodl--asee soft barrier functions. Our approach for- mulates novel bi-level optimization formulation, and develops a safe RL algorithm that jointly leams the generative model, soft barrier function, and policy optimization. Experiments demon- strate that our approach can significantly improve empirical system safe rate over CMDP-based baselines and also provide a practical lower bound of safety probability.

---

Under review as a conference paper at ICLR 2023
REFERENCES
M Mehdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender systems: A survey. ACM Computing Surveys (CSUR), 2021.
Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural com- plexity and representation learning of low rank mdps. Advances in neural information processing systems, 11-20095-201000 2020a.
Alekh Agarwal, Sham Kakade, and Lin F Yang. Model-based reinforcement learning with a gener- ative model minimax optimal. In Conference on Learning Theory, pp. 67-83, PMLR, 2020b.
Eitan Altman. Constrained Markov decision processes: stochastic modeling. Routledge, 1999.
Aaron Ames, Xiangru Xu, Jessy Grizzle, and Paulo Tabuada. Control barrier function based quadratic programs for safety critical systems. IEEE Transactions on Automatic Control, 62(8): 3861-3876, 2016.
Yarden As, Ilnura Usmanova, Sebastian Curi, and Andreas Krause. Constrained policy optimization via bayesian world models. In International Conference on Learning Representations, 2021.
Osbert Bastani, Shuo Li, and Anton Xu. Safe reinforcement learning via statistical model predictive shielding. In Robotics: Science and Systems, 2021.
Felix Berkenkamp, Matteo Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforeement learning with stability guarantees. Advances in neural information processing sys- tems, 30, 2017.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, arXiv preprint 1606.0/444 2016.
Ya-Chien Chang, Nima Roohi, and Sicun Gao. Neural Iуарunov control. Advances in neural infor- mation processing systems, 32, 2019.
Richard Cheng, Gáhor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforce- ment learning through barrier functions for safety-critical continuous control tasks. In Proceed- ings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3387-3395, 2019.
Jason Choi, Fernando Castaneda, Claire Tomlin, and Koushil Sreenath. Reinforcement learning for safety-critical control under model uncertainty, using control Iyapunov functions and control barrier functions. arXiv preprint 20XX::0004407584 2020.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained re- inforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18 (1):6070-6120, 2017.
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A Iyapunov- based approach to safe reinforcement learning. Advances in neural information processing sys- tems, 31, 2018.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint 1XXv::8011008777 2018.
Ruizhi Deng, Marcus A Brubaker, Greg Mori, and Andreas Lehrmann. Continuous latent process flows. Advances in Neural Information Processing Systems, 34:5162-5173, 2021.
Dongsheng Ding, Xiaohan Wei, Zhuoran Yang, Zhaoran Wang, and Mihailo Jovanovic. Provably efficient safe exploration via pdia policy optimization. International Conference on Artificial Intelligence and Statistics, pp. 2304-33122 PMLR, 2021.
Yousef Emam, Paul Glotfelter, Zsolt Kira, and Magnus Egerstedt. Safe model-based reinforcement learning using robust control barrier functions. arXiv preprint arXiv:21 2021.
10

---

Under review as a conference paper at ICLR 2023
Aria HasanzadeZonuzz, Archana Bura, Dileep Kalathil, and Srinivas Shakkottai. Learning with safety constraints: Sample complexity of reinforcement learning for constrained mdps. In Pro- ceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 7667-7674, 2021.
Wanxin Jin, Shaoshuai Mou, and George Pappas. Safe pontryagin differentiable programming. Advances in Neural Information Processing Systems, 34: 14.16034-16050 2021.
Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Breaking the sample size barrier in model-based reinforcement learning with generative model. Advances in neural information processing systems, 33: 12861-12872 2020a.
Shuo Li and Osbert Bastani. Robust model predictive shielding for safe reinforcement learning with stochastic dynamics. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 7166 1066-1172 7172. IEEE, 2020.
Xuechen Li, Ting- Kam Leonard Wong, Ricky T. Q. Chen, and David Duvenaud. Scalable gradi- ents for stochastic differential equations. International Conference Artificial Intelligence and Statistics, 2020b.
Lars Lindemann, Haimin Hu, Alexander Robey, Hanwen Zhang, Dimos Dimarogonas, Stephen Tu, and Nikolai Matni. Learning hybrid control barrier functions from data. In Conference on Robot Learning, pp. 1351-1370 PMLR, 2021.
Yuping Luo and Tengyu Ma. Learning barrier certificates: Towards safe reinforcement learning with zero training-tims violations. Advances in Neural Information Processing Systems, 34:25621- 25632, 2021.
Haitong Ma, Jianyu Chen, Shengbo Eben, Ziyu Lin, Yang Guan, Yangang Ren, and Sifa Zheng. Model-based constrained reinforcement learning using generalized control barrier function. In 2021 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 4552- 4559. IEEE, 2021.
Shin-ichi Maeda, Hayato Watahiki, Yi Ouyang, Shintarou Okada, Masanori Koyama, and Prabhat Nagarajan. Reconnaissance for reinforcement learning with safety constraints. In Joint Euro- pean Conference Machine Learning and Knowledge Discovery in Databases, pp. 567-582. Springer, 2021.
Teodor Mihai Moldovan and Pieter Abbeel. Safe exploration in markov decision processes. arXiv preprint / 420.4881, 2012.
Mihir Parmar, Mathew Halm, and Michael Posa. Fundamental challenges in deep learning for stiff contact dynamics. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5181-5188. IEEE, 2021.
Samuel Pfrommer, Mathew Halm, and Michael Posa. Contactnets: Learning discontinuous contact dynamics with smooth, implicit representations. In Conference on Robot Learning, pp. 2279- 2291. PMLR, 2021.
Stephen Prajna and Ali Jadbabaie. Safety verification of hybrid systems using barrier certificates. In International Workshop on Hybrid Systems: Computation and Control, pp. 477-47.9 Springer, 2004.
Stephen Prajna, Ali Jadbabaie, and George Pappas. Stochastic safety verification using bar- rier certificates. In 2004 43rd IEEE conference on decision and control (CDC)IEEE Cat. No. 04CH37601), volume 1, pp. 929-934, IEFE, 2004.
Zengyi Qin. Kaiqing Zhang, Yuxiao Chen, Jingkai Chen, and Chuchu Fan. Learning safe multi- agent control with decentralized neural barrier certificates. arXiv preprint 1oXiv:2101.054433 2021.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv: 19Xiv199000070000 7:1, 2019.

---

Under review as a conference paper at ICLR 2023
Yifei Simon Shao, Chao Chen, Shreyas Kousik, and Ram Vasudevan. Reachabilityy based trajectory safeguard (rts): safe and fast reinforcement learning safety layer for continuous control. IEEE Robotics and Automation Letters, 6(2):3663-3670, 2021.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):1140- 1144, 2018.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp. 9133-9143. PMLR, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Andrew Taylor, Andrew Singlctary, Yisong Yue, and Aaron Ames. Learning for safety-critical control with control barrier functions. In Learming for Dynamics and Control, pp. 708-717. PMLR, 2020.
Andrea Tirinzoni, Riccardo Poiani, and Marcello Restelli. Sequential transfer in reinforcement learning with generative model. In International Conference on Machine Learning, pp. 9481- 9492. PMLR, 2020.
Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite markov deci- sion processes with gaussian processes. Advances in Neural Information Processing Systems, 29, 2016.
Akifumi Wachi, Yanan Sui, Yisong Yue, and Masahiro Ono. Safe exploration and optimization of constrained mdps using gaussian processes. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Nolan C Wagener, Byron Boots, and Ching-An Cheng. Safe reinforcement learning using advantage-baue based intervention. In International Conference on Machine Learning, pp. 10630- 10640. PMLR, 2021.
Yixuan Wang, Chao Huang, Zhaoran Wang, Zhuoran Yang, and Qi Zhu. Joint differentiable opti- mization and verification for certified reinforcement learning, arXiv preprint arXiv:2201.12244 2022.
Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning, pp. 11480-11491. PMLR, 2021.
Zhengfeng Yang. Chao Huang, Xin Chen, Wang Lin, and Zhiming Liu. A linear programming relaxation based approach for generating barrier certificates of hybrid systems. In International Symposium on Formal Methods, pp. 721 721-738. Springer, 2016.
Dongjie Yu, Haitong Ma, Shengbo Li, and Jianyu Chen. Reachability constrained reinforcement learning. International Conference on Machine Learning, pp. 25636-2665222 25655.. PMLR, 2022.
Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 33:15338-15349, 2020.
Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep rein- forcement learning for robotics: a survey. In 2020 IEFE Symposium Series on Computational Intelligence (SSCI), pp. 737-744. IEEE, 2020.
12

---

Under review as a conference paper at ICLR 2023
Barrier AM
Barrier Value MM&M
Barrier MM
Barrier Value .M&..
B(5)
12
min BOLJ
aG8 on
mV BOL)
660 M
maKB(4)
180
04
320
100
Figure 6: Barrier function training and testing in Rocket powered landing.
Figure 7: Barrier function training and testing in Cartpole balancing.
A APPENDIX
A.1 PROOF OF LEMMA IN THE MAIN TEXT
For any E N, let Em (w EQ: Xn(w) > J. Since X, 2 0, we have:
[n|Xa] = XndP> /
XndP> 1 P(Em))
F.n
Therefore, P(Em) + 0, and then:
0 P(w Q: Xn (w) + = PP lim P(Em) + 0,
P(w Ω: (w) + 0) +0 = P(w E Ω (w) 0) - 1.
A.2 ADDITIONAL EXPERIMENTAL RESULTS
The barrier function training and testing results for the Rocket powered landing and the Cartpole balancing examples are shown here in Figures and
13


Reviews:
[{"review_id": "wKBkMe5lAS0", "reviewer": "Reviewer_MNvg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The focus of this work is on learning safe MDP policies, in the sense that a safety constraint holds across the entire trajectory of the policy. The distinction from the widely-studied CMDPs is that the constraint in the CMDP holds over a cost aggregated across the entire trajectory, and not at each point in the trajectory. The approach in this work is to learn a barrier function that is a supermartingale over the filtration defined by transitions, and thus with high probability is not violated during the execution. There is a (weak) theorem showing that in the asymptotic limit, the policy achieves safety. More compellingly, experiments on a few control benchmarks suggest that the method indeed avoids unsafe regions.", "summary_of_the_review": "The paper's presentation is rough, but the approach has some significant technical novelty and seems to be effective empirically.", "strength_and_weaknesses": "The main strengths of the paper are (1) the approach to learning the barrier function seems to be fairly novel in this setting and (2) the experiments do seem to be pretty promising. In particular, on the technical side, I did appreciate the use of the (locally-defined) supermartingale constraint on the barrier function to obtain the safety bound.\n\nThe main weakness is that the paper is a bit of a mess -- I'll discuss this below.\n\nAn asymptotic theoretical guarantee is included, which is better than nothing, but certainly much weaker than we'd want to have if the objective was to ensure safety. Remark 1 proposes a \"practical safety probability lower bound\" that is used to obtain bounds reported in the experiments, that are indeed less than the empirical fraction of safe trajectories (1), but it isn't immediately obvious that the bound is valid.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SndXg4a_p0", "reviewer": "Reviewer_YtE9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed ways to do safe RL in an unknown stochastic environment. Specifically, they model the problem in hard constraints instead of the standard CMDP constraints and derived a novel barrier function method to incorporate the hard constraints into the RL algorithm. The authors manage to show with theoretical results that such an approach asymptotically achieves safety in policy optimization in RL, and they demonstrate the effectiveness of the proposed method on several safe RL benchmark problems for which the proposed method outperform SOTA in terms of safety guarantees and return performance.\n", "summary_of_the_review": "This paper studied an important safe RL problem and proposed a novel barrier-function based method. It also showed this method's effectiveness both theoretically (in asymptotic sense) as well as empirically on several safety RL benchmarks. However, it'd be great to verify the benefits of this method by 1) testing in more complex domains and 2) comparing with more model-based safe RL methods (e.g., Turchetta, M. et al Safe exploration in finite Markov decision processes with gaussian processes. NeurIPS 16). The writing of this paper also needs significant improvements for it to be readable/understandable by the broader ML audience. The current paper is quite hard to follow.", "strength_and_weaknesses": "Strengths:\nThe barrier-based method for guaranteeing safety  is novel and is related to the work of guaranteeing safety in the form of stability in model-based RL algorithms. \nThe problem of enforcing RL policy to be safe w.r.t. to hard constraints is interesting and important (in areas e.g., robotics).\nThe algorithms proposed is justified with theoretical studies (which utilize martingale properties of the barrier function to obtain a safety results in the asymptotic sense) and its benefits are also illustrated with some experiments over standard CMDP methods. \n\nWeaknesses:\nIt seems the safety bound only holds in asymptotic, and it is unclear whether this method ensures the property of safety during training. \nThe experiments seem to be overly simple. Running this algorithm on more realistic benchmarks (e.g., Safety gym for safe RL experimentations) and doing more comparisons with other model-based safe RL methods will be better in terms of demonstrating the effectiveness of this method.\nIt is unclear whether the safety bounds provided in the theory can be justified by the experimental results \nThe quality on the writing of the proofs is currently quite low, it requires more work to explain the notations and intuitions clearly in order to make it readable.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "B80fd_6byDW", "reviewer": "Reviewer_kxrF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of safe reinforcement learning with hard constraints in an unknown stochastic environment. The key idea (and the contribution) is the proposed generative model based soft barrier functions that relax the hard safety constraints adapting to the unknown environment. This is further incorporated into the presented safe RL framework as a bi-level optimization process for jointly learning the optimal control policy, the soft barrier functions, and the initially unknown environment model, with avoidance of unsafe region in a probabilistic setting. Experimental results with two baseline approaches such as PPO-L and FOCOPS are presented to show the effectiveness of the proposed algorithm. ", "summary_of_the_review": "The paper presents an interesting topic of safe RL in an unknown and stochastic environment. While the general idea is clear, the reviewer feels that the technical part should be substantially improved given the insufficient details regarding safety guarantee, which is the core part of the presented approach. The experimental results also seems substandard without extensive trials under different environment settings. Authors are encouraged to address the comments in the future version of the paper.", "strength_and_weaknesses": "Strength:\n+ The general idea of this paper is well written with clear motivation and good overview about the high level idea of the algorithm.\n+ The bi-level optimization framework that jointly learns the policy and soft barrier functions is interesting.\n \nWeakness:\n- While the reviewer found the comparative analysis with CMDP algorithm is sound and solid, it is unclear how the algorithm could compare with other model-based safe RL algorithms, especially methods that combine RL and control theoretic approaches such as control barrier functions.\n- The related work discussion needs to be improved, as some claims seem inaccurate given existing approaches that do not require explicit known model nor an initial safe policy, e.g. [Berkenkamp et al 2017].\n- The proof of safety (Lemma 1) is substandard in absence of supplementary materials. For example, it is unclear how the conclusion of bounded probability for the ENTIRE trajectory could be reached with the presented proof, especially given the stochastic environment.\n- The experimental results are reported with few trials and no quantitative performance in terms of mean and variances are given to justify the robustness of the performance.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "--EnFN7k3E", "reviewer": "Reviewer_H8ou", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is on safe reinforcement learning (RL) with unknown stochastic environments. Under the safety definition in terms of reachability, the authors aim to enforce hard constraints rather than soft constraints that are often the main target of constrained Markov decision processes (CMDP) literature. In this paper, the authors introduce a notion of generative model based soft barrier functions to encode the hard safety constraints. The authors propose a safe RL algorithm to jointly model the environment and optimize the agent policy while satisfying the safety (reachability) constraints. Finally, this paper demonstrate the effectiveness of the proposed method in several benchmark problems.", "summary_of_the_review": "I think this paper has several merits in terms of 1) important problem settings, 2) new algorithm, 3) good experimental results (easy benchmark and limited baselines), and 4) good writing.\n\nHowever, I have several concerns in terms of 1) how to deal with existing safe model-based RL literature and 2) empirical experiments, as stated in Weakness.\nTherefore, I recommend rejection.", "strength_and_weaknesses": "### Strength\n- Important and relatively-unexplored problem settings on safe RL with **hard constraints**.\n- New algorithms based on shielding and generative models\n- Good experimental results compared to CMDP methods and FOCOPS*.\n- Overall, this paper is well-organized and easy to follow.\n\n### Weakness\n- Safe model-based RL\n    - There are significantly few references to safe model-based RL. I know there is the following sentence, but still I consider the relevant methods should be mentioned in related work section and compared to the authors' proposed method.\n    - > \"However, these methods do not apply to our setting, where the environment is uncertain and stochastic without explicit dynamics, backup safe policy, or human intervention.\"\n    - Hard constraints in safe RL has been studied in unknown (stochastic) environment. For example, Moldovan and Abbeel (2012) studied safe exploration in unknown environment in terms of ergodicity. Also, Turchetta et al. (2016) and Wachi et al. (2018) studied safe exploration problems in unknown environment while guaranteeing reachability and returnability under regularity assumptions. I think the authors should have surveyed wider safe RL methods (the three papers I mentioned are all hard safety constraint; that is, the agent is required to satisfy the safety constraint at every time step).\n    - Though the above papers are indeed different from the authors' one in terms of problem formulation or assumptions, Yarden et al. (2021) can be mentioned in related work section and compared to the authors' method in the experiments. If I understand correctly, the method in Yarden et al. (2021) can be fairly compared to the authors' one, and source code is also open-sourced (https://github.com/yardenas/la-mbda),\n\n- Experiments\n    - Benchmark problems are very easy. Hopefully, Safety-Gym should be used for testing safe **deep** RL methods.\n    - Computational time and sample efficiency should be also compared. I \"guess\" the authors' proposed method requires much more data until convergence.\n\n### Minor comments and typos\n- [Section 3.1. Equation regarding $J$] I think $t$ should be $i$ (or $i$ should be $t$).\n- [Definition 2] $\\tau(t)$ is not defined. Is $\\tau(t) := [ s(i) ]_{i=t}^T$?\n\n### References\n- Moldovan, T., and Abbeel, P. 2012. Safe exploration in Markov decision processes. In International Conference on Machine Learning (ICML).\n- Turchetta, M.; Berkenkamp, F.; and Krause, A. 2016. Safe\nexploration in finite Markov decision processes with gaussian processes. In Advances In Neural Information Processing Systems, 4305–4313.\n- Wachi, Akifumi, et al. \"Safe exploration and optimization of constrained mdps using gaussian processes.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.\n- As, Yarden, et al. \"Constrained Policy Optimization via Bayesian World Models.\" International Conference on Learning Representations. 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject