Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm
Authors: Ashish Kumar Jayant
Shalabh Bhatnagar
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Safe Reinforcement Learning
Model-based Safe Reinforcement Learning

Abstract: During initial iterations of training in most Reinforcement Learning (RL) algorithms, agents perform a significant number of random exploratory steps. In the real world, this can limit the practicality of these algorithms as it can lead to potentially dangerous behavior. Hence safe exploration is a critical issue in applying RL algorithms in the real world. This problem has been recently well studied under the Constrained Markov Decision Process (CMDP) Framework, where in addition to single-stage rewards, an agent receives single-stage costs or penalties as well depending on the state transitions. The prescribed  cost functions are responsible for mapping undesirable behavior at any given time-step to a scalar value. The goal then is to find a feasible policy that maximizes reward returns while constraining the cost returns to be below a prescribed threshold during training as well as deployment.

We propose an On-policy Model-based Safe Deep RL algorithm in which we learn the transition dynamics of the environment in an online manner as well as find a feasible optimal policy using the Lagrangian Relaxation-based Proximal Policy Optimization. We use an ensemble of neural networks with different initializations to tackle epistemic and aleatoric uncertainty issues faced during environment model learning.  We compare our approach with relevant model-free and model-based approaches in Constrained RL using the  challenging Safe Reinforcement Learning benchmark - the Open AI Safety Gym.  
We demonstrate that our algorithm is more sample efficient and results in lower  cumulative hazard violations as compared to constrained model-free approaches. Further, our approach shows better reward performance than other constrained model-based approaches in the literature. 

Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm
Ashish Kumar Jayant
Shalabh Bhatnagar
Department of Computer Science Automation Indian Institute of Science Bangalore aahishjayantdiisc.a...
Department of Computer Science Automation
Indian Institute of Science Bangalore shalabh@iisc..a. in
Abstract
During initial iterations of training in most Reinforcement Learning (RL) algo- rithms, agents perform significant number of random exploratory steps. the real world, this can limit the practicality of these algorithms as it can lead to poten- tially dangerous behavior. Hence safe exploration is critical issue in applying RL algorithms in the real world. This problem has been recently well studied under the Constrained Markov Decision Process (CMDP) Framework, where in addition single-stage rewards, agent receives single-stage depending on the state transitions. The prescribed cost functions are responsible for mapping undesirable behavior at any given time-step to scalar value. The goal then is to find a feasible policy that maximizes reward returns while constrain- ing the cost returns to be below prescribed threshold during training well deployment.
penalties well
We propose an On-policy Model-based Safe Deep RL algorithm in which we learn the transition dynamics of the environment in an online manner as well as find a feasible optimal policy using the Lagrangian Relaxation- Optimization. We use an ensemble of neural networks with different initializations to tackle epistemic and aleatoric uncertainty issues faced during environment model learning. We compare our approach with relevant model-free and model-based approaches in Constrained RL using the challenging Safe Reinforcement Learning benchmark the Open AI Safety Gym. We demonstrate that our algorithm more sample efficient and results in lower cumulative hazard violations compared constrained model-free approaches. Further, our approach shows better reward performance than other constrained model-based approaches in the literature.
Proximal Policy
Introduction
Deep Reinforcement Learning has provided exceptional results both in the case of discrete action settings Mnih et al. 2013 as well as continuous action domains such as locomotion tasks [Hаrmoja
al. 2018], Schulman al. [017]. [Schulman al.) 2016). However, most of the RL algorithms perform significant number of random exploratory steps during learning as well as deployment which
can lead agents performing undesirable and hazardous behaviour. This limits the application of
RI. algorithms in the real world. In scenarios like robot navigation [Han et al.) 2018), autonomous driving [Kendall al. 2018), healthcare (uu et al.) 2020), etc., where RL has potential applications, unsafe behavior can have hazardous consequences even on human life and property.
In García et al. 2015 the authors have provided comprehensive survey of several notions of safety and the associated problem formulations. In our work, we focus on a constraint-based notion of safety. In Constraint-baut RL, the goal maximize long- termtterm expected reward returns and keep the expected cost-returns below a prescribed threshold. This problem is known as the Safe
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Exploration problem. In [Ray et al. 2019]. the authors advocate that safety specifications should be separate from task performance specifications. It also helps in formulating Safe Exploration as a Constrained Optimization Problem and methods used in the optimization literature [Bertsekas) 1996
can be used to solve this problem. Constrained Markov Decision Process (CMDP) [Altman 1998 provides a framework to keep task performance specifications and safety specifications separate from one another. Here in addition to single-stage rewards, state transitions receive single-stage costs as well. The prescribed cost functions are responsible for mapping undesirable behavior at any given time-step to a non-negative scalar value.
The existing model-free algorithms used in the Constrained RL setting suffer from low sample efficiency in terms of enviromment interactions, i.e., these algorithms require large number of envi- ronment interactions (to converge) that in turn would lead to large number of hazardous actions due to unsafe exploration. This serves as the motivation for us to use a Model-based approach for Constrained-RI.
We propose a simple and sample efficient model-based approach for Safe Reinforcement Learning which uses Lagrangian relaxation to solve the constrained RL problem. We highlight the issue that arises due to the use of truncated horizon in Constrained RL and suggest a way to incorporate that in our setting. We demonstrate that our approach is ~ 3 - 4 times more sample efficient than its analogue Model-free Lagrangian relaxation approach and reduces its cumulative hazard violations by ~ 60%. Our approach also outperforms model-free Constrained Policy Optimization (CPO) Achiam al. 2017 terms of constraint satisfaction. We also compare our approach with model-based approach [Sikchi et al. 2021a| and we observe that our approach does better terms of reward performance and has competitive cost performance as well.
Our Contributions:
2 Related Work
Different notions of safety and their mathematical formulations are provided in Gareía al. 20155] There are several works on the lines of formulating the Safe RL problem in the setting of CMDP. In some of the early works, actor-critic algorithm for CMDP under the long-run average cost criterion is proposed in BBorkar (2005) that is however for the case of full state representations. Actor-critic algorithms with linear function approximation have been proposed in Bhatnagar and Lakshmanan
[2012). Bhatnagar et al. [2013) for the long-run average cost setting and in Bhatnagar] [2010] for the infinite horizon discounted cost scenario. The procedure in the aforementioned references involved forming a Lagrangian by relaxing the constraints. The algorithms in these papers are based on stochastic approximation with updates of the Lagrange parameter performed on the slow timescale, the policy updates on the medium timescale and the updates of the value function (for a given policy) performed on the fast timescale.
In more recent work, in Achiam et al. (2017). a trust region based constrained policy optimization (CPO) framework proposed, which involved approximation of the problem using surrogate functions for both the objective and the constraints and included a projection step on policy parameters that needed backtracking line search, making complicated and time-consuming. CPO showed near constraint satisfaction in every iteration of the policy updates in standard Mujoco environments modified for safe exploration [Achiam et al. 2017] but didn't yield a constraint satisfying policy in challenging Safe RL benchmark Safety Gym |Ray et al.) [2019]. Another work (Yu et al., 2019] involved using surrogate functions for approximation of both objective and constraint functions. Their procedure involved constructing a sequence of convex optimization problems for which they showed that the sequence of stationary points converges to the stationary point of the original non-convex problem.
In Ray et al. [019 Lagrangian relaxation of the Constrained RL problem is used and combined with PPO (Schulman et al. 2017] to give PPO-Lagrangian algorithm and with TRPO (Schulman al. 2015 to give a TRPO- Lagrannagann algorithm. These algorithms were seen to outperform CPO Achiam et al. 2017 in terms of constraint satisfaction on several environments in Safety Gym. Also, these algorithms are simpler to implement and tune. Another Lagrangian- based method, see Tessler et al. 2018). used a penalized reward function for optimizing their agent and showed convergence to optimal feasible policies using two-timescale stochastic approximation scheme where the Lagrange multiplier is updated on slower timescale as compared to the policy parameters as was the case with Borkar (2005), [Bhatnagar [2000). Bhatnagar and I Lakshmanan [012]. In Zhang et al. (2020). the authors proposed a first order constrained policy optimization (FOCOPS) method that involved
2

---

solving the optimization problem in a non parametric space and then projecting it back into the parametric space. Approaches in Tessler et al. [2018] and Zhang et al. (2020) performed poorly on Safety Gym. In [tooke et al. [020]] PID-based approach to damp oscillations in I Lagrangian methods is proposed, which is seen to minimize constraint violations. Wen and Topey (202T] J the
Cross Entropy method (de Boer et al.) 2005] is used for finding a safe policy and convergence using the ODE method is shown, however, empirical results are presented only on a primitive and less challenging environment, In Dalal et al. (2018), the authors formulated a state-wise constrained policy optimization problem where at each transition a constraint needs to be satisfied and an analytical method for correcting the unsafe action using safety layer trained using random exploration was proposed. In Chow et al. 2018]. the authors proposed constructing Lyapunov functions to guarantee safety of the behaviour policy under CMDP framework. In Chow et al. [2019]. the above Lyapunov based method was extended to continuous control but it's performance in Safery Gym environment |Ray et al. 2019] was not good in terms of rewards obtained (Sikchi al. 2021b). In Liu al. 2022], the method involved uses the tangent space of the constraint manifold to learn a safe policy but the approach is seen to be highly specific to the environment used.
For unconstrained state-of-the-ar Model-based RL algorithms in Deisenroth and Rasmussen [0011]
Luo et al. 2018 Chha et al. 2018)) Kurutach et al. (2018). Heess et al. [2015] a comparison of
the empirical performance on Open Al Gym [Brockman et al. 2016] is shown in Wang et al. (219
Sikchi Also.(Sikchi et al. [2021a] propose augmenting the planning trajectory with terminal value function to
incorporate long-horizon horizon reasoning in model-based methods, since the approaches in Deisenroth and
Rasmussen [2011) Luo et al. (2018). [Chua et al.] (2018), KKrutach et al. [2018], HHess et al.) [2015]
plan over a fixed and short horizon to avoid aggregation of error. This requires good approximation of both model well value function.
There are also several works that use model-based RL to tackle the problem of safety. In Liu et al. [2020), model based approach is proposed to learn the system dynamics and cost model. Then roll-outs from the learned model are used to optimize the policy using a modified cross-entropy based method which involves sampling from distribution of policies, sorting sample policies based on constraint functions and using them to update the policy distribution. However, their implementation involves data collection step using random policy for large number of episodes which itself is risky in real-world scenarios. In Cowen-Rivers et al. [2020), model dynamics is learned using PILCO [Disenroth and Rasmussen 2011] and instead of the discounted cost constraint function, conditional value at risk (CV) based constraint function is used [Uryasev and Rockafellar 2001 Chow and Ghavamzadeh, [014]. In (Thomas al. [2022] penalized reward functions are used instead of separate cost function, then model of the environment is learned and the soft-actor critic algorithm
[Haarnoja et al. 2018 is used to optimize the policy. In this approach safe and unsafe states are also
needed to be specified upfront.
3 Background
3.1 Constrained Markov Decision Process (CMDP)
CMDP denoted by the tuple (S, A, R, C, where S denotes the state space, A is the action space, R S x A x S - IR. is the single-stage reward function, C S x A x S - IR denotes the associated single-stage cost function (we assume a single constraint function for simplicity here), ? is the discount factor and μ signifies the initial state distribution. We assume that both R and C' are non-negative functions.
By policy = [To. TT. TI.... .}, we mean decision rule for selecting actions. It is specified as follows:
For any k 2 0 and E S, (s) E PP(s) is the probability distribution F (s) (ITA (s, a), E A(s)) where ( a) is the probability of picking action in state instant i under policy x. In the above, A(s) is the set of feasible actions in state s and so A = U(s).As)). Such a policy is also often referred to as a randomized policy. A stationary policy a randomized policy as above except with Ti = T/. Vk + l. Thus, a stationary policy selects actions according to given distribution regardless of the instant when an action is chosen according to the given policy. By an abuse of notation, we denote stationary policy itself.
3

---

We shall consider here class of stationary policies parameterized by a parameter 0. Our objective function is defined via the infinite horizon discounted reward criterion where for given xq we have
( =
Σ 7 R(st, des St+1) so a, To, VI
The (cost) constraint function is similarly specified via the following infinite horizon discounted cost:
2
(Te) E
Σ AC(ss, 8(+1) 80 nl, ar me. 1i
(2)
Then (To)) JC (m²) E R. Let > O denote a prescribed threshold below which we want J (T(o)
to lie. The constrained optimization problem then is the following:
max J (TO) s.t. (((T))) S d.
(3)
parameter will be called a feasible point if the cost constraint is satisfied for i... J (TT) S
3.2 Lagrangian Relaxation based Proximal Policy Optimization
The Lagrangian of the constrained optimization problem ( can be written as follows:
L(O, (TT)) X(J (me) d),
(4)
where À E R+ is the Lagrange multiplier and is a positive real number. In terms of the Lagrangian, the goal is to find tuple (0* X*) of the policy and Lagrange parameter such that
L(O", X*) max min A).,
(5)
Solving the max-min problem as above is equivalent to finding global optimal saddle point (0", A*) such that v0, A), the following holds:
L(O", A) L( (-)) ((0,)).
(6)
We assume that / refers to the parameter a Deep Neural Network, hence finding such a globally optimal saddle point is computationally hard. So our aim is find a locally optimal saddle point which satisfies (6) in local neighbourhood Heg., which is defined as follows:
Her A ((0,A) A) 110 0*1l S ||>
X1I S (2)
for some 52 > O. Assuming that L(O, A) is known for every (0 tuple, a gradient search procedure for finding local (0", )*) tuple would be the following:
on+1
On (n)Vo. (-L(On, An)),
(8)
On + (m) Ja (Te) - An Vo, ([
(9)
Ant1
І&, (2(n) (-L(,n, )n))
(10)
DA,. m2) (TO) d)]+-
(11)
Here [x]+ denotes max(0, .r.. This operator ensures that the Lagrange multiplier remains non- negative after each update. In (1) (n), (n) > 0 Vn are certain prescribed step-size schedules. We assume that the step-sizes 7)1 (1) 1/2 (m), n 2 satisfy the regular step-size conditions. Thus, for
i = 1,2, Σ (n)
Σ n (n) < co. Note however that JR (TT) and J (((p)) as specified in
are not a priori known quantities and need to be estimated. We discuss this in detail below.
3.2.1 Estimation
We run each episode for T time steps in experiments. Let Tt--1 = R(st, Ce, st+1) and c++1 C(st, at, 1. respectively, for simplicity. For each sample path we would have both a reward return as well as a cost return. Let R, (resp. Ć,) be the reward-to-go (resp. cost-to-go) estimate. We
compute R2 and C₁ according to: R² k=
---1- * Cr =I T- "* 7* respectively.

---

We use a neural network parameterized by ei, to estimate reward signal based value function V and neural network parameterized by Ve estimate cost signal based value function VCC. We run our simulations on N parallel workers and then sample mini-batch. M of size M s NT (Schulman jet al. 2017]. We use mean-squared loss to estimate value functions on sampled mini-batches as follows:
Loss(e,)
Σ
E
(V (8t)
R,,3
(12)
MT
tm0
T
(oss(vo)
1
Σ
(s) -,)
(13)
MT
TEM
t=
Let AAA.. t 2 O, and A t 2 O, respectively, denote the advantage estimates w.r. reward and cost value functions on the sample path. We compute them using Generalized Advantage Estimation
[Schulman et al. 2016 to balance the bias and variance tradeoff of advantage estimates, We can have a range of advantage estimates as under.
4 = r + (V(++1) - V(s),,
(14)
A = r1+1 + M4+2 + 7 2 (((+2) Vss:),
(15)
A = T1+1 + 1 TrE* + +
V(+)),
(16)
for k >> 2. The advantage estimate in (14 will have high bias but low variance while estimates in (16
with higher value of k generally have high variance but low bias. Let 8. r++1 V 9. (s
V (se) and of = C++1 + V. ($++1) - v. (s)), respectively, denote the reward and cost temporal
differences, Let À be parameter which adjusts the variance tradeoff. Generalized Advantage Estimates [Schulman et al. 2016 for A." A are then given by
A, Σ (7
(17)
A Σ
5
(18)
E0
respectively. Now PPO clipped objectives jR (To), J (TO) as follows:
(Schulman al. 2017] for estimation of
J (TT)) E. (m) clip(r, (0), e, 1 +)
(19)
_ E. (m) clip(r; (0), (,:11,,,+++)) +,1+
(20)
where r0( x=(a))))) is the ratio of the probability of selecting action a in state under (arle))
parameter 0 opposed to Gold- Further, A, and AC are the estimated advantages based on the reward and cost returns, respectively, by time t. (see above) and e is the clip-ratio which clips (0) to (1 if is less than (1 e) and clips (1 e) if it is greater than (1 + +c). This algorithm restricts the policy parameters to not change significantly between two iterations which helps in avoiding divergence. This approach is referred as PPO-Lagrangian (Ray al. 2019.
3.3 Model-based Constrained RI
We formulate a Constrained RL problem 21 using a model-based framework as follows:
max JR((T)) s.t. Juh (TT) S d, where E1
(21)
(m)) = E
2' R(s Cc, S4,11) so ~ u, sct1 ~ Pa())s)a)); at ~ ng, Vt
(22)
E
('(ss, (1, 8411) 1000 u 8441 ~ Pa(./se, (1)) To,
(23)
5

---

In the above, Pa (.|ss, at) is an a- p--armeetrrized environment model, d, is a human prescribed safety threshold for the ith constraint and II the set of all 8-parameterized stationary polices. Note that we assume the initial state 8o is sampled from the true initial state distribution μ and then 8141 Pa(-sst,a)) at); Vt > O. We would use approximation of environment Pa to create 'imaginary' roll-outs to estimate the reward and cost returns required for policy optimization algorithms.
4 Challenges in Environment Model Learning
In this section we discuss the challenges that commonly arise due to model learning in RL. We further highlight the challenge that arises from using environment model approximation in Safe RL settings.
1. Handling aleatoric and epistemic uncertainties: : Aleatoric Uncertainty refers to the notion of natural randomness in the system which leads to variability in outcomes of an experiment. This uncertainty is irreducible because is natural property of the system. Hence in such cases, giving measure of uncertainty in model's prediction is good practice. In Lakshminarayanan are used which give an idea about uncertainty in prediction as well. They learn Gaussian distribution parameterized by neural networks. Epistemic Uncertainty refers to the notion of lack of sufficient knowledge in the model as a result of which the model does not generalize. In "Lakshminarayanan et al. (207). an ensemble of uncertainty aware neural networks with different initializations is proposed to reduce epistemic uncertainty for fixed data.
al. [2017), Chua al. [2018) uncertainty-awar
neural networks
For the learning environment model, we also use an ensemble of n neural networks with random initialization. Each neural network's 's output parameterizes a multivariate normal distribution with diagonal covariance matrix. Suppose the ith neural network in the ensemble is parameterized by a, and the mean and standard deviation outputs respectively. Recall now that if a random vector X E Rd is distributed according to the multivariate normal distribution parameterized by (μ, Edxa) where μ E IRd is the mean vector and Eaxd is a d x d covariance matrix, then the probability density of X is defined as, P(z) (2m) 1111//eapp(------ (x -
Ma, and da
1)'E-(2 -1)), E Rdd.
As a choice of loss function we use the negative log-likelihood loss for minimization (i.e.. minimizing negative log of P(X)). For the ith neural network parameterized by 0g, the loss function L(a;) is given as follows:
M
M(as)= = Σ [Ma (st, ar) s141]" TT Eao (st, a4) (n) (a]laaa(s (8t, at) s1t1 +log (Ea, (8t, ail, (24) 1=1
where Pa, (St, ae) is the mean vector output of the ith neural network and Ex, (st, az) is the covariance matrix which is assumed to be diagonal matrix. Note that, using neural networks with random initialization tends to have a regularization effect. Intuitively, introduces diversity in learned models deal with more possible trajectories. Also, has been shown empirically that with an increase in the number of models, the performance tends to improve (See section 6.3 and Figure 4 of Kurutach et al. 2018)) ) but increase in number of models also leads to increase in space complexity.
2. Aggregation of Error: : In model-based RL, as we move forward along the horizon, the error due to approximation starts aggregating and predictions from the approximated model tend to diverge significantly from the true model. order to tackle this problem, most of the model- -baeed RL approaches [Deisenroth and Rasmussen 2011 Kurutach et al. 2018 Janner et al.) [019] use shorter (or truncated) horizon during the policy optimization phase and achieve similar performance as Model-Free RL approaches. We use truncated horizon in our approach.
3. Implication of using truncated horizon in Constrained RL When we use truncated horizon in Constrained RL, it leads to underestimation of cost returns (3) under the current policy and we use the prescribed constraints to threshold the returns obtained. This can lead to constraint violations in the real-environment where the cost objective is based on the infinite horizon cost return. We propose a hyperparameter-bann approach to deal with this problem in the next section.

---

Model-based PPO Lagrangian
We propose model-based algorithm obtained from relaxing the Lagrangian (see Algorithm T which alleviates the problem with obtaining a large number of samples in model-free Lagrangian based approaches and as consequence also decreases the cumulative hazard violations. It is difficult to evaluate the policy without interacting with the real environment accurately.
Algorithm Our Approach:
Model-based PPO-Lagrangian
Input: Initialize actor neural net parameter to. critic parameters 4h.o, eeco, ensemble models
Horizon Lagrange parameter Ào 2 0, cost threshold = d, Environment Horizon = =T, Model
Pa.]i= ]
2: for ior1=,,... N training epochs do
3: Collect data tuples (st; Ge, St+1), ), for the ith ensemble using policy TT. I,..., n, in the environment for T time steps over multiple ((E)) episodes ton
4: Train [= by minimizing (24) w.r.t Ols;
5: while Performance ratio P Plehreaholdd do
6: So
(Note: For first pass A))
mix of real data and imaginary roll-out data. (See Appendix
7: Collect data roll-outs de To, (-11e), 8t Pa, (-1s,, az) (At each time step 'q" is randomly selected from I,)..,n) for H time steps (H < T) Compute J Sammpee (To,) TET E PC((s at) where |E] is the number episodes
8:
9:
Compute advantage, cost-ad advatagee using (17) and (T8) respectively
10:
Update À by replacing (m(T)) with Je sampte (To.)) and using an appropriate value of [ in (26).
(Multiple gradient updates for actor and critic]
11:
for k = 1,.... K do
Compute jR (To ), JC (TTO as in 119 and 20 respectively.
13:
Update parameters O2 using (9)
Update critic parameters n and the minimizing and 13 respectively.
15:
end for
16:
Compute Performance Ratio (PR) using 25
17:
end while
18:
end for
For this we compute the Performance Ratio (PR) metric using ensemble models (see Kurutach et al. 2018) that is defined as follows:
1 PR Σ 1=1
((s, (R(as,0:-1))
(25)
7t ae St+1), 30 and Vt > 0 81+1 Pas (-1se, oz), a Tg. (- This measures the ratio of the number of models which policy is improved to the total number of models in ensemble (n). If PR PRihresholds continue training using the same model, not then we break and re-train our environment model on data collected from the new update policy.
where (R(a,, 0,)
Another challenge that we encounter is the underestimation of J (m²) resulting from using truncated horizon (of length H in step 7 of Algorithm I to reduce the aggregation of error.
7

---

16
beta-0.000
betamo oos
14
beta-00.0
beta-0.1
60
heta-0.03
beta
beta=0.001
beta-D 00001
50
beta 0001
003
-0.05
40
beta-6.00
Limit=18
DialEnvinteracts
le5
Envintniitesectts
le5
Figure I: Effect of beta parameter (3) on expected cost returns (left) and expected reward returns (right) in PointGoaa environment, (Here B = 0.1 conressonds to 7
!!
This potentially lead to constraint violating policies. So need make the safety threshold (d) stricter. might seem that changing this threshold (d) proportional to the truncated horizon, ie., setting d == d T might work better where H is the truncated horizon and T is the original environment horizon, but found that this leads constraint violations well because learning policy using data from the approximated environment model that would bring in errors. Hence the cost estimate of the policy (Step in Algorithm 1 is error-prone well. To tackle this issue, we make safety threshold stricter using hyperparameter < 3 < 1 by modifying the Lagrange multiplier update follows:
An Pm (e()((([(T))) d 3)]++
(26)
The variation of expected cost returns and reward returns with respect to is shown in Figure on Safety gym PointGoal [Ray et al., [019]] environment. We can observe that we reduce 5, the expected cost returns reduce because cost limit becomes stricter but choosing too small leads to low reward returns as well.
6 Experimental Details and Results
We test approach Safety Gym environments PointGoal and CarGoal with modified state representations as used in other model-based Safe RL approaches [Liu et al. 2020 Sikchi et al. 2021a that more helpful in model-learning. We increase the difficulty of PointGoal and CarGoal environments by increasing the number of hazards from 10 to 15. In both environments, the aim of robots is reach the goal position and have few collisions with hazards possible. We compare our approach model-free Safe RL approaches including Constrained Policy Optimization [Achiam al. 2017] PPO-Lagrangian |Ray et al. 2019] and the model-based approach safe-LOOP from Sikchi et al. 2021a
with Unconstrained PPO [Schulman et al. 2017).
The code for our approach is available here We run each algorithm with 8 random seeds for 450K environment interactions. The hyper-parameter settings and other experimental details are given in Appendix A. The performance of our approach and baseline algorithms shown in Figure for PointGoal environment and Figure for CarGoal environment.
http:://gttaa.cccm///saaa//aaayr

---

HBNO Legrangian
CPO
sefe LOOP
PPD Legriongisn
MEPFO Lngrangis
sefe-L00
20
PFO Lagrosgian
10
Figure 2: Reward Performance (Left) and Cost Performance (Right) in PointGoal Environment, where y-axis denotes Average Episode Reward Returns (left) / Cost Returns (right) and x-axis denotes total environment interacts
VSPPO Legrangian
CPO
Liit-1
HGPFO-Lagnogiae
PRO (Uecorobrained)
PPO lst converpenee -Det ene internctions
Figure 3: Reward Performance (Left) and Cost Performance (Right) in CarGoal Environment, where y-axis denotes Average Episode Reward Returns (left) Cost Returns (right) and x-axis denotes total environment interacts
The left part of the plots represent Average Episode Return (average of the episode rewards) on y-axis and total environment interactions the x-axis. The dashed lines represent performance of the PPO-Lagrangian (in black) at convergence which around 2 million environment interactions. The right part of the plots show Average Episode Cost (average of the episode costs) on y-axis and total environment interactions on the x-axis. The red dashed line this plot represents cost limit of 18. From the plots we can observe that our approach (MBPPO-Lagee (MBPPO-Lagrannian) converges to the same level of reward performance as PPO Lagrangian in just 450K environment interactions and outperforms the model-based safe-LOOP algorithm (Sikchi et al. 2021a). Also our approach gives constraint various values for both CarGoal and RC-Car Ahn 2019] in similar manner as we did for PointGoal in Figure T and present it in Appendix D. We also compare our algorithm with safe-LOOP LOOP(Siichiii et al. 202 la
adhering policies in both the tasks. addition to above, we run our algorithm
on RC-Car cuvironment and present it in Appendix D as well. We found both approaches competitive in RC-Car environment, although safe-LOOP (Sikchi et al. 202 exhibits higher variance.
We also measure cumulative hazard violations that occur till convergence for MBPPO-L (ours),
PPO-Lagrangian [Ray et al. 2019), safe-LOOP (Sikchi al. 2021a] follows
Cumalative Violations
Σ
[1(C(s;, == 1)].
(27)
Tall convergence
We use rliable library (See Sec 4.3 in Agarwal et al. [2021] for more details) for plotting 95% confidence intervals for cumulative violations in the right part of (normaziie by cumulative violations in unconstrained PPO in respective tasks) and final policy reward performance in the left part of Figure 4 (normalized by reward return of the final policy unconstrained PPO in respective tasks) using mean, median and inter-quartile mean aggregate estimates for our approach, PPO-Lagrangian and safe-LOOP because these approaches only give constraint satisfying policies at convergence. From figure 4 we can observe that our approach is competitive with regards the model-based safe-LOOP approach (Sikchi al., (2211a] in terms cumulative constraint violations but achieves better reward performance at convergence.

---

riddan
IQM
VgP0O (W00))
NEPPO
Normelized reward performance st convergence
Normalzzed cumalivve cast violasons convergence
Figure Normalized Reward Returns Convergence (left) with median, inter-quartile mean (IQM), mean estimates and Normalized Cumulative Violations (right) with median, inter-quartile mean (IQM), mean estimates. Top rows (in green) represent IPO-Lagrangian, middle rows (in orange) represent safe-LOOP and bottom rows (in blue) represent our approach.
Conclusions
We presented a sample efficient approach for Safe Reinforcement Learning and compared our approach with relevant model-free and model- baselines [Achiam et al., 2017 Sikchi
2021a Ray et al. 2019 Schulman et al. [017]. Note that we chose the above baselines because they
are specifically designed solve a constrained optimization problem of the same structure as (T] B3. In Sikchi et al. [202 approximation of the value function is used to provide long-term reasoning instead using a truncated horizon. limitation of this approach lies in the fact that approximations of reward and cost value functions are used. The most challenging part of model-based approaches is to learn the environment model. The first issue of computational resources and the time overhead that is needed. We provide comparison of the running time of algorithms and their hardware requirements (see Appendix B). Our algorithm does much better than safe-LOOP (Sikchi et al. 202 in terms of running time.
One should note that in safe exploration settings, agents do not explore as much an unconstrained agent would do. This adds to the complexity of model learning in safe exploration problems because constrained exploration leads limited representation of data points for model learning even with the same sample size. We found this more pronounced in high-dimensional environments like DoggoGoal| [Ray et al., 2019] (See Appendix C). Our algorithm depends on Lagrangian- approach which provides very simplistic way to construct safe RI. algorithm but suffers from the issue of low reward performance compared to unconstrained approaches (see Appendix A for a reward comparison). Note that we can have a similar model -baeed approach using TRPO-Lagun Lagrangann Ray et al. 2019] but involves an extra overhead of approximating the Fisher Information Matrix (FIM) and hence we chose PPO-based approach. Also TRPO-LLgrangee
and PPO-Lagrame PPO-Lagrangian have similar performance on Safety Gym benchmark (Ray et al. 2019]. Increasing reward performance of Lagrangian- based approaches and devising better ways for model-learning in high- -ineesiiinii state representations in safe RL settings where exploration is limited, can be looked at the future. Moreover, it would be interesting adapt off-policy natural actor-critic algorithms such in Bhatnagar et al. (2009), Diddigi et al. (2022) to the setting of constrained MDPs and study their performance both theoretically and empirically.
Acknowledgments and Disclosure of Funding
Ashish K. Jayant supported in his work through an MHRD scholarship from the Ministry of Education, Government of India; Indian Institute of Science, Bangalore; and Flipkart Internet Pvt. Ltd., Bangalore. S. Bhatnagar was supported in his work through the J B.Ce National Fellowship. Science and Engineering Research Board, Government of India; a project from the Department of Science and Technology under the ICPS program; project from DRDO under JATP-CoE; well as the Robert Bosch Centre for Cyber Physical Systems, Indian Institute of Science Bangalore.
10

---

References
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 2017.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34, 2021.
Edward Ahn. Towards safe reinforcement learning in the real world, July 2019.
Eitan Altman. Constrained Markov Decision Processes, volume Taylor & Francis, 1998.
Dimitri P. Bertsekas. Constrained Optimization and Lagrange Multiplier Methods ( (Optimization and
Neural Computation Series). Athena Scientific, edition, 1996. ISBN 1886529043.
Shalabh Bhatnagar. An actor-critic algorithm with function approximation for discounted cost constrained markov decision processes. Systems & Control Letters, 59:760-766, 12 2010.
Shalabh Bhatnagar and Lakshmanan. An online actor-critic algorithm with function approximation for constrained Markov decision processes. Journal of Optimization Theory and Applications, 153 (3):688-708, 2012.
Shalabh Bhatnagar, Richard Sutton, Mohammad Ghavamzadeh, and Mark Lec. Natural Actor- Critic Algorithms. Automatica, 45(11), July 2009. doi: w..1016f.automatii20000....0. URL
https://hal. forraa.f/hall-oe 00840470
Shalabh Bhatnagar, H.L. Prasad, and L.A. Prashanth. Stochastic recursive algorithms for optimiza- tion: Simultaneous perturbation methods. Stochastic Recursive Algorithms for Optimization: Simultaneous Perturbation Methods, volume 434, page 320. Springer, 2013.
Vivek S Borkar. An actor-critic algorithm for constrained Markov decision processes. Systems & control letters, 54(3):207-213, 2005.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym, 2016. URL
01540
Yinlam Chow and Mohammad Ghavamzadeh. Algorithms for CVaR optimization MDPs, 2014.
URL https://arxiv. org/ abs//40/.660 3339
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A Lyapunov- based approach to safe reinforcement learning. Advances neural information processing systems, 31, 2018.
Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzmant and Mohammad Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. arXiv preprint 1rXiv:1901.1003: 2019.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
Alexander Cowen Rivers, Daniel Palenicek, Vincent Moens, Mohammed Abdullah, Aivar Sootla. Jun Wang, and Haitham Ammar. Samba: Safe model- based & active reinforcement learning, 2020. URL pistps://arxiae org/ abs/2006. 09436
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv: 007557, 2018.
Pieter-Tjerk de Boer, Dirk P. Kroese, Shie Mannor, and Reuven Y. Rubinstein. A tutorial on the cross-entropy method. Annals of Operations Research, 134(1):19-67, Feb 2005. ISSN 1572-9338. doi: 1077000000
URL httpp:://dat - 110.00000000111111 -05-
11

---

Mare Deisenroth and Carl E Rasmussen. Pilco: A model--ased and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pages 465-472. Citeseer, 2011.
Raghuram Bharadwaj Diddigi, Prateek Jain, K.J. Prabuchandran, and Shalabh Bhatnagar. Neural net- work compatible off-policy natural actor-critic algorithm. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1-10, 2022. doi: 10.1 N0.10000CCNN5500422222222 2892233..
Javier García, Fem,, and o Feráandez. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(42):1437-1480, 2015. URL hhttp:/jjml org/
papers/v wttp://aa/aaaaaa..a.a. html
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning. pages 1861-1870. PMLR, 2018.
Seung- Ho Han, Ho-Jin Choi, Philipp Benz, and Jorge Loaiciga. Sensor-based mobile robot navigation via deep reinforcement learning. In 2018 IEEE International Conference on Big Data and Smart Computing (BigComp), pages 147-154, 2018. doi: 10. 109/BigComp..0 2.0000000..00000000.
Nicolas Heess, Greg Wayne, David Silver, Timothy Lillicrap. Yuval Tassa, and Tom Erez. Learning continuous control policies by stochastic value gradients, 2015. URL :ttpa:///arrzze //arxiv.vvv////////a///aa//aaavl org/ /
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
Alex Kendall, Jeffrey Hawke, David Janz, Przemyslaw Mazur, Daniele Reda, John-Mark Allen, Vinh-Dieu Lam, Alex Bewley, and Amar Shah. Learning to drive in a day, 2018.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations, 2018.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Puze Liu, Davide Tateo, Haitham Bou Ammar, and Jan Peters. Robot reinforcement learning on the constraint manifold. In Conference on Robot Learning, pages 1357-1366. PMLR, 2022.
Zuxin Liu, Hongyi Zhou, Baiming Chen, Sicheng Zhong, Martial Hebert, and Ding Zhao. Safe model-based reinforcement learning with robust cross-entropy method, 2020.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for -bsel deep reinforcement learning with theoretical guarantees, 2018. URL htttps://arxiv.orgg 3180/ 03888
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning, 2013.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint 19100//90 7:1, 2019.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference Learning Representations (ICLR), 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
12

---

Harshit Sikchi, Wenxuan Zhou, and David Held. Learning off- off-policy with online planning. CoRL, 2021a.
Harshit Sikchi, Wenxuan Zhou, and David Held. Lyapunov barrier policy optimization. arXiv preprint arXiv:2103.092:30 2021b.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by PID Lagrangian methods. International Conference on Machine Learning, pages 9133 PMLR, 2020.
Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization, 2018 8.
Garrett Thomas, Yuping Luo, and Tengyu Ma. Safe reinforcement learning by imagining the near future, 2022. URL
http://...rrg//////22// 07788
Stanislav Uryasev and Tyrrell Rockafellar. Conditional value-at-risk: optimization approach, In Stochastic optimization: algorithms and applications, pages 41 1-435. Springer, 2001.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning, 2019. URL. httpp:://a/: org/ ab071907 .02057
Min Wen and Ufuk Topcu. Constrained cross-entropy method for safe reinforcement learning. IEEE Transactions on Automatic Control, 66(7):3123-3137 2021. doi: 10. 10900AAA..20003115933..
Chao Yu, Jiming Liu, and Shamim Nemati. Reinforcement learning in healthcare: A survey, 2020.
Ming Yu, Zhuoran Yang, Mladen Kolar, and Zhaoran Wang. Convergent policy optimization for safe reinforcement learning, 2019.
Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 33: 33:15338-15349 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We describe limitations in Conclusion section
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have read the ethics review guidelines and ensured that our work conforms to them.
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (cither in the supplemental material or as URL)? [Yes] We provide the
github url for code repository
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Hyperparameter details are given in appendix A
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We report plots with 95% confidence intervals and use
rliable Agarwal et al. 2021 library for robust aggregate estimates
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] We share details hardware requirement in appendix B
13

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite the authors of Safety Gym benchmark and their github code repository.
(b) Did you mention the license of the assets? [Yes] Their MIT License which allows without restriction, including without limitation the rights to use, copy, modify, merge,
publish provided that license notice is included in our code repository
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "Uhqu0DGsDrn", "reviewer": "Reviewer_CLz8", "summary": "This paper presents Model-based PPO-Lagrangian (MBPPO-Lagrangian) algorithm for safe RL, which reduces epistemic and aleatoric uncertainty with an ensemble of neural networks and solves the underestimation problem of cost returns in a truncated horizon with the stricter threshold using a hyperparameter. The authors compared the algorithm in Safety Gym: PointGoal1 and CarGoal1. MBPPO-Lagrangian showed higher cumulative reward and lower cumulative cost performance than baseline methods. ", "questions": "<Explanation of experiments>\n\n- Please explain the environments you used. Also, please explain why you used PointGoal1 and CarGoal1 with more hazards, not PointGoal2 and CarGoal2. \n- The names ``PointGoal`` and ``PointGoal1`` both seem to be used to name the same environment (e.g. Figure 2), so please unify the name. The same goes for the CarGoal1 environment too. \n- Figure 4 is hard to interpret. I cannot understand what you want to say from Figure 4. Is there any reference that made you decide to show the data in this format? \n\n<More baselines>\n\nI think the comparison between yours and Liu et al.[1] should be done since you mentioned the common points in L264. Also, Liu’s paper dealt with the same environment as yours and they are also model-based. \n\n<Details>\n\n- L163: is $\\rightarrow$ in \n- I think $t+1$ should be $t+l$ in Eq. 17 and Eq. 18.\n\n[1] Zuxin Liu, Hongyi Zhou, Baiming Chen, Sicheng Zhong, Martial Hebert, and Ding Zhao. Safe model-based reinforcement learning with robust cross-entropy method, 2020.\n", "limitations": "Yes, the authors mentioned the limitation at the end of the conclusion.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors successfully applied a model-based approach to the PPO-Lagrangian method by reducing the uncertainty of state transition using an ensemble of neural networks. Also, they suggested a simple solution that uses a hyperparameter to fix an innate underestimation-of-cost problem in the model-based approach that assumes a truncated horizon.\n\nHowever, I suggest the authors strengthen the Experiments part. The environments used in experiments are not sufficiently explained and I feel the paper needs more baseline experiments. The more detailed suggestions will be described in the next section.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MIaULM1ywB-", "reviewer": "Reviewer_Wj7m", "summary": "The authors propose a new algorithm for a model-based safe RL (under the framework of CMDP). The proposed variant is modification of the PPO algorithm, namely by learning a model of the environment and introducing lagrangian relaxation to make sure the policy satisfies the safety constraints.\n", "questions": "How was the PR > 70% choice made? Did you run ablation analysis on this parameter?\n\nCan you please run Figure 1 for more environments?", "limitations": "Adressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and easy to understand. The main algorithm - Algorithm 1 - is also very helpful.\n\nThe motivation is sound, the results are interesting, and the algorithm itself seems easy to implement.\n\nWhat I find as a satisfying sanity check is the fact that unconstrained PPO does reach the same or better performance as MBPPO-Lagrangian. I understand that as the graphs (Figure 2 and 3) use as x-axis the number of environment interactions, the model-based approach is likely to do better initially. But as we keep going, PPO really needs to get there, especially as it is unconstrained and it clearly heavily violates the constraints.\n\n\nWeaknesses\n\n> [266] CarGoal1 by increasing the number of hazards from 10 to 15.\n\n  I am not a fan of this.. It’s fine if that’s another environment in the experiments, but the original unmodified environment should also be included.\n\nPR > 70% is a bit of an arbitrary choice, and I don’t see this being discussed anywhere. \n\nDue to the truncated horizon, the introduced \\beta parameter and associated Equation (26) is something that is likely to be domain-specific. I would appreciate it if the authors could run Figure1 for multiple environments to see how the effect of beta varies across domains.\n\nAll the Figures, especially Figure 1 and Figure 4  are very hard to read, please increase the font size.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "D0Jy53AC0Uf", "reviewer": "Reviewer_Y6ox", "summary": "The paper presents a model-based method for safe RL that learns; a dynamics model of the environment, a reward value function, a cost value function, and a policy. The method leverages the learned dynamic models to generate imaginary rollouts to learn the reward and cost value functions, which are then used by a lagrangian relaxation of PPO in the setting of constrained Markov decision processes. \n\nThe proposed method is evaluated in the Open AI safety Gym, where it achieved better rewards than constrained model-based baselines and it also obtained better sample efficiency with lower constraints violation than other constrained model-free approaches.", "questions": "- A static choice of the safety threshold seems very conservative. As training progresses I could imagine that the estimation of cost returns becomes less uncertain and thus relaxing the safety thresholds could allow for better performance in terms of rewards. How would you incorporate model uncertainty to set the safety threshold dynamically?\n\n- Is there any particular reason why the method trains the value and cost function using purely imaginary rollouts? When new data is collected to learn new dynamics models, would there be any benefits of using such data to also improve the learned value and cost functions?", "limitations": "Yes. Limitations of the method are mentioned in the conclusions. The potential societal impact was qualified as Not Applicable by the authors. \n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The presented method is simple and builds on well-established methods. The results obtained a balance of the benefits of constrained model-based and model-free baselines,  showcasing better sample efficiency with low constraint violation wrt to constrained model-free approaches and better rewards wrt model-based methods.\n\n- The paper is clear and well written. It presents a comprehensive related work and introduces a detailed background. \n\nWeaknesses:\n- The approach is evaluated in only two environments (PointGoal, CarGoal). Although the environments were modified to make them more challenging, the work would benefit from evaluating the method in more setups like the Doggo environment (which is partially reported in the appendix), or other variations of the Point and Car setting like the Button or Push setups from the Open AI Safety Gym.\n\n- The method seems to struggle with longer time horizon tasks, due to the compounding errors of the learned dynamics, reward value, and cost functions. A conservative safety margin was introduced to deal especially with the underestimation of cost returns. However, such a safety margin results in a strong tradeoff wrt the reward performance. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5URMZ7Chmqa", "reviewer": "Reviewer_hwMj", "summary": "This paper proposes a safe reinforcement learning method in a model-based manner. The method uses the Lagrangian relaxation of the original constrained optimization problem and then uses dual gradient descent to find the saddle point. The method also improves the sample efficiency by learning an ensemble of dynamics models. Experiment results on Safety Gym environments demonstrate its effectiveness over model-free and model-based safe RL baselines.", "questions": "- About the experiment part, why does PPO-Lagrangian perform worse than MBPPO-Lagrangian asymptomatically? Since PPO-Lagrangian uses rollouts from the real environment while MBPPO-Lagrangian learns from model rollouts, the latter can have better sample efficiency but suffer from biases. So I was wondering why MBPPO-Lagrangian can outperform PPO-Lagrangian in terms of asymptomatic performances.", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n- Proposed method uses an ensemble of dynamics models to improve sample efficiency and address the problem of epistemic uncertainties.\n- Experiment results show that MBPPO-Lagrangian significantly outperformstones in Safety Gym environments.\n- It found that the safety criteria should be tighter in order to achieve better performance when using truncated model rollouts.\n\n### Weakness\n- It might be better if the evaluation part could be more comprehensive since only the results of two tasks are reported.\n- The authors use dual gradient descent to find the local saddle point of the Lagrangian relaxation of the original constrained problem. It would be better if they can provide some mathematical analysis of the effectiveness of this approach. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept