Safe Reinforcement Learning by Imagining the Near Future
Authors: Garrett Thomas
Yuping Luo
Tengyu Ma
Venue: NeurIPS 2021 Poster (2021)
Keywords: safe reinforcement learning
model-based reinforcement learning

Abstract: Safe reinforcement learning is a promising path toward applying reinforcement learning algorithms to real-world problems, where suboptimal behaviors may lead to actual negative consequences. In this work, we focus on the setting where unsafe states can be avoided by planning ahead a short time into the future. In this setting, a model-based agent with a sufficiently accurate model can avoid unsafe states.
We devise a model-based algorithm that heavily penalizes unsafe trajectories, and derive guarantees that our algorithm can avoid unsafe states under certain assumptions. Experiments demonstrate that our algorithm can achieve competitive rewards with fewer safety violations in several continuous control tasks.

Safe Reinforcement Learning by Imagining the Near Future
Garrett Thomas
Yuping Luo Princeton University yupingl@cs wrinceton.nccoomcc edu
Tengyu Ma Stanford University
Stanford University gwthomas@stannford edu
tengyuma@stanforo edu
Abstract
Safe reinforcement learning is a promising path toward applying reinforcement learning algorithms to real-world problems, where suboptimal behaviors may lead to actual negative consequences. In this work, we focus on the setting where unsafe states be avoided by planning ahead short time into the future. In this setting, model-based agent with a sufficiently accurate model can avoid unsafe states. We devise a model-based algorithm that heavily penalizes unsafe trajectories, and derive guarantees that our algorithm can avoid unsafe states under certain assumptions. Experiments demonstrate that our algorithm can achieve competitive rewards with fewer safety violations in several continuous control tasks.
Introduction
Reinforcement learning (RL) enables the discovery of effective policies for sequential decision- making tasks via trial and error [Mnih et al. 2015, Gu et al. 2016 Bellemare et al. 2020). However, in domains such robotics, healthcare, and autonomous driving, certain kinds of mistakes pose danger to people and/or objects in the environment. Hence there is an emphasis on the safety of the policy, both at execution time and while interacting with the environment during learning. This issue, referred to as safe exploration, is considered an important problem AI safety Amodei et al. 2016).
In this work, we advocate a model-based approach to safety, meaning that we estimate the dynamics of the system to controlled and use the model for planning (or more accurately, policy improvement). The primary motivation for this is that a model-based method has the potential to anticipate safety violations before they occur. Often in real-world applications, the engineer has an idea of what states should be considered violations of safety: for example, robot colliding rapidly with itself or surrounding objects, car driving on the wrong side of the road, or a patient's blood glucose levels spiking. Yet model-free algorithms typically lack the ability to incorporate such prior knowledge and must encounter some safety violations before learning to avoid them.
We begin with the premise that in practice, forward prediction for relatively few timesteps is sufficient avoid safety violations. Consider the illustrative example Figure [T which an agent controls the acceleration (and thereby, speed) of a car by pressing the gas or brake (or nothing). Note that there is an upper bound on how far into the future the agent would have to plan to foresee and (if possible) avoid any collision, namely, the amount of time it takes to bring the car to a complete stop.
Assuming that the horizon required for detecting unsafe situations is not too large, show how to construct a reward function with the property that an optimal policy will never incur a safety violation. A short prediction horizon is also beneficial for model-based RL, as the well-known issue of compounding error plagues long-horizon prediction [Asadi et al. 2019]: imperfect predictions are fed back into the model inputs (possibly outside the distribution of inputs in the training data), leading to progressively worse accuracy as the prediction horizon increases.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

An illustrative example. The agent controls the speed car by pressing the accelerator or brake (or neither), attempting to avoid any obstacles such as other cars or people in the road. The top car has not yet come into contact with the pedestrian, position and speed, even if it brakes immediately. The bottom car can slow down before hitting the pedestrian. the bottom car plans several steps into the future, could reduce its speed avoid the "inreoveerable" situation faced by the top car.
Figure 1:
cannot avoid the pedestrian from its current
Our main contribution a model-based algorithm that utilizes a reward penalty the value of which is prescribed by our theoretical analysis to guarantee safety (under some assumptions). Experiments indicate that the practical instantiation of our algorithm, Safe Model-Based Policy Optimization (SMBPO), effectively reduces the number of safety viola- tions on several continuous control tasks, achieving a comparable performance with far fewer safety violations compared to several model-free safe RL algorithms. Code is made available at https://gi thub. com/ hetp///thoomsSS50 TwthhmassSSafe-MBBP MBPO
2
Background
In this work, we consider a deterministi Markov decision process (MDP) M (S, where S is the state space, .A the action space, T : x A S the transition dynamics, : X [rminn [max] the reward function, and 7 E [0, 1) the discount factor. A policy ग determines what action to take at each state. A trajectory is sequence of states and actions T (80, do, ro, 81, 41, TI. where 8:41 = T(*,, a:) and ri = r(st, at).
A(A)
Typically, the goal is to find policy which maximizes the expected discounted return (](T) = ET ET r-re]. The notation E denotes that actions are sampled according to a T (s:).. The initial state So is drawn from an initial distribution which we assume to be fixed and leave out of the notation for simplicity.
The Q function (s, = = E 7'r, 80 = s, ag = a] quantifies the conditional performance of a policy assuming Pa-r(e)("" (Q"((( (s, a)] averages this quantity over actions. The values of the best possible policies are denoted Q* a) = max, Q= a) and V*(s) = maxş (s)... The function Q' has the important
in specific state and takes action a, and the value function V (s(ss
property that any optimal policy πt" arg max, 77(x) must satisfy P(a" arg max, Q" ()) = 1 for all states s and actions a" ~ (s). Q* is the unique fixed point the Bellman operator
(=((saa==(ssp
max Q(5',a)) ,a) where s' = a))
In model-based RL, the algorithm estimates dynamics model T using the data observed so far, then uses the model for planning or data augmentation. The theoretical justification for model-based RL is typically based some version of the "simulation lemma", which roughly states that if i then (i())) ~ (TT)) Kearns and Singh 2002 Luo al. 2018
Determinismm makes safety essentially trivial in tabular MDPs. We focus on tasks with continuous state and/or action spaces. See Appendix A.2 for possible extension of approach to stochastic dynamics.

---

3
Method
In this work, we train safe policies by modifying the reward function to penalize safety violations. We assume that the engineer specifies Sunsafe, the set of states which are considered safety violations.
We must also account for the existence of states which are not themselves unsafe, but lead inevitably unsafe states regardless of what actions are taken. Definition 3.1. A state is said to be
a safety violation if E Sunsafe- irrecoverable if 4 Sunsate but for any sequence of actions ap, 01, O2,..., the trajectory defined by 50 s and $141 = Ts( at) for all E N satisfies st Sunsafe for some t E N.
unsafe if it is unsafe or irrecoverable, or safe otherwise.
We remark that these definitions are similar to those introduced in prior work on safe RL (Hans et al. 2008]. Crucially, we do not assume that the engineer specifies which states are (ir)recoverable, that would require knowledge of the system dynamics. However, we do assume that safety violation must come fairly soon after entering an irrecoverable region:
Assumption 3.1. There exists a horizon H" E N such that, for any irrecoverable states any sequence of actions do, 0 will lead an unsafe state. That is, i0 and st+1 T(s,, dz) for all t E t0,..., H - 1}, then E Sunsafe for some i E {l,..., 1,,,11" }.*}.
This assumption rules out the possibility that a state leads inevitably to termination but takes an arbitrarily long time do so. The implication of this assumption that perfect lookahead planner which considers the next H steps into the future can avoid not only the unsafe states, but also any irrecoverable states, with some positive probability.
3.1 Reward penalty framework
Now present reward penalty framework for guaranteeing safety. Let Mc (S,A, be an MDP with reward function and dynamics
s E Sunsafe
(f(s, (((s,a).(ss.)) (). = (--C,s
(2)
E Sunsate
where the terminal cost in that they transition back into themselves and receive the reward of - regardless of what action is taken.
a constant (more on this below). That is, unsafe states are "absorbing"
The basis of our approach is to determine how large C' must be so that the Q values of actions leading unsafe states are less than the values of safe actions.
Lemma 3.1. Suppose that Assumption 3.7 holds, and let
Tmax Tmin
7H8
Tmax-
(3)
Then for any state 8, if is safe action (i.e. T(s,a) is safe state) and al is an unsafe action (i.e. T(s,a) a) is unsafe), holds that Q'
as) (s
(s, d'), where Q" is the Q" function for the MDP Mc.
Proof. Since is unsafe, it leads to an unsafe state at most H" steps by assumption. Thus the discounted reward obtained is at most
=-
Tmax (1- -
Σ
C((-)-=
(4)
1-7
By comparison, the safe action leads to another safe state, where it can be guaranteed to never encounter a safety violation. The reward of staying within the safe region forever must be least Thus, it suffices to choose C' large enough that
Fmax 7 " Ct"
Tmin 1-7
(5)
Rearranging, arrive at the condition stated.
3

---

The important consequence of this result is that an optimal policy for this MDP M will always take safe actions. However, in practice we cannot compute Q* without knowing the dynamics model T. Therefore we extend our result to the model-based setting where the dynamics are imperfect.
3.2 Extension to model-based rollouts
We prove safety for the following theoretical setup. Suppose we have a dynamics model that outputs sets of states ((s, a) C S to account for uncertainty.
Definition 3.2. We say that set-valued dynamics model f P x A + P(S is calibrated if T(8,a) a) E (s, a) for all (s,a) E x A.
We define the Bellmin operator:
B'Q (s, = F(s, + min max Q(s', a')
(6)
of (x,o)
Lemma 3.2. The Bellmin operator B* is a 7-contraction the co-norm.
The proof is deferred to Appendix As consequence Lemma 3.2 and Banach's fixed-point
theorem, E has unique fixed point Q which can be obtained by iteration. This fixed point is lower bound on the true Q function the model is calibrated:
Lemma 3.3. T is calibrated in the sense of Definition 3.2 then Q"(s, S Q² as) a) for all (s, a).
Proof. Let B* denote the Bellman operator with reward function F. First, observe that for any Q, : S x A - RR Q I Q' pointwise implies B S BBQQ pointwise because we have F(s. a) + + max.' Q(s' < F(s.a) + max,' ('ss', a') pointwise and the min defining B includes the true 5' = a)).
Now let Qo be any inital Q function. Define @x (B+))Qu and Q2 (B*)+&o. An inductive
argument coupled with the previous observation shows that Q2 S Qn pointwise for all & E N. Hence, taking the limits Q" = limg-+oc Q2 and Q" = lim Q2 we obtain Q < Q pointwise.
Now we are ready to present our main theoretical result.
Theorem 3.1. Let T be a calibrated dynamics model and π²" (s) = arg max, Q" (s, a) the greedy policy with respect to Q" Assume that Assumption 3.7 holds. Then any E S, if there exists an action a such that Q* (8, a) Z Tmin I-q"
then x² (8)) is safe action.
Proof. Lemma 3.2 implies that (' s Q" (s, a) for all (s, a) E A.
As shown in the proof of Lemma 3.1 any unsafe action a' satisfies
(s, s ('(sa')) Tmax (1 -) Cy
(7)
1-7
Similarly if (8,
(8, I-q
we also have
Tmin
ass (s,a)
(8)
1-?
so is a safe action. Taking C' in inequality guarantees that Q* > (*, so the greedy policy x² will choose over a'.
O
This theorem gives us a way to establish safety using only short-horizon predictions. The conclusion conditionally holds for any state s, but for far from the observed states, we expect that f(s, a) likely has contain many states in order to satisfy the assumption that it contains the true next state, so that Q" (8, a) will be very small and we may not have any action such that Q' (s, a) 2 famm However, it is plausible to believe that there can be such an a for the set of states in the replay buffer. {s (s,a, r, s') E D).
1-7
(x)(X) is the powerset of X.

---

Algorithm Safe Model-Based Policy Optimization (SMBPO)
Require: Horizon
I:
Initialize empty buffers D and D, ensemble of probabilistic dynamics (Te, J policy na. critic Qu. Collect imitial data using random policy, add D.
3: for episode 1, 2.... do
4: Collect episode using add the samples to D. Let l be the length of the episode.
5: Re-fit models (oo, L.N by several epochs of SGD on (9.) defined in (
6: Compute empirical Tmin and Tmax, and update C' according to (
7: for & times do
for Thollou times (in parallel) do Sample ~ DD
9:
Startin from roll out H steps using ne and (To.:: add samples to D.
for Noanrr times
12:
Draw samples from D D.
13:
Update Q by SGD on (v)) defined in 10 and target parameters & according to
12
14:
Update To by SGD T, (0) defined in
3.3
Practical algorithm
Based (mostly) on the framework described in the previous section, we develop deep model- based RL algorithm. We build on practices established in previous deep model-based algorithms, particularly MBPO [Janner et al. 2019] a state-of-the-art model-based algorithm (which does not emphasize safety).
The algorithm, dubbed Safe Model-Based Policy Optimization (SMBPO), is described in Algo- rithm M It follows a common pattern used by online model-based algorithms: alternate between collecting data, re-fitting the dynamics models, and improving the policy.
Following prior work [Chua et al. 2018 Janner et al. 2019]. we employ an ensemble of (diagonal) Gaussian dynamics models (To, } where (s a) = N(H, (s, a), diag(o (s, a))), in an attempt capture both aleatoric and epistemic uncertainties. Each model trained via maximum likelihood on all the data observed so far:
(0.) = -B(e.,,r.)))- log To. (8', al
However, random differences in initialization and mini batch order while training lead to different models. The model ensemble can be used to generate uncertainty-aware predictions. For example, set-valued prediction can be computed using the means T(ss, a) {ue, (s, a)
The models are used to generate additional samples for fitting the Q function and updating the policy. In MBPO, this takes the form of short model-based rollouts, starting from states in D, reduce the risk of compounding error. At each step in the rollout, a model T; is randomly chosen from the ensemble and used to predict the next state. The rollout horizon H is chosen as a hyperparameter, and ideally exceeds the (unknown) H" from Assumption 3.1 In principle, one can simply increase H to ensure it is large enough, but this increases the opportunity for compounding error.
MBPO is based on the soft actor-critic (SAC) algorithm, widely used off off-policy maximum-entroppy actor-critic algorithm [Haarnoja et al. 2018a The Q function is updated by taking one or more SGD steps the objective
LQ(D)) B (aars))_usll00ffs tV (s))))
(10)
where Vo(5') (-c/(1 (
s' E Sunsafe
(11)
log na(a' 8')]
s' Sunsafe
The scalar a hyperparameter of SAC which controls the tradeoff between entropy and reward. We tune a using the procedure suggested by HHarnoja et al. (0118].
The & are parameters of "target" Q function which is updated via exponential moving average towards vh:
(12)
for a hyperparameter 7 E (0, which often chosen small, e-g. 0.005. This is a common practice used to promote stability in deep RL, originating from Lillicrap et al. 2015 We also employ the
5

---

(a) Hopper
(b) Chectah-no-fip
(c) Ant
(d) Humanoid
Figure 2: We show examples of failure states for the control tasks considered in experiments.
clipped double-Q method (Fujimoto al. 2018 in which two copies of the parameters (1/1 and 4/2) and target parameters (1) and (i2) are maintained, and the target value in equation is computed using min;=1,2 1,2 Q (s',a').
Note that TOP. we are fitting to the average TD target across models, rather than the min, even though we proved Theorem 3.2 using the Bellmin operator. We found that taking the average worked better empirically, likely because the min was overly conservative and harmed exploration.
The policy is updated by taking one more steps to minimize
La (6) = B.DDDD la log (al s) (s
(13)
4 Experiments
In the experimental evaluation, we compare our algorithm to several model-free safe RL algorithms, as well as MBPO, on various continuous control tasks based on the MuJoCo simulator (Todorov) et al. 2012]. Additional experimental details, including hyperparameter selection, are given in Appendix A.3
4.1 Tasks
The tasks are described below:
Standard hopper environment from OpenAl Gym, except with the "alive bonus" (a constant) removed from the reward that the task reward does not implicitly encode the safety objective. The safety condition is the usual termination condition for this task, which corresponds to the robot falling over.
Hopper:
Cheetah-no-flip: The standard half-cheetah environment from OpenAl Gym, with safety condi- tion: the robot's head should not come into contact with the ground.
Ant, Humanoid: Standard ant and humanoid environments from OpenAl Gym, except with the alive bonuses removed, and contact forces removed from the observation (as these are difficut to model). The safety condition is the usual termination condition for this task, which corresponds to the robot falling over.
For all of the tasks, the reward corresponds to positive movement along the r-axis (minus some small cost on action magnitude). and safety violations cause the current episode terminate. See Figure for visualizations of the termination conditions.
4.2 Algorithms
We compare against the following algorithms:
MBPO: Corresponds to SMBPO with C 0.
MBPO+bonus: The same as MBPO, except adding back in the alive bonus which was subtracted out of the reward.

---

hopper
cheetah-no-flip
3000
12000
2500
10000
2000
8000
Return
Return
1500
6000
1000
4000
500
2000
250
500
750
1000
1250
0
50
100
150
200
250
300
Total violations
Total violations
ant
humanoid
4000
3000
SMBPO
3000
2500
MBPO
2000
MBPO+bonus
Return
2000
Return
1500
RRL MF
RCPO
1000
1000
SQRL
LR
500
500
1000
1500
2000
0
1000
2000
3000
4000
5000
Total violations
Total viclations
Figure 3: Undiscounted return of policy vs. total safety violations. We rum 5 seeds for each algorithm independently and average the results. The curves indicate mean of different seeds and the shaded areas indicate one standard deviation centered at the mean.
Recovery RL, model-free (RRL-MF): Trains critic to estimate the safety separately from the reward, as well as a recovery policy which invoked when the critic predicts risk of a safety violation.
Lagrangian relaxation (LR): Forms Lagrangian implement constraint on the risk, updating the dual variable via dual gradient descent.
Safety Q-functions for RI (SQRL): Also formulates Lagrangian relaxation, and uses filter to reject actions which are too risky according to the safety critic.
Reward constrained policy optimization (RCPO): Uses policy gradient to optimize a reward function which is penalized according to the safety critic.
of the above algorithms except for MBPO are as implemented in the Recovery RL paper TTananjeyyan et al. 2020) and its publicly available codebase We follow the hyperparameter tuning procedure described their paper; see Appendix |A.3 for more details. A recent work [Bharadhwa] al. 2020 can also serve bascline but the code has not been released.
Our algorithm requires very little hyperparameter tuning. We use 7 0.99 in all experiments. We tried both H and H 10 and found that H = 10 works slightly better, so we use H = 10 in all experiments.
4.3 Results
The main criterion in which we are interested is performance (return) vs. the cumulative number of safety violations. The results are plotted in Figure g We that our algorithm performs favorably compared to model-free alternatives in terms of this tradeoff, achieving similar or better performance with fraction of the violations.
7

---

Performance during training (cheetah-no-flip)
Total violations during training (cheetah-no-ff)
200
C-10
12000
175
10000
150
C 8.0
violations
125
Average
6000
100
4000
C- 2.0
C= 4.0
25
0
200
400
600
800
1000
0
200
400
600
800
1000
episodes
episodes
(a) Performance with varying C
(b) Cumulative safety violations with varying C
Figure 4: Increasing the terminal cost C makes exploration more conservative, leading fewer safety violations but potentially harming performance. This is what we expected: A larger C' focuses more on the safety requirement and learns more conservatively. Note that an epoch corresponds to 1000 samples.
MBPO is competitive in terms of sample efficiency but incurs more safety violations because it isn't designed explicitly to avoid them.
We also show in Figure that hard-coding the value of C' leads to an intuitive tradeoff between performance and safety violations. With larger C, SMBPO incurs substantially fewer safety violations, although the total rewards are learned slower.
5 Related Work
Safe Reinforcement Learning Many of the prior works correct the action locally, that is, changing
action when the action is detected to lead to an unsafe state. Dalal et al. (2018) linearizes the
dynamics and adds layer the top of the policy for correction. Bharadhwaj et al.
2020 uses
rejection sampling to ensure the action meets the safety requirement. Thananjeyan et al. 2020 either
trains a backup policy which is only used guarantee safety, to find the best action sequence. MPC could also be applied in the short-horizon setting that we consider here, but it involves high runtime cost that may not be acceptable for real-time robotics control. Also, MPC only optimizes for rewards under the short horizon and can lead to suboptimal performance tasks that require longer-term considerations (Tamrr et al. 2017
model-predictive control (MPC)
Other works aim to solve the constrained MDP more efficiently and better, with Lagrangian methods being applied widely. The Lagrangian multipliers can be a fixed hyperparameter, or adjusted by the algorithm Tessler et al., 2018 Stooke et al. 2020 The policy training might also have issues. The issue that the policy might change too fast so that it's 's no longer safe is addressed by building trust region of policies [Achiam et al. 2017 Zanger et al. 202T] and further projecting to a safer policy (Yang et al. 20200. and another issue of too optimistic policy is addressed by Bharadhwaj et al. (2220] by using conservative policy updates. Expert information can greatly improve the training- time safety. Srinivasan et al. (2020), (Thananjeyan al. [2020] provided offline data, while Turchetta al. [020] is provided interventions which are invoked at dangerous states and achieves zeros safety violations during training.
Returnability is also considered by Eysenbach et al. 2018 in practice, which trains policy to return to the initial state, by Roderick et al. [021 in theory, which designs PAC algorithm to train a policy without safcty violations. [Bansal et al. 2017 gives a brief overview of Hamilton-Jacol Reachability and its recent progress.
Model-based Reinforcement Learning Model-based reinforcement learning, which additionally learns the dynamies model, has gained its popularity due to its superior sample efficiency. KKrutach al. 2018 uses an ensemble of models to produce imaginary samples to regularize leaerning and

---

reduce instability. The use of model ensemble is further explored by Chua et al. different methods to sample trajectories from the model ensemble. Based on Chua et al. (2018). Wang and Ba 2019) combines policy networks with online learning. Luo et al. bound of the policy in the real environment given its performance in the learned dynamics model, and then optimizes the lower bound stochastically. Our work is based on Janner et al. 2019). which shows the learned dynamics model doesn't I generalize well for long horizon and proposes to use short model-generated rollouts instead of full episodes. Dong et al. [2020] studies the expressivity of Q function and model and shows that at some environments, the model is much easier to learn than the Q function.
2018]] which studies
2019 derives a lower
6 Conclusion
We consider the problem of safe exploration in reinforcement learning, where the goal discover a policy that maximizes the expected return, but additionally desire the training process to incur minimal safety violations.
this work, we assume access user-specified function which can be queried to determine whether not a given state is safe. We have proposed model-based algorithm that can exploit this information to anticipate safety violations before they happen and thereby avoid them. Our theoretical analysis shows that safety violations could be avoided with a sufficiently large penalty and accurate dynamics model. Empirically, our algorithm compares favorably to state-of-the model-free safe exploration methods in terms of the tradcoff between performance and total safety violations, and in terms sample complexity.
Acknowledgements
TMM acknowledges support of Google Faculty Award, NSF IIS 2045685, the Sloan Fellowship, and I..com. YI. is supported by NSF, ONR, Simons Foundation, Schmidt Foundation, DARPA and SRC.
References
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International Conference Machine Learning, pages 22-31. PMLR, 2017.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in safety. arXiv preprint wxXi::6606.06665, 2016.
Kavosh Asadi, Dipendra Misra, Seungchan Kim, and Michael L. Littman. Combating the compounding-error problem with a multi-step model. arXiv preprint, abs/1905. 13320, 2019.
URL http:http:/artee org/oagggooo....o Bbergg/b///aa abs/1905.... 1905. 13320
Somil Bansal, Mo Chen, Sylvia Herbert, and Claire J Tomlin. Hamilton-jacobi reachability: A brief overview and recent advances. In 20/7 IEEE 56th Annual Conference on Decision and Control (CDC), pages 2242-2253. IERE, 2017.
Marc G. Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C. Machado, Subhodeep Moitra, Sameera Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning, page 77-82, 2020.
Homanga Bharadhwaj. Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Animesh Garg. Conservative safety critics for exploration. arXiv preprint arXiv:2010.144977, 2020.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn ing in handful of trials using probabilistic dynamics models. arXiv preprint arXiv: 10.111801..111 2018.
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval 2018.
Tassa. Safe exploration in continuous action spaces. arXiv preprint
Kefan Dong, Yuping Luo, Tianhe Yu, Chelsea Finn, and Tengyu Ma. On the expressivity of neural networks for deep reinforcement learning. International Conference on Machine Learning, pages 2627-2637. PMLR, 2020.

---

B Eysenbach. S Gu, J Ibarz, and S Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. 6th International Conference on Learning Representations (ICLR 2018). OpenReview. net, 2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. International Conference on Machine Leurning, pages 1587-1596. PMLR, 2018.
Shixiang Gu, Ethan Holly, Timothy P. Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation. abs/1610.00633, 2016. UtLLtttp:///xxxx.rrr////a//////
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. In International Conference on Machine Learning, pages 1861-1870, 2018b.
Alexander Hans, Daniel SchneegaB, Anton Maximilian Schäfer, and Steffen Udluft. Safe exploration for reinforcement learning. In ESANN, pages 148-148. Citeseer, 2008.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. arXiv preprint urXiv:1906 08253, 2019.
Michael Kearns and Satinder Singh. Netr-optimal reinforcement learning in polynomial time.
Machine learning, 49(2-3):209-22) 2002.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv:/802.10550 2018.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: /509 02990,297,, 2015.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. arXiv preprint 1807v:8807.0338 2018.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In Interna- fional Conference Learning Representations, 2019. WRRL.tttps://pperreeie http://///peeeeeeeeeeeefotoota
JJ= 1E2R5XX
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 51 518(7540):599533, 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Viti,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché- Buc. E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. uRL.tttp: htt.:pppr......pppatee.eeeeeeeep pdf
Melrose Roderick, Vaishnavh Nagarajan, and Zico Kolter. Provably safe pac-mdp exploration using analogies. In International Conference on Artificial Intelligence and Statistics, pages 1216-1224. PMLR, 2021.
10

---

Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha. Jie Tan, and Chelsea Finn. Learning to be safe: Deep rl with safety critic. arXiv preprint 2Xiv:2010.146013 2020.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety reinforcement learning by pid lagrangian methods, In International Conference Machine Learning, pages 9133-9143.
PMLR, 2020.
Aviv Tamar, Garrett Thomas, Tianhao Zhang, Sergey Levine, and Pieter Abbeel. Learning from the hindsight plan episodic mpee improvement. In 2017 IEEE International Conference on Roboties and Automation (ICRA), pages 336-343, 2017. doi: 20.100//CRA
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805.11000 2018.
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. arXiv preprint arXiv:2010. /5920, 2020.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 5026-5033 IEEE, 2012.
Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe rein- forcement learning via curriculum induction. arXiv preprini arXiv:2006. /2136, 2020.
Tingwu Wang and Jimmy Ba. Exploring model-based planning with policy networks. arXiv preprint orXiv:1906.08664 2019.
Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Accelerating safe reinforcement learning with constraint-mismatciie policies. arXiv preprint 1rX1::2006.11645, 2020.
Moritz A Zanger, Karam Daaboul, and Marius Zöllner. Safe continuous control with constrained model-based policy optimization. arXiv preprins arXiv:2/04.066 2021.


Reviews:
[{"review_id": "MCmmZNtbSAr", "reviewer": "Reviewer_Fek3", "summary": "This paper improves sample collection in a model-based exploration scenario, by incorporating a learned reward penalty that prevents unsafe behaviors. The experiments demonstrate improvements in safety during sampling, and sample-efficiency, and policy performance on par with state-of-the-art methods.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Model-based policy optimization uses short rollouts from states in the replay buffer to improve the sample efficiency during learning without harming the final policy performance. This paper expands such benefits, to improve the safety of this model-based learning approach, by incorporating a reward penalty over states that are unsafe. \n\nThis paper is well-written. The domains are well motivated, and the experimental results demonstrate both improved sample complexity and good final performance. I would like to recommend for acceptance.\n\nThat said, it would be good to expand on the number of domains and experiments in the camera-ready of this paper. For example, it would be interesting to see variants of some of the deep mind control suite domains.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours split over two days"}, "extra_text": {}}, {"review_id": "Bv20fBjSqMp", "reviewer": "Reviewer_Fboi", "summary": "The paper deals with one manifestation of safe exploration. Starting from a given safety function that classifies states as safe or unsafe, a modification of MBPO is used to change the Q-function in such a way that unsafe states should not be visited. The method is tested on two benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\\\nThe topic is relevant.\\\nThe paper does not make exaggerated claims.\n\nWeaknesses:\\\nTerms such as safe state and irrecoverable state are introduced, which were introduced very similarly already in [1], without mentioning that and citing [1].\\\nThe experiments are based on only three replicates (3 random seeds).\n\n\nOther comments:\\\nThe expression \"for any irrecoverable state s, any sequence of actions [...] will lead to an unsafe state\" seems to assume deterministic MDPs. In general, stochastic MDPs it depends on random chance which states are visited. It should then rather be a) \"will lead to an unsafe state with a probability greater than 0\" or b) \"will lead to an unsafe state with probability 1\". Where a) is more in line with the concept of super-critical or irrecoverable states.\\\nPage 2, line 54: $\\eta$ is used without having been introduced.\n\nTypos / style:\\\nThere is a missing reference on page 9, line 243\\\n„Startin“ -> „Starting“\\\n„the min“ -> „the minimum“\\\n„leaerning“ -> „learning“\n\n[1] Alexander Hans et al., Safe Exploration for Reinforcement Learning. In Proceedings of the European Symposium on Artificial Neural Network, pages 143–148, 2008.\n\n**After feedback** \\\nThe authors addressed my concerns to my full satisfaction and I agree with the authors that in continuous state spaces even with a restriction to deterministic environments the topic is not trivial. Therefore, I think the paper is a valuable contribution and I increased the score to 6.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "0eNaPcOl3Yl", "reviewer": "Reviewer_hu21", "summary": "This paper studies *safe* reinforcement learning (RL), in which an RL algorithm is tasked with learning a high-value behavioral policy while simultaneously minimizing the number of safety violations. To frame the work, the paper takes on a few key perspectives. First, that predicting only a few timesteps into the future is all that is needed to prevent safety violations. Second, that safety is defined in terms of the occupancy of set of states, as determined by the \\textsc{Unsafe} predicate designed by an engineer. With this predicate in hand, a natural notion of an irrecoverable state is introduced, defined as those states that always lead to an unsafe state under any policy. Third, the paper assumes that environments are deterministic for the main analysis, and captures transition model uncertainty in the form of a next-state-set prediction.\n\nWith these perspectives in hand, the work establishes two primary contributions. The first is the formation of a safe proxy MDP, $\\tilde{M}$, that uses a proxy reward and transition function. The proxy reward produces $-C$ for all unsafe states, where $C$ is a well chosen constant (to be returned to shortly). The proxy transition function replaces any unsafe transition with a self-loop. Hence, in reasoning with this proxy MDP, acting in an unsafe state will produce $-C$ reward indefinitely, thereby accumulating $C / (1-\\gamma)$ value. By consequence, Prop. 1 illustrates that the Q values under the proxy MDP ensure safe actions always have higher values than unsafe actions. Then, section 3.2 takes this idea and extends it to the case when a model is being learned. In particular, it is assumed that the form of transition model learned is a \"set-valued dynamics model\" that maps each $(s,a)$ pair to a non-empty set of possible next states. With this definition of model, two Lemmas are introduced (1 and 2) that illustrate two things: First, that the cutely named \"Bellmin\" operator (which takes the s' achieved by minimizing Q w.r.t. possible next states in the set) is a $\\gamma$-contraction (Lemma 1). Second, that a \"calibrated\" dynamics model (defined in Definition 2 as a set-valued model that always contains the true next-state in its next-state set) will ensure that the proxy Q function upper bounds the set-valued model's Q function. These two lemmas are used to Prove Theorem 1, which states that under a calibrated dynamics model, if there is an action with value greater than $r_{\\min}/(1-\\gamma)$, the greedy policy (w.r.t. $\\underline{Q}$) will choose a safe action.\n\nThen, following this analysis, the paper shifts focus on developing a practical, safe, model-based based algorithm (Algorithm 1, SMBPO). Experiments are conducted in two navigation tasks, hopper, and a cheetah variant, contrasting a variety of RL algorithms (many of which I was unfamiliar with). These include: MBPO, RRL-MF (recovery RL), LR (Lagrangian relaxation), SQRL (Safety Q-functions ), RCPO (Reward constrained policy optimization), and RSPO (Risk-sensitive policy optimization). The main quality explored here is the trade off each algorithm makes between performance and the number of safety violations.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n__Pros: Motivation, Clarity.__ There is a lot to like about this paper. The motivation is clearly articulated and easy to sympathize with: In order for RL agents to avoid safety violations while learning, it is sensible for agents to acquire a forward predictive model that can steer agent's behavior away from unsafe future states. This backdrop ideology frames much of the paper, which makes the definitions, method, and results fit very neatly with each other. I found definitions to be clear and succinct, as were proofs, assumptions, and most of the exposition in the paper. The decision to introduce irrecoverable states was quite sensible.\n\n__Con: Modelling Choices.__ Some aspects of the technical setup, while quite clearly presented, were more difficult to get behind. In particular, I was surprised to see that the introduced definition of an MDP only allows for deterministic transition functions. This seems like a dramatic simplification, particularly in the case of safe RL. If an agent only ever needs to experience a transition once to learn it, why should we worry about the compounding error phenomenon? Moreover, we need not worry about adding extra pseudo-rewards to prevent agents from landing in irrecoverable states, as in order to experience the pseudo-reward the agent will need to take the action that moves it to the irrecoverable state already. Since the transitions are deterministic, this single experience is enough (with or without the pseudo-rewards) to allow the agent to avoid such decisions in the future. For this reason I found the setup of the analysis quite puzzling. Another consequence of this decision to be mindful of is that it does not force a trade-off between safety and value. That is, there is no notion of risk. In a stochastic environment we might imagine that good behavior might trade off with the potential for a safety violation (which do we as designers care more about? Is safety a hard or soft constraint?). I believe these questions are important to address, but due to deterministic transitions they can be avoided entirely as a safety violation is effectively assigned the lowest reward and that's that. Perhaps this is okay, as the work might build toward the general stochastic case, but I did not see any mention of the choice to stick with deterministic dynamics, nor suggestions for how to relaxat to stochastic dynamics in the analysis. This is what puzzled me the most about the paper. Moreover, the environments used in Section 4. (Experiments) are not themselves deterministic, so I found quite a start contrast between the framing from the previous sections and the experimental section.\n\n__Writing Suggestions.__\n\n\n1. Introduction\n- \"spiking.Yet\" --> \"spiking. Yet\"\n\n3. Method\n- I found the definition of \"irrecoverable\" to be slightly confusing in the way that it is worded. I believe the phrase \"for all $t \\in \\mathbb{N}$ satisfies $Unsafe(s_t)$ for some $t \\in \\mathbb{N}$\". Having both quantifiers for some $t$ and for all $t$ is confusing. I see, on parsing this a few more times, it seems as though the first quantifier expresses the construction of the trajectory ($s_{t+1} = T(s_t, a_t)$), while the second quantifier expresses the existence of one unsafe state. I would suggest revising the wording here to make this more clear.\n- Similarly, in assumption 1, it would be useful to explicitly state that $s_0$ is irrecoverable (this happens in the first sentence of the assumption, but not the second). So: \"That is, if $s_0 = s$ is irrecoverable, then...\"\n- In Proposition 1, it is stated that \"if $a$ is a safe action (i.e. $T(s,a)$ is a safe state)\", but as I read it $T(s,a)$ should be a distribution over next states. Ah, I see, the transition function is actually defined to be deterministic. This seems to be quite a major limitation. I would have expected this to be emphasized early on: The compounding error problem cited at the beginning should not really be an issue for deterministic transitions, as the agent only needs to see a single transition per $(s,a)$ pair to learn the model.\n- From Prop 1.'s proof, it turns out that $C$ is actually lower than the minimal reward. This also seems potentially odd: We reserve a particularly low quantity of reward for unsafe states. This seems okay, but it is likely worth calling attention to the fact that C is less than $r_{\\min}$.\n\n- Why consider model-based algorithms that output a set of states? Are you aware of any algorithms that actually do this, and are themselves designed for deterministic environments? Why would this be a valid way to represent uncertainty in a deterministic environment, since a single experience of a transition will fully resolve any uncertainty?\n\n- Some of the references are off. Or, there is an odd convention at play that I found hard to follow. Definition 2 is labeled as such, but is referred to as Definition 3.2 (presumably since it appears in Section 3). I would suggest aligning these names. So either use \\numberwithin to enforce \"Definition 3.2\" to appear, or when referring to the definition, use the listed number.\n\n\n=============\n== Update after rebuttal\n\nIn light of the author's rebuttal, I believe the focus on deterministic-but-continuous environments is indeed suitably interesting. The linked pdf also provides a partial path toward the stochastic setting (though I believe more work is needed to adequately expose and handle risk, exploration, and other considerations). In light of this, I believe the paper will be much improved if it takes time to highlight the nature of the setting, motivating why deterministic-but-continuous environments are an interesting case for safety (and bringing closer attention to the choice of deterministic continuous MDPs). Consequently, I am raising my score from 4 to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As stated, the most apparent limitation of the analysis is its restriction to deterministic transition models. I do not see how the core insights from Theorem 1 translate to the stochastic case, particularly due to the model of uncertainty chosen for the transition dynamics. A more suitable approach would be minimax over the worst case transition function, I suppose.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a model-based policy optimization method that reduces the frequency that unsafe states are visited. All three reviewers recommend acceptance, one strongly. The primary concerns of the reviewers centered around whether the proposed method would extend to interesting problem settings. After the discussion, the reviewers were all convinced that the covered settings are of interest, and the extension to even more settings (stochastic transitions) may be feasible as an avenue for future work. The AC recommends that the authors work to improve the clarity of these points in the paper to ensure that future readers do not run into the same points of confusion that tripped up the reviewers. 
Decision: Accept (Poster)