Look Before You Leap: Safe Model-Based Reinforcement Learning with Human Intervention
Authors: Yunkun Xu
Zhenyu Liu
Guifang Duan
Jiangcheng Zhu
Xiaolong Bai
Jianrong Tan
Venue: CoRL2021 Poster (2021)
Keywords: Safety RL
Model-based RL
Model Predict Control

Abstract: Safety has become one of the main challenges of applying deep reinforcement learning to real world systems. Currently, the incorporation of external knowledge such as human oversight is the only means to prevent the agent from visiting the catastrophic state. In this paper, we propose MBHI, a novel framework for safe model-based reinforcement learning, which ensures safety in the state-level and can effectively avoid both local and non-local catastrophes. An ensemble of supervised learners are trained in MBHI to imitate human blocking decisions. Similar to human decision-making process, MBHI will roll out an imagined trajectory in the dynamics model before executing actions to the environment, and estimate its safety. When the imagination encounters a catastrophe, MBHI will block the current action and use an efficient MPC method to output a safety policy. We evaluate our method on several safety tasks, and the results show that MBHI achieved better performance in terms of sample efficiency and number of catastrophes compared to the baselines.


Look Before You Leap: Safe Model-Based Reinforcement Learning with Human Intervention
Yunkun Xu Zhenyu Liu", Guifang Duan', Jian',,Jaannnnnnn Zhu", Xiaolong , Jianrong Tan Sttte Key Laboratory of CAD&CG, Zhejiang University
"Huawei Cloud
fxuyunkun, liuzy, gfduan egi} @zju.edu.cn, fzhujiangcheng, baixiaolong } @ huawei.com
Abstract: Safety has become one of the main challenges of applying deep rein- forcement learning real world systems. Currently, the incorporation of external knowledge such as human oversight is the only means to prevent the agent from visiting the catastrophic state. In this paper, we propose MBHI, novel frame- work for safe model-based reinforcement learning, which ensures safety in the state-level and can effectively avoid both "local" and "non-local" catastrophes. An ensemble of supervised learners are trained in MBHI to imitate human block- ing decisions. Similar to human decision- maaing process, MBHI will roll out an imagined trajectory in the dynamics model before executing actions to the envi- ronment, and estimate its safety. When the imagination encounters a catastrophe, MBHI will block the current action and use efficient MPC method to output a safety policy. We evaluate our method on several safety tasks, and the results show that MBHI achieved better performance in terms of sample efficiency and number of catastrophes compared to the baselines.
Keywords: Safety RL., Model-based RL, Model Predict Control
Introduction
Deep reinforcement learning (DRL) is proposed as an automated framework for intelligent decision-making problems, and has shown tremendous progress in various domains such as video games [1, 2]. board games [3, 4] and robotic control [5, 6]. However, most of these success were achieved in the idcalized simulator where agents can interact with the environment without restric- tion. In real-world applications, random exploration may lead to equipment damage or even worse results [7]. Consequently, when applied in the real world, DRI. algorithms need ensure the secu- rity throughout the whole learning process, especially in safety-critical scenarios.
This has encouraged the research on safety in DRL, which aims to maximize the cumulative rewards while minimizing safety violations during the training processes [8, 9]. Safe RL can be divided into two categories: vajectory-based [9, 10, II] and state-based [12, 13, 14]. The former uses the cost function to evaluate the safety of the whole trajectory, while state-based methods place restrictions on state-wise basis. Safe RL al- gorithms such as CPO [10] and HIRL [12] have shown that is possible to significantly reduce the number of constraint violations in safety-critical tasks.
Safe Control Rsafe
Learned
Policy ft
Imaginary
Rollout
Unfortunately, most of existing methods have high sample complexity, which means that large number of interactions with the real world are required, resulting in the increase of the number of catastrophe. Another shortcoming is that most of existing approaches use the explicitly defined constraint or safety cost function, while safety is subjective and always vague. For example, in autonomous driv- ing, the overseer brakes when judges that the car going to crash. The overseer makes judgement based on subjective feelings rather than vehicle dynamic model. Finally, it is more strict and natural
Imaginary safety detection and replacement of catastrophic actions in MBHI.
Figure 1:
Corresponding author
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

to define safety over the state. However, most of existing state- state-based methods are hard to effectively
avoid potential disasters in the future [12. 13, 15]. or need suboptimal demonstrations [14, 16].
To address the above-described challenges, in this paper, we introduce novel safety-aware model- based framework, called Safe Model-based Re- inforcement Learning with Human Intervention (MBHI), which incorporates safe RL with model- based methods and human knowledge. Intuitively, before excuting an action, people will estimate po- tential dangers the future. In MBHI, an ensemble of deep neural networks is used to model a super- vised Blocker to imitate the human overseer or ex- plicit constraints, output the catastrophe probability of the state and evaluate the model uncertainty. As shown in Figure I, at each time step, we roll out a trajectory with a certain number of steps in the learned dynamics to check the safety of the agent in the future. If dangerous, the MPC controller will take over and output a safe action to avoid possible future catastrophes in advance. Furthermore, safe active learning is integrated to enable grounded exploration only in safe regions, thereby accelerat- ing model training on the premise of safety.
Policy update
Agent
Med.i-based
RL
Replay Buffer
Dynamies model
Formand collout
Human overseer
Blocker
MPC collout
Catostoohhe probabiliy
MPC Control
Whether intercept ?....
Transition data (r.)
dir
Figure 2: Overall framework of MBHI. The interaction phase shown in blue and the policy learning in red.
To summarize, the primary contributions of our work are as follows: (1) novel safe model-based RL framework with human intervention, (2) safe active learning mechanism that only encourages agents to explore in safe regions, (3) a state-based action correction mechanism that perceive pos- sible catastrophes the future and avoid them in advance, (4) efficient MPC solving method that outputs safe alternative policy in continuous action space.
2 Related Work
Safe RL: An interaction process called safe if the agent rarely accesses the dangerous state [9, 11, 13]. Introducing external knowledge is the only way to ensure safety [12, 17]. Re- cently, various kinds of explicit constraints are used to impose some form of safety measures. To name some, constraints
expectation [9, 10], constraints on uncertainty [7, 18], and Conditional Value-at-Risk [11, 19, 20). Unlike the above trajectory-based safety guarantees, [13] and [21] solve an optimization problem in the policy-level to ensure the safety of states. CARL [22] first trains the agent in non-safety -afect-critiical critical environments, and then adapts to the target environment. In the setting of implicit constraints, HIRL [12] and [15] train a supervised learner imitate the human's 's inter- vention decisions and use it to evaluate the safety of the current state-action pairs. DDPGID [16] and SAVED [14] learn the policy efficiently and safely in the sparse reward environment by using suboptimal human demonstrations. This work also focuses on learning implicit safety constraints from human knowledge. The main difference of MBHI that (1) we image in the leaned dynamics to avoid potential hazards in the future, (2) the human knowledge only contains the judgment about catastrophes, not suboptimal demonstrations, and (3) non-safety-critice source environments are not available.
Model-based RL: Model-based RL exhibits low sample complexity and fast convergence. The learned dynamics model can be used for planning [4, 23], value expansion [24, 25). or imaginary training [26. 27, 28]. In planning approaches. MBMF [29] and PETS [23] use Model Predictive Control (MPC) to select actions based on the learned model. MVE [25] and STEVE [24] incorpo- rate value expansion
improve the targets for temporal difference learning. ME-TRPO [26] and MBPO [27] are directly trained on imaginary data to accelerate policy learning. To limit the influ- ence of the model error, the uncertainty estimation is widely used in MBRL, For instance, PETS [23] use uncertainty-awan tween different imaginary rollout lengths to favor those targets with lower uncertainty. M2AC [30] uses the masking mechanism to only use the model when the model error is small.
models for trajectory propagation and planning. STEVE [24) interpolates be-

---

In this work, STEVE is used as the base learning algorithm, but our framework is generally appli- cable other model-based algorithms with uncertainty quantification, such as MBPO, M2AC, etc. We choose STEVE for three main reasons. Firstly, it explicitly estimates uncertainty for both dy- namics and Q-function. Secondly, if the dynamics model cannot be learned or the Q-value network converges better, STEVE can reduce to model-free algorithm instead of learning from inaccurate imaginary data. Thirdly, since STEVE is an off-policy algorithm, the safety detection process in MBHI can be integrated with value expansion each time-step to reduce calculation amount.
3 Preliminaries
3.1 Markov Decision Processes
We focus Markov Decision Process (MDP) with continuous states and action spaces, defindd as (S,A, Po T), in which is the reward function, Po S - IR+ is the initial state distribution, T S x A + S is the dynamics function, and y [0, 1] is the discount factor. The agent starts from an initial state A in state 8, E S according policy (8r) parameterized with 0, transitions to a successor state 81+1 = (st, at) and receives reward Tt R(s,, de, S1+1). The goal of reinforcement learning is to find optimal policy ge- that maximize the expected discounted sum of rewards l = arg maxy Eea 12==0 r-r.).
the state space, A is the action space,
x S TR.
so Po. at each time step, the agent chooses
action
3.2 Stochastic Ensemble Value Expansion
In model-based RL, the agent learns the world model from the replay buffer to approximate the true dynamics function. STEVE, the dynamics model consists of the following modules:
Transition model 8, [ a1-1)
Reward model : rt ~ a, 8c-1)
Termination model de ~ D& (delss, at, 8141).
An ensemble of parameters is maintained estimate uncertainty in the learned dynamics (i.e. { (5 Nr o { {v: N)) The entire dynamics model is optimized follows:
min B|||s++1 as)ss,,))l" + ||r, Ro(St, ae, st+1)l1 Hid,, Do ae 8141)))
(2)
E.
where the eross-entropy, and the expectation is calculated over data {(s,, ae S441, Tr, d:)).
For actor-critic methods or Q-learning methods, action-value function Qu is critical quantity to guide policy updating. The Temporal-Difference target is written as:
TTP S1+1) = re + 7(1 - dz) d,) (1s411TT(ss)
(3)
where fp means target network. STEVE improves TD targets by combining the short-term value estimate using the dynamics and long-term value estimate using Q.p- An ensemble of Q-functions is also maintained to evaluate the relative uncertainty between the dynamics and Q-function. Through these uncertainty estimates, STEVE dynamically interpolates between imagined rollouts of different horizon lengths, favoring those candidate targets with lower uncertainty, and give up utilizing them when significant errors are introduced. Thus. the interpolated target 1 EVVEV is written as
T.SEEVE 8141) ΣnT Th (TH))
/ (
(4)
h=0
8t() + Σ
Di 2 Rog (8 a4: 844141) 814i1) + Dht17h+1 (814h41, a-n-1):
(5)
1
where D. d - Ds attj. S(+++1),, st-h1 Te ac+h). Tn is the value
expansion target with length h, Ty and TØ² are the empirical mean and variance for each Tí. If the environment is too complex or noise, STEVE will ignore the dynamics and reduce to model-free algorithm immediately.
3

---

4 Methods
In this section, present novel model- safe RL framework, MBHI, shown in Figure 2. In the interaction phase, at each time-step, we use the current policy network xy to roll out multiple steps in the learned dynamics, and check the safety of the imaginary trajectories with the supervised Blocker. If the catastrophe probability of the imaginary future rollout is greater than the threshold, the output action will be intercepted and replaced by the MPC controller. In policy learning phase, the model-based method is used to update the policy. Moreover, to overcome catastrophic forgetting problem of the network [12], we split replay buffer into safe and unsafe categories.
4.1 Catastrophe Prediction Network
Similar to HIRL [12], a supervised learner is trained to imitate the human overseer and block actions that are unsafe, but we maintain an ensemble of parameters for the Blocker μ = { to evaluate the uncertainty. In concrete terms, the human-imitator method has the following advantages. 1) Blocker can learn from data logs, human intervention data or explicit safety constraints by super- vised learning. 2) Blocker is modular, it can be applied in distributed system and different tasks. 3) As binary network, the output value the Blocker can be regarded as the distribution distance between the input state and the catastrophe zone. 4) Blocker can provide gradient information.
During human oversight phase, we store (st, Qe; St+1, Pt, de, ct) at each time-step, where o a binary label for whether the current state is dangerous defined by humans. Thus, the Blocker can be trained a supervised manner by minimizing the cross entropy loss as:
Ly. = H(t,, Bug (St, Ot, 81+1))
1 i Nn-
Each By, is initialized with various weights and trained on different input sequences. At this stage, for model-based RL, the collected data can also be used to train the dynamics using Eq (2). For the sake of fairness, in all experiments, we did not pre-train the dynamics model during the human oversight phase.
4.2 Safe Active Exploration
In order to reduce model uncertainty, we aim to query those points with large entropy, or where the disagreement would be most reduced in posterior model. In our settings, the dynamics model takes the form of ensemble, and the disagreement can be quantified by calculating the empirical variance across the output of each model. Therefore, the optimization objective can be written as:
To -mxx E
Σ ((1-))r
where ea is the coefficient of the active learning reward and À is policy dependent weighting. The second term of Eq can also be regarded as maximizing the differential entropy of the transition model. The model outputs is in the form of empirical Gaussian distribution, its differential entropy can be easily obtained, given by H (x) equivalent maximizing model entropy. It is also possible maximize the variance of the predicted reward, as proposed in [31], but we find it perform poorly in the sparse reward environment.
(In(2r0²) + 1). Therefore, maximizing variance is
However, for safe RL, do not want agents to visit unsafe regions, even if these areas have greater uncertainty. A simple yet effective method is to mask the exploration reward near the dangerous area. The catastrophe probability predicted by the Blocker
be regarded as the distribution distance to the unsafe region. Thus, Eq (7) can be rewritten in the form of safety-aware.
((1- To = max ( - X)ra + Aco 2 (T)(s),):)) - F. (B 0t, 8+11))
(8)
t=
where the exponent controls the masking rate. À determines the level of exploration guided by the learned transition model, should tend towards stability with the convergence of the policy and the full exploration of the environment. In this work, we use the uncertainty measure of the dynamics to
(F)(
adjust À adaptively. Formally, A is defined as λ 2-maxx (2s())) where o; denotes the mean
variance. This restricts À to [0, 0.5) and make the exploration to be guided by the reward signal.

---

4.3 Replacement of Catastrophic Actions
Before the agent execute the action to the environment, a C step trajectory from current state is rolled out in each dynamics. The catastrophe probability of the imagination is computed as follows.
p max Ei-Na.j~N,r [B (6 mo(%) Te, (,, To -)) eEC
If p is greater than the safety threshold, it is considered that the agent will have disaster in certain number of steps under the current policy, and necessary to replace the action in advance. In HIRL [12], the author selects the alternative action based on a lookup table or the logit score ranking. However, these methods are not suitable for continuous action space and "non-local" catastrophes. In this work,
Control is used as expert to output safe actions. Given state St, the MPC prediction horizon H and the action sequence H1:++-11, each dynamics model is used to predict the resulting trajectories ((t++i:(t+i-))) 11) At each timestep t, the MPC controller solves the following optimization problem
max OlE
AMPC(SLth atth, S144+1)
s.t. Sith+1 Te(ssth: (10)
where ÍMC is the MPC reward. There are two requirements of it. First, guide the agent com- task. Second, guide the agent away from the unsafe regions. In
plete the consequence, PM contains the predicted external reward and the predicted disaster probability. In addition, further introduce Leave-One-Our safety estimation to make MPC samples far away from the unsafe region as possible. Given the unsafe dataset Dus and the MPC sampled state The value of ue+h is close to zero, when p(- (Dus U 8eth) and and (Ps)) gradually match each other, which indicates that the sampled state is dangerous. Therefore, we aim to maximize Σ U++a to keep the whole MPC sampled sequence away from danger. The MPC return can be formulated as
(D...p(.IP.... St+h, the estimator is defined u4+h Dx [p(-) U Se+A (lp(p))pppP))))))
PMMPC(5th: Oc+hi Sc+h+1) E6 R&g (Stthy asths 9144+1]]+1,, [B,,(B+tt ah S1+ )] uc+h++
(11)
Finally, the controller executes the first action a₂ the environment, advances the next time-step. and recheck the safety of the agent.
To improve the efficiency and performance of MPC, especially in high-dimensional action space, as shown in Figure 3, the gradient of the dynamics is used to guide the update of CEM sampling distribution. Each iteration consists of three steps: (1) Sample action se- quences from CEM sampling distribution, and calculate MPC return with Eq (11). (2) Fix the dynamics parameters, and update action sequences via gradient ascent method. (3) Up- date the parameters of CEM sampling distri- bution. The action replacement mechanism is summarized in algorithm in Appendix A.
corplin (aaaa-a Free
CEM sampling distribution
Upskte (fooe i with BP
Uider sampling distitation
Uplated insepting (tian- ae)
Figure 3: Schematic of MPC optimization method.
negative reward must be given to guide the agent to learn a safe policy. The interception mecha- nism in MBHI considers the danger of the current policy in state 8, rather than the state-action pair. Thus, the negative reward should solely depend on the state. In this work, an intrinsic reward signal generated based on how far the agent is from the unsafe region to encourage the agent to stay away from these areas. Specifically, the output of Blocker is served as this kind of measurement. Besides, increase the punishment near the unsafe area, a boundary value Bound is introduced. When the predicted catastrophe probability is greater than this value, a larger scaling factor is used increase the negative reward. The intrinsic reward signal is computed as
EB, at, s141)] if E, (S at 8(41)] Bound else,
('(s ar St-1) =
(12)
E,, (B, 0e 81+1)]
where & and on are scaling factors, which control the penalties in different areas. Bound is the safety bound, a smaller bound will increase the complexity of the environment and obtain a more
5

---

conservative policy. Finally, the overall optimization problem can be written as
0* arg max E (Fx)) ( ()(re) (5 4r S(+1)) + Ar" (5e Oe: St+1))]
(13)
0
where r is the active learning reward in Eq (8). The full procedure is outlined in Algorithm 2 in Appendix A.
Experiments
5.1 Experiment Setting
In this section, we evaluate our approach and various baselines over five safety-critical continuous control tasks. 1) PuckWorld-L & PuckWorld-H: they differ in acceleration. The catastrophe of PuckWorld-H is locally avoidable, while PuckWorld-L is not. The agent rewarded for reaching the target and is constrained to not touch the barrier. 2) Reacher: the agent is rewarded for reaching target without touching the vertical bar. 3) Ant-Limit: the agent needs to move forward between two parallel barriers. 4) Ant- Ant-Block: the agent needs to move forward and avoid obstacles in the path. The last three tasks are adapted from PyBullet's 's environments [32]. The visualization and details of these environments are given in Appendix B. All experiments
evaluated over random seeds.
We compare our proposed MBHI against both the model-based and model-free baselines, PPO, DDPG and STEVE, and human intervention method, HIRL [12]. we extend HIRL to the continuous action space and also use STEVE as the basic algorithm. To ensure fair comparison, HIRL and MBHI have the same Blocker and action replacement mechanism. HIRL also splits the replay buffer like MBHI. The final implementation differences between HIRL and MBHI are summarized as follows: (1) HIRL only receives negative reward when catastrophic action is blocked, while penalty is introduced in MBHI in the form of Eq (12); (2) HIRL does not employ active exploration; (3) HIRL does not image in the learned dynamics to detect potential disasters in the future. In all experiments, we use the same network structure of the policy, value, dynamics and Blocker for all algorithms, the number of ensemble models is 4. Details can be found in Appendix H.
5.2 Comparison of Performance
New Maril
/02
-00
113
(8)
(c)
MBHI
HIRL
STEVE
DDPG
PPO
(d)
(0)
Figure 4: Learning curves of MBHI and baselines on five continuous control tasks over time (mean and standard deviation). PorkWWrl (b) PuckWorld-IH, (c) Reacher, (d) Ant-Limit, (e) Ant-Block. The x-axis is environment steps.
The comparison results in Fig 4 demonstrate that the sample efficiency of MBHI is better than baselines, especially DDPG and PPO. MBHI achieves similar better performances in all tasks

---

2004
60
12
16
(a)
(c)
MBHI
HIRL
STEVE
DDPG
PPO
(d)
(0)
Figure 5: Cumulative catastrophes of MBHI and baselines on five continuous control tasks over time (mean and standard deviation). (a) PuckWorld-L. (b) PuckWorld-H, (c) Reacher, (d) Ant-Limit, (c) Ant-Block. The x-axis is environment steps.
than baselines. In Puck WorldH,, the parameters of MBHI and HIRL in safety inspection are exactly the same, but MBHI converge faster, which illustrates the importance of active exploration. Frequent action shielding increases the complexity of policy learning. However, as an off-policy method, we do not find perceptible negative effects for MBHI.
The experiment results of STEVE and DDPG in Fig 5 demonstrate that optimal unsupervised behav- iors result in large number of catastrophes. Note that DDPG and PPO has fewer catastrophes in some environments because the agent has not learned how to complete the task, and tends to move near the initial state. For example, in Ant- Ant-Block, the agent does not successfully move to the place where it could encounter the obstacle, which can explain why the number of catastrophes of PPO suddenly rises in Fig 5e. Additionally, HIRL is able successfully resolve locally avoidable catastrophes, but failed in other environments. By contrast, MBHI effectively avoids both "local" and "non-local" catastrophes during training. In the simple environment like PuckWorld, MBHI almost achieves zero catastrophes.
In tasks other than WorlddH,, HIRL cannot effectively save the agent. However, from Figure 5, we can see that the number of catastrophes HIRL is significantly less than STEVE. This is because divide the replay buffer in HIRL into safe and unsafe parts. Besides, we can also find that even though STEVE has converged, the number catastrophes is still rising. Our findings are in line with recent results catastrophic forgetting [12, 33). MBHI can well restrain this problem.
5.3 Ablation on Safety Detection Rollout Length
Intuitively, the longer the safety detection distance, the more conservative the policy. We evaluate different safety detection rollout lengths on Puck and PuckWorld-H The result is shown in Figure 6. It shows that, surprisingly, when the safety detection distance is sufficient, our method is not so sensitive to the length of safety detection rollouts. This also proves that safety detection in the imagination can effectively avoid the catastrophe that will happen in advance. In addition, the long detection distance does not further reduce the cumulative catastrophes significantly, but it will introduce unnecessary calculation costs.
5.4 Comparison of Action Replacement Methods
We study the impact of different MPC optimization techniques and reward settings. The compared optimizers include the uniform random search [29] and the cross-entropy method [23]. The MPC
7

---

60
12
24
Figure 6: Ablation studies of the safety detection rollout length. (a) Average return in PuckWorld-L: (b) Average return Puck World-H; (c) Cumulative catastrophes. Error bars are 95% bootstrap confidence intervals.
Random
CEM
Grad+CEM (r)
GradHCEM (rBB)
GradHCEM (r-Bru)
(n)
(b)
Figure 7: Interception suecess rate of different methods. (a) Converged dynamics model, (b) pre-trained dy-
namies model. For fair comparison, random shooting sampled 3200 times, CEM has 25 iterations of sampling 128 candidate actions, and Grad+CEM has iterations plus gradient steps of sampling 128 candidate actions. Shown is mean performance, with enror bars showing 95% bootstrap confidence intervals over runs.
reward settings include three types: 1) the reward ê predicted by the learned reward function; 2) and the catastrophe probability B predicted by the Blocker; f, B, and the distribution distance between MPC samples and unsafe samples. For the sake of fairness, MPC reward in Random and CEM. We sample ten points uniformly around the catastrophe as the initial position, and make the agent converged dynamics model and the pre-trained model are used as the simulator of MPC to evaluate the robustness of the proposed method. The results of the interception success rate shown in Figure 7. It shows that our method has better interception performance and is more robust.
also use Ñ + B + the
toward the catastrophe the fastest speed. In addition, the
6 Conclusion
In this paper, we proposed state-based safety oversight mechanism MBHI, which is based on the model-based RL algorithm with active learning method. MBHI is forward-looking, it considers the safety of the current policy a certain time-step the future, replaces dangerous actions in advance, and realizes the prevention of both "local" and "non-local" catastrophes. We evaluated the proposed algorithm on five continuous safety control tasks, the experiment results demonstrate that MBHI can significantly reduce catastrophes and accelerate training. Furthermore, solution acts directly at the policy level, so is independent the learning algorithm and can plugged into any existing model-based and model-free continuous control algorithms.
MBHI effectively alleviates the problem of "non-local" catastrophes, but it needs to have a certain understanding of the task. The safety detection distance needs to be long enough to avoid catastro- phes in advance. Besides, hyperparameters are sensitive the performance of the Blocker. Going forward, the safety detection length should be adaptive to the specific task, rather than set fixed value. Moreover, in future research, we plan to study the theoretical guarantee MBHI and improve the algorithm's is robustness to the hyperr-arameteer. In terms of computation, since safety evaluation inevitably increases the cost of training process, we plan to study more effective methods to estimate the potential danger of the agent in the future and to integrate these estimates in model-based policy optimization.

---

Acknowledgments
We would like to thank Yang Li and our anonymous reviewers for their insightful comments and suggestions, This work was supported in part by the National Key Research and Development Pro- gram of China (grant No. 2019YFBI 10YFB131100 312600), in part by the National Natural Science Foundation of China (grant No. 52075480), in part by the High-level Talent Special Support Plan of Zhejiang Province, China (grant No. 2020R52004) and in part by the Natural Science Foundation of Zhejiang Province, China under Grant Y19E050078.
References
[1] S. Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney. Recurrent experience replay in distributed reinforcement learning. In ICLR, 2019.
[2] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 2013.
[3] D. Silver, J. Schrittwieser, K. Simonyan, Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354 359, 2017.
[4] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, Graepel, T. Lillicrap, and D. Silver. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588 7839:604- 11066006699906 2020.
[5] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: (ate::/09900999 2015.
[6] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
[7] B. Lütjens, M. Everett, and J. How. Safe reinforcement learning with model uncertainty esti- mates. 2019 International Conference on Robotics and Automation (ICRA), pages 8662-8668, 2019.
[8] J. Garcia and F. Fernández. A comprehensive survey on safe reinforcement learning. J. Mach. Learn. Res., 16:1437-1480, 2015.
[9] J. Fan and W. Li. Safety-guided deep reinforcement learning via online gaussian process estimation. ArXiv, abs/1903. 020526, 2019.
[10] J. Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained policy optimization. In International Conference on Machine Learning, pages 22-31, 2017.
[11] A. Cowen-Rivers D. Palenicek. V. Moens, M. Abdullah, A. Sootla, J. Wang, and H. Ammar. Samba: Safe model-based & active reinforcement learning. ArXiv, abs/2006.09436, 2020.
[12] W. Saunders, G. Sastry, A. Stuhlmüller, and O. Evans. Trial without error: Towards safe reinforcement learning via human intervention. In AAMAS, 2018.
[13] G. Dalal, K. Dvijotham, M. Vecerík, Hester, C. Paduraru, and Tassa. Safe exploration in continuous action spaces. ArXiv, abs/1801
2018.
[14] B. Thananjeyan, A. Balakrishna, U. Rosolia, F. Li, R. McAllister, J. Gonzalez, S. Levine, Borrelli, and K. Goldberg. Safety augmented value estimation from demonstrations (saved); Safe deep model-based rl for sparse cost robotic tasks, IEEE Robotics and Automation Letters, 5:3612-3619 2020.
[15] B. Prakash, M. Khatwani, N. R. Waytowich, and T. Mohsenin. Improving safety in reinforce- ment learning using model-based architectures and human intervention. FLAIRS Confer- ence, 2019.

---

[16] M. Vecerík, T. Hester, J. Scholz, F. Wang. O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. A. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. ArXiv, abs/1707.08817, 2017.
[17] M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer. S. Niekum, and U. Topcu. Safe reinforce- ment learning via shielding. In AAAI, 2018.
[18] G. Kahn, A. Villaflor, V. H. Pong. P. Abbeel, and S. Levine. Uncertainty-aware reinforcement learning for collision avoidance. ArXiv, abs/1702.01182, 2017.
[19] A. PrashanthL. Policy gradients for cvar-constrained mdps. ALT, 2014.
[20] Y. Chow, M. Ghavamzadeh, L. Janson, and M. Pavone. Rosttcons reinforcement leamn ing with percentile risk criteria. ArXiv, abs/ abs/1512.01629, 2017.
[21] T. Pham, G. Magistris, and R. Tachibana. Optlayer- practicall constrained optimization for deep reinforcement learning the real world. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6236 6243,,623,222 2018.
[22] J. Zhang, B. Cheung, C. Finn, S. Levine, and D. Jayaraman. Cautious adaptation for rein- forcement learning in safety-critical settings. H. D. III and A. Singh, editors, Proceed- ings of the 37th International Conference on Machine Learning, volume 119 of Proceed- ings of Machine Learning Research, pages 11055-11065. PMLR, 13-18 Jul 2020. URL btttps//proceediine
mlr ptwss//119//aaaggggeeer httml. httl..
[23] K. Chua, R. Calandra, R. McAllister, and Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In NeurlPS, 2018.
[24] J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee. Sample-efficient reinforcement learning with stochastic ensemble value expansion. In NeurIPS, 2018.
[25] V. Feinberg. A. Wan, I. Stoica, M. Jordan, J. E. Gonzalez, and S. Levine. Model-based value estimation for efficient model-free reinforcement learning. ArXiv, abs/1803.00101, 2018.
[26] T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization. In ICLR, 2018.
[27] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. In NeurIPS, 2019.
[28] G. Kalweit and J. Boedecker. Uncertainty-driven imagination for continuous deep reinforce- ment learning, In CoRL, pages 195-206, 2017.
[29] A. Nagabandi, G. Kahn, R. Fearing, and S. Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning, 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 7559-7566, 2018.
[30] F. Pan, J. He, D. Tu, and Q. He. Trust the model when it is confident: Masked model-based actor-critic. In NeurlPS, 2020.
[31] P. Ball, J. Marker-Holded A. Pacchiano, K. Choromanski, and S. Roberts. Ready policy one: World building through active learning. In ICML, 2020.
[32] E. Coumans and Y. Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http::/pybuulee org, 2016-2019.
[33] R. Kemker, A. Abitino, M. McClure, and C. Kanan. Mcasuring catastrophic forgetting in neural networks. In AAAI, 2018.
10


Reviews:
[{"review_id": "QIztJBh3FMS", "reviewer": "Reviewer_rwsy", "summary": "This paper presents a method to avoid potential future catastrophes in a reinforcement learning setting.\nIt uses the framework STEVE because it explicitly estimates the uncertainty for the dynamics but any other MBRL with uncertainty quantification could be used. \n\nThe method works by checking for failures in the imaginary rollouts and then replacing the faulty actions with an MPC controller. The catastrophe prediction is done with an ensemble of networks trained by supervised learning with human labels, called the Blocker Network.\nThe loss for model learning consists of:\n- the loss used in STEVE\n- an additional cross entropy loss for their binary network which classifies whether the current transition is safe or not (supervised learning with human labels)\n\nThe expected reward optimized by the policy network consists of three terms:\n- the standard reward from the environment \n- an exploration term (model entropy) that is proportional to the probability of *not* being in a dangerous region\n- a safe-learning reward that explicitly penalizes unsafe areas\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nThis method is able to avoid non-local catastrophes, meaning that the decision to avoid failure is done many steps ahead rather than right before the critical point. \nThe single components are not novel but their combination is novel and works better than the presented baselines.\n\n### Weaknesses\nThe method consists of many components and it is not very easy to put everything in place, the presentation is a bit confusing and it could be improved. \n\n\nI believe the authors could compare their work to other baselines that also employ active exploration.\nBy reading the paper it seems that this work is the first one to have both active exploration and safety awareness. I would rephrase the introduction/related work and add more comparisons to existing methods. The presented model-based ones are more an ablation of MBHI.\nThe authors could better summarize the necessary additions - that are now scattered in the paper - and present some take home messages. For example, if separating the buffer is an important addition (which is briefly mentioned in line 244).\n\nAdditional comments listed in Issues.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I believe the paper could be better organized and optimized for clarity. Some details are missing and should be included. Some baselines are reported in the results but nothing that specifically includes active exploration and risk awareness. \nI think the paper has potential: the idea of avoiding future catastrophes in advance and to include human intervention is valid. I will modify my score upon improvement of the paper.\n\nEdit: Updated to Strong Accept after rebuttal", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Regarding Figure 9 in the Appendix: what we see are the replaced actions or the original ones? What are the unsafe areas in these cases? \n\n\nYou don’t really show how accurate the blocker network is. It could be interesting to add a plot with the labeled areas and compare to the regions where actions are replaced. As a reader, it is not clear what the labels are. Are only the obstacles labeled as unsafe?\n\nIt is not clear to me if the obstacles are modeled by the dynamics.\n\nInstead of using a blocker network, did you try explicitly adding a penalty cost as a distance to the obstacles?\n\nWhat is the final difference between your modification of HIRL and MBHI? Is it the blocker network and the buffer split? This information should be made explicit.\n\nLine 158, what do you mean by “asymptotically consistent”?\nLine 162, did you try to plot lambda over time? It could be interesting to see.\nLine 189, it is mostly clear what is the use of the KL divergence term, namely to sample away from unsafe regions, but it could be better explained.\n\n\nMany typos, please correct.\ne.g.:\nline 43: the catastrophe of locally avoidable\nor line 260: sampling samples\nLine 263: and the model only be pre-trained\n"}}, {"review_id": "2S_dRZxa0C", "reviewer": "Reviewer_wEtd", "summary": "The paper proposes a method which utilizes a policy guided by dynamics and value-function predictions and uncertainties over unsafe states for safer exploration and learning.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Paper Strengths**\nTechnical contribution: The method is an amalgamation of different ideas that results in a well-performing algorithm which demonstrates improvements over a number of baselines. While no part itself is particularly novel, the combination of everything makes the method a novel contribution.\n\nExperiments: Experimental evaluation was solid with respect to the baselines and environments tested. The method performs well in terms of catastrophe avoidance, and offers mixed to strong improvements over baselines on final reward.\n\nPaper Writing: Given the complexity of the method, the method is generally described well.\n\n**Paper Weaknesses**\nMissing related work: Cautious Adaptation for RL in Safety-Critical Settings (Zhang et al 2020) also targets state-based safety through an ensemble of dynamics models with MPC planning, and SAVED (Thananjeyan et al 2020) performs safe RL from human demonstrations, with a similar value-function and dynamics-model based cost function. These are important recent works the authors should cite and contrast their method with in the Related Works/intro section.\n\nClaim: “However, existing state-based approaches only consider the catastrophe of locally avoidable, that is, catastrophes can be avoided by choosing the appropriate action just before the catastrophe would have happened.” I don’t think this can be stated as fact. Model-based methods with MPC planning will be able to avoid catastrophes over longer horizons as long as the horizon of the planner is long enough. This is even more true for Dynamics + Value-function based const functions which exist in both STEVE and the aforementioned SAVED work.\n\nClarity: $T_{STEVE}^H$ seems to come out of nowhere (Eq 5) and its purpose is not explained immediately, and $T_h^\\mu$ also should be mentioned.\n\nAblations: The authors introduce a method that has many different design choices, such as the leave-one-out estimation, gradient updates for CEM, having c_l and c_h, the safety bound, etc. There should be more ablation studies and sensitivity analysis on these parameters in general.\n\nExperiments: I can’t find where the paper mentions the number of random seeds the results are reported over. And as hinted to in the related works weaknesses, there are baselines that can be extended to work on the scenario in the paper that would be nice to compare against.\n\n\n**Minor Issues**\nTypo: Safe Active Explorationm (4.2), Puck-Workd (L214),\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend acceptance of this paper due to the solid performance and the technical contribution of the method. However, I do have some issues listed in the paper weaknesses section that I would like to be addressed during the rebuttal.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See weaknesses/minor issues section.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "2JdkIlXGJXC", "reviewer": "Reviewer_QSAz", "summary": "The paper presents a way to perform model-based RL with catastrophe action blocking and active correction. It builds on the MBRL algorithm STEVE (which comes with uncertainty quantification), and also human-in-the-loop method HIRL (which trains a human blocker model), with the additional technique of unrolling imaginary trajectories and optimize the future trajectory. The algorithm is able to outperform several baselines in catastrophe prevention and reward maximization. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think the main strengths of the paper are: \n- Novel technique of unrolling imagined trajectory\n- Good literature review of the field\n- Promising experimental results\n\nThe weaknesses:\n- Experiment details: There could be more details on the algorithm and experiment settings. For example, it is unclear what's the human workload during the human supervision phase.\n- Baseline comparison: Since this method involves human blocking, there could be other types of human-in-the-loop algorithms serving as baseline comparisons. For example, instead of naive DDPG, DDPGfD* might also serve as a fair comparison since that involves human demonstrations rather than just RL from scratch.\n\n* Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards\nMel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, Martin Riedmiller", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I would recommend the paper because of its insight and novelty, the introduction of human supervision and blocking in MBRL and extensive experimental results. It could potentially benefit from more algorithmic details and baseline comparisons as mentioned above but overall I think it is an insightful work.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "- Has the authors considered other types of human-in-the-loop algorithms besides HIRL for catastrophe detection? Would be curious for any other existing work in the field\n- More details could be given in the Appendix / supplementary materials section on algorithm and experiment details.\n"}}, {"review_id": "0q6fNhOwyrw", "reviewer": "Reviewer_VQ9u", "summary": "This work deals with the safe RL problem.  The authors propose a method where a model of the environment is learned and used to simulate fixed-length rollouts from a current state.  With these rollouts, a supervised learning framework is derived where it is predicted if the rollout is dangerous (based on human-defined labels).  If the rollout is dangerous, MPC control is used to avoid the dangerous state (select safe actions).  The work is highly relevant to robot learning and is validated on the standard suite of MuJoCo safe RL environments.  Compared to other models such as HIRL, this work deals with longer simulated trajectories rather than a single transition so it can reason about non-local danger.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This method is well validated on highly relevant tasks where safety is an important aspect and results are promising.  This method performs well in terms of avoiding catastrophes, but performance in terms of return is not highly significant in most tasks compared with a baseline model-based approach (especially in the non-local catastrophe environment).  There is an absence of real robotic simulations, but I do not think this diminishes the quality of the work. The ablation experiments are useful and highlight the specific impact of the method, though more could be added on parameter sensitivity.\n\nThe work appears to be sound in terms of the method. The improvements are simple/incremental over HIRL but the work is shown to be empirically useful for the problem it attacks, avoiding non-local danger.  It is validated on the non-local danger tasks explicitly.  The novelty of the method overall is lacking, but this work combines many different models to form a useful and relevant method.\n\nMore comparisons against traditional safe RL algorithms could be present, such as a policy optimization method with conditional value at risk objective (CVaR). Comparing against CVaR would make a useful comparison against a method not requiring human intervention, which is difficult to obtain.  I would also have liked to see some comparisons against methods with more constrained updates such as TRPO or PPO as opposed to DDPG.  \n\nThe text is generally polished.  There are a few minor details to point out: \n- The authors alternate between the notation s’ and s_{t+x} between the method and related work section.  \n- \\alpha in (9) is superscript and appears to be a power instead of a weight.\n- In (14) tau is written instead of the symbol.\n- The parameters for each part of the dynamics model could be combined into a single set with its own symbol?\n- Notation could be simplified at many steps such as (13) where the bound is not stated.  \n- It is unclear how many random seeds are used.\n\nOverall, the visuals are very useful in understanding the method. The paper reads clearly.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this work is borderline.  The method is relevant to CoRL but is lacking some key evaluation baselines and novelty in general. The authors should be able to make minor adjustments to the text and overall, the paper is relatively easy to understand.  \n\n**Post Response**:  While I still think this work is borderline, due to the revisions from the authors I am now leaning towards acceptance.  I believe the additional baseline and ablation analysis solidify the usefulness of the method and believe the paper should be \"accepted if there is room\".", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See above", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a safe reinforcement learning method, which switches to a safe MPC controller if unsafe events are predicted to happen. Reviewers generally agree that the paper makes good technical contributions to the important problem of safe learning. While most of the components of this algorithm are not novel, reviewers acknowledge that the combination of them all is novel and useful. The original concerns from reviewers about presentation, literature reviews, and baselines comparisons are well addressed in the rebuttal.
Decision: Accept (Poster)