Guided Safe Shooting: model based reinforcement learning with safety constraints
Authors: Giuseppe Paolo
Jonas Gonzalez-Billandon
Albert Thomas
Balázs Kégl
Venue: Submitted to ICLR 2023 (2023)
Keywords: Model-based Reinforcement learning
Safe-RL
Evolutionary method
Planning

Abstract: In the last decade, reinforcement learning successfully solved complex control
tasks and decision-making problems, like the Go board game. Yet, there are few
success stories when it comes to deploying those algorithms to real-world scenarios.
One of the reasons is the lack of guarantees when dealing with and avoiding unsafe
states, a fundamental requirement in critical control engineering systems. In this
paper, we introduce Guided Safe Shooting (GuSS), a model-based RL approach
that can learn to control systems with minimal violations of the safety constraints.
The model is learned on the data collected during the operation of the system in an
iterated batch fashion, and is then used to plan for the best action to perform at each
time step. We propose three different safe planners, one based on a simple random
shooting strategy and two based on MAP-Elites, a more advanced divergent-search
algorithm. Experiments show that these planners help the learning agent avoid
unsafe situations while maximally exploring the state space, a necessary aspect
when learning an accurate model of the system. Furthermore, compared to model-
free approaches, learning a model allows GuSS reducing the number of interactions
with the real-system while still reaching high rewards, a fundamental requirement
when handling engineering systems.

GUIDED SAFE SHOOTING: MODEL BASED REINFORCE- MENT LEARNING WITH SAFETY CONSTRAINTS
Anonymous authors Paper under double-blind review
ABSTRACT
In the last decade, reinforcement learning successfully solved complex control tasks and docision-making problems, like the Go board game. Yet, there few success stories when comes to deploying those algorithms to real-world scenarios. One of the reasons is the lack of guarantees when dealing with and avoiding unsafe states, a fundamental requirement in critical control engineering systems. In this paper, we introduce Guided Safe Shooting (GuSS), model-based based RI approach that can learn to control systems with minimal violations of the safety constraints. The model learned on the data collected during the operation of the system in an iterated batch fashion, and is then used to plan for the best action to perform at each time step. We propose three different safe planners, one based on simple random shooting strategy and two based on MAP-Elites, more advanced divergent-search algorithm. Experiments show that these planners help the learning agent avoid unsafe situations while maximally exploring the state space, necessary aspect when learning an accurate model of the system. Furthermore, compared to model- free approaches, learning model allows GuSS reducing the number of interactions with the real-system while still reaching high rewards, a fundamental requirement when handling engineering systems.
INTRODUCTION
Initial state
In-modd planning phuse
Tesjacory excrtion
Figure 1: An illustrative example of planning with model-based approach on the Acrobot environment. The agent controls the torque on the first joint with the goal of getting its end effector as high as possible, avoiding the unsafe zone (red area). Starting the rest position (left) the agent uses its model to find the best plan (middle) that will maximize the reward while satisfying the safety constraint and execute on the real system (right). The example is especially relevant to applications in which safety and reward are traded off.
In recent years, deep Reinforcement Learning (RL) solved complex sequential Decision-making problems in a variety of domains, such controlling robots, and video and board games (Mnih et al., 2015; Andrychowicz et al., 2020; Silver al., 2016). However, in the majority of these cases, success is limited to simulated world. The application of these RL solutions to real-world systems is still yet to come. The main reason for this gap is the fundamental principle of RL of learning by trial and error maximize a reward signal (Sutton & Barto, 2018). This framework requires unlimited access to the system to explore and perform actions possibly leading to undesired outcomes. This is not always possible. For example, considering the task of finding the optimal control strategy for a data center cooling problem (Lazic et al., 2018), the RL algorithm could easily take actions leading high temperatures during the learning process, affecting and potentially breaking the

---

system. Another domain where safety is crucial is robotics. Here unsafe actions could not only break the robot but could potentially also harm humans. This issue, known as safe exploration, is a central problem in Al safety (Amodei et al., 2016). This is why most achievements in RL are in simulated environments, where the agents can explore different behaviors without the risk of damaging the real system. However, those simulators not always accurate enough, if available at all, leading to suboptimal control strategies when deployed on the real-system (Salvato et al., 2021).
With the long-term goal of deploying RL algorithms on real engineering systems, it imperative to overcome those limitations. A straightforward way to address this issue is to develop algorithms that can be deployed directly on the real system that provide guarantees in terms of constraints, such as safety, to ensure the integrity of the system. This could potentially have great impact, as many industrial systems require complex decision-making, which efficient RI. systems can easily provide. Going towards this goal, in this paper we introduce Guided Safe Shooting (GuSS), a safe Model Based Reinforcement Learning (MBRL) algorithm that learns a model of the system and uses it to plan for safe course of actions through Model Predictive Control (MPC) (Garcia et al., 1989). GuSS learns the model in an iterated batch fashion (Matsushima et al., 2021; Kégl et al., 2021), allowing for minimal real-system interactions. This is desirable property for safe RL approaches. as fewer interactions with the real-system mean less chance of entering unsafe states, a condition difficult to attain with model-free safe RL methods (Achiam et al., 2017; Ray et al., 2019b; Tessler et al., 2018). Moreover, by learning a model of the system, this allows flexibility and safety guarantees as using the model we can anticipate unsafe actions before they occur. Consider the illustrative example in Fig. 1: the agent, thanks to the model of its dynamics, can perform "mental simulation" and select the best plan to attain its goal while avoiding unsafe zones. This contrasts with many of the methods in the literature that address the problem of finding safe course of action through Lagrangian optimization or by penalizing the reward function (Webster & Flach, 2021; Ma al., 2021: Cowen- Rivers et al., 2022). GuSS avoids unsafe situations by discarding trajectories that are deemed unsafe using the model predictions. Within this framework, we propose three different safe planners, one based on simple random shooting strategy and two based MAP-Eliin (ME) (Mouret & Clune, 2015), a more advanced Quality-Diversse and select the safest actions with the highest rewards. Using divergent-search methods for planning allows the agent to more widely explore the possible courses of actions. This leads to both a safer and more efficient search, while covering higher portion of the state space, an important factor when learning model, given that more exploratory data lead to better models (Yarats et al., 2022).
(QD) algorithm. These planners are used to generate, evaluate,
We test GuSS on three different environments. The presented results highlight how the model and planners can easily find strategies reaching high rewards with minimal costs, even when the two metries are antithetical, as is the case for the Safe Acrobot environment.
To recap, the contributions of the paper are the following:
We introduce Guided Safe Shooting (GuSS), an MBRL method capable of efficiently learning to avoid unsafe states while optimizing the reward;
We propose the of quality-diversity evolutionary methods as MAP-Elites (ME) as planning techniques in MBRL approaches;
We present 3 different planners, Safe Random Shooting (S-RS), Safe MAP-Elites (S-ME), Pareto Safe MAP-Elites (PS-ME), that can generate wide array of action sequences while discarding the ones deemed unsafe during planning.
2 RELATED WORK
Some of the most common techniques addressing safety in RL rely on solving a Constrained Markov Decision Process (CMDP) (Altman, 1999) through model-free RL methods (Achiam et al., 2017; Ray et al., 2019b; Tessler et al., 2018; 8; Hsu et al., 2021; Zhang et al., 2020). Among these approaches, a well-known method is CPO (Achiam et al., 2017) which adds constraints to the policy optimization process in a fashion similar to TRPO (Schulman et al., 2015). A similar approach is taken by PCPO (Yang et al., 2020b) and its extension (Yang et al., 2020a). The algorithm works by first optimizing the policy with respect to the reward and then projecting it back on the constraint set in an iterated two-step process. A different strategy consists in storing all the "recovery" actions that the agent took to leave unsafe regions in separate replay buffer (Hsu al., 2021). This buffer is used whenever
2

---

the agent enters an unsafe state by selecting the most similar transition in the safe replay buffer and performing the same action to escape the unsafe state.
Model-free RL methods need many interactions with the real-system in order to collect the data necessary for training. This can be a huge limitation in situations in which safety is critical, where increasing the number of samples increases the probability of entering unsafe states. MBRL limits this problem by learning a model of the system that can then be used to learn a safe policy. This allows increased flexibility in dealing with unsafe situations, even more if the safety constraints change in time.
Many of these methods work by modifying either the cost or the reward function to push the algorithm away from unsafe areas. The authors of Uncertainty Guided Cross-Entropp Methods (CEM) (Webster & Flach, 2021) extend PETS (Chua et al., 2018) by modifying the objective function of the CEM- based planner to avoid unsafe areas. In this setting, an unsafe area is defined as the set of states for which the ensemble of models has the highest uncertainty. A different strategy is inflate the cost penalty function, as done in CAP (Ma et al., 2021). This cost
function with uncertainty-awail change can be applied to any MBRL algorithm and its conservativeness is automatically tuned through the use of PI controller. Another approach, SAMBA (Cowen-Rivers et al., 2022), uses Gaussian Processes (GP) to model the environment. This model is then used to train a policy by including the safety constraint in the optimization process through Lagrangian multipliers. Closer to GuSS are other methods using a trajectory sampling approach to select the safest trajectories generated by CEM (Wen & Topcu, 2018; Liu et al., 2020). This lets deal with the possible uncertainties of the model predictions that could lead to consider a trajectory safe when it is not.
3 BACKGROUND
In this section, introduce the concepts of safe-RL and QD algorithms on which our method builds.
3.1 SAFE REINFORCEMENT LEARNING
Reinforcement learning problems are usually represented as a Markov decision process (MDP) M (S, Tr..))... where S is the state space, A is the action space, T S x A + S is the transition dynamics, r S Sx x IR the reward function and y é [0, 1] is the discount factor. Let A(X) denote the family of distributions over set X. The goal is to find policy S + A(A) which maximizes (Sutton & Barto, 2018). This
the expected discounted return π = arg max E, [Σ,
formulation can be easily accommodated to incorporate constraints, for example representing safety requirements. To do so, similarly to the reward function, we define a new cost function C : S x A + IR which, in our case, is a simple indicator for whether an unsafe interaction has occurred (C, = the state is unsafe and C, O otherwise). The new goal is then to find an optimal policy x² with high expected reward E, Σ, (r(((1, [t)] and a low safety cost En [E, C(st, a.) One way to solve this new problem to rely on constrained Markov Decision processes (CMDPs) (Altman, 1999) by adding constraints on the expectation (La & Ghavamzadeh, 2013) or on the variance of the return (Chow et al., 2017).
3.2 MODEL BASED REINFORCEMENT LEARNING
In this work, we address the issue of respecting safety constraints through an MBRL ap- proach (Moerland et al., 2021). In this setting, the transition dynamics Preal are estimated us- ing the data collected when interacting with the real system. The objective is to learn a model a(st,a)) St-1" to predict 8t-1 given 8t and a and use it to learn an optimal policy =². In this work, we considered the iterated-batch learning approach (also known as growing batch (Lange et al., 2012) or semi-batch (Singh et al., 1994)). In this setting, the model is trained and evaluated on the real-system through an alternating two-step process consisting in: (i) ap- plying and evaluating the learned policy on the environment for a whole episode and (ii) then training the model on the growing data of transitions collected during the evaluation T itself Tr, {($), a,, T1, C1, $1)............................. (s), Gt Ft, Ct2 $)) ..... (3), aj C1, s1)" ...., (..,,,,,,,,,,, ai Te, Cr s
Wee use to denote both probabilistic and deterministic mapping.
3

---

and updating the policy. The process is repeated until given number of evaluations certain model precision is reached.
3.3 QUALITY DIVERSITY
QD methods are family of Evolution Algorithms (EAs) performing divergent search with the goal of generating collection diverse but highly performing policies (Pugh et al., 2016; Cully & Demiris, 2017). The divergent search is performed over a, usually hand-designed, behavior space B in which the behavior of the evaluated policies represented. Some of these methods have been shown, given enough iterations, to be able to uniformly cover the whole behavior space (Doncieux et al., 2019). Combining these methods with RL approaches then allows to greatly increase the exploration abilities of RL algorithms.
Among the different QD methods introduced in the literature (Lchman Stanley, 2011; Paolo et al., 2021; Mouret & Clune, 2015), in this work the ME algorithm (Mouret & Clune, 2015) due to its simplicity and power. detailed description of how ME works and the related pseudo-code are presented in Appendix A.
4 METHOD
In this work, we use model of the environment to plan safe trajectories maximizing the reward function. In many real settings it is possible to assume that the set of unsafe situations specified by the engineer. For this reason we consider that the cost function is given and the cost is calculated from the states the system is in.
In this section, describe in detail how GuSS trains the model and uses the planners. An overview of the method is presented Alg. The code is available at: <URL hidden for review>.
Algorithm 1: Guided Safe Shooting (GuSS)
INPUT: real-system Preal- number episodes M, number of action sequences for planning step N. planning horizon length h, episode length T, initial random policy 2 RESULT: learned model p(-) and planner;
Initialize empty trace Tr Ø;;
50 Preal
Sample initial state from real system
for 1 in [0, T) do
Og x (8:) Tr Tr=
Generate random action Store transition in trace Apply action on real system and get new state
85+1 + Preat (31, at))
for i in [I...,MI& do
D¹ TRAIN(1 f Tr) T do ar ~ PLAN(p St, h,, N) T TrU
Train model
for in 10,
11
12
D Use planner to generate next action Store transition in trace
14
St-1 *v Preat S1, 01)
L Apply action on real system and get new state
4.1 TRAINING THE MODEI.
Let Tr, ((11, a1, C1, $1) ()...(s;, (st, (t, 0()3()} be a system trace consisting t steps and (st, az) state-action tuple. The goal is to train model to predict 8 given the previous (st, at). This model can be learned in a supervised fashion given the history trace Trı. We chose as p deterministic deep mixture density network (Bishop, 1994), which has proven to have good properties when used in MPC (Kégl et al., 2021). More details on the model parameters and training can be found in Appendix E.
4

---

4.2 PLANNING FOR SAFETY
The trained model p(s at) ) is used in an MPC fashion to select the action a, to apply on the real system. This means that every time step / the model evaluate N action sequences by simulating the trajectories of length h from state 85- For each action sequence, the return R and the cost C
h R = 3
h C-
(1)
are evaluated, where 85 is the state generated by the model p(-) and y the discount factor. GuSS then selects the first action of the best sequence, with respect to both the return and the cost. In this work, we assume that both the reward function r(s, a) and the cost function c(s, a) are given. reality, this is not limiting, in many real engineering settings both the reward and the cost are known and given by the engineer. We tested three different approaches to generate and evaluate safe action sequences planning time.
4.2. SAFE RANDOM SHOOTING (S-RS)
Based on the Random Shooting planner used in Kégl et al. (2021), Safe Random Shooting (S-RS) generates N random sequences of actions a; A of length h. These sequences are then evaluated on the model starting from state st. The next action a, to apply on the real system is selected from the action sequence with the lowest cost. If multiple action sequences have the same cost, 0t is selected from the with the highest reward among them. The pseudocode of the planner is shown in Appendix B.I.
4.2.2 SAFE MAP-ELITED (S-ME)
Safe MAP-Elites (S-ME) is a safe version of the ME algorithm detailed in Appendix A. Rather than directly generating sequences of actions done by S-RS, here generate the weights & of small Neural Networks (NNs) that are then used to generate actions depending on the state provided as input: (xx:) d₁- This removes the dependency space present in S-RS. After the evaluation of a policy di this is added to the collection AME. g- If another policy with the same behavior descriptor has already been found, S-ME only keeps the policy with the lowest cost. If the costs are the same, the one with the highest reward will be stored.
the horizon length h of the size of the search
Moreover, each generation, the algorithm samples K policies á from the collection to generate K new policies 6. For this step, only the policies with C are considered. If there are enough policies in the collection satisfying this requirement, the probability of sampling each policy is weighted by its reward. On the contrary, if only k policies with C are in the collection, the missing K are randomly generated. This increases the exploration and can be useful in situations in which it is difficult not to incur in any cost. The pseudocode of the planner shown Appendix B.2.
4.2.3 PARETO SAFE MAP- HAP-ELITES (PS-ME)
Another safe version of the ME algorithm. Contrary S-ME, which only samples policies for which C == 0, PS-ME sorts all the policies present in the collection into non non-dominated fronts. The K policies are then sampled from the best non-dominated front. In case less than K policies are present on this front, PS-ME samples them from the other non-dominated front, in decreasing order of non-domina,ion, until all K policies are selected.
This strategy takes advantage of the search process operated until that point even when not enough safe solutions are present, rather than relying on random policies as done with S-ME, The pseudo code of the planner is shown in Appendix B.3.
EXPERIMENTS
5.1 ENVIRONMENTS
We test GuSS on two different OpenAl gym environments with safety constraints (pendulum swing- up, Acrobot) and the OpenAl SafeCar-Gool safety-gym environment (Ray et al., 2019a). In the
5

---

environment design follow previous work (Cowen-Rivers al., 2022; Ray al., 2019b) and delegate the details to Appendix C. Moreover, we use the stochastic version of the SafeCar-Gou environment with the position of the unsafe areas randomly resampled at the beginning of each episode. This makes the environment much harder, not allowing the agents to overfit the position of the unsafe zones. To our knowledge, we the first where other works focused on the casier version with no layout randomization used by Yang et al. (2021).
this version test MBRL approach,
5.2 RESULTS
We compare GuSS with the three different planners introduced in Sec. 4.2 against various baselines. To have a baseline about how much different are the performances of safe methods with respect to unsafe ones, we compared against two unsafe versions of GuSS: RS and ME. RS performs random shooting to plan for the next action, while ME uses vanilla MAP-Elites as a planner, without taking into account any safety requirement. We also compared against the safe MBRL approach RCEM (Liu et al.. 2020) and its respective unsafe version, labeled CEM. Moreover, to show the efficiency of model-based approaches when dealing with safety requirements, compared against three model-free baselines: CPO (Achiam et al., 2017), TRPO lag and PPO lag; all of them come from the Safety-Gym benchmark (Ray et al., 2019b).
Imodel hee)
PAN 2-877
22272024
Hich Eltea (43)
Rancor Stopding (82)
FRD A lagardhnn
Cho
SyTEe SECaND sapp tnecel beadd
1ystem SCCIS Meep (model trest
10055
79000
Alataalll 0.35%
Neeto Safe PAP (PS-HD)
MW
Aluccafii - D.0%
Sale 9A7 Die 15 MEE
#lussafi- 1.0
Crua Ehtragy Mathed ICEM
Aisecalii- 2005
TRPO ti lag undlan
systems ECCENS step tmodet beand)
FRO wit boraoobn
Alvooii Fit.
Figure 2: Mean reward and probability percentage of unsafe for Safe Pendulum environment. Dashed curves indicate Model-free baselines and plain one Model-based approaches. Thicker lines represent the proposed algorithms. The red dashed line indicates the random unsafe probability. All curves are calculated over random seed.
The algorithms are compared according to four metrics: Mean Asymptotic Reward (MAR), Mean Reward Convergence Pace (MRCP). Probability percentage of unsafe (p(unsafe) % l) and transient probability percentage of unsafe (p(unsafe) % Itrans). The details on how these metrics are calculated are defined in Appendix D.
The results shown in Table 1. The MAR scores and the p(unsafe) for the pendulum system are shown in Fig. 2, while the ones for the acrobot system are in Fig. 3. Finally, Fig. 4 shows the results for the SafeCar-Goal environment. Additional plots are presented in Appendices F and G.

---

spitem ROCESS SOp UrOSE FreS!
30
10000
12
2.0
Robust Erorpy Nothod
A44 -2 07
00:00
Davsso 5ate Mihh Eltcs MS-
Salo Mcpooiee
1641 - 0:0
legranciinn H40
ONO
oytten sccess Inccel
1000
2000
30000
19400
50000
603000
Sale 103 Diten MC) (luncal)) 10am Shooting #tvioaf)
Fareta
stnnsdole AM7
Robudt cous sRrOW Nethhe (KGOP)
Fandom Shoating (15)
(tsesilie 14.0%
Entrazy Mathod ICEM)
pluncall)
FPO horergian
Kystem
pesecol)) CRD
Figure 3: Mean reward and probability percentage of unsafe for Safe Acrobot environment. Dashed curves indicate Model-free baselines and plain one Model-based approaches. Thicker lines represent the proposed algorithms. The red dashed line indicates the random unsafe probability. All curves are calculated over random seed.
moddal
HO0 38.01
Shoating
1007 choode
Palrad Cray fray /MAlia (RCEM)
FTO with anganngtin
ligrangen
2000
800
E300
Ramell Dres Entregs Peffhe OICIN
Crass Darrpp Method clamaal1 1.1%
Salt ISC tii
dlamaay) 1995
CPO
lagurgan gan
Figure 4: Mean reward and probability percentage of unsafe for SafeCar-Goal environment. Dashed one Model-based approaches. Thicker lines represent
curves indicate Model-free baselines and plain the proposed algorithms. The red dashed line indicates the random unsafe probability. All curves are calculated over random seed.
DISCUSSION
The results presented Sec. 5.2 show how GuSS can reach high performances while keeping safety cost low of the three environments tested. As expected, Acrobot, safe methods reach
7

---

Table Summary of the different methods on the different environments Pendulum and Acrobot. MAR is the mean asymptotic reward, MRCP(rah~) is the number of system access steps needed achieve Tthr of the optimum reward, p(unsafe)( and p(unsafe)[% [tr]ra are the probability percentage of being unsafe during epochs. All the metrics are average over all epochs and seeds and and f mean lower and higher the better, respectively. The best safe methods with respect to each metric are highlighted in bold. All values are 90% Gaussian confidence interval.
Method
MAR
MRCPx10 p(unsafe
p(unsafe) trans [%)
Safe Pendulum Fshr 225
GuSS(S-RS)
-2.09 + 0.09
2.12
+ 1.45
0.35 = 0.63
1.23 = 1.28
GuSS(S-ME)
-2.2 0.16
2.0
+ 0.9
0.63 0.57
0.54 0.57
GuSS(PS-ME)
-2.27 0.17
2.10
+
0.58 = 0.52
0.6 = 0.56
RCEM
-6.12 0.15
6.7
0.81
0.63 0.64
0.84 0.7
RS
-2.7 0.14
2.60
+ 1.2
1.49 = 1.0
1.36 = 0.61
ME
-2.33 # 0.19
1.90
# 0.7
3.01 2.78
2.01 1.95
CEM
-2.9 0.15
1.64
= 0.41
1.59 = 0.89
1.43 = 0.85
CPO
-6,06
+ 0.04
22
# 0.0
1.73 = 0.92
1.59 = 0.78
PPO lag
4.1 0.12
138
# 65
2.05 1.75
2.15 1.48
TRPO lag
-7.02 + 0.03
161
+ 55
1.31 = 0.91
2.07 # 0.89
Safe Acrobot Fthy
1.6
GuSS(S-RS)
1.35
0.07
1.6
+ 0.26
1.1 1.05
1.85
1.56
GuSS(S-ME
1.64
0.01
1.36
= 0.25
1.1
1.22
2.45
1.92
GuSS(PS-ME)
1.65
+ 0.01
1.56
# 0.11
1.42
1.62
3.75
2.77
RCEM
1.68
0.01
1.60
# 0.37
2.01 = 1.4
2.85 # 1.84
RS
2.07
+ 0.01
1.28
# 0.29
24.8
8.3
10.85 + 7.36
ME
2.12
0.02
1.12
0.24
25.6
7.91
12.18 7.95
CEM
2.12
+ 0.02
1.40
# 0.43
24.9
= 8.91
9.14 # 7.17
CPO
0.94
+ 0.01
87
+: 59
5.43
1.74
4.35 2.49
PPO lag
0.94
0.01
24
#
3.7
= 1.94
4.02 = 2.46
TRPO lag
1.02
0.01
37
22
4.2
2.0
4.31 2.53
SafeCar Goal Pihr 10
GuSS(S-RS)
-0.29 0.22
0.69 1.04
0.49 0.83
GuSS(S-ME)
15.25 + 1.65
32
4 0.0
1.99 # 3.52
0.55 # 0.9
GuSS(PS-ME)
10.01
1.49
12.33
+ 4.54
1.95 2.71
0.76 1.10
RCEM
-1.41 + 0.21
0.50 0.83
0.63 = 0.99
CEM
0.37 1.77
58
9.31
1.21 2.47
0.67 1.26
CPO
3.95
0.09
297
153.6
0.55 1.11
0.42 0.68
PPO lag
5.36
+ 0.1
231
# 121.24
0.66 = 1.51
0.34 = 0.58
TRPO lag
3.66
0.07
108
0.0
0.65 1.58
0.39 0.72
lower MAR scores compared to unsafe ones due to these last ones ignoring the safety constraints. At the same time, this leads much higher p( unsafe) for unsafe approaches. Interestingly, this is not the case on Pendulum where unsafe approaches tend to have lower MAR scores compared to safe methods even while
respecting safety criteria. The best unsafe method, ME, has in fact MAR of -2.53 0.19 while the worst MBRL safe method, PS-ME, has a MAR of 2.27 = 0.17 (p le 08). This is likely due to the unsafe region effectively halving the search space. The left hand side fact provides lower "safe rewards" compared to the right hand side, pushing the safe algorithms to focus more on swinging the arm towards the right than the left. This is particularly visible in Appendices between the safe and unsafe methods where the unsafe states. Moreover, while the safe methods, and in particular ME cover larger part the state space, the safe ones tend focus much
cleanly cuts the distribution of visited
the high rewarding state.
8

---

When comparing the performances of the different safe planners for GuSS, it is possible to notice how the simpler S-RS planner outperforms the more complex S-ME and PS-ME on Pendulum with respect to both MAR (p 0.02) and p (uusaffe) (p < 0.05). This is not the case the more complex Acrobot, where the improved search strategies performed by S-ME and PS-ME reach higher MAR scores compared S-RS (p < 0.02), while being comparable with respect to the safety cost. This hints at the fact that while a simple strategy like S-RS is very efficient in simple set-ups, more advanced planning techniques are needed in more complex settings, RCEM the closest method to GuSS, failed on Pendulum but reached higher MAR on Acrobot, however with twice the safety cost.
The need for more complex planners is even more evident on the hardest environment we tested: SafeCar- Goal. Here, S-ME and PS-ME clearly outperforms all other methods, even the model-free approaches, in terms of MAR. At the same time, both methods show to be the most unsafe. In fact, while on the pl (unsafe) trans metric S-ME and PS-ME perform similarly to the model-free approaches, this is not the case for the p(unsafe) which increases to surpass the random unsafe probability before decreasing to acceptable levels. This can be explained by the increased complexity of the environment due to the layout randomization of the unsafe areas at the beginning of each episode, which requires for the model to generalize in order to safely navigate the environment. This effect is supported looking at S-ME and PS-ME where respectively at around 50k steps p (unssff) drops while the reward increases. A similar effect can be observed with CEM but here there is no reward increase. Moreover, while the high y (unsafe) during the training seems to be consistent with simply ignoring the safety, closer inspection shows that this is not the case. The planning agent mostly avoids unsafe areas, with the exception of when it does an error and traverses them. In that case, it tends to get stuck on them without being able leave. This is possible due to the nature of the observables inside the unsafe regions and requires a more in-depth investigation.
At the same time, while model-free methods tend remain safer than GuSS, they require at least an order of magnitude more real-system interactions while still obtaining low MAR scores.
These results seem to confirm how the increased exploration of QD methods helps with the collection of more informative data, crucial when learning the model in the MBRL setting. This is particularly visible in the Safety-gym environment where no other baselines methods manage to solve the environment.
7 CONCLUSION AND FUTURE WORK
In this study, we proposed GuSS, model based planning method for safe reinforcement learning. We tested the method on three environments with safety constraints and the results show that, while being simple, GuSS provides good results in terms of both safety and reward trade-off with minimal computational complexity. Moreover, we observed that in some settings, like Safe Pendulum, the introduction of safety constraints can lead to better completion of the task. Further experiments are needed to confirm this effect, but if this holds, it could be possible to take advantage of it in a curriculum learning fashion: starting from strict safety constraints and incrementally relaxing them to get to the final task and safety requirement. This could help even more in the application of RL methods to real engineering systems, allowing the engineers handling the system to be more confident in the performances of the methods.
Notwithstanding the great results obtained by our proposed approach, the performances of GuSS are still tied to the accuracy of the model. If the model is wrong, it could easily lead the agent to unsafe states. This is partieularly noticeable in the SafeCar-Goal environment, where the unsafe areas layout randomization renders the environment particularly hard for model based approaches. This led our method to obtain high reward but also high cost. As discussed in Section 6, these results require more in depth analysis, but also open the path to new directions of research. A possible solution to the problem of the model inaceuracy is to have uncertainty aware models that can predict their own uncertainty the sampled trajectories and use it to be more conservative in terms of safety when learning the model. At the same time, the ME- based safe planners we proposed also suffer from the limitation of many QD approaches, namely the need hand-design the behavior space. While some works have been proposed to address this issue (Cully, 2019; Paolo et al., 2020), these add another layer of complexity to the system, possibly reducing performance with respect to the safety constraints.

---

REPRODUCIBILITY STATEMENT
In order to ensure reproducibility we will release the code at <URL hi hidden for review>, once the paper has been accepted. Moreover, the code is also attached the submission supplementary material.
Finally, all the hyperparameters of the algorithms listed in Appendix and the detailed pseudocode is shown in Appendix B.
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pp. 22-31. PMLR, 2017.
Eitan Altman. Constrained Markov decision processes: stochastic modeling. Routledge, 1999.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in safety. arXiv preprint arXiv:1606.06566 2016.
OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Christopher M Bishop. Mixture density networks. 1994.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:/606.00) (11/6006001400. 2016.
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1):6070-6120, 2017.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
Alexander Cowen-Rivers. Daniel Palenicek, Vincent Moens, Mohammed Amin Abdullah, Aivar Sootla, Jun Wang, and Haitham Bou-Ammar. Samba: Safe model-based active reinforcement learning. Machine Learning, pp. 1-31, 2022.
Antoine Cully. Autonomous skill discovery with quality-diversee and unsupervised descriptors. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 8 81-89, 2019.
Antoine Cully and Yiannis Demiris. Quality and diversity optimization: A unifying modular framework. IEEE Transactions on Evolutionary Computation, 22(2):245-259, 2017.
Stephane Doncieux, Alban Laflaquière, and Alexandre Coninx. Novelty search: theoretical perspective. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 90-106, 2019.
Carlos E Garcia, David M Prett, and Manfred Morari. Model predictive control: Theory and practice survey. Automatica, 25(3):335 348, 1989.
Hao-Lun Hsu, Qiuhua Huang, and Sehoon Ha. Improving safety in deep reinforcement learning using unsupervised action planning. arXiv preprint arXiv:2.09.1432.. 2021.
Balázs Kégl, Gabriel Hurtado, and Albert Thomas. Model-based micro-data reinforcement learning: what are the crucial model properties and which model to choose? arXiv preprint 107 2021.
Prashanth La and Mohammad Ghavamzadeh. Actor-critic algorithms for risk-sensitive mdps. Ad- vances neural information processing systems, 26, 2013.
10

---

Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning, pp. 45-73. Springer, 2012.
Nevena Lazic, Craig Boutilier, Tyler Lu, Eehern Wong, Binz Roy, MK Ryu, and Greg Imwalle. Data center cooling using model-predictive control. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 3/, pp- 3814-3823, Curran Associates, Inc., 2018.
Joel Lehman and Kenneth O Stanley. Evolving diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th annual conference on Genetic and evolutionary computation, pp. 21 1-2188 2011.
Zuxin Liu, Hongyi Zhou, Baiming Chen, Sicheng Zhong, Martial Hebert, and Ding Zhao. Con- strained model-based reinforcement learning with robust cross-entropy method. arXiv preprint arXiv:2010.07968, 2020.
Carlos Luis. SwingUp Pendulum. https:
Yecheng Jason Ma, Andrew Shen, Osbert Bastani, and Dinesh Jayaraman. Conservative and adaptive penalty for model-based safe reinforcement learning. arXiv preprint arXiv:2112.07701, 2021.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. In International Conference on Learning Representations, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-35 533, 2015.
Thomas M. Moerland, Joost Broekens, and Catholijn M. Jonker. Model-based reinforcement learning: A survey, arXiv preprint arXiv:2006.16712. 2021.
Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909 2015.
Giuseppe Paolo, Alban Laflaquiere, Alexandre Coninx, and Stephane Doncieux. Unsupervised learning and exploration of reachable outcome space. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 2379-2385. IEEE, 2020.
Giuseppe Paolo, Alexandre Coninx, Stéphane Doncieux, and Alban Laflaquière. Sparse reward exploration via novelty search and emitters. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 154-162, 2021.
Justin K Pugh. Lisa B Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolutionary computation. Frontiers Roboties and AI, 3:40, 2016.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking Safe Exploration in Deep Reinforce- ment Learning. 2019a.
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement 7:1, 2019b.
learning. arXiv preprint
Erica Salvato, Gianfranco Fenu, Eric Medvet, and Felice Andrea Pellegrino. Crossing the reality gap: survey on sim-to-real transferability of robot controllers in reinforcement learning. IEEE Access, 2021.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
11

---

Satinder Singh, Tommi Jaakkola, and Michael Jordan. Reinforcement learning with soft state aggregation. Advances in neural information processing systems, 1994.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv: 1074, 2018.
Stefan Radic Webster and Peter Flach. Risk sensitive model-based reinforcement learning using uncertainty guided planning. arXiv preprint arXiv:2111.04972, 2021.
Min Wen and Ufuk Topeu. Constrained cross-entropy method for safe reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
Qisong Yang, Thiago D Simão, Simon H Tindemans, and Matthijs TJ Spaan. Wesac: Worst-case soft actor critic for safety- anfetycconstrained reinforcement learning. In AAAI, pp. 1N639-10646 2021.
Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter Ramadge. Accelerating safe reinforcement learning with constraint-missatceed policies. arXiv preprint arXiv:2006.16645, 2020a.
Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter Ramadge. Projection-basee constrained policy optimization. arXiv preprint arXiv:2010.03/522 2020b..
Denis Yarats. David Brandfonbrener, Hao Liu, Michael Laskin, Picter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't 't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv:2201. 13425, 2022.
Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems,
2020.
12

---

A MAP-ELITES
In MAP-Elites (ME) the behavior space B is discretized through a grid with the algorithm trying to fill every cell of the grid (Mouret & Clune, 2015). ME starts by sampling the parameters policies from a random distribution and evaluating them in the environment. The behavior descriptor B policy Φ; is then calculated from the sequence of states traversed by the system during the policy evaluation. This descriptor is then assigned to the corresponding cell in the discretized behavior space. If no other policy with the same behavior descriptor is discovered, Φ) is stored as part of the collection of policies AME returned by the method. On the contrary, the algorithm only stores, in the collection, the policy with the highest reward among those with the same behavior descriptor. This allows the gradual increase of the quality of the policies stored in AME At this point, ME randomly samples policy from the collection, and uses it to generate a new policy 6; to evaluate.
of M
The generation of d; is done by adding random noise to its parameters through variation function V(-). The cycle repeats until the given evaluation budget N depleted. The pseudo-code ME is shown in Alg. 2.
Algorithm 2: MAP-Elites
INPUT: real system Preat, initial state so, evaluation budget N, parameter space 4 discretized behavior space variation function VV-), number of initial policies M, episode length H: RESULT: collection of policies AME;
AME 0
Initialize empty collection Sample initial M policies b Evaluate policies in the real system
I + SAMPLE( M)
R, C, Treal ROLLOUT (Preal: Su. 6s H), Vo, E Vos E T and VTreal E &Treal
b, = -fodi,
Calculate policies behavior descriptors
AME -bi VØ; ET
> Store policies in collection
while N not depleted do
r SELECT(AME, K,
Select K policies from collection
10 F & V(T)
> Generate new policies
R, C,Treal ROLLOUT(Peaat 80, Le H), VoL ef system
Evaluate new policies in the real
12
b= f Treat):
Vo, e I and VTr E Treal
Calculate behavior descriptors
AME hi, VΦ: e
D Store policies in collection
13

---

B PLANNERS PSEUDOCODE
This appendix contains the pseudo code of the three planners introduced in the paper. Each planner evaluates a policy on the model through a ROLLOUT function detailed in Alg. 3.
Algorithm 3: ROLLOUT
INPUT: model p, initial state so, policy é, horizon length /;
RESULT: collected reward R, collected cost C, simulated trace model.
Tmodeel = Ø
Initialize empty trace
for in [0, .... ht] do
0j +v (88))
b Draw action from policy
8j-1 (()
Draw predicted state
model = model ag)
>> Update trace
R Σj-0 7'r(s).
D Calculate reward of action sequence
C' E (3)
> Calculate cost of action sequence
B. SAFE RANDOM SHOOTING
Algorithm 4 shows the pseudocode of the Safe Random Shooting (S-RS) planner.
Algorithm 4: S-RS
INPUT: model p. current real-system state 8t, planning horizon h, evaluated action sequences N, action space A;
RESULT: action to perform dz:
AS Ø
Initialize empty collection of action sequences
For i in [O...., NJ do
Φ; [at,.... a+th] SAMPLE(A,A) R, C, T Tmodee RROLLOUT(p, 8t, P.. h)
Sample action sequence
D Evaluate action sequence Store evaluated action sequence
AS = AS ASU((i,, R; C;)
ASIc * AS
D Get action sequences with lowest cost
Øbest ASIc
D> Get action sequence with highest reward
10 4v Prese (t
>> Get next action
B.2 SAFE MAP-ELITES
Algorithm 6 shows the pseudocode of the Safe MAP-Elites (S-ME) method. The SELECT function shown in Alg.
Algorithm SELECT function S-ME planner
INPUT: Collection of policies AME, Number of policies to select K, Policy parameter space +
RESULT: set of selected policies I;
T Ø
Initialize empty of selected policies Select policies with C' 0 from collection
I' < AME[ [C 0] size(T') K then [=TUSAMPLE(3)
Sample missing policies from parameter space
14

---

Initialize empty collection of policies Sample initial M policies Evaluate policies in the model Algorithm 6: S-ME
INPUT: model p. current real-system state st, planning horizon h, evaluated action sequences N, discretized behavior space B, variation function V(.), number of initial policies M, number of policies per iteration K, policy parameter space p;
RESULT: action perform dz;
AME Ø
I SAMPLE( (
R. C. Tmodel ROLLOUT(P,. St. hi, VOL ET
b; =[(d; (ibieToode::
VΦ; E and VTmode model
Calculate policies behavior descriptors
AME + Øi, VØ, E
D Store policies in collection
while N not depleted do SELECT(AME;
D Select K policies with C'= 0 from collection
10
r V(T)
>> Generate new policies
11
R. C, Tmodel= ROLLOUT(P,, St: h), h),
Evaluate new policies in the model
12
b-= f(di, T Tmodel),
Vis: E and VT Toodel
D Calculate behavior descriptors
AME oi Vol e
D Store policies in collection
AS 15 Obest *o I'
D Get policies with lowest cost
D Get policy with highest reward
ag Uness (zt)
D
Get next action
B.3 PARETO SAFE MAP-ELITES
The Pareto Safe MAP-Elites (PS-ME) planner works similarly to the S-ME one with the exception of the SELECT function at line of Alg. 6. So for this planner we just report the pseudocode of this function in Alg. 7.
Algorithm 7: SELECT function PS-ME planner
INPUT: Collection policies AME, Number of policies to select K, Policy parameter space + RESULT: set of selected policies I;
D Initialize empty set of selected policies
[I .ND. *> NON_DOMINATED (ORT(AME) SAS fronts
D Sort collection into non dominated
==0
while size( (T) < K do T N
Select policies from best front
i=itl
15

---

C ENVIRONMENTS
C.I SAFE PENDULUM
safe version of OpenAI's swing up pendulum Luis, which the unsafe region corresponds to the angles in the [20", ,330' range, shown in red in Fig. 5, The task consists in swinging the pendulum up without crossing the unsafe region. The agent controls the torque applied to the central joint and receives a reward given by -02 0.162 + 0.001a², where 0 is the angle of the pendulum and the action generated by the agent. Every time-step in which 0 [20", 30" leads cost penalty of 1. The state observations consists of the tuple
sin()))ssinn))))))) Each episode has length of T 200.
Figure 5: Pendulum upright task.
C.2 SAFE ACROBOT
A safe version of the Acrobot environment (Brockman al., 2016), shown Fig. 6. It consists in an underactuated double pendulum which the agent can control the second joint through discrete torque actions {-,,0,1}. The state of the system observed through six observables s = (Cos(Oo), sin(Oo), cos(01). sin(())))i(()) (0,0,). responds to the height of the tip of the double pendulum with respect the hanging position. The unsafe area corresponds to each point for which the height the tip
01). The reward
[0, 4] cor-
double pendulum is above with respect to the hanging position, shown in red in Fig. 6. Each episode has length of T 200 and each time-step spent in the unsafe region leads to a cost penalty of
Figure 6: Acrobot up- right task.
For this environment, the constraint directly goes against the maximization of the reward. This is similar to many real-world setups in which one performance metric needs to optimized while being careful not go out of safety limits. An example of this is agent controlling the cooling system of a room whose goal is to reduce the total amount power used while also keeping the temperature under certain level. The lowest power is used when the temperature is highest, but this would render the room unusable. This means that an equilibrium has to be found between the amount of used power and the temperature of the room, similarly how the tip of the acrobot has to be as high as possible while still being lower than 3.
C.3 SAFECAR-Goo
Figure 7: Safety gym environment
Introduced in Safety Gym (Ray et al., 2019b) it consists of two-wheeled robot with differential drive that has to reach a goal area in position randomly selected on the plane, represented as the green cylinder in Fig. 7.(c). When goal is reached, another one is spawned in random location. This
16

---

repeats until the end of the episode is reached (at 1000 time-steps). On the plane there multiple unsafe areas, shown as blue circles in Fig. 7.(c), that the robot has to avoid. The placement of these areas is randomly chosen at the beginning of each episode.
The agent can control the robot by setting the wheels speed, with a E A= The observations consists of the data collected from multiple sensors: accelerometer, gyro, velocimeter, magnetometer, a 10 dimensional lidar providing the position of the unsafe areas and the current position of the robot with respect to the goal. This leads an observation space of size 22.
In the original environment, the cost is computed using the robot position with respect to the position of the different hazards areas. However, as these information not available in the observations change the safety function using the Lidar readings and a predefined threshold of 0.9 above which cost of incurred.
17

---

D METRICS
In this appendix we discuss the details on how the metrics used to compare the algorithms are defined. Mean Asymptotic Reward (MAR). Given trace Tr and reward r(st, a.) obtained each step we define the mean reward R(Tr) + Σ rt. The mean reward in iteration is then MR) R(Tr("). The measure of asymptotic performance (MAR), is the mean reward in the second half of the epochs (we set N so that the algorithms converge after less than N/2 epochs) MAR = N E==N/22 (.()).
Mean Reward Convergence Pace (MRCP). To assess the speed of convergence, we define the MRCP the number of steps needed achieve environment--pecifee reward threshold rthr- The unit of MRCP(r is real-system access steps, to make it invariant to epoch length, and because better translates the sample efficiency of the different methods.
Probability percentage of unsafe (p(unsafe) [%]). To compare the safety cost of the different algorithms, we compute the probability percentage of being unsafe during each episode as punnsffe) 100+ + Σx=o C& where T' the number steps per episode. We also compute the transient probability percentage p(unsafe) trans [%] as measure to evaluate safety the beginning of the training phase, usually the riskiest part of the training process, It is computed by taking the mean of p( (unsafe) on the first 15% training epochs.
18

---

E TRAINING AND HYPER-PARA PARAMETERS SELECTION
All the training of model-based and model-free methods have been perform in parallel with CPU servers. Each server had 16 Intel(R) (R) Gold CPU's and 32 gigabytes of RAM.
E. MODEL BASED METHOD
For the Safe Acrobot and Safe Pendulumn environments we used as model deterministic deep auto-regressive mixture density network (D (DAMeNae)) while for the Safety-gym environment we used the same architecture but without auto-regressivity (DMeN..)..
Table Model and Agent Syyper-paramtter
Safe Pendulum
Safe Acrobot
SafeCar-Goal
Model
Optimizer
Adam
Adam
Adam
Learning
le-3
le-3
1e-3
D(AR)MDN
Nb layers
2
2
2
Neurons per layer Nb epochs
50
50
50
300
300
300
Planning Agents
CEM and RCEM
Horizon
10
10
30
Nb actions sequence
20
20
3000
Nb elites
10
10
12
S-RS and RS
Horizon
10
10
30
Nb actions sequence
100
100
3000
ME, S-ME, PS-ME
Horizon Nb policies
10
10
30
100
100
525
Nb initial policies
25
25
25
Nb policies per iteration
10
Behavior space grid size Nb policy params.
50 x 50
50 50
20 20
26
83
77
Nb policy hidden layers Nb policy hidden size Policy activation func.
1
5
Sigmoid
Sigmoid
Sigmoid
E.2 MODEL FREE METHOD
All model free algorithms implementation and hyper-parameters were taken from the http://gihabboooooeeeniiiiitttts repository.
19

---

F OPTIMALITY
Safe-RL algorithms have to optimize the reward while minimizing the cost. Choosing which algorithm the best is multi-objective optimization problem. Fig. shows where each the methods tested in this paper resides with respect to the MAR score and the p(unsafe). Methods labeled with the same number belong to the same optimality front with respect to the two metrics. A lower label number indicates higher performance of the method.
Sah fendur (5 San Han (GRR
25
thiiii 1-MM
Crea Larsgy Nethie
Nentre in aatigess Osu betay Netiod ICDN
Gart Farcoe Sheciing (-A
70
DC
2.0
1.5
10
1.0
0.5
Meen Asymptotic Reward (MA2)
Moan Asymptote (MAR)
PENDULUM
SAFENO PENDULUM_SAFE-VO
ACROBOT, SEROBBT SINCOS
SINEAV
SAFE-V
foote Macheet (Gind)
Hill EoIl
1.8
mlimsafl1 [1)]
1.4
1.2
1.0
0.6
0.0
2.5
15.0 3.0
7.5
20.0
12.5
Mean Asymptatic Reward (MAR)
Figure 8
G EXPLORATION PLOTS
In this section we present plots about the explored state distribution for the different algorithms on the two environments.
The State Coverage plots, leftmost one in the figures, represents the density of times a state has been explored. The unsafe area is highlighted in red.
The Reward Heatmap plots, center ones, show the distribution of the visited states overimposed to the reward landscape. The unsafe area is highlighted in red.
20

---

The Reward bins plots, rightmost ones, show the histogram of visited stated with respect to the reward of that state. The histograms are generated by dividing the interval between the minimum and maximum reward possible in the environment into 10 buckets and counting how many visited states obtain that reward.
It is possible see the reward plots for the Safe Acrobot environment in Appendix G.2 how the safety constraint limits the number of times the states with r are visited by safe methods compared unsafe ones.
G.1 SAFE PENDULUM
GUS525-R5)
SE
GUSSIS-NEI
SUSS|SP-ME)
4.09
Tarts
Figure 9: Explored states for Safe Pendulum.For Safe Pendulum state corresponds to the angular position and velocity of the pendulum (0,0). The states are color coded according to the corresponding reward. The red-shaded area corresponds to the unsafe area. Each point represent an explored state. Red points are safe states while blue points unsafe ones. The size of the point is proportional to the number of times that state has been visited.
Reward bins
8.00
coverage
18:00
3300
3200
3.6
2500
0.00
-8
2200
10
1202
4.20
0000
-12
2000
022
-14
500
3.14
137
6:0
357
332
15.24
1137
0:00
157
331
Figure 10: Exploration plot for RS on Safe Pendulum.
Reward bins
Stato coverage
Reward heatmap
3.00
0.00
4000
4.32
34:00
Uoeta de.
qumtor
4.30
02
-11
8,30
1.5
-1.57
6.00
150
8.00
-3.14
-1.57
0.50 Thets
157
1114
-16
Figure 11: Exploration plot for GuSS(S-RS) Safe Pendulum.
21

---

Reward bins
State coverage
0.6
0.00
Numiib
4.20
100
-12
0.2
1000
5.14
1.57
6:01
157
514
1..1
1.37
0.00
157
-15
Figure 12: Exploration plot for ME on Safe Pendulum.
2.0
40022
4,30
4.6
0.00
0.00
2200
0.0
Theta
-10
Reward
Figure 13: Exploration plot for GuSS(S-ME) on Safe Pendulum.
Reward
3.00
State coverage
0.00
Reward heatmap
0.5
4.30
1.00
2000
0.4
1002
4:30
1.4
-1.57
Thele
1.52
3.34
0.0
0.00
-3.14
Theik
0.50
1.57
-15
-15
-10
Figure 14: Exploration plot for GuSS(PS-MBB on Safe Pendulum.
Reward bins
Stato coverage
Reward heatmap
3.00
0.00
4.32
2500
Uoata de.
0.6
2200
4,30
-12
1000
02
-11
200
8,30
14
-1.77
6.60
150
8.00
3..14
-1.57
0.50 Thets
157
1114
-16
Figure 15: Exploration plot for CEM on Safe Pendulum.
22

---

Reward bins
3.00
coverage
3300
3200
0.6
0.00
2200
4.20
100
-12
022
-11
200
3.14
1.57
6.05
157
314
1..14
1.37
0.00
157
-15
Figure 16: Exploration plot for RCEM on Safe Pendulum.
G.2 SAFE ACROBOT
Reward bins
State COVE
Reward heatmap
2.00
2,00
31
4002
1.5
-3.30
-.000
42
-2.70
00
1.00
0.00
1.00
2
-2.00
2.00
3,200
1.00
2.00
Figure 17: Exploration plot for RS on Safe Acrobot.
State COVD age
Reward heatmap
11
1.30
100
29
00
0.00
2.0
414
1.3
3000
1.3
2000
23
1000
0.0
60
Figure 18: Exploration plot for GuSS(S-RS) on Safe Acrobot.
Reward bins
2.00
State Tage
2,50
Rewend heatmap
31
4000
1.30
2000
1500
-3.30
1.000
1000
02
-2.05
00
1.00
0.00
1.0
2
-2.00
3..00
-3.20
1.09
2000
Figure 19: Exploration plot for ME Safe Acrobot.
23

---

Reward bins
sage
35
06
233
0.00
0.00
2/3
1.3
3.30
+..00
002
-2.35
0.00
1..00
0.00
1.00
2,00
-2.00
+3.00
-3.20
0:00
3.00
2.00
Figure 20: Exploration plot for GuSS(S-ME) Safe Acrobot.
Reward bins
COOD ago
2.00
2.00
11
4000
3000
0.
-3.00
1.00
02
2.85
-1.M
0.00
6.0
-2.00
+1.20
2002
Figure 21: Exploration plot for GuSS(PS-ME) Safe Acrobot.
Reward bins
State cove age
Reward heatmap
2.00
2:00
4005
0.8
3.30
3000
2500
-3.30
3.35
5..0
-1.00
0.00
1.00
2.50
2.00
+2.00
-3.50
1.00
2.00
300
Figure 22: Exploration plot for CEM Safe Acrobot.
State COVE Tage
Reward heatmap
2.00
4005
3.30
1.00
3000
35
2300
0.00
30
-4
1309
1.2
3.35
2:60
85
0.0
86
Reware
Figure 23: Exploration plot for RCEM on Safe Acrobot.
24


Reviews:
[{"review_id": "zYmR1pqzvd", "reviewer": "Reviewer_5gos", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method to obtain safe model-based RL by using a guided safe shooting method (with variations) and applies it to 3 toy tasks.\n\nThe paper a quality-diversity method as an optimization method method with some variations for safe planning. ", "summary_of_the_review": "The paper looks at an important problem, but in the current state, as the paper is empirical, the evaluation is limited to pretty simple environments. Also a simple baseline, for identifying which part is really important is missing. Apart from that, there is a prior work that was not mentioned that solves the tackled problem.\n", "strength_and_weaknesses": "Strengths:\n- important subject\n- bringing quality-diversity planners into MBRL\n\nWeaknesses:\n- missing baselines\n- no statistical analysis\n- only toy environments\n- not really clear which improvement the decisive one \n- too small fonts in the graphics\n\nDetails:\nSince you are not dealing with uncertainty while planning, I am wondering what about a standard approach where the costs term is simply substracted from the reward with a big constant factor (referred to as \"big M\" approach in control). This should be a baseline: so ME with this r - M*c as a reward with large M. Then we would see whether the particular optimization improvements that you do really are crucial. How many rounds of ME are you performing, every timestep. Is there any shift initialiation etc?  \n\nI suspect that most of the performance gain of the paper comes from the diversity term in the ME optimizer. However, there was prior work [1] that does all of this already: epistemic uncertainty to collect data needed for model learning, safety awareness including aleatoric uncertainty (which is not so important here, as you have only deterministic and simple environments) and probabistic safety constraints, also in highdimensional control problems.\n\nI cannot find the description of the \"behavior space\" used. \n\n[1] Risk-Averse Zero-Order Trajectory Optimization, CoRL 2021\nhttps://openreview.net/forum?id=WqUl7sNkDre", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mxknkF3P-w", "reviewer": "Reviewer_uMvM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper takes a particular view on safe reinforcement learning in the form of model-predictive control. In particular, the authors present an approach, called ‘guided safe shooting (GuSS), that combines model-based planning with RL, towards a minimal violation of safety. Technically, a constrained MDP approach is realized, together with three different planning methods. The approach is evaluated by means of three OpenAI environments.", "summary_of_the_review": "Good idea and approach, but the novelty is in doubt. \n", "strength_and_weaknesses": "Strengths\n\n- The topic is very relevant and well-fitting for ICLR.\n- The concept of combining model-based control with RL is natural. \n- The evaluation considers state-of-the-art environments.\n\nWeaknesses\n- The novelty of the approach is unclear, and the related work is not discussed appropriately.\n- The paper seems a bit unpolished at parts. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kkei2rSf6fv", "reviewer": "Reviewer_WaLB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a model based safe-RL algorithm that uses custom planners belonging to the family of Quality Diversity methods. Three such custom planners are proposed. The resulting RL approach is tested in three different simulation setups, with randomized safety constraints. ", "summary_of_the_review": "Overall the paper should be rejected as it is lacking a clear methodological contribution, see comments above.", "strength_and_weaknesses": "Strengths:\n\n* The introduced QD-planners are extensively tested in 3 different environments against several competing approaches.\n\nWeaknesses:\n\n* The paper does not introduce a new method per se, as in a new model-based RL algorithm that e.g. learns a model in a particular way, or that implements a new model structure. Rather, three ad-hoc QD-planners are introduced but not analyzed rigorously/theoretically: the heuristics are shown to work well in the chosen experimental scenarios, but what are the failure points? When would they be suboptimal and when could the resulting policies violate safety constraints? These are not discussed theoretically or verbally.\n\n* As the authors themselves mention at the conclusion, \"if the model is wrong, it could easily lead the agent to unsafe states\". Indeed the focus of model-based RL approaches introduced to safety critical environments, need to consider very critically how the learned model could mislead the agent, or how to learn robust models that do not do so. I would suggest the authors to look up the literature on robust control or the more recent uncertainty-aware learning controllers (using GPs, etc.).\n\n* If the contribution of the paper is not a new model-based RL agent but rather the custom planners, then as mentioned above, they would need to be analyzed critically and rigorously, e.g. as is often done in the MPC literature. I would suggest the authors to look up e.g. stochastic/robust/learning MPC formulation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "h6nMaJq-qU", "reviewer": "Reviewer_G6tK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new method for safe model-based RL that minimizes safety constraint violations during planning.\nThree variants of the method are proposed (S-RS, S-ME, PS-ME) and experiments are performed to show their effectiveness in terms of safety, state space exploration, and rewards earned.", "summary_of_the_review": "The paper works in an interesting direction and shows some promising results, but the method itself and how it works is unclear. Several methods are proposed, but without a clear winner or recommendation of which method to apply for future work.\n\nAs it is, I do not think the paper is sufficiently ready for publication.", "strength_and_weaknesses": "The use of quality diversity methods is an interesting approach to increase the exploration of the agent, which is an important aspect to collect information about the safety conditions in every part of the state space and to collect alternative planning configurations depending on the policy behaviour.\nThe experiments show some of the variants to be competitive with the baseline methods, although not consistently, they however still outperform the model-free alternatives, as we should expect.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject