Do Androids Dream of Electric Fences? Safety-Aware Reinforcement Learning with Latent Shielding
Authors: Chloe He
Borja G. León
Francesco Belardinelli
Venue: ICLR 2022 Submitted (2022)
Keywords: Model-Based Reinforcement Learning
Safety-Aware Reinforcement Learning
Shielding
World Models

Abstract: The growing trend of fledgling reinforcement learning systems making their way into real-world applications has been accompanied by growing concerns for their safety and robustness. In recent years, a variety of approaches have been put forward to address the challenges of safety-aware reinforcement learning; however, these methods often either require a handcrafted model of the environment to be provided beforehand, or that the environment is relatively simple and low-dimensional. We present a novel approach to safety-aware deep reinforcement learning in high-dimensional environments called latent shielding. Latent shielding leverages internal representations of the environment learnt by model-based agents to "imagine" future trajectories and avoid those deemed unsafe. We experimentally demonstrate that this approach leads to improved adherence to formally-defined safety specifications.

Under review as a conference paper at ICLR 2021
Do ANDROIDS DREAM OF ELECTRIC FENCES? SAFETY-AWARE REINFORCEMENT LEARNING WITH LATENT SHIELDING
Anonymous authors Paper under double-bling blind review
ABSTRACT
The growing trend of fledgling reinforcement learning systems making their way into real-world applications has been accompanied by growing concerns for their safety and robustness. In recent years, variety of approaches have been put forward to address the challenges of safety-aware reinforcement learning; how- ever, these methods often either require handcrafted model of the environment to be provided beforehand, or that the environment is relatively simple and low- dimensional. We present a novel approach to safety-aware deep reinforcement learning in high-dimensional environments called latent shielding. Latent shield- ing leverages internal representations of the environment learnt by model- model-based agents to "imagine" future trajectories and avoid those deemed unsafe. We exper- imentally demonstrate that this approach leads to improved adherence to formally- defined safety specifications.
INTRODUCTION
The steady trickle of reinforcement learning (RL) systems making their way out of the lab and into the real world has cast spotlight on the safety and robustness of RL agents. The motivation behind this should be relatively easy to grasp: when training an agent in real-world settings, it is desirable that some states are never reached as they could, for instance, cause permanent damage to the hardware the agent controlling. We can thus informally define the notion of safety-aware RI in terms of the classical RL setup with the added requirement that the number of unsafe states visited be minimised. Under this definition, however, has been found that many state-of-the-ar RL algorithms unnecessarily enter unsafe states despite safe alternatives being available and there being a positive correlation between avoiding such states and reward (Giacobbe et al. 2021).
The field of safety-aware RL encompasses multitude of approaches ranging from constrained pol- icy optimisation (Chow et al.] 2017 Achiam et al. [0177] Yang et al.) (000)) to safety critics (Srini)
vasan et al. 2020 Bharadhwaj et al. 2021 TTaaaajeyyan et al. 2021 to meta-learning (Turchetta et al. 2020). In this work, we focus on a particular family of approaches known as shielding (Al-
shickh et al. 2018 Anderson al. 2020; Giacobbe et al. 2021 ElSayed-Aly al. [202]] Pranger et al.) 2021 Central to shielding is the notion of a shield, a filter that checks actions proposed by the agent's existing policy with reference to a model of the environment's dynamics and some formal safety specification. The shield overrides actions that may lead to an unsafe state using some other safe (but by no means optimal) policy. A key advantage of many shielding approaches that the resulting shielded policies are formally verifiable; however, a shortcoming is that they require a model of environmental dynamics typically handcrafted to be provided in advance. Provid- ing such model may prove difficult for complex real-world environments, with inaccuracies and human biases creeping into handcrafted models.
In this work, propose safe RL agent that makes uses of latent shielding, approach to shield- ing in environments where a formally-specifii dynamics model is not available advance. At an intuitive level, the agent uses data-driven approach to learn its own latent world model (a com- ponent of which is a dynamics model) which is then leveraged by a shield. The shield then uses the agent's model to "imagine" trajectories arising from different actions, forcing the agent avoid

---

Under review as a conference paper at ICLR 2021
those it foresees leading to unsafe states. In addition, the agent can be trained within its latent world model thus reducing the number of safety violations seen during training.
Contributions The main contribution of this work is framework for shielding agents in complex, stochastic and high-dimensional environments without knowledge of environmental dynamics a pri- ori. We further introduce a new method to aid exploration when training shielded agents. Though our framework loses the formal safety guarantees associated with traditional symbolic shielding ap- proaches, our experiments illustrate that latent shielding reduces unsafe behaviour during training and achieves testing performance comparable to previous symbolic approaches.
2 PRELIMINARIES
In this section, we cover some relevant background topics. We begin by introducing our problem setup for safety-aware RL and give overview of the specification language used in this work. This is followed by an outline of the latent world model we make use of in this work as well as discussion on shielding.
2.1 PROBLEM SETUP
We consider an agent interacting with an environment & modelled as partially observable Markov decision process (POMDP) with states s E Sg, observations O, Og, -eneratgee actions a, E Ag and scalar rewards Tt E I over discrete time steps [0, I..,, 1]. We assume the environment has been augmented with labelling function L Se + {safe, unsafe) that, at each time step, informs us whether violation has occurred with respect to some formal safety specification ø. For the avoidance of doubt, we define a violation to have occurred whenever & does not hold. This is a weaker assumption than previous works in shielding (which assume access to an abstraction of the environment) and can be thought as secondary safety-focused reward function with a binary output. Intuitively, the goal of the agent is to learn policy и that maximises its expected cumulative reward while minimising the number violations
2.2 SYNTACTICALLY CO-SAFE LINEAR TEMPORAL LOGIC
In this work, use syntactically co-safe Linear Temporal Logic (scLTL) (Kupferman & Vardi 2001) as our specification language. Valid scLTL formulae over some set of atomic propositions AP can be constructed according the following grammar:
Φ ::= true -d @V é A Oo 6 u oΦ
(1)
where E AP, (negation). V (disjunction), A (conjunction) are the familiar operators from propositional logic, and O (next), (until) and o feventually) are temporal operators. We can monitor co-safe LTL specification using technique known as progression (Bacchus & Kabanza 2000
2.3 RECURRENT STATE-SPACE MODELS
We refer the predictive model of an environment maintained by a model-based agent as its world model. World models can be learnt from experience and be used both as a substitute for the environ- ment during training (Ha Schmidhuber 2018 Hafner et al.) 202T) and for planning at run-time (Hafner et al. 2019b). Though many realisations of the notion of world model exist, the world model used in this work based on the recurrent state-space model (RSSM) proposed by Hafner et al. 2019b).
An RSSM composed of three key components: a latent dynamics model, a reward model, and an observation model. These components act on compact states formed from the concatenation of deterministic latent state ht and stochastic latent state zt.
The latent dynamics model is made up of number of smaller mod- els. First, the recurrent model h = fh, 2, a.-1) is used to compute the deterministic latent state based the previous compact state and action. From by and the current observation or,
Latent Dynamics Model
2

---

Under review as a conference paper at ICLR 2021
distribution q(2ç or) over posterior stochastic latent states 2, is computed by the representation model. At the same time, a distribution p(2g)he) over prior stochastic latent states & is computed by the transition model, based only h₂ During training, the transition model attempts to min- imise the Kullback Leibler (KI.) divergence between the prior and posterior stochastic latent state distributions. In doing this, the RSSM learns to predict future latent states (using the recurrent and transition models) without access to future observations.
Observation Model The observation model computes the distribution p(or 2t) over observa- tions 6, for particular state. Though not strictly needed, the observation model can prove useful for visualising predicted future states and providing richer training signal.
Reward Model particular state.
The reward model computes the distribution p(fe|le, =x) over rewards it for
In practice, the distributions p and are implemented with neural networks pa and ga respectively, parameterised by some set of parameters 0. These latent dynamics models define fully-observable Markov decision process (MDP) as the latent states in the agent's ' own internal model can always be observed by the agent (Hafner et al. (019a). We denote the state space of this MDP (comprised of compact latent states) as ST.
2.4 SHIELDING
The classical formulation of shielding in RL is given by Alshiekh et al. (2018). It assumes access to two ingredients: an LTL safety specification and abstraction (a MDP model of the environment that captures the aspects of the environment relevant for plamning ahead with respect the safety specification). These ingredients are used to construct a formally verifiable reactive system that monitors the agent's 's actions, overriding those which lead to violation states.
Proposed by Giacobbe et al. (021), bounded prescience shielding (BPS) avoids the need for hand- crafted abstractions by exploiting the fact that some agents trained in computer simulations. The shield operates by leveraging access to the program underlying the simulation to look ahead into future states within some finite horizon. Using BPS over classical shielding does, however, come with few disadvantages. Firstly, it requires access to the simulation at run run-time which may prove difficult provide (especially in cases where running the simulation is computationally expensive). Moreover, an agent using BPS, even when starting from safe state, can find itself entering unsafe states in cases where the number of steps between a violation being caused by an action and the violation state itself exceeds the shield's look-ahead horizon. This is not the case for classical shielding which resembles BPS with an infinite horizon.
2.5 BOUNDED SAFETY
The notion of safety used by BPS is defined over MDPs. For an arbitrary MDP with states S and actions A, a bounded trajectory p of length H is sequence of states and actions dgg ب
On-1, S,, comprised of no more than H states and with the final state s, either being a terminal state = H - 1. We further denote the set of all finite trajectories starting from some arbitrary state s E S by e(s) and the set of all bounded trajectories length H that start from 8 by (s))..
We say bounded trajectory p of length H satisfies H-bounded safety with respect to safety speci- fication o, written Su (p. Φ), if and only if for all E p, [g(s)) safe. Moreover, we extend the notion of bounded safety over the set policies: a policy π is H-bounded safe with respect o, denoted Su (T, Φ), if and only if for all E S.
either there exists some p E eu (s) such that Sy (p, Φ) and (+6)) = Og:
or for all p E (s),., (p. o).
In other words, the policy will choose a safe trajectory as long as one exists. Finally, we formally define violation of & to be inevitable in state so S if and only if for all p E e(so), (p,(p,)) Φ).
3

---

Under review as a conference paper at ICLR 2021
ho
fz
hg
Za
o
22
23
Oo
01
6
69. 82
ōg. P3
Figure I: Safety RSSM. In this example, the model observes the environment for two time steps and predicts the subsequent two time steps. at denotes the action taken step Os the observation, he the deterministic component of the latent state, & the posterior stochastic component, & the prior stochastic component, Å the violation prediction (our novel contribution, highlighted in yellow), o the predicted observation and f the predicted reward. Circles represent stochastic variables whereas squares represent deterministic variables. An arrow from one shape to another indicates that the source is used the calculation of the destination.
SHIELDED DREAMS
We introduce the notion of latent shielding, novel class of shielding approaches that replace ab- stractions used by shields with learned latent dynamics models thus allowing the enforcement of formal safety specifications while avoiding the need for explicitly-defnned abstraction of the vironment. We further introduce the first such approach, approximate bounded prescience shielding, framework for latent shielding that leverages latent world models learnt by model-based deep RL (DRL) agents. In this work, our model-based agent of choice Dreamer (Hafner et al. 2021 | which we modify to incorporate shielding into its data collection, training and deployment phases.
3.1 SAFETY RSSM
We augment the standard RSSM with a labelling function L - {safe, unsafe} which maps latent states s E SI whether they correspond states violation of As with the other models, LÝ is implemented with a neural network with a categorical output la also parameterised by This yields enhanced RSSM (illustrated in Figure D which we will refer to as safety RSSM (SRSSM).
We train la along with the other components of the SRSSM with the objective
min Lmodel Lobsereation Creward + LKL Lesolation
where the first three terms are as described in (Hafner et al. 2019a 2021) and, for convenience, can be found in Appendix B Lvialation is term that introduce that acts weighted binary cross-entropy loss over predictions by lo
Lusolation
Σ (a)log(lo(b, + (1 -1 (16(, =))) [DAl ((..))
where Dx C S (0. 1} is dataset of state-label pairs (with 0 corresponding safe and 1 to unsafe) and a E [0, 1] is a weighting term. The inclusion of the weighting hyperparameter a is motivated by the observation that the number of observations safe and unsafe states will generally be quite imbalanced. Thus, to prevent predictions by lo from collapsing to a single class, we assign greater weight the resserepeeedntee class.
3.2 APPROXIMATE BOUNDED PRESCIENCE SHIELDING
We now integrate the SRSSM as part of latent shielding approach which we shall refer to as approximate bounded prescience shielding (ABPS). Though ABPS is inspired by BPS, it differs in
In practice, any model-based based agent with latent dynamics model be used.
4

---

Under review as a conference paper at ICLR 2021
two key aspects: (1) approximate the labelling function Lg and environmental dynamics using SRSSM; and (2) we sample a fixed number of potential future trajectories as opposed to exhaustively exploring all possibilities.
Thus, our approach can be thought of as an approximation of some "ideal" bounded prescience shield that uses the true environmental dynamies and labelling function. The advantage of the first difference should be obvious: it enables the shield to learn its own abstraction, removing the need for hand-crafting or access
digital environment's 's underlying program. Why the second difference is advantageous may be slightly less obvious - it's a heuristic that allows us to increase the horizon H. By directing the sampling of trajectories accordance with states and actions the policy is biased towards (as opposed to uniformly), it may be possible to achieve comparable performance to a standard BPS in foreseeing unsafe states. The intuition behind this that by not sampling trajectories the agent is unlikely to take, more of our computational budget can be dedicated into ensuring that the agent's most likely trajectories are safe. other words, we will not spend time planning to correct actions that the agent is unlikely to take.
More formally, our shielded policy nt can be written:
x² (8t) [s()): if P(lo(s(+1) 1|-(s()) <e [ole(S)), otherwise
(4)
where (s;) is the agent's policy without shielding; E Sy is some compact latent state; and Tale an alternative policy that ensures that, if violation isn't 't already inevitable, the agent avoids predicted unsafe state. In our implementation, Watt simply considers all the other actions until safe trajectory found. In the event that no safe action is found, the agent takes a random action.
Since the SRSSM models stochasticity, we can sample multiple futures arising from action being taken and derive probabilistic estimates of whether an action will lead to violation. this way, we can estimate the safety of an action taken in given state by checking whether the probability of violation occurring (inferred by sampling) exceeds a fixed threshold e (see Algorithm I Moreover, the sampling process can, in practice, be augmented to sample wider range of trajectories (less likely to be taken by the agent) by adding a small amount of noise to actions suggested by the policy during sampling.
Algorithm 1: Approximate Bounded Prescience Shielding in latent space.
Current compact latent state (h, z), unshielded action a, horizon H, number of trajectories sample N, alternative policy Talt and threshold Output: A tuple containing an action and whether or not the shield had to interfere. Initialise array of zeros A with length N.
Input:
for n I..N do
Imagine trajectory p = (h, S (h1, 21) "1 aM (hH, 2H) where a, = =T(hi 2:). 3i E 21, H| such that lo (o(,, zi) = unsafe then A(m] :
end
inter = false. N E>EA A e then
a, dninterfere fered Talt (h,, 21), truc.
end
return (a.interfered).
3.3 TRAINING REGIME
We extend the training regime proposed in Hafner et al. (019a) to include the training and applica- tion of ABPS. Though the full training procedure is detailed in Algorithm we provide an overview below.
5

---

Under review as a conference paper at ICLR 2021
3.3.1 OVERVIEW
The training procedure be split into three phases: data collection, latent world model training and agent training (lines 10-22, 4-6 and 7-8 in Algorithm respectively). These phases are cycled through until convergence.
Data Collection this phase, the agent interacts with the real environment to collect a dataset D S X O R {0, 1 of states, actions, observations, rewards and violations with which latent world model can be learned. At the very start training, we collect trajectories from S seed episodes using random policy; all other times, of the components of the agent active during this phase can be found in Figure in Appendix
the agent's shielded policy. An illustration
Latent World Model Training The goal of this phase is to improve the model of the world so that the policy has an accurate imagined environment train in. To this end, train the latent world model with respect to the objective in Equation On B data sequences of length L sampled from D. An illustration of the components of the agent active during this phase can be found in Figure [ in Appendix
Agent Training Agent training is composed of two steps. First, starting from states from the same B data sequences from the latent world model training phase, the agent imagines trajectories of length H with actions chosen from its current policy. Next, the unshielded policy is updated as in Hafner et al. (2019a 2021).
3.3.2 KBY CHANGES AND CONTRIBUTIONS
We describe and discuss the key changes we have made that differentiate our agent from previous approaches.
Experience Dataset. Elements in the experience dataset D now contain a binary variable represent- ing whether a violation has occurred '
Latent Shielding. Before being sent to the environment ε, actions generated by the unshielded policy π are routed through our newly-proposed latent shield (described Algorithm (T.. The shield returns a new approximately H bouudded safe action at-
Intrinsic Punishment. Whenever a violation occurs (whether detected by the latent shield or by the environment), we override the environment's reward function by instead assigning reward p < O. This discourages the agent's unshielded policy π from taking actions that lead to unsafe states through the standard RL setup. Over time, this means that the shield will have to interfere less as π becomes biased away from unsafe states. This is standard practice in the shielding literature Alshiekh et al., 2018), however new the Dreamer family of agents (Hafner al. 2019a 2021
Shield Introduction Schedule We introduce the novel notion of a shield introduction schedule Il which can enable and disable shielding during training. The rationale behind this is that a shield backed by an inaccurate world model will incorrectly label some safe states as unsafe, and vice versa. In some cases, this may prove detrimental to the training process: suppose lg happens to be initialised in such a way that all states are labelled as unsafe. As a result, the shield can prevent the agent from exploring the environment and improving its internal model of the environment. This, in turn, can prevent the labelling function from learning to correctly differentiate safe states from unsafe states.
To give the agent time to learn good world model before restricting exploration through learned shield, we augment the training procedure with II which aims to gradually introduce shielding. To our knowledge, this is the first time such a system has been proposed. Though there exist wide range of possible implementations of II, in this work we use a simple schedule that seems to work well empirically: start training with shielding disabled and enable shielding once the world model loss begins to plateau (see Appendix H for more details). A detailed comparison of different shield introduction schedules, however, is beyond the scope of this study and left for further work.

---

Under review as a conference paper at ICLR 2021
Algorithm 2: Training Dreamer with Approximate Bounded Prescience Shielding.
Input:
Punishment for violation "panish. shield imagination horizon H, number of steps to imagine ahead for training 7, number of seed episodes S, number of training steps per episode C, number of real environment interaction steps per episode T, sequence length
L, batch size B, environment &, labelling function Lg and shield introduction schedule II.
Initialise dataset with S seed episodes and neural network parameters 0 randomly. while converged do
/* Learning the internal world model and agent pol ley.
co
1..C
)++) Draw B data sequences {(ae; 0t r,, Az) ) D. Compute model states h2, and &4
Update 8 using representation learning.
Imagine trajectory p of length / for each state in each data sequence.
Update the unshielded policy æ based on the imagined trajectories. end
/* Collecting data from the environment. : do
10
Compute model states he 2, from history.
12
Select action a, with the unshielded policy, adding exploration noise desired. if IT has enabled shielding then
14
Pass h4 24 and at into Algorithm
| to obtain the tuple (d,, inter interferedy
else
16
de inter f fered: == at. Rt..fall false.
end
18
Tt, == EEstep (a/).
Check if Ig has detected violation and store the result 0 or in A
20
if Xr = 1 or inter fered then T2 := Tpumish.
21
end
Add experience to dataset D = J ((ot, Ot, tt, AT
22
23 end
EXPERIMENTS
In this section, compare our ABPS agent against Dreamer without shielding (Hafner al.) 2019a 2021 Dreamer with BPS (Giacobbe et al.) 2021), and CPO (Achiam et al. (017). We also em- pirically investigate some aspects of the internal workings of our agent. brief overview of the environments used can be found below with more detailed a discussion in Appendix
Visual Grid World The Visual Grid World (VGW) environment is simple deterministic naviga- tion benchmark with high-dimensiona (64 x 64 x 3) visual observations (see Figure 2 and discrete actions. The agent's (in green) task is navigate randomly-placed targets (in black) while avoid- ing any unsafe locations (in red). We experiment with both fixed and procedurally generated grids.
Cliff Driver The Cliff Driver (CD) environment is a symbolic benchmark with stochastic dynamics and continuous actions. The agent controls the for- ward acceleration of car and is tasked with driving to the edge of a cliff as quickly as possible without overshooting and falling into the sea. Stochastic- ity comes from the fact that at each time step there is a probability Patick that car's controls become "stuck" (Machado et al. 2018).
4.1 PERFORMANCE EVALUATION
Figure 2: Sample visual observation from the VGW environment.
We evaluate our agents on three metrics: (1) average reward per episode at test-time; (2) average number of violations per episode at test-time; and (3)
7

---

Under review as a conference paper at ICLR 2021
Flavour
Metric
Latent
Unshielded
BPS
CPO
Testing Reward
15067 (434)
13148 (249)
12468 (620)
-2925 (1065)
Fixed
Testing Violations
0.30 (0.76)
2.25 (1.60
0 (0)
13.43 (19.25)
Visual
Training
(172)
(833)
(1435)
Grid World
Testing Reward
8084 (2221)
6825 (1427)
1938 (3552)
-1888 (2051)
Procedural
Testing Violations
4.50 (3.59)
33.7 (16.28)
0 (0)
19:60 (13.83)
Training Violations
14018 (1852)
15309 (4686)
0 (0)
18705 (3756)
Testing Reward
8.57 (2.96)
10.76 (3.29)
10.50 (3.28)
7.56 (2.86)
Patick 0.1
Testing Violations
(0)
(0)
3.40 (1.91)
Cliff
Training Violations
(9.60)
90.0 (9.10)
24.0 (13.02)
(3577))
Driver
Testing Reward
8,10 (4.99)
6.63 (8.07)
7.10 (9.52)
6.44 (3.00)
Putick 0.5
Testing Vinlations
0.18 (0.84)
0.54 (1.53)
0.22 (1.18)
0.48 (1.24)
Training Violations
91.8 (16.85)
157.6 (18.4)
80.4 (17.43)
3126 (2823)
Table Comparison of trained agents with standard deviations in parentheses where applicable.
Unshielded
Unshielded BPS
1.5
Rewend (Ted)
0.0
-0.5
0.-
-1.0
-05
-15
1.5
2.0
Figure 3: Reward curves for the model-based agents on fixed (left) and procedurally generated (right) instances of the VGW environment. The shaded area denotes one standard deviation.
total number of violations during training. Results calculated by averaging over five trained versions of each agent. The latent shield horizon H was and 6 for the VGW and CD environments respectively. Moreover, since the action space of the CD environment continuous, we discretise the actions into four bins when performing BPS (see Appendix D.5). Such modifications, however, are not needed for our agent. Full details on training and evaluation can be found in Appendix We also provide further evaluation of our agents on environment-spedee metrics in Appendix F
As can be seen in Table 4 our agent collected more reward test-time than both the unshielded and BPS agents in the VGW environment (reward curves during training can be seen in Figure 3 with further training curves showcased in Appendix E). Moreover, our agent saw a seven-fold reduction in test-time violations compared to the unshielded agent across both the fixed and proce- durally generated environments, averaging at less than one violation per episode for the fixed VGW. This reduction in training violations is even more dramatic when compared to the model-free CPO agent. Our agent outperformed the other agents at test-time in the most stochastic CD environment (Pstick 0.5). This is possibly due to the SRSSM used by our latent shield being better able to capture non-determinism than BPS. Nevertheless, the result is still rather impressive given that the agent using BPS sampled 1024 trajectories every time step whereas our agent only sampled 20.
4.2 OTHER EXPERIMENTS
Qualitative Evaluation of Learned Latent Dynamics We evaluate the quality of each agent's s latent dynamies model by observing its trajectory predictions given a particular starting state and action sequence. Each agent is given the same starting observation and predefined sequence of 10 actions. The agents take these actions in their respective latent world models with the states traversed being decoded for inspection. Results can be seen in Appendix G and indicate that, qualitatively, our agent's 's decoder performed the best, accurately predicting 10 frames into the future. One possible reason for this observation is that the inclusion of the violation detection objective in the SRSSM encourages the model to focus on accurately capturing violation states. Another potential factor is that the BPS agent never actually enters unsafe states and thus finds it difficult to represent them.
8

---

Under review as a conference paper at ICLR 2021
Do Shield Introduction Schedules Actually Help? We perform ablation study with shield in- troduction schedules in the VGW environment. Results can be found in Appendix and demonstrate decreased performance in the absence of shield introduction schedules.
RELATED WORK AND DISCUSSION
Latent World Models Learning latent world models from visual observations has seen growing
interest from the RL community (Wahlström et al. 2013 Watter et al. 2013 Racanière al. 2017
Ha & Schmidhuber 2018 Hafnnr et al.] 2019a b Schrittwieser et al. 2020 Hafner et al.] 2021
One key trend in the literature is that of training agents in their own learned world models (Ha &
Schmidhuber 2018 Hafner et al. 2019a 2021). In this work, we extend the world model formula-
tion used in Hafner et al. ( 2021 to encode notion of safety into state representations.
Safety By Filtering Actions Overriding unsafe actions with safe ones has been a popular approach to safety-aware RL. Introduced into the RL scene by Alshiekh et al. (2018) shielding has received much research interest and has seen applications in real-world settings (Nikou al. 202T). Various works have attempted to address some of its shortcomings of the original formulation. These include allowing the shield to be updated to aid exploration or correct model imperfections (Anderson al,,
2020
Pranger et al. 2021); improving performance in non-deterministic environments (Jansen et al. Li & Bastani 2020); and extending shielding to multi-agent RL (ElSayed-Aly et al. 2021
2020
To our knowledge, few works (Srinivasan al. 2020; Thananjeyan et al., [222] Bharadhwaj al.,
2021
Giacobbe et al. (202)) focus on removing the need for handcrafting an abstraction (among the
most time-cossmmin and error arror-prone aspects of shielding), and only one of them (Giacobbe al.,
2021 achieves this without getting rid of the abstraction altogether, albeit by providing the agent with access to the program that controls the environment.
In contrast, latent shielding tackles the problem head-on by directly learning an abstraction for use by the shield. In this work, the abstraction we use is an SRSSM, which captures stochasticity by design, a useful property for non-deterministic environments. Though our latent shield satisfies an approximation of H H-bounded safety (see Section 2.4 with respect to the learned abstraction, its safety with respect to the true environmental dynamics is not guaranteed and instead relies on the fidelity with which the true dynamics are captured. Furthermore, unlike for its formally-verified predecessors, it is necessary sacrifice that the agent visits unsafe states during training in order to learn a notion of safety (unless, of course, pre-training is possible). Nevertheless, a learned abstraction may be advantageous in settings where handcrafting abstraction is not feasible and privileged access to some simulation (as in Giacobbe et al. (202T)) cannot be assumed. Moreover, latent shielding may provide greater degree of explainability over model-free methods which get rid of the abstraction altogether (Srinivasan et al.) 2020) Thananjeyan et al.) [202]] [Bharadhwaj et al.] 2 if the shield overrides an action, one can reconstruct the imagined unsafe trajectories that led to the interference.
6 CONCLUSIONS
In this paper, we have presented latent shielding, new framework for shielding DRL agents with- out the need for a handcrafted abstraction of the environment. Using this framework, we have designed novel shield and demonstrated that this method not only leads to improved adherence to safety specifications on two benchmark environments with respect to an unshielded agent, but also works out-of-the-boxx on both continuous and discrete environments (unlike its predecessor. BPS). Furthermore, we have demonstrated for the first time that shielding at inappropriate times may ad- versely impact the performance of model-based DRL agents and showed how this phenomenon can be counteracted using novel notion of shield introduction schedules.
Our work opens several exciting avenues for future work. For instance, this work uses a very simple shield introduction schedule; future work may provide richer investigation into the properties of different schedules. Moreover, though demonstrating promising empirical results, our realisation of latent shielding loses the formally verifiable safety guarantees enjoyed by many symbolic shielding approaches whether it is possible to construct a verifiable implementation of latent shielding, or compensate for the loss of formal guarantees, are open problems.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 22-31. PMLR, 06-11 Aug 2017. URLpttts:: http:///coeeeenngg..... mwr wtwpp/ppwwe/////ccaaaoaaa/ee ott.o//ooooooooiiiiooooooooaaaaa
html
Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Greg Anderson, Abhinav Verma, Isil Dillig, and Swarat Chaudhuri. Neurosymbolic reinforce- ment learning with formally verified exploration. In Advances in Neural Information Processing Systems, volume 33, pp. 6172-6183, 2020. URL htt.//ttttss/prroeeeediiee
neurips cc/ pdf
Fahiem Bacchus and Froduald Kabanza. Using temporal logics to express search control knowl- edge for planning. Artificial Intelligence. 116(1):123-111, 2000. ISSN 0004-3702. doi: 10.
1016/500004-30022990000000777700
URL https::/tt// hettps://ww... wwc.eeeneeiiecte www.eeecttcccmmeeeeeeeeeeeeeeeee com/science/
articleep11/5004477990000
Homanga Bharadhwaj. Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian Shkurti, and Ani- mesh Garg. Conservative safety critics for exploration. In International Conference on Learning Representations, 2021. URL URLhttps:
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained rein- forcement learning with percentile risk criteria. J. Mach. Learn. Res., 18( (1)6:607066 January 2017. ISSN 1532-4435.
Ingy ElSayed-Aly, Suda Bharadwaj, Christopher Amato, Rüdiger Ehlers, Ufuk Topeu, and Lu Feng. Safe multi-agent reinforcement learning via shielding. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgen Systems, AAMAS '21, pp. 483-491, Rich- land, SC, 2021. International Foundation for Autonomous Agents and Multiagent Systems. ISBN
9781450383073.
Mirco Giacobbe, Mohammadhosein Hasanbeig, Daniel Kroening, and Hjalmar Wijk. Shielding atari games with bounded prescience. In Proceedings of the 20th International Conference Autonomous Agents and MultiAgent Systems, pp. 1507-1509, 2021. ISBN 978 1503833733
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Pro- ceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 1455-2467,20
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: ing behaviors by latent imagination. In International Conference on Learming Representations, 2019a.
Learn-
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Proceedings of the 36th International Conference on Machine Learning, pp.
PMLR, 2019b. URL
http::/proccee www.nng........mmmmmmmmm.mm..m.m www.sssss......777.7... wherraafneeerrrr 19 .t..mmmlll
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021. URL
https://openrevieer net htte://ooummimddobaaaaaa
Nils Jansen, Bettina Könighofer, Sebastian Junges, Alex Serban, and Roderick Bloem. Safe Re- inforcement Learning Using Probabilistic Shields (Invited Paper). Igor Konnov and Laura Kovács (eds.), 31st International Conference Concurrency Theory (CONCUR 2020), vol- ume 171 Leibniz International Proceedings in Informatics (LIPlcs), pp. 3:1-3:16 Dagstubl, Germany, 2020. Schloss Leriuuhll-Liiiiii--eeee für Informatik. ISBN 978-3-95977-160. 3. doi: www.:///L.PIcc.CCOCC..20020000 URL https: http://drppp..oo dagstuhl. de/oups//e//////////////op vol Voll1textee2022/2221
10

---

Under review as a conference paper at ICLR 2021
Orna Kupferman and Moshe Y. Vardi. Formal Methods in System Design, 19(3):291-33 -144,, 2001. doi: 10. 10.1023241011225422223.
Shuo Li and Osbert Bastani. Robust model predictive shielding for safe reinforcement learning with stochastic dynamics. In 2020 IEKE International Conference Robotics and Automation (ICRA), pp. 7166-7172, 2020. doi: 10.1 N..109//CRA40945..00 2020.911922009
Marlos C. Machado, Mare G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. J. Artif. Int. Res., 61(1):223-562, January 2018. ISSN 1076-9757.
Alexandros Nikou, Anusha Mujumdar, Marin Orlić, and Aneta Vulgarakis Feljan. Symbolic re- inforcement learning for safe ran control. In Proceedings of the 20th International Confer- ence on Autonomous Agents and MultiAgent Systems, AAMAS '21, pp. 1782-1784, Richland, SC. 2021. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450383073.
Stefan Pranger, Bettina Könighofer, Martin Tappler, Martin Deixelberger, Nils Jansen, and Roderick Bloem. Adaptive shielding under uncertainty. In 2021 American Control Conference (ACC), pp- 3467-3474, 2021.
doi: 10.23919AAC55051 1.2051 1.948889.
Sébastien Racanière, Théophane Weber, David P. Reichert, Lars Buesing, Arthur Guez, Danilo Rezende, Adria Puigdomênech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, Razvan Pascanu, Peter Battaglia, Demis Hassabis, David Silver, and Daan Wierstra. for deep reinforcement learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 5694 5705, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
agents
Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint
2019.
Julian Schrittwieser, Toannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588 (7839):604-609, 2020. doi:
K. Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. Learning to be safe: Deep rl with safety critic. ArXiv, abs/2010.14603, 2020.
Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E. Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, (3):4915-4922, 2021.
doi: 10. 109/LRAA
Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe rein- forcement learning via curriculum induction. Advances in Neural Information Processing Sys- tems, 33, 2020.
Niklas Wahlström, Thomas B. Schön, and Mare Peter Deisenroth. From pixels to torques: Policy learning with deep dynamical models, 2015.
Manuel Watter, Jost Tobias Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 2, pp- 2746 2754, 2015.
Tsung- Yen Yang. Justinian Rosca, Karthik Narasimhan, and Peter Ramadge. Projection-based constrained policy optimization. In International Conference on Learning Representations, 2020. URL

---

Under review as a conference paper at ICLR 2021
A FURTHER HELPFUL DIAGRAMS
022
17
π
π
π
ho
h1
h2
20
2
222
E
E
E
Xp
X1
X2
Figure 4: Interacting with the real environment. Observations Fy from the environment are encoded and latent representation is obtained. The representation is passed to and the shield to obtain an H-bounded safe (with respect to the agent's world model) action áz. The circle with the plus sign represents the concatenation operation.
A
So
1
$1
f2
4g
D
ag
4g
D
a
Lg
hg
h
h
2
Zu
Z1
22
Z
E
E
E
X2
Figure 5: Training world model. Observations H, from the experience dataset D encoded by E and used to generate a posterior stochastic state 2i. is concatenated with the deterministic state his and used to predict the reward fr. original observation à; (through decoder D). and whether or not violation has taken place X; (though network 1g). prior stochastic state é, also generated with view to matching the posterior state as closely as possible. The circle with the plus sign represents the concatenation operation.
12

---

Under review as a conference paper at ICLR 2021
B DREAMER Loss FUNCTIONS
Given some dataset of experience De S x Og x
I::
Lobservation
Σ
logp(o),,
[Del (())))
Creward
Σ
logp(r h2
[Del ((b.))...rEE.
LKL
Σ
3LL (1- (1- (2ll=)
(5)
[Del
((b.)).O)ED.
where /3 is hyperparameter for KL balaneing (Hainer et al. 2021 Here, minimising Lobservation and Creward corresponds to maximising the log-likelihood of and é respectively; minimising LKL
corresponds to minimising the KL divergence between the z and
C ENVIRONMENT DETAILS
C.I VISUAL GRID WORLD
The Visual Grid World simple visual navigation benchmark. The environment (see Figure 2 made up of the agent (green circle), target (black circle) and number of unsafe locations (red squares). The actions the agent can take correspond to moving up, down, left, right and staying still; visual observations from the environment are encoded in 64 64 x arrays. The agent's task is navigate to the target, avoiding any unsafe locations. This yields relatively straightforward safety specification:
agent in red square U episode ended
where agent pvent_in_red. red__squaee is if and only if the agent is in an unsafe location, and episode ended is only true at the end of an episode.
The reward function (see Equation 6 used also quite simple, with a small penalty term each time step to encourage movement (though indeed different forms of intrinsic motivation may be also used).
100,
agent reaches target,
- 40,
agent enters unsafe state,
R R(s, 0z)
=
6
10,
agent does not move,
otherwise
We train and test our models on randomised 4x4 instances of the environment, each with either one unsafe states.
C.2 CLIFF DRIVER
The Cliff Driver environment is simple symbolic benchmark with stochastic dynamics. The agent controls car on one mi--dmmeniionl road with continuous actions E [-1, 1] which correspond the acceleration of the car. The car cannot move backwards and its speed v, is lower-bounded at 0 (thus < 0 corresponds to braking as opposed to reversing). At the end of the road, there cliff. The agent starts each episode stationary at fixed distance 2o from the edge of the cliff with its distance subsequent time steps being written Furthermore, at each time step, there is a probability Patick that the car's controls get "stuck" meaning that the current action is ignored and replaced with the previous action.
The agent's goal is to drive the car to the edge of the cliff as quickly as possible, without overshooting and falling into the sea. Observations from the environment are given two-dimensional vectors encoding the distance from the edge of the cliff in one component and the speed of the car in the other. The safety specification thus be written:
13

---

Under review as a conference paper at ICLR 2021
-agent.fell.off.clfff U episode
where agent www.fofffffffff.... ccliff is true if and only if the agent has overshot the cliff, and episode ended is only true at the end of an episode.
Finally, the reward function is given as:
1
agent has not fell off cliff,
R(s,, az)
-5, z0
otherwise
D EVALUATION DETAILS
D.I GENERAL
Agents were trained from different seeds on machine with a single NVIDIA RTX 2080 GPU, an AMD Ryzen 7 2700X processor and 64GB of RAM. All agents trained using the Adam optimiser (see Appendix D.6 for hyperparameters). Each agent was trained five times, from differ- random seeds (except the CPO agent in the VGW environments, which, due computational limitations, was only trained from three seeds). For each environment, the model-based agents were trained for the same number of steps; the CPO agent was allowed train for more steps (2x longer for the VGW environments and 5x longer for the CD environments).
D.2 EVALUATION PROTOCOL
Agents trained on the fixed VGW were evaluated over ten 500-step episodes of the same environ- ment configuration they were trained in. Agents trained on the procedurally generated VGW were evaluated over random environment configurations.
Agents trained on the CD environments were evaluated over ten episodes, each with different random seed.
D.3 NETWORK ARCHITECTURES
We use the network architectures proposed Hafner et al.) (2019a). The number of nodes in each layer varied depending the environment and can be found in Appendix D.6] The size of the fully-connected layers and output the observation encoders and decoders can found under Observation Embedding Size; the size of the fully-connected layers in the reward, value, observa- tion and violation models can be found under NN Hidden Layer Size. We implement our encoder for symbolic observations from the CD environment connected hidden layers and ReLU activations. For the CPO agent, we use the same observation encoders and policy networks mentioned above.
feed-forward network with three fully-
D.4 SHIELD INTRODUCTION SCHEDULES
For details on how we constructed these schedules, see Appendix H
Visual Grid World Start with shield disabled. After 10 episodes (including the 5 seed episodes), enable shielding every third episode. After 20 episodes, enable shielding every other episode. After 30 episodes, enable shielding all the time. Linearly decay the unsafe threshold c form 0.5 to 0.125 over the entire course of training.
Cliff Driver Start with shield disabled. After 60 episodes (including the seed episodes), enable shielding all the time.
14

---

Under review as a conference paper at ICLR 2021
D.5 BOUNDED PRESCIENCE SHIELD FOR CONTINUOUS ACTION SPACES
In our experiments in the CD environment, make the action space amenable BPS by discretis- ing actions into four bins. When the agent outputs a continuous action, we allocate the action to bin by rounding the action the nearest action in the f-1, 0011, 0.1, This rounded action is then sent to the shield, which exhaustively searches through the bounded trajectories generated by taking the actions defined the aforementioned set.
D.6 HYPERPARAMETERS
Hyperparameter values were kept constant for the base Dreamer model in all agents in each envi- ronment. For the CPO agent, use the same hyperparameters as Ray et al. (2019).
Hyperparameter
Symbol
Value
Grid World
Cliff Driver
Deterministic State Size
200
8
Stochastic State Size
30
16
NN Hidden Layer Size
200
16
Observation Embedding Size
1024
32
Discount Factor
7
0.99
0.99
Action Repeat
1
Seed Episodes
S
50
Episode Length
T
500
20
Batch Size
50
250
Sequence Length
L
50
10
Training Steps
C
100
100
Exploration Noise Variance
0.3
0.3
Imagination Horizon
15
KL Balancing Ratio
B: (1 3)
4:1
4:1
Violation Balancing Ratio
a: a:
1:3
1:3
Model Learning Rate
le-3
le-4
Policy Learning Rate
8e-5
8e-5
Value Learning Rate
8c-5
8c-7
Bit Depth
Adam Epsilon
lc-7
le-7
Adam Beta
0.9, 0.999
0.9, 0.999
Latent Shield Horizon
6
Latent Shield Sampled Trajectories
N
20
10
Latent Shield Unsafe Threshold
0.15
0.15
CPO Cost Limit
5
0
E TRAINING CURVES
Shaded regions in the plots denote single standard deviation. Note that rewards for the shielded agents may look lower due intrinsic punishment. That is, the shields can assign punishments (to discourage potentially unsafe behaviour) in addition to the environment's 's reward function. Further- more, since the CPO agents were were trained for many more steps than the others, we plot their final performance single dotted lines.
E.1
VISUAL GRID WORLD REWARD
See Figure
15

---

Under review as a conference paper at ICLR 2021
E.2
VISUAL GRID WORLD VIOLATIONS
Unshielded Latent
Unshieided
BPS
CPO (405 steps)
Volittone
60
20
0.0
0.
100
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Enviromeent Steps (1e5)
Environment Steps (1e5)
Figure 6: Violations per episode for agents fixed (left) and procedurally generated (right) in- stances of the VGW environment.
E.3
CLIFF DRIVER REWARD
10
Reward
Unshielded
Unshielded
Latent
Latent
BPS
-30
steps)
1.5
3.5
(1c3)
Enviiromment
Figure 7: Training reward curves for agents on Psticky = 0.1 (left) and Psticky = 0.5 (right) in- stances of the CD environment.
E.4 CLIFF DRIVER VIOLATIONS
Unshielded
6-
BPS
CPO (2e4 steps)
Wolatoon
Unshielded
CPO (2e4 steps)
0.
1.5
2.0
3.5
2.5
3.0
Environment Steps (1e3)
Enveronment Steps (1e3)
Figure 8: Violations per episode for agents on Psticky 0.1 (left) and Patinky 0.5 (right) instances of the CD environment.
F ADDITIONAL EVALUATION METRICS
We evaluate the agents two further environment-specdn evaluation metrics below. The first is the average number of targets reached during a VGW episode. The second is the average time taken travel 90% of the distance the edge of the cliff during CD episode.
16

---

Under review as a conference paper at ICLR 2021
Flavour
Metrie
Latent
Unshielded
BPS
CPO
Visual Grid World
Fixed
Targets Reached
154,16 (13.09)
136.52 (11.92)
108.68 (10:74)
2.53 (2.69)
Procedural
Targets Reached
64.36 (32,97)
84.82 (19.8))
55.30 (39.81)
9.94 (10.37)
Cliff Driver
Patick 0.1
Time Till 90%
19.4 (4.42)
15.96 (4.12)
15.90 (4,49)
14.24
Patsoh
0.5
Time Till 90%
18.70 (4.69)
18.32 (4.88)
18.78 (4.80)
17.38 (4.27)
Table 2: Comparison of trained agents with standard deviations in parentheses where applicable.
G QUALITY OF LATENT DYNAMICS
Figure 9: Comparison of latent trajectories in the agents' learned models of the environment. All trajectories begin from the leftmost observation and time progresses from left to right. The bottom row of arrows denote the actions taken just before each time step. From top to bottom, the three rows of images correspond to trajectories in the latent space of the BPS agent, the unshielded agent, and our agent.
H METHOD FOR CONSTRUCTING SHIELD INTRODUCTION SCHEDULE
The method used construct our shield introduction schedules is follows:
1. Train the agent without shielding and plot the violation loss Lviolation (see Equation 3
2. Look for the "elbow" in the plot (see Figure T0 for example).
3. Disable the shield until around the time of the "elbow". Once the elbow has been reached, enable shielding. If the violation curve does not plateau as nicely as in Figure IO consider introducing the shield more gradually (e.g. once every episodes, where n eventually decays to 1).
0.66
0.50
65
20
Taloodes
Figure 10: Plot of violation loss Lviolation from the CD environment. As you can see, the "elbow" the curve occurs at around 60 episodes.
17

---

Under review as a conference paper at ICLR 2021
I ABLATION STUDY WITH SHIELD INTRODUCTION SCHEDULES
We compare the first 100 training episodes of our agent in the fixed VGW environment with and without the shield introduction schedule outlined in Appendix D.4 As in the other experiments, results are calculated over five trained agents. Though both agents start at roughly the same reward, the agents with a shield introduction schedule consistently outperforms their counterparts without shield introduction schedules.
With Without
Reward Dett
Steps (104)
Figure 11: Training reward curves of agents using latent shielding with and without a shield intro- duction schedule (SIS).
J A NOTE ON THE TITLE
Yes, the title of this work is indeed a spin off Do Androids Dream of Electric Sheep?. Here, androids dreaming of electric fences refers RL agents dreaming (i.e. using latent imagination) about being punished for entering forbidden parts of an environment' 's state space.
18


Reviews:
[{"review_id": "ckFdz2wMMGN", "reviewer": "Reviewer_Lq99", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution is minor. The proposed approach is mostly is a straightforward combination of existing methods. The experiments are limited in complexity and thoroughness. The baseline methods such as Dreamer (Hafner 2021) have been evaluated over a wide range of test scenarios. Therefore two environments for evaluation are not sufficient. Furthermore, for fair comparison methods from Safe Reinforcement Learning should also be included. ", "summary_of_the_paper": "This paper proposes a safe Reinforcement Learning approach using Latent Shielding. The agent learns a  data-driven recurrent state-space model representing a world model based on an existing approach (Hafner et al., 2019b). The learned latent world model allows agents to foresee various trajectories arising from different actions and avoid unsafe states. In their setting, the environment also provides binary labeling over states indicating an occurrence of violation in each step, formulated as syntactically co-safe Linear Temporal Logic (scLTL) (Kupferman & Vardi, 2001). The proposed approach is evaluated on two environments (1) Visual Grid World and (2) Cliff driver and compared against Dreamer agent with no shielding (Hafner et al., 2019a; 2021) and a Dreamer agent with Bounded Prescience Shield (Giacobbe et al., 2021).\n", "summary_of_the_review": "The paper contribution is minor. Their experiments are limited in terms of test environments and benchmark methods.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FGia_prfhE6", "reviewer": "Reviewer_eXDd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed approach is simple, and intuitive, which is a major advantage. Fig. 1 clearly describes the contribution of the paper on top of Dreamer, by incorporating a safety constraint violation predictor. The violation predictor is learned based on an approximate shielding approach inspired by classical notions of a shield. The shield checks the probability of constraint violation given the policy outputs a certain action, and if this is the case, then an alternate policy based on rejection sampling is activated. The paper is well-written and easy to follow, with detailed algorithms mentioning the flow of the approach. \n\n\nI have a few questions and several concerns regarding the paper which I have listed below:\n\n- An important limitation of the approach is that since visiting unsafe states are discouraged during exploration (by penalizing the agent with -negative rewards), the agent is unlikely to visit unsafe states, which is good for being safety but bad for learning an accurate dynamics model in those regions. Hence, dynamics model rollouts are likely to be very inaccurate around unsafe parts of the state-space, thereby making the planning process inaccurate. \n\n- Another limitation of the approach is that it necessarily requires visiting a lot of unsafe states during training, for learning the world model, and hence can never be completely safe. This is also reflected in the results in Table 1 where the training violations are significantly high. It would be helpful to have a discussion about acknowledgement of this limitation in the paper. \n\n- There are no guarantees of how accurate the safety shield is given errors in the learned dynamics model. It will be helpful to formally quantify the safety constraint satisfaction as some function of model-error, or provide intuitions about the extent to which inaccuracies in the model are likely to affect safety violations. \n\n- Since $\\pi_{\\text{alt}}$ in equation 4 outputs a \"safe\" action by rejection sampling, there is no guarantee that a safe action if it exists will be found. This is because the learned model is inaccurate, so at any given time, when a safe action is not found, we can't be sure whether this is because a safety violation is inevitable, or because there exists a safe action which is not feasible under the current (inaccurate) dynamics. \n\n-  There are no comparisons to model-free safe RL approaches (for example the constrained optimization and safety critics papers cited in the 2nd paragraph of the introduction). In comparison to prior safe RL works that learn a safe policy without learning a safe dynamics+rewards model i.e. in comparison to safe model-free approaches, the proposed approach will likely suffer from a compounding of errors. This is likely to happen because the proposed method has a large number of learned components (parts of the model) and relies on the model being accurate for planning. Hence, a comparison with some prior model-free safe RL methods is important. \n\n- A very important limitation is the lack of sufficient experiments. Experiments are only on toy settings. It is necessary to evaluate safe behavior on more complex dynamical systems (for example robot manipulation, locomotion, navigation environments). Many of the prior safe RL papers cited evaluate on such environments. \n\n- [clarification] Is the unshielded variant (in blue in the plots), normal Dreamer? If so, then how does the proposed method compare with Dreamer in the normal DeepMind Control Suite environments? It is necessary to see this in order to understand whether the proposed method achieves safe behavior while compromising on task performance in environments where safety is not explicitly important. \n\n- [minor] I am curious if the title has any relevance to the contents of the paper? I understand that it is a spin-off of \"Do Androids Dream of Electric Sheep\" and the connection to the paper in my understanding is: Dream --> Dreamer; and Fences --> Shield. If this is the case / there is more to it, then a one line explanation of it somewhere in the paper / in the appendix might be helpful.", "summary_of_the_paper": "This paper proposes a framework for safe model-based RL through latent shielding. The key contribution is extension of an existing MBRL algorithm Dreamer with shielding such that the agent is penalized for taking unsafe actions during exploration; and during planning, the agent can sample multiple plausible futures through the learned dynamics model and avoid taking actions that lead to unsafe states.", "summary_of_the_review": "This paper modifies a prior model-based RL framework, Dreamer to incorporate a safety violation predictor based on latent shielding. While this idea is simple, easy to implement, there are several issues with the paper with regard to severe safety violations during training, compounding errors due to model error and policy error, and lack of experiments + comparisons. As such, I don't think the paper is ready for publication, and does not provide concrete reliable takeaways. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0rkNtfQcLw7", "reviewer": "Reviewer_HtqJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and structured and accessible to the reader.\n\nThe new latent shielding has one major drawback and weakness, which is of course mention and acknowledged in the paper, but still a concern to some degree: The guarantees that other shielding techniques have are lost as the method only works on the latent model.\nThis is advantage and disadvantage at the same time. The advantage is the non-requirement of hand-crafted rules. However, violation information is required during training which also needs to be modeled from somewhere, so the experts are still needed, although it might be easier to identify a violation than to mitigate it from rules. The disadvantage is that the model cannot be deployed alone for RL safety, but at least for other applications or as the main safety technique with an additional external shield as fallback.\n\nI'm a bit disappointed by the alternative action in case of a violation. Here I would expect that the world model would allow to perform better planning than just selecting any non-violating action.\n\nThe experimental evaluation is okay. A more complicated environment could have been chosen to really highlight the advantage of not needing hand-crafted rules could have been chosen, e.g. car racing or also one from Alshiekh 2018 for comparison.\n\nThe method itself is sound, although it is based on many existing contributions and it is a judgement call whether the contributions over the existing techniques are sufficiently high, but I think they are and it is an interesting contribution and basis for future work.", "summary_of_the_paper": "The paper introduces shielding technique for RL safety into world model RL agents, like the Dreamer model.\nThe shielding technique works on the latent space and is trained from violation information only, i.e. without the hand-crafted design of safety rules.\nAn experimental evaluation is performed on two environments and compared against the unshielded version and BPS from the literature.", "summary_of_the_review": "The paper explores an interesting direction for RL safety and shielding and even though it cannot provide performance guarantees, it is helpful to the agent's performance and reduces some violations.\nThe contribution will be helpful to the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-n9tesipiOe", "reviewer": "Reviewer_WV6o", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this was an interesting paper, but I have several concerns regarding the experiments.\n - One issue is that the difference in performance between “Latent” and “Unshielded” does not seem significant (i.e., the error bars are substantially overlapping in Figure 3 and in several rows of Table 1). \n - It’s difficult to interpret the numerical values of the test rewards. It would be helpful to report more interpretable performance metrics, such as the fraction of targets reached in the VGW environment, and the time-to-edge-of-cliff in the CD environment.\n - “Unshielded” does not seem like a fair baseline, since it does not make use of the labelling function (while the “Latent” method has access to the labelling function at training time). A stronger baseline might be [Trial Without Error](https://arxiv.org/abs/1707.05173), which also trains a classifier to predict whether or not states are unsafe using labels from an oracle (in this case, a human supervisor), and trains an RL agent to avoid unsafe states.\n\nI’m also concerned about the novelty of the method: it is essentially Dreamer, but with a safety constraint on the planned trajectories. I may have missed the algorithmic contribution of this paper, and am happy to revise this comment during the discussion period.\n\nUpdate\n----\nThank you to the authors for the thoughtful response. I have increased my score.", "summary_of_the_paper": "The paper proposes a safe model-based reinforcement learning algorithm that trains a classifier to predict whether or not a state is unsafe, and plans trajectories that avoid unsafe states. The key idea is to train the classifier on latent representations of states given by the latent dynamics model in a recurrent state-space model. Experiments in simulated navigation tasks show that in some environments, the proposed method achieves higher reward and violates safety constraints less often than an ablated variant that does not constrain planned trajectories to be safe.", "summary_of_the_review": "Weak experimental results and lack of comparisons to prior methods", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a modification of the Dreamer method to account for safety constraints.
There is agreement among the majority of reviewers that the paper lacks in novelty (with respect to Dreamer), in the safety analysis, and in convincing experiments.
I encourage the authors to take the detailed reviews into account when improving their work.
Decision: Reject