Do Androids Dream of Electric Fences? Safety-Aware Reinforcement Learning with Latent Shielding

The growing trend of fledgling reinforcement learning systems making their way into real-world applications has been accompanied by growing concerns for their safety and robustness . In recent years , a variety of approaches have been put forward to address the challenges of safety-aware reinforcement learning ; however , these methods often either require a handcrafted model of the environment to be provided beforehand , or that the environment is relatively simple and lowdimensional . We present a novel approach to safety-aware deep reinforcement learning in high-dimensional environments called latent shielding . Latent shielding leverages internal representations of the environment learnt by model-based agents to “ imagine ” future trajectories and avoid those deemed unsafe . We experimentally demonstrate that this approach leads to improved adherence to formallydefined safety specifications . 1 INTRODUCTION . The steady trickle of reinforcement learning ( RL ) systems making their way out of the lab and into the real world has cast a spotlight on the safety and robustness of RL agents . The motivation behind this should be relatively easy to grasp : when training an agent in real-world settings , it is desirable that some states are never reached as they could , for instance , cause permanent damage to the hardware the agent is controlling . We can thus informally define the notion of safety-aware RL in terms of the classical RL setup with the added requirement that the number of unsafe states visited be minimised . Under this definition , however , it has been found that many state-of-the-art RL algorithms unnecessarily enter unsafe states despite safe alternatives being available and there being a positive correlation between avoiding such states and reward ( Giacobbe et al. , 2021 ) . The field of safety-aware RL encompasses a multitude of approaches ranging from constrained policy optimisation ( Chow et al. , 2017 ; Achiam et al. , 2017 ; Yang et al. , 2020 ) to safety critics ( Srinivasan et al. , 2020 ; Bharadhwaj et al. , 2021 ; Thananjeyan et al. , 2021 ) to meta-learning ( Turchetta et al. , 2020 ) . In this work , we focus on a particular family of approaches known as shielding ( Alshiekh et al. , 2018 ; Anderson et al. , 2020 ; Giacobbe et al. , 2021 ; ElSayed-Aly et al. , 2021 ; Pranger et al. , 2021 ) . Central to shielding is the notion of a shield , a filter that checks actions proposed by the agent ’ s existing policy with reference to a model of the environment ’ s dynamics and some formal safety specification . The shield overrides actions that may lead to an unsafe state using some other safe ( but by no means optimal ) policy . A key advantage of many shielding approaches is that the resulting shielded policies are formally verifiable ; however , a shortcoming is that they require a model of environmental dynamics - typically handcrafted - to be provided in advance . Providing such a model may prove difficult for complex real-world environments , with inaccuracies and human biases creeping into handcrafted models . In this work , we propose a safe RL agent that makes uses of latent shielding , an approach to shielding in environments where a formally-specified dynamics model is not available in advance . At an intuitive level , the agent uses a data-driven approach to learn its own latent world model ( a component of which is a dynamics model ) which is then leveraged by a shield . The shield then uses the agent ’ s model to “ imagine ” trajectories arising from different actions , forcing the agent to avoid those it foresees leading to unsafe states . In addition , the agent can be trained within its own latent world model thus reducing the number of safety violations seen during training . Contributions The main contribution of this work is a framework for shielding agents in complex , stochastic and high-dimensional environments without knowledge of environmental dynamics a priori . We further introduce a new method to aid exploration when training shielded agents . Though our framework loses the formal safety guarantees associated with traditional symbolic shielding approaches , our experiments illustrate that latent shielding reduces unsafe behaviour during training and achieves testing performance comparable to previous symbolic approaches . 2 PRELIMINARIES . In this section , we cover some relevant background topics . We begin by introducing our problem setup for safety-aware RL and give an overview of the specification language used in this work . This is followed by an outline of the latent world model we make use of in this work as well as a discussion on shielding . 2.1 PROBLEM SETUP . We consider an agent interacting with an environment E modelled as a partially observable Markov decision process ( POMDP ) with states s ∈ SE , observations ot ∈ OE , agent-generated actions at ∈ AE and scalar rewards rt ∈ R over discrete time steps t ∈ [ 0 , 1 , ... , T − 1 ] . We assume the environment has been augmented with a labelling function LφE : SE → { safe , unsafe } that , at each time step , informs us whether a violation has occurred with respect to some formal safety specification φ . For the avoidance of doubt , we define a violation to have occurred whenever φ does not hold . This is a weaker assumption than previous works in shielding ( which assume access to an abstraction of the environment ) and can be thought as a secondary safety-focused reward function with a binary output . Intuitively , the goal of the agent is to learn a policy π that maximises its expected cumulative reward while minimising the number of violations of φ . 2.2 SYNTACTICALLY CO-SAFE LINEAR TEMPORAL LOGIC . In this work , we use syntactically co-safe Linear Temporal Logic ( scLTL ) ( Kupferman & Vardi , 2001 ) as our specification language . Valid scLTL formulae over some set of atomic propositions AP can be constructed according to the following grammar : φ : := true | d | ¬d | φ ∨ φ | φ ∧ φ | ©φ | φ ∪ φ | φ ( 1 ) where d ∈ AP , ¬ ( negation ) , ∨ ( disjunction ) , ∧ ( conjunction ) are the familiar operators from propositional logic , and © ( next ) , ∪ ( until ) and ( eventually ) are temporal operators . We can monitor a co-safe LTL specification using a technique known as progression ( Bacchus & Kabanza , 2000 ) . 2.3 RECURRENT STATE-SPACE MODELS . We refer to the predictive model of an environment maintained by a model-based agent as its world model . World models can be learnt from experience and be used both as a substitute for the environment during training ( Ha & Schmidhuber , 2018 ; Hafner et al. , 2021 ) and for planning at run-time ( Hafner et al. , 2019b ) . Though many realisations of the notion of a world model exist , the world model used in this work is based on the recurrent state-space model ( RSSM ) proposed by Hafner et al . ( 2019b ) . An RSSM is composed of three key components : a latent dynamics model , a reward model , and an observation model . These components act on compact states formed from the concatenation of a deterministic latent state ht and stochastic latent state zt . Latent Dynamics Model The latent dynamics model is made up of a number of smaller models . First , the recurrent model ht = f ( ht−1 , zt−1 , at−1 ) is used to compute the deterministic latent state based on the previous compact state and action . From ht and the current observation ot , a distribution q ( zt|ht , ot ) over posterior stochastic latent states zt is computed by the representation model . At the same time , a distribution p ( ẑt|ht ) over prior stochastic latent states ẑt is computed by the transition model , based only on ht . During training , the transition model attempts to minimise the Kullback Leibler ( KL ) divergence between the prior and posterior stochastic latent state distributions . In doing this , the RSSM learns to predict future latent states ( using the recurrent and transition models ) without access to future observations . Observation Model The observation model computes the distribution p ( ôt|ht , zt ) over observations ôt for a particular state . Though not strictly needed , the observation model can prove useful for visualising predicted future states and providing a richer training signal . Reward Model The reward model computes the distribution p ( r̂t|ht , zt ) over rewards r̂t for a particular state . In practice , the distributions p and q are implemented with neural networks pθ and qθ respectively , parameterised by some set of parameters θ . These latent dynamics models define a fully-observable Markov decision process ( MDP ) as the latent states in the agent ’ s own internal model can always be observed by the agent ( Hafner et al. , 2019a ) . We denote the state space of this MDP ( comprised of compact latent states ) as SI . 2.4 SHIELDING . The classical formulation of shielding in RL is given by Alshiekh et al . ( 2018 ) . It assumes access to two ingredients : an LTL safety specification and abstraction ( a MDP model of the environment that captures the aspects of the environment relevant for planning ahead with respect to the safety specification ) . These ingredients are used to construct a formally verifiable reactive system that monitors the agent ’ s actions , overriding those which lead to violation states . Proposed by Giacobbe et al . ( 2021 ) , bounded prescience shielding ( BPS ) avoids the need for handcrafted abstractions by exploiting the fact that some agents are trained in computer simulations . The shield operates by leveraging access to the program underlying the simulation to look ahead into future states within some finite horizon . Using BPS over classical shielding does , however , come with a few disadvantages . Firstly , it requires access to the simulation at run-time which may prove difficult to provide ( especially in cases where running the simulation is computationally expensive ) . Moreover , an agent using BPS , even when starting from a safe state , can find itself entering unsafe states in cases where the number of steps between a violation being caused by an action and the violation state itself exceeds the shield ’ s look-ahead horizon . This is not the case for classical shielding which resembles BPS with an infinite horizon . 2.5 BOUNDED SAFETY . The notion of safety used by BPS is defined over MDPs . For an arbitrary MDP with states S and actions A , a bounded trajectory ρ of length H is a sequence of states and actions s0 a0−→ s1 a1−→ . . . an−1−−−→ sn comprised of no more than H states and with the final state sn either being a terminal state or n = H − 1 . We further denote the set of all finite trajectories starting from some arbitrary state s ∈ S by % ( s ) and the set of all bounded trajectories of length H that start from s by % H ( s ) . We say a bounded trajectory ρ of length H satisfies H-bounded safety with respect to safety specification φ , written SH ( ρ , φ ) , if and only if for all si ∈ ρ , LφE ( si ) = safe . Moreover , we can extend the notion of H-bounded safety over the set of policies : a policy π is H-bounded safe with respect to φ , denoted as SH ( π , φ ) , if and only if for all s ∈ S , • either there exists some ρ ∈ % H ( s ) such that SH ( ρ , φ ) and π ( s0 ) = a0 ; • or for all ρ ∈ % H ( s ) , ¬SH ( ρ , φ ) . In other words , the policy will choose a safe trajectory as long as one exists . Finally , we formally define a violation of φ to be inevitable in state s0 ∈ S if and only if for all ρ ∈ % ( s0 ) , ¬SH ( ρ , φ ) .

Summary: This paper proposes a safe Reinforcement Learning approach using Latent Shielding. The agent learns a  data-driven recurrent state-space model representing a world model based on an existing approach (Hafner et al., 2019b). The learned latent world model allows agents to foresee various trajectories arising from different actions and avoid unsafe states. In their setting, the environment also provides binary labeling over states indicating an occurrence of violation in each step, formulated as syntactically co-safe Linear Temporal Logic (scLTL) (Kupferman & Vardi, 2001). The proposed approach is evaluated on two environments (1) Visual Grid World and (2) Cliff driver and compared against Dreamer agent with no shielding (Hafner et al., 2019a; 2021) and a Dreamer agent with Bounded Prescience Shield (Giacobbe et al., 2021).