Safe Reinforcement Learning by Imagining the Near Future

1 Introduction . Reinforcement learning ( RL ) enables the discovery of effective policies for sequential decisionmaking tasks via trial and error [ Mnih et al. , 2015 , Gu et al. , 2016 , Bellemare et al. , 2020 ] . However , in domains such as robotics , healthcare , and autonomous driving , certain kinds of mistakes pose danger to people and/or objects in the environment . Hence there is an emphasis on the safety of the policy , both at execution time and while interacting with the environment during learning . This issue , referred to as safe exploration , is considered an important problem in AI safety [ Amodei et al. , 2016 ] . In this work , we advocate a model-based approach to safety , meaning that we estimate the dynamics of the system to be controlled and use the model for planning ( or more accurately , policy improvement ) . The primary motivation for this is that a model-based method has the potential to anticipate safety violations before they occur . Often in real-world applications , the engineer has an idea of what states should be considered violations of safety : for example , a robot colliding rapidly with itself or surrounding objects , a car driving on the wrong side of the road , or a patient ’ s blood glucose levels spiking.Yet model-free algorithms typically lack the ability to incorporate such prior knowledge and must encounter some safety violations before learning to avoid them . We begin with the premise that in practice , forward prediction for relatively few timesteps is sufficient to avoid safety violations . Consider the illustrative example in Figure 1 , in which an agent controls the acceleration ( and thereby , speed ) of a car by pressing the gas or brake ( or nothing ) . Note that there is an upper bound on how far into the future the agent would have to plan to foresee and ( if possible ) avoid any collision , namely , the amount of time it takes to bring the car to a complete stop . Assuming that the horizon required for detecting unsafe situations is not too large , we show how to construct a reward function with the property that an optimal policy will never incur a safety violation . A short prediction horizon is also beneficial for model-based RL , as the well-known issue of compounding error plagues long-horizon prediction [ Asadi et al. , 2019 ] : imperfect predictions are fed back into the model as inputs ( possibly outside the distribution of inputs in the training data ) , leading to progressively worse accuracy as the prediction horizon increases . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Our main contribution is a model-based algorithm that utilizes a reward penalty – the value of which is prescribed by our theoretical analysis – to guarantee safety ( under some assumptions ) . Experiments indicate that the practical instantiation of our algorithm , Safe Model-Based Policy Optimization ( SMBPO ) , effectively reduces the number of safety violations on several continuous control tasks , achieving a comparable performance with far fewer safety violations compared to several model-free safe RL algorithms . Code is made available at https : //github.com/gwthomas/Safe-MBPO . 2 Background . In this work , we consider a deterministic1 Markov decision process ( MDP ) M = ( S , A , T , r , ) , where S is the state space , A the action space , T : S ⇥A ! S the transition dynamics , r : S ⇥A ! [ rmin , rmax ] the reward function , and 2 [ 0 , 1 ) the discount factor . A policy ⇡ : S ! ( A ) determines what action to take at each state . A trajectory is a sequence of states and actions ⌧ = ( s0 , a0 , r0 , s1 , a1 , r1 , . . . ) where st+1 = T ( st , at ) and rt = r ( st , at ) . Typically , the goal is to find a policy which maximizes the expected discounted return ⌘ ( ⇡ ) = E⇡ [ P1 t=0 t rt ] . The notation E⇡ denotes that actions are sampled according to at ⇠ ⇡ ( st ) . The initial state s0 is drawn from an initial distribution which we assume to be fixed and leave out of the notation for simplicity . The Q function Q⇡ ( s , a ) = E⇡ [ P1 t=0 t rt | s0 = s , a0 = a ] quantifies the conditional performance of a policy ⇡ assuming it starts in a specific state s and takes action a , and the value function V ⇡ ( s ) = Ea⇠⇡ ( s ) [ Q⇡ ( s , a ) ] averages this quantity over actions . The values of the best possible policies are denoted Q⇤ ( s , a ) = max⇡ Q⇡ ( s , a ) and V ⇤ ( s ) = max⇡ V ⇡ ( s ) . The function Q⇤ has the important property that any optimal policy ⇡⇤ 2 argmax⇡ ⌘ ( ⇡ ) must satisfy P ( a⇤ 2 argmaxa Q⇤ ( s , a ) ) = 1 for all states s and actions a⇤ ⇠ ⇡⇤ ( s ) . Q⇤ is the unique fixed point of the Bellman operator B⇤Q ( s , a ) = r ( s , a ) + max a0 Q ( s0 , a0 ) where s0 = T ( s , a ) ( 1 ) In model-based RL , the algorithm estimates a dynamics model bT using the data observed so far , then uses the model for planning or data augmentation . The theoretical justification for model-based RL is typically based some version of the “ simulation lemma ” , which roughly states that if bT ⇡ T then ⌘̂ ( ⇡ ) ⇡ ⌘ ( ⇡ ) [ Kearns and Singh , 2002 , Luo et al. , 2018 ] . 1Determinism makes safety essentially trivial in tabular MDPs . We focus on tasks with continuous state and/or action spaces . See Appendix A.2 for a possible extension of our approach to stochastic dynamics . 3 Method . In this work , we train safe policies by modifying the reward function to penalize safety violations . We assume that the engineer specifies Sunsafe , the set of states which are considered safety violations . We must also account for the existence of states which are not themselves unsafe , but lead inevitably to unsafe states regardless of what actions are taken . Definition 3.1 . A state s is said to be • a safety violation if s 2 Sunsafe . • irrecoverable if s 62 Sunsafe but for any sequence of actions a0 , a1 , a2 , . . . , the trajectory defined by s0 = s and st+1 = T ( st , at ) for all t 2 N satisfies st̄ 2 Sunsafe for some t̄ 2 N. • unsafe if it is unsafe or irrecoverable , or safe otherwise . We remark that these definitions are similar to those introduced in prior work on safe RL [ Hans et al. , 2008 ] . Crucially , we do not assume that the engineer specifies which states are ( ir ) recoverable , as that would require knowledge of the system dynamics . However , we do assume that a safety violation must come fairly soon after entering an irrecoverable region : Assumption 3.1 . There exists a horizon H⇤ 2 N such that , for any irrecoverable states s , any sequence of actions a0 , . . . , aH⇤ 1 will lead to an unsafe state . That is , if s0 = s and st+1 = T ( st , at ) for all t 2 { 0 , . . . , H⇤ 1 } , then st̄ 2 Sunsafe for some t̄ 2 { 1 , . . . , H⇤ } . This assumption rules out the possibility that a state leads inevitably to termination but takes an arbitrarily long time to do so . The implication of this assumption is that a perfect lookahead planner which considers the next H steps into the future can avoid not only the unsafe states , but also any irrecoverable states , with some positive probability . 3.1 Reward penalty framework . Now we present a reward penalty framework for guaranteeing safety . Let fMC = ( S , A , eT , r̃ , ) be an MDP with reward function and dynamics r̃ ( s , a ) , eT ( s , a ) = ⇢ ( r ( s , a ) , T ( s , a ) ) s 62 Sunsafe ( C , s ) s 2 Sunsafe ( 2 ) where the terminal cost C 2 R is a constant ( more on this below ) . That is , unsafe states are “ absorbing ” in that they transition back into themselves and receive the reward of C regardless of what action is taken . The basis of our approach is to determine how large C must be so that the Q values of actions leading to unsafe states are less than the Q values of safe actions . Lemma 3.1 . Suppose that Assumption 3.1 holds , and let C > rmax rmin H ⇤ rmax . ( 3 ) Then for any state s , if a is a safe action ( i.e . T ( s , a ) is a safe state ) and a0 is an unsafe action ( i.e . T ( s , a ) is unsafe ) , it holds that eQ⇤ ( s , a ) > eQ⇤ ( s , a0 ) , where eQ⇤ is the Q⇤ function for the MDP fMC . Proof . Since a0 is unsafe , it leads to an unsafe state in at most H⇤ steps by assumption . Thus the discounted reward obtained is at most H ⇤ 1X t=0 t rmax + 1X t=H⇤ t ( C ) = rmax ( 1 H ⇤ ) C H⇤ 1 ( 4 ) By comparison , the safe action a leads to another safe state , where it can be guaranteed to never encounter a safety violation . The reward of staying within the safe region forever must be at least rmin 1 . Thus , it suffices to choose C large enough that rmax ( 1 H ⇤ ) C H⇤ 1 < rmin 1 ( 5 ) Rearranging , we arrive at the condition stated . The important consequence of this result is that an optimal policy for this MDP fM will always take safe actions . However , in practice we can not compute eQ⇤ without knowing the dynamics model T . Therefore we extend our result to the model-based setting where the dynamics are imperfect . 3.2 Extension to model-based rollouts . We prove safety for the following theoretical setup . Suppose we have a dynamics model that outputs sets of states bT ( s , a ) ✓ S to account for uncertainty . Definition 3.2 . We say that a set-valued dynamics model bT : S ⇥ A ! P ( S ) 2 is calibrated if T ( s , a ) 2 bT ( s , a ) for all ( s , a ) 2 S ⇥A . We define the Bellmin operator : B⇤Q ( s , a ) = r̃ ( s , a ) + min s02bT ( s , a ) max a0 Q ( s0 , a0 ) ( 6 ) Lemma 3.2 . The Bellmin operator B⇤ is a -contraction in the1-norm . The proof is deferred to Appendix A.1 . As a consequence Lemma 3.2 and Banach ’ s fixed-point theorem , B⇤ has a unique fixed point Q⇤ which can be obtained by iteration . This fixed point is a lower bound on the true Q function if the model is calibrated : Lemma 3.3 . If bT is calibrated in the sense of Definition 3.2 , then Q⇤ ( s , a ) eQ⇤ ( s , a ) for all ( s , a ) . Proof . Let B̃⇤ denote the Bellman operator with reward function r̃ . First , observe that for any Q , Q 0 : S ⇥ A ! R , Q Q0 pointwise implies B⇤Q B⇤Q0 pointwise because we have r̃ ( s , a ) + maxa0 Q ( s0 , a0 ) r̃ ( s , a ) + maxa0 Q0 ( s0 , a0 ) pointwise and the min defining B⇤ includes the true s0 = T ( s , a ) . Now let Q0 be any inital Q function . Define eQk = ( B̃⇤ ) kQ0 and Q k = ( B⇤ ) kQ0 . An inductive argument coupled with the previous observation shows that Q k eQk pointwise for all k 2 N. Hence , taking the limits eQ⇤ = limk ! 1 eQk and Q⇤ = limk ! 1 Q k , we obtain Q⇤ eQ⇤ pointwise . Now we are ready to present our main theoretical result . Theorem 3.1 . Let bT be a calibrated dynamics model and ⇡⇤ ( s ) = argmaxa Q⇤ ( s , a ) the greedy policy with respect to Q⇤ . Assume that Assumption 3.1 holds . Then for any s 2 S , if there exists an action a such that Q⇤ ( s , a ) rmin1 , then ⇡ ⇤ ( s ) is a safe action . Proof . Lemma 3.2 implies that Q⇤ ( s , a ) eQ⇤ ( s , a ) for all ( s , a ) 2 S ⇥A . As shown in the proof of Lemma 3.1 , any unsafe action a0 satisfies Q ⇤ ( s , a0 ) eQ⇤ ( s , a0 ) rmax ( 1 H ⇤ ) C H⇤ 1 ( 7 ) Similarly if Q⇤ ( s , a ) rmin1 , we also have rmin 1 Q ⇤ ( s , a ) eQ⇤ ( s , a ) ( 8 ) so a is a safe action . Taking C as in inequality ( 3 ) guarantees that Q⇤ ( s , a ) > Q⇤ ( s , a0 ) , so the greedy policy ⇡⇤ will choose a over a0 . This theorem gives us a way to establish safety using only short-horizon predictions . The conclusion conditionally holds for any state s , but for s far from the observed states , we expect that bT ( s , a ) likely has to contain many states in order to satisfy the assumption that it contains the true next state , so that Q⇤ ( s , a ) will be very small and we may not have any action such that Q⇤ ( s , a ) rmin1 . However , it is plausible to believe that there can be such an a for the set of states in the replay buffer , { s : ( s , a , r , s0 ) 2 D } . 2P ( X ) is the powerset of a set X . Algorithm 1 Safe Model-Based Policy Optimization ( SMBPO ) Require : Horizon H 1 : Initialize empty buffers D and bD , an ensemble of probabilistic dynamics { bT✓i } Ni=1 , policy ⇡ , critic Q . 2 : Collect initial data using random policy , add to D. 3 : for episode 1 , 2 , . . . do 4 : Collect episode using ⇡ ; add the samples to D. Let ` be the length of the episode . 5 : Re-fit models { bT✓i } Ni=1 by several epochs of SGD on L bT ( ✓i ) defined in ( 9 ) 6 : Compute empirical rmin and rmax , and update C according to ( 3 ) . 7 : for ` times do 8 : for nrollout times ( in parallel ) do 9 : Sample s ⇠ D. 10 : Startin from s , roll out H steps using ⇡ and { bT✓i } ; add the samples to bD . 11 : for nactor times do 12 : Draw samples from D [ bD . 13 : Update Q by SGD on LQ ( ) defined in ( 10 ) and target parameters ̄ according to ( 12 ) . 14 : Update ⇡ by SGD on L⇡ ( ) defined in ( 13 ) .

Summary: The paper deals with one manifestation of safe exploration. Starting from a given safety function that classifies states as safe or unsafe, a modification of MBPO is used to change the Q-function in such a way that unsafe states should not be visited. The method is tested on two benchmarks.