Safe Reinforcement Learning by Imagining the Near Future

1 Introduction . Reinforcement learning ( RL ) enables the discovery of effective policies for sequential decisionmaking tasks via trial and error [ Mnih et al. , 2015 , Gu et al. , 2016 , Bellemare et al. , 2020 ] . However , in domains such as robotics , healthcare , and autonomous driving , certain kinds of mistakes pose danger to people and/or objects in the environment . Hence there is an emphasis on the safety of the policy , both at execution time and while interacting with the environment during learning . This issue , referred to as safe exploration , is considered an important problem in AI safety [ Amodei et al. , 2016 ] . In this work , we advocate a model-based approach to safety , meaning that we estimate the dynamics of the system to be controlled and use the model for planning ( or more accurately , policy improvement ) . The primary motivation for this is that a model-based method has the potential to anticipate safety violations before they occur . Often in real-world applications , the engineer has an idea of what states should be considered violations of safety : for example , a robot colliding rapidly with itself or surrounding objects , a car driving on the wrong side of the road , or a patient ’ s blood glucose levels spiking.Yet model-free algorithms typically lack the ability to incorporate such prior knowledge and must encounter some safety violations before learning to avoid them . We begin with the premise that in practice , forward prediction for relatively few timesteps is sufficient to avoid safety violations . Consider the illustrative example in Figure 1 , in which an agent controls the acceleration ( and thereby , speed ) of a car by pressing the gas or brake ( or nothing ) . Note that there is an upper bound on how far into the future the agent would have to plan to foresee and ( if possible ) avoid any collision , namely , the amount of time it takes to bring the car to a complete stop . Assuming that the horizon required for detecting unsafe situations is not too large , we show how to construct a reward function with the property that an optimal policy will never incur a safety violation . A short prediction horizon is also beneficial for model-based RL , as the well-known issue of compounding error plagues long-horizon prediction [ Asadi et al. , 2019 ] : imperfect predictions are fed back into the model as inputs ( possibly outside the distribution of inputs in the training data ) , leading to progressively worse accuracy as the prediction horizon increases . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Our main contribution is a model-based algorithm that utilizes a reward penalty – the value of which is prescribed by our theoretical analysis – to guarantee safety ( under some assumptions ) . Experiments indicate that the practical instantiation of our algorithm , Safe Model-Based Policy Optimization ( SMBPO ) , effectively reduces the number of safety violations on several continuous control tasks , achieving a comparable performance with far fewer safety violations compared to several model-free safe RL algorithms . Code is made available at https : //github.com/gwthomas/Safe-MBPO . 2 Background . In this work , we consider a deterministic1 Markov decision process ( MDP ) M = ( S , A , T , r , ) , where S is the state space , A the action space , T : S ⇥A ! S the transition dynamics , r : S ⇥A ! [ rmin , rmax ] the reward function , and 2 [ 0 , 1 ) the discount factor . A policy ⇡ : S ! ( A ) determines what action to take at each state . A trajectory is a sequence of states and actions ⌧ = ( s0 , a0 , r0 , s1 , a1 , r1 , . . . ) where st+1 = T ( st , at ) and rt = r ( st , at ) . Typically , the goal is to find a policy which maximizes the expected discounted return ⌘ ( ⇡ ) = E⇡ [ P1 t=0 t rt ] . The notation E⇡ denotes that actions are sampled according to at ⇠ ⇡ ( st ) . The initial state s0 is drawn from an initial distribution which we assume to be fixed and leave out of the notation for simplicity . The Q function Q⇡ ( s , a ) = E⇡ [ P1 t=0 t rt | s0 = s , a0 = a ] quantifies the conditional performance of a policy ⇡ assuming it starts in a specific state s and takes action a , and the value function V ⇡ ( s ) = Ea⇠⇡ ( s ) [ Q⇡ ( s , a ) ] averages this quantity over actions . The values of the best possible policies are denoted Q⇤ ( s , a ) = max⇡ Q⇡ ( s , a ) and V ⇤ ( s ) = max⇡ V ⇡ ( s ) . The function Q⇤ has the important property that any optimal policy ⇡⇤ 2 argmax⇡ ⌘ ( ⇡ ) must satisfy P ( a⇤ 2 argmaxa Q⇤ ( s , a ) ) = 1 for all states s and actions a⇤ ⇠ ⇡⇤ ( s ) . Q⇤ is the unique fixed point of the Bellman operator B⇤Q ( s , a ) = r ( s , a ) + max a0 Q ( s0 , a0 ) where s0 = T ( s , a ) ( 1 ) In model-based RL , the algorithm estimates a dynamics model bT using the data observed so far , then uses the model for planning or data augmentation . The theoretical justification for model-based RL is typically based some version of the “ simulation lemma ” , which roughly states that if bT ⇡ T then ⌘̂ ( ⇡ ) ⇡ ⌘ ( ⇡ ) [ Kearns and Singh , 2002 , Luo et al. , 2018 ] . 1Determinism makes safety essentially trivial in tabular MDPs . We focus on tasks with continuous state and/or action spaces . See Appendix A.2 for a possible extension of our approach to stochastic dynamics . 3 Method . In this work , we train safe policies by modifying the reward function to penalize safety violations . We assume that the engineer specifies Sunsafe , the set of states which are considered safety violations . We must also account for the existence of states which are not themselves unsafe , but lead inevitably to unsafe states regardless of what actions are taken . Definition 3.1 . A state s is said to be • a safety violation if s 2 Sunsafe . • irrecoverable if s 62 Sunsafe but for any sequence of actions a0 , a1 , a2 , . . . , the trajectory defined by s0 = s and st+1 = T ( st , at ) for all t 2 N satisfies st̄ 2 Sunsafe for some t̄ 2 N. • unsafe if it is unsafe or irrecoverable , or safe otherwise . We remark that these definitions are similar to those introduced in prior work on safe RL [ Hans et al. , 2008 ] . Crucially , we do not assume that the engineer specifies which states are ( ir ) recoverable , as that would require knowledge of the system dynamics . However , we do assume that a safety violation must come fairly soon after entering an irrecoverable region : Assumption 3.1 . There exists a horizon H⇤ 2 N such that , for any irrecoverable states s , any sequence of actions a0 , . . . , aH⇤ 1 will lead to an unsafe state . That is , if s0 = s and st+1 = T ( st , at ) for all t 2 { 0 , . . . , H⇤ 1 } , then st̄ 2 Sunsafe for some t̄ 2 { 1 , . . . , H⇤ } . This assumption rules out the possibility that a state leads inevitably to termination but takes an arbitrarily long time to do so . The implication of this assumption is that a perfect lookahead planner which considers the next H steps into the future can avoid not only the unsafe states , but also any irrecoverable states , with some positive probability . 3.1 Reward penalty framework . Now we present a reward penalty framework for guaranteeing safety . Let fMC = ( S , A , eT , r̃ , ) be an MDP with reward function and dynamics r̃ ( s , a ) , eT ( s , a ) = ⇢ ( r ( s , a ) , T ( s , a ) ) s 62 Sunsafe ( C , s ) s 2 Sunsafe ( 2 ) where the terminal cost C 2 R is a constant ( more on this below ) . That is , unsafe states are “ absorbing ” in that they transition back into themselves and receive the reward of C regardless of what action is taken . The basis of our approach is to determine how large C must be so that the Q values of actions leading to unsafe states are less than the Q values of safe actions . Lemma 3.1 . Suppose that Assumption 3.1 holds , and let C > rmax rmin H ⇤ rmax . ( 3 ) Then for any state s , if a is a safe action ( i.e . T ( s , a ) is a safe state ) and a0 is an unsafe action ( i.e . T ( s , a ) is unsafe ) , it holds that eQ⇤ ( s , a ) > eQ⇤ ( s , a0 ) , where eQ⇤ is the Q⇤ function for the MDP fMC . Proof . Since a0 is unsafe , it leads to an unsafe state in at most H⇤ steps by assumption . Thus the discounted reward obtained is at most H ⇤ 1X t=0 t rmax + 1X t=H⇤ t ( C ) = rmax ( 1 H ⇤ ) C H⇤ 1 ( 4 ) By comparison , the safe action a leads to another safe state , where it can be guaranteed to never encounter a safety violation . The reward of staying within the safe region forever must be at least rmin 1 . Thus , it suffices to choose C large enough that rmax ( 1 H ⇤ ) C H⇤ 1 < rmin 1 ( 5 ) Rearranging , we arrive at the condition stated . The important consequence of this result is that an optimal policy for this MDP fM will always take safe actions . However , in practice we can not compute eQ⇤ without knowing the dynamics model T . Therefore we extend our result to the model-based setting where the dynamics are imperfect . 3.2 Extension to model-based rollouts . We prove safety for the following theoretical setup . Suppose we have a dynamics model that outputs sets of states bT ( s , a ) ✓ S to account for uncertainty . Definition 3.2 . We say that a set-valued dynamics model bT : S ⇥ A ! P ( S ) 2 is calibrated if T ( s , a ) 2 bT ( s , a ) for all ( s , a ) 2 S ⇥A . We define the Bellmin operator : B⇤Q ( s , a ) = r̃ ( s , a ) + min s02bT ( s , a ) max a0 Q ( s0 , a0 ) ( 6 ) Lemma 3.2 . The Bellmin operator B⇤ is a -contraction in the1-norm . The proof is deferred to Appendix A.1 . As a consequence Lemma 3.2 and Banach ’ s fixed-point theorem , B⇤ has a unique fixed point Q⇤ which can be obtained by iteration . This fixed point is a lower bound on the true Q function if the model is calibrated : Lemma 3.3 . If bT is calibrated in the sense of Definition 3.2 , then Q⇤ ( s , a ) eQ⇤ ( s , a ) for all ( s , a ) . Proof . Let B̃⇤ denote the Bellman operator with reward function r̃ . First , observe that for any Q , Q 0 : S ⇥ A ! R , Q Q0 pointwise implies B⇤Q B⇤Q0 pointwise because we have r̃ ( s , a ) + maxa0 Q ( s0 , a0 ) r̃ ( s , a ) + maxa0 Q0 ( s0 , a0 ) pointwise and the min defining B⇤ includes the true s0 = T ( s , a ) . Now let Q0 be any inital Q function . Define eQk = ( B̃⇤ ) kQ0 and Q k = ( B⇤ ) kQ0 . An inductive argument coupled with the previous observation shows that Q k eQk pointwise for all k 2 N. Hence , taking the limits eQ⇤ = limk ! 1 eQk and Q⇤ = limk ! 1 Q k , we obtain Q⇤ eQ⇤ pointwise . Now we are ready to present our main theoretical result . Theorem 3.1 . Let bT be a calibrated dynamics model and ⇡⇤ ( s ) = argmaxa Q⇤ ( s , a ) the greedy policy with respect to Q⇤ . Assume that Assumption 3.1 holds . Then for any s 2 S , if there exists an action a such that Q⇤ ( s , a ) rmin1 , then ⇡ ⇤ ( s ) is a safe action . Proof . Lemma 3.2 implies that Q⇤ ( s , a ) eQ⇤ ( s , a ) for all ( s , a ) 2 S ⇥A . As shown in the proof of Lemma 3.1 , any unsafe action a0 satisfies Q ⇤ ( s , a0 ) eQ⇤ ( s , a0 ) rmax ( 1 H ⇤ ) C H⇤ 1 ( 7 ) Similarly if Q⇤ ( s , a ) rmin1 , we also have rmin 1 Q ⇤ ( s , a ) eQ⇤ ( s , a ) ( 8 ) so a is a safe action . Taking C as in inequality ( 3 ) guarantees that Q⇤ ( s , a ) > Q⇤ ( s , a0 ) , so the greedy policy ⇡⇤ will choose a over a0 . This theorem gives us a way to establish safety using only short-horizon predictions . The conclusion conditionally holds for any state s , but for s far from the observed states , we expect that bT ( s , a ) likely has to contain many states in order to satisfy the assumption that it contains the true next state , so that Q⇤ ( s , a ) will be very small and we may not have any action such that Q⇤ ( s , a ) rmin1 . However , it is plausible to believe that there can be such an a for the set of states in the replay buffer , { s : ( s , a , r , s0 ) 2 D } . 2P ( X ) is the powerset of a set X . Algorithm 1 Safe Model-Based Policy Optimization ( SMBPO ) Require : Horizon H 1 : Initialize empty buffers D and bD , an ensemble of probabilistic dynamics { bT✓i } Ni=1 , policy ⇡ , critic Q . 2 : Collect initial data using random policy , add to D. 3 : for episode 1 , 2 , . . . do 4 : Collect episode using ⇡ ; add the samples to D. Let ` be the length of the episode . 5 : Re-fit models { bT✓i } Ni=1 by several epochs of SGD on L bT ( ✓i ) defined in ( 9 ) 6 : Compute empirical rmin and rmax , and update C according to ( 3 ) . 7 : for ` times do 8 : for nrollout times ( in parallel ) do 9 : Sample s ⇠ D. 10 : Startin from s , roll out H steps using ⇡ and { bT✓i } ; add the samples to bD . 11 : for nactor times do 12 : Draw samples from D [ bD . 13 : Update Q by SGD on LQ ( ) defined in ( 10 ) and target parameters ̄ according to ( 12 ) . 14 : Update ⇡ by SGD on L⇡ ( ) defined in ( 13 ) .

Summary: This paper studies *safe* reinforcement learning (RL), in which an RL algorithm is tasked with learning a high-value behavioral policy while simultaneously minimizing the number of safety violations. To frame the work, the paper takes on a few key perspectives. First, that predicting only a few timesteps into the future is all that is needed to prevent safety violations. Second, that safety is defined in terms of the occupancy of set of states, as determined by the \textsc{Unsafe} predicate designed by an engineer. With this predicate in hand, a natural notion of an irrecoverable state is introduced, defined as those states that always lead to an unsafe state under any policy. Third, the paper assumes that environments are deterministic for the main analysis, and captures transition model uncertainty in the form of a next-state-set prediction.  With these perspectives in hand, the work establishes two primary contributions. The first is the formation of a safe proxy MDP, $\tilde{M}$, that uses a proxy reward and transition function. The proxy reward produces $-C$ for all unsafe states, where $C$ is a well chosen constant (to be returned to shortly). The proxy transition function replaces any unsafe transition with a self-loop. Hence, in reasoning with this proxy MDP, acting in an unsafe state will produce $-C$ reward indefinitely, thereby accumulating $C / (1-\gamma)$ value. By consequence, Prop. 1 illustrates that the Q values under the proxy MDP ensure safe actions always have higher values than unsafe actions. Then, section 3.2 takes this idea and extends it to the case when a model is being learned. In particular, it is assumed that the form of transition model learned is a "set-valued dynamics model" that maps each $(s,a)$ pair to a non-empty set of possible next states. With this definition of model, two Lemmas are introduced (1 and 2) that illustrate two things: First, that the cutely named "Bellmin" operator (which takes the s' achieved by minimizing Q w.r.t. possible next states in the set) is a $\gamma$-contraction (Lemma 1). Second, that a "calibrated" dynamics model (defined in Definition 2 as a set-valued model that always contains the true next-state in its next-state set) will ensure that the proxy Q function upper bounds the set-valued model's Q function. These two lemmas are used to Prove Theorem 1, which states that under a calibrated dynamics model, if there is an action with value greater than $r_{\min}/(1-\gamma)$, the greedy policy (w.r.t. $\underline{Q}$) will choose a safe action.  Then, following this analysis, the paper shifts focus on developing a practical, safe, model-based based algorithm (Algorithm 1, SMBPO). Experiments are conducted in two navigation tasks, hopper, and a cheetah variant, contrasting a variety of RL algorithms (many of which I was unfamiliar with). These include: MBPO, RRL-MF (recovery RL), LR (Lagrangian relaxation), SQRL (Safety Q-functions ), RCPO (Reward constrained policy optimization), and RSPO (Risk-sensitive policy optimization). The main quality explored here is the trade off each algorithm makes between performance and the number of safety violations.