Safe Reinforcement Learning by Imagining the Near Future
Year: 2022

Abstract:
Safe reinforcement learning is a promising path toward applying reinforcement learning algorithms to real-world problems, where suboptimal behaviors may lead to actual negative consequences. In this work, we focus on the setting where unsafe states can be avoided by planning ahead a short time into the future. In this setting, a model-based agent with a sufficiently accurate model can avoid unsafe states. We devise a model-based algorithm that heavily penalizes unsafe trajectories, and derive guarantees that our algorithm can avoid unsafe states under certain assumptions. Experiments demonstrate that our algorithm can achieve competitive rewards with fewer safety violations in several continuous control tasks.

heading: 1 Introduction
text: Reinforcement learning (RL) enables the discovery of effective policies for sequential decisionmaking tasks via trial and error [Mnih et al., 2015, Gu et al., 2016, Bellemare et al., 2020]. However, in domains such as robotics, healthcare, and autonomous driving, certain kinds of mistakes pose danger to people and/or objects in the environment. Hence there is an emphasis on the safety of the policy, both at execution time and while interacting with the environment during learning. This issue, referred to as safe exploration, is considered an important problem in AI safety [Amodei et al., 2016].
In this work, we advocate a model-based approach to safety, meaning that we estimate the dynamics of the system to be controlled and use the model for planning (or more accurately, policy improvement). The primary motivation for this is that a model-based method has the potential to anticipate safety violations before they occur. Often in real-world applications, the engineer has an idea of what states should be considered violations of safety: for example, a robot colliding rapidly with itself or surrounding objects, a car driving on the wrong side of the road, or a patient’s blood glucose levels spiking.Yet model-free algorithms typically lack the ability to incorporate such prior knowledge and must encounter some safety violations before learning to avoid them.
We begin with the premise that in practice, forward prediction for relatively few timesteps is sufficient to avoid safety violations. Consider the illustrative example in Figure 1, in which an agent controls the acceleration (and thereby, speed) of a car by pressing the gas or brake (or nothing). Note that there is an upper bound on how far into the future the agent would have to plan to foresee and (if possible) avoid any collision, namely, the amount of time it takes to bring the car to a complete stop.
Assuming that the horizon required for detecting unsafe situations is not too large, we show how to construct a reward function with the property that an optimal policy will never incur a safety violation. A short prediction horizon is also beneficial for model-based RL, as the well-known issue of compounding error plagues long-horizon prediction [Asadi et al., 2019]: imperfect predictions are fed back into the model as inputs (possibly outside the distribution of inputs in the training data), leading to progressively worse accuracy as the prediction horizon increases.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Our main contribution is a model-based algorithm that utilizes a reward penalty – the value of which is prescribed by our theoretical analysis – to guarantee safety (under some assumptions). Experiments indicate that the practical instantiation of our algorithm, Safe Model-Based Policy Optimization (SMBPO), effectively reduces the number of safety violations on several continuous control tasks, achieving a comparable performance with far fewer safety violations compared to several model-free safe RL algorithms. Code is made available at https://github.com/gwthomas/Safe-MBPO.

heading: 2 Background
text: In this work, we consider a deterministic1 Markov decision process (MDP) M = (S,A, T, r, ), where S is the state space, A the action space, T : S ⇥A! S the transition dynamics, r : S ⇥A! [rmin, rmax] the reward function, and 2 [0, 1) the discount factor. A policy ⇡ : S ! (A) determines what action to take at each state. A trajectory is a sequence of states and actions ⌧ = (s0, a0, r0, s1, a1, r1, . . . ) where st+1 = T (st, at) and rt = r(st, at).
Typically, the goal is to find a policy which maximizes the expected discounted return ⌘(⇡) = E⇡[ P1 t=0 t rt]. The notation E⇡ denotes that actions are sampled according to at ⇠ ⇡(st). The initial state s0 is drawn from an initial distribution which we assume to be fixed and leave out of the notation for simplicity. The Q function Q⇡(s, a) = E⇡[ P1
t=0 t rt | s0 = s, a0 = a] quantifies the conditional performance
of a policy ⇡ assuming it starts in a specific state s and takes action a, and the value function V ⇡(s) = Ea⇠⇡(s)[Q⇡(s, a)] averages this quantity over actions. The values of the best possible policies are denoted Q⇤(s, a) = max⇡ Q⇡(s, a) and V ⇤(s) = max⇡ V ⇡(s). The function Q⇤ has the important property that any optimal policy ⇡⇤ 2 argmax⇡ ⌘(⇡) must satisfy P(a⇤ 2 argmaxa Q⇤(s, a)) = 1 for all states s and actions a⇤ ⇠ ⇡⇤(s). Q⇤ is the unique fixed point of the Bellman operator
B⇤Q(s, a) = r(s, a) + max a0 Q(s0, a0) where s0 = T (s, a) (1)
In model-based RL, the algorithm estimates a dynamics model bT using the data observed so far, then uses the model for planning or data augmentation. The theoretical justification for model-based RL is typically based some version of the “simulation lemma”, which roughly states that if bT ⇡ T then ⌘̂(⇡) ⇡ ⌘(⇡) [Kearns and Singh, 2002, Luo et al., 2018].
1Determinism makes safety essentially trivial in tabular MDPs. We focus on tasks with continuous state and/or action spaces. See Appendix A.2 for a possible extension of our approach to stochastic dynamics.

heading: 3 Method
text: In this work, we train safe policies by modifying the reward function to penalize safety violations. We assume that the engineer specifies Sunsafe, the set of states which are considered safety violations. We must also account for the existence of states which are not themselves unsafe, but lead inevitably to unsafe states regardless of what actions are taken. Definition 3.1. A state s is said to be
• a safety violation if s 2 Sunsafe.
• irrecoverable if s 62 Sunsafe but for any sequence of actions a0, a1, a2, . . . , the trajectory defined by s0 = s and st+1 = T (st, at) for all t 2 N satisfies st̄ 2 Sunsafe for some t̄ 2 N.
• unsafe if it is unsafe or irrecoverable, or safe otherwise.
We remark that these definitions are similar to those introduced in prior work on safe RL [Hans et al., 2008]. Crucially, we do not assume that the engineer specifies which states are (ir)recoverable, as that would require knowledge of the system dynamics. However, we do assume that a safety violation must come fairly soon after entering an irrecoverable region: Assumption 3.1. There exists a horizon H⇤ 2 N such that, for any irrecoverable states s, any sequence of actions a0, . . . , aH⇤ 1 will lead to an unsafe state. That is, if s0 = s and st+1 = T (st, at) for all t 2 {0, . . . , H⇤ 1}, then st̄ 2 Sunsafe for some t̄ 2 {1, . . . , H⇤}.
This assumption rules out the possibility that a state leads inevitably to termination but takes an arbitrarily long time to do so. The implication of this assumption is that a perfect lookahead planner which considers the next H steps into the future can avoid not only the unsafe states, but also any irrecoverable states, with some positive probability.

heading: 3.1 Reward penalty framework
text: Now we present a reward penalty framework for guaranteeing safety. Let fMC = (S,A, eT , r̃, ) be an MDP with reward function and dynamics
r̃(s, a), eT (s, a) = ⇢ (r(s, a), T (s, a)) s 62 Sunsafe ( C, s) s 2 Sunsafe
(2)
where the terminal cost C 2 R is a constant (more on this below). That is, unsafe states are “absorbing” in that they transition back into themselves and receive the reward of C regardless of what action is taken.
The basis of our approach is to determine how large C must be so that the Q values of actions leading to unsafe states are less than the Q values of safe actions. Lemma 3.1. Suppose that Assumption 3.1 holds, and let
C > rmax rmin
H ⇤ rmax. (3)
Then for any state s, if a is a safe action (i.e. T (s, a) is a safe state) and a0 is an unsafe action (i.e. T (s, a) is unsafe), it holds that eQ⇤(s, a) > eQ⇤(s, a0), where eQ⇤ is the Q⇤ function for the MDP fMC .
Proof. Since a0 is unsafe, it leads to an unsafe state in at most H⇤ steps by assumption. Thus the discounted reward obtained is at most
H ⇤ 1X
t=0
t rmax +
1X
t=H⇤
t( C) = rmax(1
H ⇤ ) C H⇤
1 (4)
By comparison, the safe action a leads to another safe state, where it can be guaranteed to never encounter a safety violation. The reward of staying within the safe region forever must be at least rmin 1 . Thus, it suffices to choose C large enough that
rmax(1 H ⇤ ) C H⇤
1 < rmin 1 (5)
Rearranging, we arrive at the condition stated.
The important consequence of this result is that an optimal policy for this MDP fM will always take safe actions. However, in practice we cannot compute eQ⇤ without knowing the dynamics model T . Therefore we extend our result to the model-based setting where the dynamics are imperfect.

heading: 3.2 Extension to model-based rollouts
text: We prove safety for the following theoretical setup. Suppose we have a dynamics model that outputs sets of states bT (s, a) ✓ S to account for uncertainty. Definition 3.2. We say that a set-valued dynamics model bT : S ⇥ A ! P(S)2 is calibrated if T (s, a) 2 bT (s, a) for all (s, a) 2 S ⇥A.
We define the Bellmin operator: B⇤Q(s, a) = r̃(s, a) + min
s02bT (s,a) max a0
Q(s0, a0) (6)
Lemma 3.2. The Bellmin operator B⇤ is a -contraction in the1-norm.
The proof is deferred to Appendix A.1. As a consequence Lemma 3.2 and Banach’s fixed-point theorem, B⇤ has a unique fixed point Q⇤ which can be obtained by iteration. This fixed point is a lower bound on the true Q function if the model is calibrated: Lemma 3.3. If bT is calibrated in the sense of Definition 3.2, then Q⇤(s, a)  eQ⇤(s, a) for all (s, a).
Proof. Let B̃⇤ denote the Bellman operator with reward function r̃. First, observe that for any Q,Q
0 : S ⇥ A ! R, Q  Q0 pointwise implies B⇤Q  B⇤Q0 pointwise because we have r̃(s, a) + maxa0 Q(s0, a0)  r̃(s, a) + maxa0 Q0(s0, a0) pointwise and the min defining B⇤ includes the true s0 = T (s, a).
Now let Q0 be any inital Q function. Define eQk = (B̃⇤)kQ0 and Q k = (B⇤)kQ0. An inductive argument coupled with the previous observation shows that Q k  eQk pointwise for all k 2 N. Hence, taking the limits eQ⇤ = limk!1 eQk and Q⇤ = limk!1 Q k , we obtain Q⇤  eQ⇤ pointwise.
Now we are ready to present our main theoretical result. Theorem 3.1. Let bT be a calibrated dynamics model and ⇡⇤(s) = argmaxa Q⇤(s, a) the greedy policy with respect to Q⇤. Assume that Assumption 3.1 holds. Then for any s 2 S , if there exists an action a such that Q⇤(s, a) rmin1 , then ⇡ ⇤(s) is a safe action.
Proof. Lemma 3.2 implies that Q⇤(s, a)  eQ⇤(s, a) for all (s, a) 2 S ⇥A.
As shown in the proof of Lemma 3.1, any unsafe action a0 satisfies
Q ⇤(s, a0)  eQ⇤(s, a0)  rmax(1
H ⇤ ) C H⇤
1 (7)
Similarly if Q⇤(s, a) rmin1 , we also have rmin
1  Q ⇤(s, a)  eQ⇤(s, a) (8)
so a is a safe action. Taking C as in inequality (3) guarantees that Q⇤(s, a) > Q⇤(s, a0), so the greedy policy ⇡⇤ will choose a over a0.
This theorem gives us a way to establish safety using only short-horizon predictions. The conclusion conditionally holds for any state s, but for s far from the observed states, we expect that bT (s, a) likely has to contain many states in order to satisfy the assumption that it contains the true next state, so that Q⇤(s, a) will be very small and we may not have any action such that Q⇤(s, a) rmin1 . However, it is plausible to believe that there can be such an a for the set of states in the replay buffer, {s : (s, a, r, s0) 2 D}.
2P(X) is the powerset of a set X .
Algorithm 1 Safe Model-Based Policy Optimization (SMBPO) Require: Horizon H 1: Initialize empty buffers D and bD, an ensemble of probabilistic dynamics { bT✓i}Ni=1, policy ⇡ , critic Q . 2: Collect initial data using random policy, add to D. 3: for episode 1, 2, . . . do 4: Collect episode using ⇡ ; add the samples to D. Let ` be the length of the episode. 5: Re-fit models { bT✓i}Ni=1 by several epochs of SGD on L bT (✓i) defined in (9) 6: Compute empirical rmin and rmax, and update C according to (3). 7: for ` times do 8: for nrollout times (in parallel) do 9: Sample s ⇠ D. 10: Startin from s, roll out H steps using ⇡ and { bT✓i}; add the samples to bD. 11: for nactor times do 12: Draw samples from D [ bD. 13: Update Q by SGD on LQ( ) defined in (10) and target parameters  ̄ according to (12). 14: Update ⇡ by SGD on L⇡( ) defined in (13).

heading: 3.3 Practical algorithm
text: Based (mostly) on the framework described in the previous section, we develop a deep modelbased RL algorithm. We build on practices established in previous deep model-based algorithms, particularly MBPO [Janner et al., 2019] a state-of-the-art model-based algorithm (which does not emphasize safety).
The algorithm, dubbed Safe Model-Based Policy Optimization (SMBPO), is described in Algorithm 1. It follows a common pattern used by online model-based algorithms: alternate between collecting data, re-fitting the dynamics models, and improving the policy.
Following prior work [Chua et al., 2018, Janner et al., 2019], we employ an ensemble of (diagonal) Gaussian dynamics models { bT✓i}Ni=1, where bTi(s, a) = N (µ✓i(s, a), diag( 2✓i(s, a))), in an attempt to capture both aleatoric and epistemic uncertainties. Each model is trained via maximum likelihood on all the data observed so far:
LbT (✓i) = E(s,a,r,s0)⇠D log bT✓i(s 0 , r | s, a) (9)
However, random differences in initialization and mini-batch order while training lead to different models. The model ensemble can be used to generate uncertainty-aware predictions. For example, a set-valued prediction can be computed using the means bT (s, a) = {µ✓i(s, a)}Ni=1. The models are used to generate additional samples for fitting the Q function and updating the policy. In MBPO, this takes the form of short model-based rollouts, starting from states in D, to reduce the risk of compounding error. At each step in the rollout, a model bTi is randomly chosen from the ensemble and used to predict the next state. The rollout horizon H is chosen as a hyperparameter, and ideally exceeds the (unknown) H⇤ from Assumption 3.1. In principle, one can simply increase H to ensure it is large enough, but this increases the opportunity for compounding error.
MBPO is based on the soft actor-critic (SAC) algorithm, a widely used off-policy maximum-entropy actor-critic algorithm [Haarnoja et al., 2018a]. The Q function is updated by taking one or more SGD steps on the objective
LQ( ) = E(s,a,r,s0)⇠D[ bD[(Q (s, a) (r + V ̄(s 0))2] (10)
where V  ̄ (s0) = ⇢ C/(1 ) s0 2 Sunsafe Ea0⇠⇡(s0)[Q ̄(s0, a0) ↵ log ⇡ (a0 | s0)] s0 62 Sunsafe
(11)
The scalar ↵ is a hyperparameter of SAC which controls the tradeoff between entropy and reward. We tune ↵ using the procedure suggested by Haarnoja et al. [2018b].
The  ̄ are parameters of a “target” Q function which is updated via an exponential moving average towards :  ̄ ⌧ + (1 ⌧) ̄ (12) for a hyperparameter ⌧ 2 (0, 1) which is often chosen small, e.g., 0.005. This is a common practice used to promote stability in deep RL, originating from Lillicrap et al. [2015]. We also employ the
clipped double-Q method [Fujimoto et al., 2018] in which two copies of the parameters ( 1 and 2) and target parameters ( ̄1 and  ̄2) are maintained, and the target value in equation (11) is computed using mini=1,2 Q ̄i(s 0 , a 0).
Note that in (10), we are fitting to the average TD target across models, rather than the min, even though we proved Theorem 3.2 using the Bellmin operator. We found that taking the average worked better empirically, likely because the min was overly conservative and harmed exploration.
The policy is updated by taking one or more steps to minimize
L⇡( ) = Es⇠D[ bD,a⇠⇡ (s)[↵ log ⇡ (a | s) Q (s, a)]. (13)

heading: 4 Experiments
text: In the experimental evaluation, we compare our algorithm to several model-free safe RL algorithms, as well as MBPO, on various continuous control tasks based on the MuJoCo simulator [Todorov et al., 2012]. Additional experimental details, including hyperparameter selection, are given in Appendix A.3.

heading: 4.1 Tasks
text: The tasks are described below:
• Hopper: Standard hopper environment from OpenAI Gym, except with the “alive bonus” (a constant) removed from the reward so that the task reward does not implicitly encode the safety objective. The safety condition is the usual termination condition for this task, which corresponds to the robot falling over.
• Cheetah-no-flip: The standard half-cheetah environment from OpenAI Gym, with a safety condition: the robot’s head should not come into contact with the ground.
• Ant, Humanoid: Standard ant and humanoid environments from OpenAI Gym, except with the alive bonuses removed, and contact forces removed from the observation (as these are difficut to model). The safety condition is the usual termination condition for this task, which corresponds to the robot falling over.
For all of the tasks, the reward corresponds to positive movement along the x-axis (minus some small cost on action magnitude), and safety violations cause the current episode to terminate. See Figure 2 for visualizations of the termination conditions.

heading: 4.2 Algorithms
text: We compare against the following algorithms:
• MBPO: Corresponds to SMBPO with C = 0. • MBPO+bonus: The same as MBPO, except adding back in the alive bonus which was subtracted
out of the reward.
• Recovery RL, model-free (RRL-MF): Trains a critic to estimate the safety separately from the reward, as well as a recovery policy which is invoked when the critic predicts risk of a safety violation.
• Lagrangian relaxation (LR): Forms a Lagrangian to implement a constraint on the risk, updating the dual variable via dual gradient descent.
• Safety Q-functions for RL (SQRL): Also formulates a Lagrangian relaxation, and uses a filter to reject actions which are too risky according to the safety critic.
• Reward constrained policy optimization (RCPO): Uses policy gradient to optimize a reward function which is penalized according to the safety critic.
All of the above algorithms except for MBPO are as implemented in the Recovery RL paper [Thananjeyan et al., 2020] and its publicly available codebase3. We follow the hyperparameter tuning procedure described in their paper; see Appendix A.3 for more details. A recent work [Bharadhwaj et al., 2020] can also serve as a baseline but the code has not been released.
Our algorithm requires very little hyperparameter tuning. We use = 0.99 in all experiments. We tried both H = 5 and H = 10 and found that H = 10 works slightly better, so we use H = 10 in all experiments.

heading: 4.3 Results
text: The main criterion in which we are interested is performance (return) vs. the cumulative number of safety violations. The results are plotted in Figure 3. We see that our algorithm performs favorably compared to model-free alternatives in terms of this tradeoff, achieving similar or better performance with a fraction of the violations.
3https://github.com/abalakrishna123/recovery-rl
MBPO is competitive in terms of sample efficiency but incurs more safety violations because it isn’t designed explicitly to avoid them.
We also show in Figure 4 that hard-coding the value of C leads to an intuitive tradeoff between performance and safety violations. With a larger C, SMBPO incurs substantially fewer safety violations, although the total rewards are learned slower.

heading: 5 Related Work
text: Safe Reinforcement Learning Many of the prior works correct the action locally, that is, changing the action when the action is detected to lead to an unsafe state. Dalal et al. [2018] linearizes the dynamics and adds a layer on the top of the policy for correction. Bharadhwaj et al. [2020] uses rejection sampling to ensure the action meets the safety requirement. Thananjeyan et al. [2020] either trains a backup policy which is only used to guarantee safety, or uses model-predictive control (MPC) to find the best action sequence. MPC could also be applied in the short-horizon setting that we consider here, but it involves high runtime cost that may not be acceptable for real-time robotics control. Also, MPC only optimizes for rewards under the short horizon and can lead to suboptimal performance on tasks that require longer-term considerations [Tamar et al., 2017].
Other works aim to solve the constrained MDP more efficiently and better, with Lagrangian methods being applied widely. The Lagrangian multipliers can be a fixed hyperparameter, or adjusted by the algorithm [Tessler et al., 2018, Stooke et al., 2020]. The policy training might also have issues. The issue that the policy might change too fast so that it’s no longer safe is addressed by building a trust region of policies [Achiam et al., 2017, Zanger et al., 2021] and further projecting to a safer policy [Yang et al., 2020], and another issue of too optimistic policy is addressed by Bharadhwaj et al. [2020] by using conservative policy updates. Expert information can greatly improve the training-time safety. Srinivasan et al. [2020], Thananjeyan et al. [2020] are provided offline data, while Turchetta et al. [2020] is provided interventions which are invoked at dangerous states and achieves zeros safety violations during training.
Returnability is also considered by Eysenbach et al. [2018] in practice, which trains a policy to return to the initial state, or by Roderick et al. [2021] in theory, which designs a PAC algorithm to train a policy without safety violations. Bansal et al. [2017] gives a brief overview of Hamilton-Jacobi Reachability and its recent progress.
Model-based Reinforcement Learning Model-based reinforcement learning, which additionally learns the dynamics model, has gained its popularity due to its superior sample efficiency. Kurutach et al. [2018] uses an ensemble of models to produce imaginary samples to regularize leaerning and
reduce instability. The use of model ensemble is further explored by Chua et al. [2018], which studies different methods to sample trajectories from the model ensemble. Based on Chua et al. [2018], Wang and Ba [2019] combines policy networks with online learning. Luo et al. [2019] derives a lower bound of the policy in the real environment given its performance in the learned dynamics model, and then optimizes the lower bound stochastically. Our work is based on Janner et al. [2019], which shows the learned dynamics model doesn’t generalize well for long horizon and proposes to use short model-generated rollouts instead of a full episodes. Dong et al. [2020] studies the expressivity of Q function and model and shows that at some environments, the model is much easier to learn than the Q function.

heading: 6 Conclusion
text: We consider the problem of safe exploration in reinforcement learning, where the goal is to discover a policy that maximizes the expected return, but additionally desire the training process to incur minimal safety violations. In this work, we assume access to a user-specified function which can be queried to determine whether or not a given state is safe. We have proposed a model-based algorithm that can exploit this information to anticipate safety violations before they happen and thereby avoid them. Our theoretical analysis shows that safety violations could be avoided with a sufficiently large penalty and accurate dynamics model. Empirically, our algorithm compares favorably to state-of-the-art model-free safe exploration methods in terms of the tradeoff between performance and total safety violations, and in terms of sample complexity.

heading: Acknowledgements
text: TM acknowledges support of Google Faculty Award, NSF IIS 2045685, the Sloan Fellowship, and JD.com. YL is supported by NSF, ONR, Simons Foundation, Schmidt Foundation, DARPA and SRC.

Review summaries:
The paper deals with one manifestation of safe exploration. Starting from a given safety function that classifies states as safe or unsafe, a modification of MBPO is used to change the Q-function in such a way that unsafe states should not be visited. The method is tested on two benchmarks.

This paper studies *safe* reinforcement learning (RL), in which an RL algorithm is tasked with learning a high-value behavioral policy while simultaneously minimizing the number of safety violations. To frame the work, the paper takes on a few key perspectives. First, that predicting only a few timesteps into the future is all that is needed to prevent safety violations. Second, that safety is defined in terms of the occupancy of set of states, as determined by the \textsc{Unsafe} predicate designed by an engineer. With this predicate in hand, a natural notion of an irrecoverable state is introduced, defined as those states that always lead to an unsafe state under any policy. Third, the paper assumes that environments are deterministic for the main analysis, and captures transition model uncertainty in the form of a next-state-set prediction.  With these perspectives in hand, the work establishes two primary contributions. The first is the formation of a safe proxy MDP, $\tilde{M}$, that uses a proxy reward and transition function. The proxy reward produces $-C$ for all unsafe states, where $C$ is a well chosen constant (to be returned to shortly). The proxy transition function replaces any unsafe transition with a self-loop. Hence, in reasoning with this proxy MDP, acting in an unsafe state will produce $-C$ reward indefinitely, thereby accumulating $C / (1-\gamma)$ value. By consequence, Prop. 1 illustrates that the Q values under the proxy MDP ensure safe actions always have higher values than unsafe actions. Then, section 3.2 takes this idea and extends it to the case when a model is being learned. In particular, it is assumed that the form of transition model learned is a "set-valued dynamics model" that maps each $(s,a)$ pair to a non-empty set of possible next states. With this definition of model, two Lemmas are introduced (1 and 2) that illustrate two things: First, that the cutely named "Bellmin" operator (which takes the s' achieved by minimizing Q w.r.t. possible next states in the set) is a $\gamma$-contraction (Lemma 1). Second, that a "calibrated" dynamics model (defined in Definition 2 as a set-valued model that always contains the true next-state in its next-state set) will ensure that the proxy Q function upper bounds the set-valued model's Q function. These two lemmas are used to Prove Theorem 1, which states that under a calibrated dynamics model, if there is an action with value greater than $r_{\min}/(1-\gamma)$, the greedy policy (w.r.t. $\underline{Q}$) will choose a safe action.  Then, following this analysis, the paper shifts focus on developing a practical, safe, model-based based algorithm (Algorithm 1, SMBPO). Experiments are conducted in two navigation tasks, hopper, and a cheetah variant, contrasting a variety of RL algorithms (many of which I was unfamiliar with). These include: MBPO, RRL-MF (recovery RL), LR (Lagrangian relaxation), SQRL (Safety Q-functions ), RCPO (Reward constrained policy optimization), and RSPO (Risk-sensitive policy optimization). The main quality explored here is the trade off each algorithm makes between performance and the number of safety violations.