SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Milan Ganai
UC San Diego  
mganai@ucsd.edu
Zheng Gong  
UC San Diego  
zhgong@ucsd.edu
Chenning Yu  
UC San Diego  
chy010@ucsd.edu
Sylvia Herbert  
UC San Diego  
erbert@ucsd.edu
Sicun Gao  
UC San Diego  
sicung@ucsd.edu
Ensuring safety is important for the practical deployment of reinforcement learning (RL). Various challenges must be addressed, such as handling stochasticity in the environments, providing rigorous guarantees of persistent state-wise safety satisfaction, and avoiding overly conservative behaviors that sacrifice performance. We propose a new framework, Reachability Estimation for Safe Policy Optimization (RESPO), for safety-constrained RL in general stochastic settings. In the feasible set where there exist violation-free policies, we optimize for rewards while maintaining persistent safety. Outside this feasible set, our optimization produces the safest behavior by guaranteeing entrance into the feasible set whenever possible with the least cumulative discounted violations. We introduce a class of algorithms using our novel reachability estimation function to optimize in our proposed framework and in similar frameworks such as those concurrently handling multiple hard and soft constraints. We theoretically establish that our algorithms almost surely converge to locally optimal policies of our safe optimization framework. We evaluate the proposed methods on a diverse suite of safe RL environments from Safety Gym, PyBullet, and MuJoCo, and show the benefits in improving both reward performance and safety compared with state-of-the-art baselines.
Safety-Constrained Reinforcement Learning is important for a multitude of real-world safety-critical applications [1, 2]. These situations require guaranteeing persistent safety and achieving high performance while handling environment uncertainty. Traditionally, methods have been proposed to obtain optimal control by solving within the Constrained Markov Decision Process (CMDP) framework, which constrains expected cumulative violation of the constraints along trajectories to be under some threshold. Trust-region approaches [3, 4, 5] follow from the Constrained Policy Optimization algorithm [6] which tries to guarantee monotonic performance improvement while satisfying the cumulative constraint requirement. However, these approaches are generally very computationally expensive, and their first or second order Taylor approximations may have repercussions on performance [2]. Several primal-dual approaches [7, 8, 9] have been proposed with better computational efficiency and use multi-timescale frameworks [10] for asymptotic convergence to a local optimum. Nonetheless, these approaches inherit CMDP's lack of crucial rigorous safety guarantees that ensure persistent safety. Particularly, maintaining a cost budget for hard constraints like collision avoidance is insufficient. Rather, it's crucial to prioritize obtaining safe, minimal-violation policies when possible.
There have been a variety of proposed control theoretic functions that provide the needed guarantees on persistent safety for such hard constraints, including Control Barrier Functions (CBFs) [11] and Hamilton-Jacobi (HJ) reachability analysis [12, 13]. CBFs are energy-based certification functions, whose zero super-level set is control invariant [14, 15]. The drawbacks are the conservativeness of the feasible set and the lack of a systematic way to find these functions for general nonlinear systems. Under certain assumptions, approximate energy-based certificates can be learned with sufficient samples [16, 17, 18, 19, 20, 21]. HJ reachability analysis focuses on finding the set of initial states such that there exists some control to avoid and/or reach a certain region. It computes a value function, whose zero sub-level set is the largest feasible set, together with the optimal control. The drawback is the well known 'curse of dimensionality,' as HJ reachability requires solving a corresponding Hamilton-Jacobi-Bellman Partial Differential Equation (HJBPDE) on a discrete grid recursively. Originally, both approaches assume the system dynamics and environment are known, and inputs are bounded by some polytopic constraints, while recent works have applied them to model-free settings [22, 23, 24]. For both CBF and HJ reachability, controllers that optimize performance and guarantee safety can be synthesized in different ways online [25]. There also exist hard-constraint approaches like [26] have theoretical safety guarantees without relying on control theory; however, they require learning or having access to the transition-function dynamics.
Consequently, it can be beneficial to combine CMDP-based approaches to find optimal control and the advantages of control-theoretic approaches to enable safety. The recent framework of Reachability Constrained Reinforcement Learning (RCRL) [27] proposed an approach that guarantees persistent safety and optimal performance when the agent is in the feasible set. However, RCRL is limited to learning deterministic policies in deterministic environments. In the general RL setting, stochasticity incurs a variety of challenges including how to define membership in the feasible set since it is no longer a binary question but rather a probabilistic one. Another issue is that for states outside optimal feasible set, RCRL optimization cannot guarantee entrance into the feasible set when possible. This may have undesirable consequences including indefinitely violating constraints with no recovery.
In light of the above challenges, we propose a new framework and class of algorithms called Reachability Estimation for Safe Policy Optimization (RESPO). Our main contributions are:
- We introduce reachability estimation methods for general stochastic policies that predict the likelihood of constraint violations in the future. Using this estimation, a policy can be optimized such that (i) when in the feasible set, it maintains persistent safety and optimizes for reward performance, and (ii) when outside the feasible set, it produces the safest behavior and enters the feasible set if possible. We also demonstrate how our optimization can incorporate multiple hard and soft constraints while even prioritizing the hard constraints.
- We provide a novel class of actor-critic algorithms based on learning our reachability estimation function, and prove our algorithm converges to a locally optimal policy of our proposed optimization.
- We perform comprehensive experiments showing that our approach achieves high performance in complex, high-dimensional Safety Gym, PyBullet-based, and MuJoCo-based environments compared to other state-of-the-art safety-constrained approaches while achieving small or even 0 violations. We also show our algorithm's performance in environments with multiple hard and soft constraints.
2.1 Constrained Reinforcement Learning
Safety-constrained reinforcement learning has been addressed through various proposed learning mechanisms solving within an optimization framework. CMDP [28, 29] is one framework that augments MDPs with the cost function: the goal is to maximize expected reward returns while constraining cost returns below a manually chosen threshold. Trust-region methods [3, 4, 5, 6] and primal-dual based approaches [7, 9, 30, 31] are two classes of CMDP-based approaches. Trust-region approaches generally follow from Constrained Policy Optimization (CPO) [6], which approximates the constraint optimization problem with surrogate functions for the reward objective and safety constraints, and then performs projection steps on the policy parameters with backtracking line search. Penalized Proximal Policy Optimization [32] improves upon past trust-region approaches via the clipping mechanism, similar to Proximal Policy Optimization's [33] improvement over Trust Region Policy Optimization [34]. Constraint-Rectified Policy Optimization [35] takes steps toward improving reward performance if constraints are currently satisfied else takes steps to minimize
constraint violations. PPO Lagrangian [30] is a primal-dual method combining Proximal Policy Optimization (PPO) [33] with lagrangian relaxation of the safety constraints and has relatively low complexity but outperforms CPO in constraint satisfaction. [7] uses a penalized reward function like PPO Lagrangian, and demonstrates convergence to an optimal policy through the multi-timescale framework, introduced in [10]. [36] proposes mapping the value function constraining problem as constraining densities of state visitation. Overall, while these algorithms provide benefits through various learning mechanisms, they inherit the problems of the CMDP framework. Specifically, they do not provide rigorous guarantees of persistent safety and so are generally unsuitable for state-wise constraint optimization problems.
2.2 Hamilton-Jacobi Reachability Analysis
HJ reachability analysis is a rigorous approach that verifies safety or reachability for dynamical systems. Given any deterministic nonlinear system and target set, it computes the viscosity solution for the HJBPDE, whose zero sub-level set is the backward reachable set (BRS), meaning there exists some control such that the states starting from this set will enter the target set in future time (or stay away from the target set for all future time). However, the curse of dimensionality and the assumption of knowing the dynamics and environment in advance restrict its application, especially in the model-free RL setting where there is no access to the entire observation space and dynamics at any given time. Decomposition [37], warm-starting [38], sampling-based reachability [39], and Deepreach [40] have been proposed to solve the curse of dimensionality. [23, 24] proposed methods to bridge HJ analysis with RL by modifying the HJBPDE. The work of [27] takes framework into a deterministic hard-constraint setting. There are additionally model-based HJ reachability approach for reinforcement learning-based controls. [41] is a model-based extension of [27]. Approaches like [42, 43, 44] use traditional HJ reachability for RL control while learning or assuming access to the system dynamics model. In stochastic systems, finding the probability of reaching a target while avoiding certain obstacles are key problems. [45, 46] constructed the theoretic framework based on dynamic programming and consider finite and infinite time reach-avoid problems. [47, 48, 49] propose computing the stochastic reach-avoid set together with the probability in a tractable manner to address the curse of dimensionality.
3.1 Markov Decision Processes
Markov Decision Processes (MDP) are defined as  $\mathcal{M} \coloneqq \langle \mathcal{S}, \mathcal{A}, P, r, h, \gamma \rangle$ .  $\mathcal{S}$  and  $\mathcal{A}$  are the state and action spaces respectively.  $P: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \mapsto [0,1]$  is the transition function capturing the environment dynamics.  $r: \mathcal{S} \times \mathcal{A} \mapsto \mathbb{R}$  is the reward function associated with each state-action pair,  $h: \mathcal{S} \mapsto \mathbb{R}_0^+$  is the safety loss function that maps a state to a non-negative real value, which is called the constraint value, or simply cost.  $H_{\mathrm{min}}$  is the minimum non-zero value of function  $h$ ;  $H_{\mathrm{max}}$  is upper bound on function  $h$ .  $\gamma$  is a discount factor in the range  $(0,1)$ .  $S_I$  is initial state set,  $d_0$  is initial state distribution, and  $\pi(a|s)$  is a stochastic policy that is parameterized by the state and returns an action distribution from which an action can be sampled and affects the environment defined by the MDP. In unconstrained RL, the goal is to learn an optimal policy  $\pi^*$  maximizing expected discounted sum of rewards, i.e.  $\pi^* = \arg \max_{\pi} \mathbb{E}_{s \sim d_0} V^\pi(s)$ , where  $V^\pi(s) \coloneqq \mathbb{E}_{\tau \sim \pi, P(s)}[\sum_{s_t \in \tau} \gamma^t r(s_t, a_t)]$ . Note:  $\tau \sim \pi$ ,  $P(s)$  indicates sampling trajectory  $\tau$  for horizon  $T$  starting from state  $s$  using policy  $\pi$  in MDP with transition model  $P$ , and  $s_t \in \tau$  is the  $t^{th}$  state in trajectory  $\tau$ .
3.2 Constrained Markov Decision Process
CMDP attempts to optimize the reward returns  $V^{\pi}(s)$  under the constraint that the cost return is below some manually chosen threshold  $\chi$ . Specifically:
$$
\max  _ {\pi} \mathbb {E} _ {s \sim d _ {0}} \left[ V ^ {\pi} (s) \right], \text {s u b j e c t t o} \mathbb {E} _ {s \sim d _ {0}} \left[ V _ {c} ^ {\pi} (s) \right] \leq \chi , \tag {CMDP}
$$
where cost return function  $V_{c}^{\pi}(s)$  is often defined as  $V_{c}^{\pi}(s) \coloneqq \mathbb{E}_{\tau \sim \pi, P(s)}[\sum_{s_t \in \tau} \gamma^{t} h(s_t)]$ .
While many approaches have been proposed to solve within this framework, CMDPs have several difficulties: 1. cost threshold  $\chi$  often requires much tuning while using prior knowledge of the environment; and 2. CMDP often permits some positive average cost which is incompatible with state-wise hard constraint problems, since  $\chi$  is usually chosen to be above 0.
4 Stochastic Hamilton-Jacobi Reachability for Reinforcement Learning
Classic HJ reachability considers finding the largest feasible set for deterministic environments. In this section, we apply a similar definition in [45, 46] and define the stochastic reachability problem.
4.1 Persistent Safety and HJ Reachability for Stochastic Systems
The instantaneous safety can be characterized by the safe set  $S_{s}$ , which is the zero level set of the safety loss function  $h: S \mapsto \mathbb{R}_0^+$ . The unsafe (i.e. violation) set  $S_{v}$  is the complement of the safe set.
Definition 1. Safe set and unsafe set:  $\mathcal{S}_s \coloneqq \{ s \in \mathcal{S} : h(s) = 0 \}$ ,  $\mathcal{S}_v \coloneqq \{ s \in \mathcal{S} : h(s) > 0 \}$ .
We will write  $\mathbb{1}_{s\in S_v}$  as the instantaneous violation indicator function, which is 1 if the current state is in the violation set and 0 otherwise. Note that the safety loss function  $h$  is different from the instantaneous violation indicator function since  $h$  captures the magnitude of the violation at the state.
It is insufficient to only consider instantaneous safety. When the environment and policy are both deterministic, we easily have a unique trajectory for starting from each state (i.e. the future state is uniquely determined) under Lipschitz environment dynamics. In classic HJ reachability literature [13], for a deterministic MDP's transition model  $P_{d}$  and deterministic policy  $\pi_d$ , the set of states that guarantees persistent safety is captured by the zero sub-level set of the following value function:
Definition 2. Reachability value function  $V_h^\pi : \mathcal{S} \mapsto \mathbb{R}_0^+$  is:  $V_h^\pi(s) \coloneqq \max_{s_t \in \tau \sim \pi_d, P_d(s)} h(s_t)$ .
However, when there's a stochastic environment with transition model  $P(\cdot | s, a)$  and policy  $\pi (\cdot | s)$ , the future states are not uniquely determined. This means for a given initial state and policy, there may exist many possible trajectories starting from this state. In this case, instead of defining a binary function that only indicates the existence of constraint violations, we define the reachability estimation function (REF), which captures the probability of constraint violation:
Definition 3. The reachability estimation function (REF)  $\phi^{\pi}:\mathcal{S}\mapsto [0,1]$  is defined as:
$$
\phi^ {\pi} (s) := \underset {\tau \sim \pi , P (s)} {\mathbb {E}} \max  _ {s _ {t} \in \tau} \mathbb {1} _ {(s _ {t} | s _ {0} = s, \pi) \in \mathcal {S} _ {v}}.
$$
In a specific trajectory  $\tau$ , the value  $\max_{s_t \in \tau} \mathbb{1}_{(s_t | s_0 = s, \pi) \in S_v}$  will be 1 if there exist constraint violations and 0 if there exists no violation, which is binary. Taking expectation over this binary value for all the trajectories, we get the desired probability. We define optimal REF based on an optimally safe policy  $\pi^* = \arg \min_{\pi} V_c^\pi(s)$  (note that this policy may not be unique).
Definition 4. The optimal reachability estimation function  $\phi^{*}:\mathcal{S}\mapsto [0,1]$  is:  $\phi^{\ast}(s)\coloneqq \phi^{\pi^{*}}(s)$
Interestingly, we can utilize the fact the instantaneous violation indicator function produces binary values to learn the REF function in a bellman recursive form. The following will be used later:
Theorem 1. The REF can be reduced to the following recursive Bellman formulation:
$$
\phi^ {\pi} (s) = \max  \{\mathbb {1} _ {s \in \mathcal {S} _ {v}}, \underset {s ^ {\prime} \sim \pi , P (s)} {\mathbb {E}} \phi^ {\pi} (s ^ {\prime}) \},
$$
where  $s' \sim \pi$ ,  $P(s)$  is a sample of the immediate successive state (i.e.,  $s' \sim P(\cdot | s, a \sim \pi(\cdot | s))$ ) and the expectation is taken over all possible successive states. The proof can be found in the appendix.
Definition 5. The feasible set of a policy  $\pi$  based on  $\phi^{\pi}(s)$  is defined as:  $S_f^\pi \coloneqq \{s \in S : \phi^\pi(s) = 0\}$ .
Note, the feasible set for a specific policy is the set of states starting from which no violation is reached, and the safe set is the set of states at which there is no violation. We will use the phrase likelihood of being feasible to mean the likelihood of not reaching a violation, i.e.  $1 - \phi^{\pi}(s)$ .
4.2 Comparison with RCRL
The RCRL approach [27] uses reachability to optimize and maintain persistent safety in the feasible set. Note, in below formulation,  $S_{f}$  is the optimal feasible set, i.e. that of a policy  $\arg \min_{\pi}V_h^{\pi}(s)$ . The RCRL formulation is:
$$
\max  _ {\pi} \mathbb {E} _ {s \sim d _ {0}} \left[ V ^ {\pi} (s) \cdot \mathbb {1} _ {s \in \mathcal {S} _ {f}} - V _ {h} ^ {\pi} (s) \cdot \mathbb {1} _ {s \notin \mathcal {S} _ {f}} \right], \text {s u b j e c t t o} V _ {h} ^ {\pi} (s) \leq 0, \forall s \in \mathcal {S} _ {I} \cap \mathcal {S} _ {f}. \quad (\text {R C R L})
$$
The equation RCRL considers two different optimizations. When in the optimal feasible set, the optimization produces a persistently safe policy maximizing rewards. When outside this set, the
optimization produces a control minimizing the maximum future violation, i.e.  $\arg \min_{\pi} V_h^{\pi}(s)$ . However, this does not ensure (re)entrance into the feasible set even if such a control exists.
RCRL performs constraint optimization on  $V_h^{\pi}$  with a neural network (NN) lagrange multiplier with state input [9]. When learning to optimize a Lagrangian dual function, the NN lagrange multiplier should converge to small values for states in the optimal feasible set and converge to large values for other states. Nonetheless, learning  $V_h$  provides a weak signal during training: if there is an improvement in safety along the trajectory not affecting the maximum violation,  $V_h^{\pi}$  remains the same for all states before the maximum violation in the trajectory. These improvements in costs can be crucial in guiding the optimization toward a safer policy. And optimizing with  $V_h(s)$  can result in accumulating an unlimited number of violations smaller than the maximum violation. Also, a major issue with this approach is that it's limited to deterministic MDPs and policies because its reachability value function in the Bellman formulation does not directly apply to the stochastic setting. However, in general stochastic settings, estimating feasibility cannot be binary since for a large portion of the state space, even under the optimal policy, the agent may enter the unsafe set with a non-zero probability, rendering such definition too conservative and impractical.
5 Iterative Reachability Estimation for Safe Reinforcement Learning
In this paper, we formulate a general optimization framework for safety-constrained RL and propose a new algorithm to solve our constraint optimization by using our novel reachability estimation function. We present the deterministic case in Section 5.1 and build our way to the stochastic case in Section 5.2. We present our novel algorithm to solve these optimizations, involving our new reachability estimation function, in Section 5.3. We introduce convergence analysis in Section 5.4.
5.1 Iterative Reachability Estimation for Deterministic Settings
All state transitions and policies happen with likelihood 0 or 1 for the deterministic environment. Therefore, the probability of constraint violation for policy  $\pi$  from state  $s$ , i.e.,  $\phi^{\pi}(s)$ , is in the set  $\{0,1\}$ . According to Definition 4, if there exists some policy  $\pi$  such that  $\phi^{\pi}(s) = 0$ , we have  $\phi^{*}(s) = 0$ . Otherwise,  $\phi^{*}(s) = 1$ . Notice that this captures definitive membership in the optimal feasible set  $\phi^{*}(s) = \mathbb{1}_{s \in S_{f}^{\pi s}}$ , which is the feasible set of some safest policy  $\pi_{s} = \arg \min_{\pi} V_{c}^{\pi}(s)$ . Now, we divide our optimization in two parts: the infeasible part and the feasible part.
For the infeasible part, we want the agent to incur the least cumulative damage (discounted sum of costs) and, if possible, (re)enter the feasible set. Different from previous Reachability-based RL optimizations, by using the discounted sum of costs  $V_{c}^{\pi}(s)$  we consider both magnitude and frequency of violations, thereby improving learning signal. The infeasible portion takes the form:
$$
\max  _ {\pi} \mathbb {E} _ {s \sim d _ {0}} \left[ - V _ {c} ^ {\pi} (s) \right]. \tag {1}
$$
For the feasible part, we want the policy to ensure the agent stays in the feasible set and maximize reward returns. This produces a constraint optimization where the cost value function is constrained:
$$
\max  _ {\pi} \mathbb {E} _ {s \sim d _ {0}} \left[ V ^ {\pi} (s) \right], \text {s u b j e c t t o} V _ {c} ^ {\pi} (s) = 0, \forall s \in \mathcal {S} _ {I}. \tag {2}
$$
The following propositions justify using  $V_{c}^{\pi}$  as the constraint. Both proofs are in the appendix.
Proposition 1. The cost value function  $V_{c}^{\pi}(s)$  is zero for state  $s$  if and only if the persistent safety is guaranteed for that state under the policy  $\pi$ .
We define here  $S_f \coloneqq S_f^{\pi_s}$ , the feasibility set of some safest policy. Now, the above two optimizations can be unified with the use of the feasibility function  $\phi^*(s)$ :
$$
\max  _ {\pi} \mathbb {E} _ {s \sim d _ {0}} \left[ V ^ {\pi} (s) \cdot \left(1 - \phi^ {*} (s)\right) - V _ {c} ^ {\pi} (s) \cdot \phi^ {*} (s) \right], \text {s u b j e c t} V _ {c} ^ {\pi} (s) = 0, \forall s \in \mathcal {S} _ {I} \cap \mathcal {S} _ {f}. \tag {3}
$$
Unlike other reachability based optimizations like RCRL, one particular advantage in Equation 3 is, with some assumptions, the guaranteed entrance back into feasible set with minimum cumulative discounted violations whenever a possible control exists. More formally, assuming infinite horizon:
Proposition 2. If  $\exists \pi$  that produces trajectory  $\tau = \{(s_i), i \in \mathbb{N}, s_1 = s\}$  in deterministic MDP  $\mathcal{M}$  starting from state  $s$ , and  $\exists m \in \mathbb{N}, m < \infty$  such that  $s_m \in S_f^\pi$ , then  $\exists \epsilon > 0$  where if discount factor  $\gamma \in (1 - \epsilon, 1)$ , then the optimal policy  $\pi^*$  of Equation 3 will produce a trajectory  $\tau' = \{(s_j'), j \in \mathbb{N}, s_1' = s\}$ , such that  $\exists n \in \mathbb{N}, n < \infty$ ,  $s_n' \in S_f^{\pi^*}$  and  $V_c^{\pi^*}(s) = \min_{\pi'} V_c^{\pi'}(s)$ .
5.2 Iterative Reachability Estimation for Stochastic Settings
In stochastic environments, for each state, there is some likelihood of entering into the unsafe states under any policy. Thus, we adopt the probabilistic reachability Definitions 3 and 4. Rather than using the binary indicator in the optimal feasible set to demarcate the feasibility and infeasibility optimization scenarios, we use the likelihood of infeasibility of the safest policy. In particular, for any state  $s$ , the optimal likelihood that the policy will enter the infeasible set is  $\phi^{*}(s)$  from Definition 4.