Title: Iterative Reachability Estimation for Safe Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Constraints

Safety

Hamilton Jacobi Reachability

Deep Reinforcement Learning

Robotics
Abstract: Ensuring safety is important for the practical deployment of reinforcement learning (RL). Various challenges must be addressed, such as handling stochasticity in the environments, providing rigorous guarantees of persistent state-wise safety satisfaction, and avoiding overly conservative behaviors that sacrifice performance. We propose a new framework, Reachability Estimation for Safe Policy Optimization (RESPO), for safety-constrained RL in general stochastic settings. In the feasible set where there exist violation-free policies, we optimize for rewards while maintaining persistent safety. Outside this feasible set, our optimization produces the safest behavior by guaranteeing entrance into the feasible set whenever possible with the least cumulative discounted violations. We introduce a class of algorithms using our novel reachability estimation function to optimize in our proposed framework and in similar frameworks such as those concurrently handling multiple hard and soft constraints. We theoretically establish that our algorithms almost surely converge to locally optimal policies of our safe optimization framework. We evaluate the proposed methods on a diverse suite of safe RL environments from Safety Gym, PyBullet, and MuJoCo, and show the benefits in improving both reward performance and safety compared with state-of-the-art baselines.

Iterative Reachability Estimation for Safe Reinforcement Learning

Abstract

Ensuring safety is important for the practical deployment of reinforcement learning (RL). Various challenges must be addressed, such as handling stochasticity in the environments, providing rigorous guarantees of persistent state-wise safety satisfaction, and avoiding overly conservative behaviors that sacrifice performance. We propose a new framework, Reachability Estimation for Safe Policy Optimization (RESPO), for safety-constrained RL in general stochastic settings. In the feasible set where there exist violation-free policies, we optimize for rewards while maintaining persistent safety. Outside this feasible set, our optimization produces the safest behavior by guaranteeing entrance into the feasible set whenever possible with the least cumulative discounted violations. We introduce a class of algorithms using our novel reachability estimation function to optimize in our proposed framework and in similar frameworks such as those concurrently handling multiple hard and soft constraints. We theoretically establish that our algorithms almost surely converge to locally optimal policies of our safe optimization framework. We evaluate the proposed methods on a diverse suite of safe RL environments from Safety Gym, PyBullet, and MuJoCo, and show the benefits in improving both reward performance and safety compared with state-of-the-art baselines.

1 Introduction

Safety-Constrained Reinforcement Learning is important for a multitude of real-world safety-critical applications [1, 2]. These situations require guaranteeing persistent safety and achieving high performance while handling environment uncertainty. Traditionally, methods have been proposed to obtain optimal control by solving within the Constrained Markov Decision Process (CMDP) framework, which constrains expected cumulative violation of the constraints along trajectories to be under some threshold. Trust-region approaches [3, 4, 5] follow from the Constrained Policy Optimization algorithm [6] which tries to guarantee monotonic performance improvement while satisfying the cumulative constraint requirement. However, these approaches are generally very computationally expensive, and their first or second order Taylor approximations may have repercussions on performance [2]. Several primal-dual approaches [7, 8, 9] have been proposed with better computational efficiency and use multi-timescale frameworks [10] for asymptotic convergence to a local optimum. Nonetheless, these approaches inherit CMDP's lack of crucial rigorous safety guarantees that ensure persistent safety. Particularly, maintaining a cost budget for hard constraints like collision avoidance is insufficient. Rather, it's crucial to prioritize obtaining safe, minimal-violation policies when possible.

There have been a variety of proposed control theoretic functions that provide the needed guarantees on persistent safety for such hard constraints, including Control Barrier Functions (CBFs) [11] and Hamilton-Jacobi (HJ) reachability analysis [12, 13]. CBFs are energy-based certification functions, whose zero super-level set is control invariant [14, 15]. The drawbacks are the conservativeness of the feasible set and the lack of a systematic way to find these functions for general nonlinear systems. Under certain assumptions, approximate energy-based certificates can be learned with sufficient samples [16, 17, 18, 19, 20, 21]. HJ reachability analysis focuses on finding the set of initial states such that there exists some control to avoid and/or reach a certain region. It computes a value function, whose zero sub-level set is the largest feasible set, together with the optimal control. The drawback is the well known 'curse of dimensionality,' as HJ reachability requires solving a corresponding Hamilton-Jacobi-Bellman Partial Differential Equation (HJBPDE) on a discrete grid recursively. Originally, both approaches assume the system dynamics and environment are known, and inputs are bounded by some polytopic constraints, while recent works have applied them to model-free settings [22, 23, 24]. For both CBF and HJ reachability, controllers that optimize performance and guarantee safety can be synthesized in different ways online [25]. There also exist hard-constraint approaches like [26] have theoretical safety guarantees without relying on control theory; however, they require learning or having access to the transition-function dynamics.

Consequently, it can be beneficial to combine CMDP-based approaches to find optimal control and the advantages of control-theoretic approaches to enable safety. The recent framework of Reachability Constrained Reinforcement Learning (RCRL) [27] proposed an approach that guarantees persistent safety and optimal performance when the agent is in the feasible set. However, RCRL is limited to learning deterministic policies in deterministic environments . In the general RL setting, stochasticity incurs a variety of challenges including how to define membership in the feasible set since it is no longer a binary question but rather a probabilistic one. Another issue is that for states outside optimal feasible set, RCRL optimization cannot guarantee entrance into the feasible set when possible . This may have undesirable consequences including indefinitely violating constraints with no recovery.

In light of the above challenges, we propose a new framework and class of algorithms called Reachability Estimation for Safe Policy Optimization ( RESPO ). Our main contributions are:

We introduce reachability estimation methods for general stochastic policies that predict the likelihood of constraint violations in the future. Using this estimation, a policy can be optimized such that (i) when in the feasible set, it maintains persistent safety and optimizes for reward performance, and (ii) when outside the feasible set, it produces the safest behavior and enters the feasible set if possible. We also demonstrate how our optimization can incorporate multiple hard and soft constraints while even prioritizing the hard constraints.

We provide a novel class of actor-critic algorithms based on learning our reachability estimation function, and prove our algorithm converges to a locally optimal policy of our proposed optimization.

We perform comprehensive experiments showing that our approach achieves high performance in complex, high-dimensional Safety Gym, PyBullet-based, and MuJoCo-based environments compared to other state-of-the-art safety-constrained approaches while achieving small or even 0 violations. We also show our algorithm's performance in environments with multiple hard and soft constraints.

2 Related Work

2.1 Constrained Reinforcement Learning

Safety-constrained reinforcement learning has been addressed through various proposed learning mechanisms solving within an optimization framework. CMDP [28, 29] is one framework that augments MDPs with the cost function: the goal is to maximize expected reward returns while constraining cost returns below a manually chosen threshold. Trust-region methods [3, 4, 5, 6] and primal-dual based approaches [7, 9, 30, 31] are two classes of CMDP-based approaches. Trust-region approaches generally follow from Constrained Policy Optimization (CPO) [6], which approximates the constraint optimization problem with surrogate functions for the reward objective and safety constraints, and then performs projection steps on the policy parameters with backtracking line search. Penalized Proximal Policy Optimization [32] improves upon past trust-region approaches via the clipping mechanism, similar to Proximal Policy Optimization's [33] improvement over Trust Region Policy Optimization [34]. Constraint-Rectified Policy Optimization [35] takes steps toward improving reward performance if constraints are currently satisfied else takes steps to minimize

constraint violations. PPO Lagrangian [30] is a primal-dual method combining Proximal Policy Optimization (PPO) [33] with lagrangian relaxation of the safety constraints and has relatively low complexity but outperforms CPO in constraint satisfaction. [7] uses a penalized reward function like PPO Lagrangian, and demonstrates convergence to an optimal policy through the multi-timescale framework, introduced in [10]. [36] proposes mapping the value function constraining problem as constraining densities of state visitation. Overall, while these algorithms provide benefits through various learning mechanisms, they inherit the problems of the CMDP framework. Specifically, they do not provide rigorous guarantees of persistent safety and so are generally unsuitable for state-wise constraint optimization problems.

2.2 Hamilton-Jacobi Reachability Analysis

HJ reachability analysis is a rigorous approach that verifies safety or reachability for dynamical systems. Given any deterministic nonlinear system and target set, it computes the viscosity solution for the HJBPDE, whose zero sub-level set is the backward reachable set (BRS), meaning there exists some control such that the states starting from this set will enter the target set in future time (or stay away from the target set for all future time). However, the curse of dimensionality and the assumption of knowing the dynamics and environment in advance restrict its application, especially in the model-free RL setting where there is no access to the entire observation space and dynamics at any given time. Decomposition [37], warm-starting [38], sampling-based reachability [39], and Deepreach [40] have been proposed to solve the curse of dimensionality. [23, 24] proposed methods to bridge HJ analysis with RL by modifying the HJBPDE. The work of [27] takes framework into a deterministic hard-constraint setting. There are additionally model-based HJ reachability approach for reinforcement learning-based controls. [41] is a model-based extension of [27]. Approaches like [42, 43, 44] use traditional HJ reachability for RL control while learning or assuming access to the system dynamics model. In stochastic systems, finding the probability of reaching a target while avoiding certain obstacles are key problems. [45, 46] constructed the theoretic framework based on dynamic programming and consider finite and infinite time reach-avoid problems. [47, 48, 49] propose computing the stochastic reach-avoid set together with the probability in a tractable manner to address the curse of dimensionality.

3 Preliminaries

3.1 Markov Decision Processes

Markov Decision Processes (MDP) are defined as M := ⟨S , A , P, r, h, γ ⟩ . S and A are the state and action spaces respectively. P : S × A × S ↦→ [0 , 1] is the transition function capturing the environment dynamics. r : S × A ↦→ R is the reward function associated with each state-action pair, h : S ↦→ R + 0 is the safety loss function that maps a state to a non-negative real value, which is called the constraint value, or simply cost. H min is the minimum non-zero value of function h ; H max is upper bound on function h . γ is a discount factor in the range (0 , 1) . S I is initial state set, d 0 is initial state distribution, and π ( a | s ) is a stochastic policy that is parameterized by the state and returns an action distribution from which an action can be sampled and affects the environment defined by the MDP. In unconstrained RL, the goal is to learn an optimal policy π ∗ maximizing expected discounted sum of rewards, i.e. π ∗ = arg max π E s ∼ d 0 V π ( s ) , where V π ( s ) := E τ ∼ π,P ( s ) [ ∑ s t ∈ τ γ t r ( s t , a t )] . Note: τ ∼ π, P ( s ) indicates sampling trajectory τ for horizon T starting from state s using policy π in MDP with transition model P , and s t ∈ τ is the t th state in trajectory τ .

3.2 Constrained Markov Decision Process

CMDP attempts to optimize the reward returns V π ( s ) under the constraint that the cost return is below some manually chosen threshold χ . Specifically:

where cost return function V π c ( s ) is often defined as V π c ( s ) := E τ ∼ π,P ( s ) [ ∑ s t ∈ τ γ t h ( s t )] .

While many approaches have been proposed to solve within this framework, CMDPs have several difficulties: 1 . cost threshold χ often requires much tuning while using prior knowledge of the environment; and 2 . CMDP often permits some positive average cost which is incompatible with state-wise hard constraint problems, since χ is usually chosen to be above 0.

4 Stochastic Hamilton-Jacobi Reachability for Reinforcement Learning

Classic HJ reachability considers finding the largest feasible set for deterministic environments. In this section, we apply a similar definition in [45, 46] and define the stochastic reachability problem.

4.1 Persistent Safety and HJ Reachability for Stochastic Systems

The instantaneous safety can be characterized by the safe set S s , which is the zero level set of the safety loss function h : S ↦→ R + 0 . The unsafe (i.e. violation) set S v is the complement of the safe set.

Definition 1. Safe set and unsafe set: S s := { s ∈ S : h ( s ) = 0 } , S v := { s ∈ S : h ( s ) > 0 } .

We will write ✶ s ∈S v as the instantaneous violation indicator function , which is 1 if the current state is in the violation set and 0 otherwise. Note that the safety loss function h is different from the instantaneous violation indicator function since h captures the magnitude of the violation at the state.

It is insufficient to only consider instantaneous safety. When the environment and policy are both deterministic, we easily have a unique trajectory for starting from each state (i.e. the future state is uniquely determined) under Lipschitz environment dynamics. In classic HJ reachability literature [13], for a deterministic MDP's transition model P d and deterministic policy π d , the set of states that guarantees persistent safety is captured by the zero sub-level set of the following value function:

Definition 2. Reachability value function V π h : S ↦→ R + 0 is: V π h ( s ) := max s t ∈ τ ∼ π d ,P d ( s ) h ( s t ) .

However, when there's a stochastic environment with transition model P ( ·| s, a ) and policy π ( ·| s ) , the future states are not uniquely determined. This means for a given initial state and policy, there may exist many possible trajectories starting from this state. In this case, instead of defining a binary function that only indicates the existence of constraint violations, we define the reachability estimation function (REF), which captures the probability of constraint violation:

Definition 3. The reachability estimation function (REF) ϕ π : S ↦→ [0 , 1] is defined as:

In a specific trajectory τ , the value max s t ∈ τ ✶ ( s t | s 0 = s,π ) ∈S v will be 1 if there exist constraint violations and 0 if there exists no violation, which is binary. Taking expectation over this binary value for all the trajectories, we get the desired probability. We define optimal REF based on an optimally safe policy π ∗ = arg min π V π c ( s ) (note that this policy may not be unique).

Definition 4. The optimal reachability estimation function ϕ ∗ : S ↦→ [0 , 1] is: ϕ ∗ ( s ) := ϕ π ∗ ( s ) .

Interestingly, we can utilize the fact the instantaneous violation indicator function produces binary values to learn the REF function in a bellman recursive form. The following will be used later:

Theorem 1. The REF can be reduced to the following recursive Bellman formulation:

where s ′ ∼ π, P ( s ) is a sample of the immediate successive state (i.e., s ′ ∼ P ( ·| s, a ∼ π ( ·| s )) ) and the expectation is taken over all possible successive states. The proof can be found in the appendix.

Definition 5. The feasible set of a policy π based on ϕ π ( s ) is defined as: S π f := { s ∈ S : ϕ π ( s ) = 0 } .

Note, the feasible set for a specific policy is the set of states starting from which no violation is reached, and the safe set is the set of states at which there is no violation. We will use the phrase likelihood of being feasible to mean the likelihood of not reaching a violation, i.e. 1 -ϕ π ( s ) .

4.2 Comparison with RCRL

The RCRL approach [27] uses reachability to optimize and maintain persistent safety in the feasible set. Note, in below formulation, S f is the optimal feasible set, i.e. that of a policy arg min π V π h ( s ) . The RCRL formulation is:

The equation RCRL considers two different optimizations. When in the optimal feasible set, the optimization produces a persistently safe policy maximizing rewards. When outside this set, the

optimization produces a control minimizing the maximum future violation, i.e. arg min π V π h ( s ) . However, this does not ensure (re)entrance into the feasible set even if such a control exists.

RCRL performs constraint optimization on V π h with a neural network (NN) lagrange multiplier with state input [9]. When learning to optimize a Lagrangian dual function, the NN lagrange multiplier should converge to small values for states in the optimal feasible set and converge to large values for other states. Nonetheless, learning V h provides a weak signal during training: if there is an improvement in safety along the trajectory not affecting the maximum violation, V π h remains the same for all states before the maximum violation in the trajectory. These improvements in costs can be crucial in guiding the optimization toward a safer policy. And optimizing with V h ( s ) can result in accumulating an unlimited number of violations smaller than the maximum violation. Also, a major issue with this approach is that it's limited to deterministic MDPs and policies because its reachability value function in the Bellman formulation does not directly apply to the stochastic setting. However, in general stochastic settings, estimating feasibility cannot be binary since for a large portion of the state space, even under the optimal policy, the agent may enter the unsafe set with a non-zero probability, rendering such definition too conservative and impractical.

5 Iterative Reachability Estimation for Safe Reinforcement Learning

In this paper, we formulate a general optimization framework for safety-constrained RL and propose a new algorithm to solve our constraint optimization by using our novel reachability estimation function. We present the deterministic case in Section 5.1 and build our way to the stochastic case in Section 5.2. We present our novel algorithm to solve these optimizations, involving our new reachability estimation function, in Section 5.3. We introduce convergence analysis in Section 5.4.

5.1 Iterative Reachability Estimation for Deterministic Settings

All state transitions and policies happen with likelihood 0 or 1 for the deterministic environment. Therefore, the probability of constraint violation for policy π from state s , i.e., ϕ π ( s ) , is in the set { 0 , 1 } . According to Definition 4, if there exists some policy π such that ϕ π ( s ) = 0 , we have ϕ ∗ ( s ) = 0 . Otherwise, ϕ ∗ ( s ) = 1 . Notice that this captures definitive membership in the optimal feasible set ϕ ∗ ( s ) = ✶ s ∈ S πs f , which is the feasible set of some safest policy π s = arg min π V π c ( s ) . Now, we divide our optimization in two parts: the infeasible part and the feasible part.

For the infeasible part, we want the agent to incur the least cumulative damage (discounted sum of costs) and, if possible, (re)enter the feasible set. Different from previous Reachability-based RL optimizations, by using the discounted sum of costs V π c ( s ) we consider both magnitude and frequency of violations, thereby improving learning signal. The infeasible portion takes the form:

For the feasible part, we want the policy to ensure the agent stays in the feasible set and maximize reward returns. This produces a constraint optimization where the cost value function is constrained:

The following propositions justify using V π c as the constraint. Both proofs are in the appendix.

Proposition 1. The cost value function V π c ( s ) is zero for state s if and only if the persistent safety is guaranteed for that state under the policy π .

We define here S f := S π s f , the feasibilty set of some safest policy. Now, the above two optimizations can be unified with the use of the feasibility function ϕ ∗ ( s ) :

Unlike other reachability based optimizations like RCRL, one particular advantage in Equation 3 is, with some assumptions, the guaranteed entrance back into feasible set with minimum cumulative discounted violations whenever a possible control exists. More formally, assuming infinite horizon:

Proposition 2. If ∃ π that produces trajectory τ = { ( s i ) , i ∈ ◆ , s 1 = s } in deterministic MDP M starting from state s , and ∃ m ∈ ◆ , m < ∞ such that s m ∈ S π f , then ∃ ϵ > 0 where if discount factor γ ∈ (1 -ϵ, 1) , then the optimal policy π ∗ of Equation 3 will produce a trajectory τ ′ = { ( s ′ j ) , j ∈ ◆ , s ′ 1 = s } , such that ∃ n ∈ ◆ , n < ∞ , s ′ n ∈ S π ∗ f and V π ∗ c ( s ) = min π ′ V π ′ c ( s ) .

5.2 Iterative Reachability Estimation for Stochastic Settings

In stochastic environments, for each state, there is some likelihood of entering into the unsafe states under any policy. Thus, we adopt the probabilistic reachability Definitions 3 and 4. Rather than using the binary indicator in the optimal feasible set to demarcate the feasibility and infeasibility optimization scenarios, we use the likelihood of infeasibility of the safest policy. In particular, for any state s , the optimal likelihood that the policy will enter the infeasible set is ϕ ∗ ( s ) from Definition 4.

We again divide the full optimization problem in stochastic settings into infeasible and feasible ones similar to Equations 1 and 2. However, we consider the infeasible formulation with likelihood the current state is in a safest policy's infeasible state, or ϕ ∗ ( s ) . Similarly, we account for the feasible optimization formulation with likelihood the current state is in a safest policy's feasible set, 1 -ϕ ∗ ( s ) . The complete Reachability Estimation for Safe Policy Optimization (RESPO) can be rewritten as:

(RESPO)

In sum, the RESPO framework provides several benefits when compared with other constrained Reinforcement Learning and reachability-based approaches. Notably, 1) it maintains persistent safety when in the feasible set unlike CMDP-based approaches, 2) compared with other reachability-based approaches, RESPO considers performance optimization in addition to maintaining safety, 3) it maintains the behavior of a safest policy in the infeasible set and even reenters the feasible set when possible, 4) RESPO employs rigorously defined reachability definitions even in stochastic settings.

5.3 Overall Algorithm

We describe our algorithms by breaking down the novel components. Our algorithm predicts reachability membership to guide the training toward optimizing the right portion of the optimization equation (i.e., feasibility case or infeasibility case). Furthermore, it exclusively uses the discounted sum of costs as the safety value function - we can avoid having to learn the reachability value function while having the benefit of exploiting the improved signal in the cost value function.

Optimization in infeasible set versus feasible set. If the agent is in the infeasible set, this is the simplest case. We want to find the optimal policy that maximizes -V π c ( s ) . This would be the only term that needs to be considered in optimization.

On the other hand, if the agent is in the feasible set, we must solve the constraint optimization max π V π ( s ) subject to V π c ( s ) = 0 . This could be solved via a Lagrangian-based method:

Now what remains is obtaining the reachability estimation function ϕ ∗ . First, we address the problem of acquiring optimal likelihood of being feasible. It is nearly impossible to accurately know before training if a state is in a safest policy's infeasible set. We propose learning a function guaranteed to converge to this REF (with some discount factor for γ -contraction mapping) by using the recursive Bellman formulation proved in Theorem 1.

We learn a function p ( s ) to capture the probability ϕ ∗ ( s ) . It is trained like a reachability function:

where S v is the violation set, s ′ is the next sampled state, and γ is a discount parameter 0 ≪ γ < 1 to ensure convergence of p ( s ) . Furthermore, and crucially, we ensure the learning rate of this REF is on a slower time scale than the policy and its critics but faster than the lagrange multiplier.

Bringing the concepts covered above, we present our full optimization equation:

We show the design of our algorithm RESPO in an actor-critic framework in Algorithm 1. Note that the V and V c have corresponding Q functions: V π ( s ) = E a ∼ π ( ·| s ) Q ( s, a ) and V π c ( s ) = E a ∼ π ( ·| s ) Q c ( s, a ) . The gradients' definitions are found in the appendix. We use operator Γ Θ to indicate the projection of vector θ ∈ ❘ n to the closest point in compact and convex set Θ ⊆ ❘ n . Specifically, Γ Θ = arg min ˆ θ ∈ Θ || ˆ θ -θ || 2 . Γ Ω is similarly defined.

Algorithm 1 RESPO Actor Critic

Require: Randomly initialized policy π θ 's parameters θ 0 , reward critic Q 's parameters η 0 , cost critic Q c 's parameters κ 0 , REF p 's parameters ξ 0 , Lagrange multiplier λ 's parameters ω 0 , horizon T Require: Convex projection operators Γ Θ and Γ Ω , and reward and cost critic learning rate ζ 1 ( k ) , policy learning rate ζ 2 ( k ) , REF learning rate ζ 3 ( k ) , lagrange multiplier learning rate ζ 4 ( k )

1: for k = 0 , 1 , 2 , ... do

2: for i = 0 , 1 , 2 , ... do

3: Sample trajectories τ i : { ( s j , a j , s ′ j , r j , h j ) } ∼ π θ

4: Rew. Update η k +1 = η k -ζ 1 ( k ) ∇ η Q ( s t , a t ) · [ Q ( s t , a t ) -( r ( s t , a t ) + γQ ( s t +1 , a t +1 ))]

5: Cost Update κ k +1 = κ k -ζ 1 ( k ) ∇ κ Q c ( s t , a t ) · [ Q c ( s t , a t ) -( h ( s t ) + γQ c ( s t +1 , a t +1 ))]

6: Policy Update θ k +1 =

8: REF Update ξ k +1 = ξ k -ζ 3 ( k ) ∇ ξ p ( s t ) · [ p ( s t ) -max { ✶ h ( s t ) > 0 , γp ( s t +1 ) } ]

9: Lagrange multiplier Update ω k +1 = Γ Ω ( ω k -ζ 4 ( k ) Q c ( s t , a t )(1 -p ( s t )) ∇ ω λ

10: end for

11: end for

Unsafe

region

5.4 Convergence Analysis

We provide convergence analysis of our algorithm for Finite MDPs (finite bounded state and action space sizes, maximum horizon T , reward bounded by R max , and cost bounded by H max ) under reasonable assumptions. We demonstrate our algorithm almost surely finds a locally optimal policy for our RESPO formulation, based on the following assumptions:

A2 (Strict Feasibility): ∃ π ( ·|· ; θ ) such that ∀ s ∈ S I where ϕ ∗ ( s ) = 0 , V π θ c ( s ) ≤ 0 .

The reward returns and cost returns critic value functions must follow the fastest schedule ζ 1 ( k ) , the policy must follow the second fastest schedule ζ 2 ( k ) , the REF must follow the second slowest schedule ζ 3 ( k ) , and finally, the lagrange multiplier should follow the slowest schedule ζ 4 ( k ) .

A3 (Differentiability and Lipschitz Continuity): For all state-action pairs ( s, a ) , we assume value and cost Q functions Q ( s, a ; η ) , Q c ( s, a ; κ ) , policy π ( a | s ; θ ) , and REF p ( s, a ; ξ ) are continuously differentiable in η, κ, θ, ξ respectively. Furthermore, ∇ ω λ ω and, for all state-action pairs ( s, a ) , ∇ θ π ( a | s ; θ ) are Lipschitz continuous functions in ω and θ respectively.

The detailed proof of the following result is provided in the appendix.

Theorem 2. Given Assumptions A1 -A3 , the policy updates in Algorithm 1 will almost surely converge to a locally optimal policy for our proposed optimization in Equation RESPO.

6 Experiments

Baselines. The baselines we compare are CMDP-based or solve for hard constraints. The CMDP baselines are Lagrangian-based Proximal Policy Optimization ( PPOLag ) based on [7], ConstraintRectified Policy Optimization ( CRPO ) [35], Penalized Proximal Policy Optimization ( P3O ) [32], and Projection-Based Constrained Policy Optimization ( PCPO ) [3]. The hard constraints baselines are RCRL [27], CBF with constraint ˙ h ( s ) + ν · h ( s ) ≤ 0 , and Feasibile Actor-Critic ( FAC ) [9]. We classify FAC among the hard constraint approaches because we make its cost threshold χ = 0 in order to better compare using NN lagrange multiplier with our REF approach in RESPO . We include the unconstrained Vanilla PPO [33] baseline for reference.

)

Benchmarks. We compare RESPO with the baselines in a diverse suite of safety environments. We consider high-dimensional environments in Safety Gym [30] (namely PointButton and CarGoal), Safety PyBullet [50] (namely DroneCircle and BallRun), and Safety MuJoCo [51], (namely Safety HalfCheetah and Reacher). We also show our algorithm in a multi-drone environment with multiple hard and soft constraints . More detailed experiment explanations and evaluations are in the appendix.

6.1 Main Experiments in Safety Gym, Safety PyBullet, and MuJoCo

We compare our algorithm with SOTA benchmarks on various high-dimensional (up to 76 D observation space), complex environments in the stochastic setting, i.e., where the environment and/or policy are stochastic. Particularly, we examine environments in Safety Gym, Safety PyBullet, and Safety MuJoCo. The environments provide reward for achieving a goal behavior or location, while the cost is based on tangible (e.g., avoiding quickly moving objects) and non-tangible (e.g., satisfying speed limit) constraints. Environments like PointButton require intricate behavior where specific buttons must be reached while avoiding multiple moving obstacles, stationary hazards, and wrong buttons.

Overall, RESPO achieves the best balance between optimizing reward and minimizing cost violations across all the environments. Specifically, our approach generally has the highest reward performance (see the red lines from the top row of Figure 2) among the safety-constrained algorithms while maintaining reasonably low to 0 cost violations (like in HalfCheetah). When RESPO performs the second highest, the highest-performing safety algorithm always incurs several times more violations than RESPO - for instance, RCRL in PointButton or PPOLag in Drone Circle. Non-primal-dual CMDPapproaches, namely CRPO , P3O , and PCPO generally satisfy their cost threshold constraints, but their reward performances rarely exceed that of PPOLag . RCRL generally has extremes of high reward and high cost, like in BallRun, or low reward and low cost, like in CarGoal. FAC and CBF generally have conservative behavior that sacrifices reward performance to minimize cost.

0

2

4

6

8

1e6

10

0

10

20

30

CarGoal Cumulative

Rewards

RCRL

FAC

CBF

Vanilla PPO

CRPO

P3O

PCPO

0

2

4

6

8

1e6

0

10

20

PointButton Cumulative

Rewards

0

2

4

6

8

1e6

0.0

1.0

2.0

3.0

×10 3

BallRun Cumulative

Rewards

0

2

4

6

8

1e6

0.0

0.2

0.4

0.6

0.8

1.0

×10 3

DroneCircle Cumulative

Rewards

0

2

4

6

8

0

20

40

60

CarGoal Cumulative

Costs

0

2

4

6

8

0.0

0.5

1.0

1.5

×10 2

PointButton Cumulative

Costs

0

2

4

6

8

0.0

0.5

1.0

1.5

2.0

2.5

×10 2

BallRun Cumulative

Costs

0

2

4

6

8

0

10

20

30

40

DroneCircle Cumulative

Costs

1e6

1e6

1e6

1e6

6.2 Hard and Soft Constraints

Wealso demonstrate RESPO 's performance in an environment with multiple hard and soft constraints. The environment requires controlling two drones to pass through a tunnel one at a time while respecting certain distance requirements. The reward is given for quickly reaching the goal positions. The two hard constraints involve (H1) ensuring neither drone collides into the wall and (H2) the distance between the two drones is more than 0.5 to ensure they do not collide. The soft constraint is that the two drones are within 0.8 of each other to ensure real-world communication. It is preferable to prioritize hard constraint H1 over hard constraint H2 , since colliding with the wall may have more serious consequences to the drones rather than violations of an overly precautious distance constraint.

Our approach, in the leftmost of Figure 3, successfully reaches the goal while avoiding the wall obstacles in all time steps. We are able to prioritize this wall avoidance constraint over the second hard constraint. This can be seen particularly in between the blue to cyan time period where the higher

1.5

1.0

0.5

0.0

0.5

1.0

1.5

0.6

0.4

0.2

0.0

0.2

0.4

0.6

RESPO

(ours)

0

10

20

30

40

50

60

70

Trajectory Step Number

1.5

1.0

0.5

0.0

0.5

1.0

1.5

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0

20

40

60

80

Trajectory Step Number

1.5

1.0

0.5

0.0

0.5

1.0

1.5

0.6

0.4

0.2

0.0

0.2

0.4

0.6

RCRL

0

10

20

30

40

50

60

70

80

Trajectory Step Number

1.5

1.0

0.5

0.0

0.5

1.0

1.5

0.6

0.4

0.2

0.0

0.2

0.4

0.6

FAC

0

20

40

60

80

100

120

Trajectory Step Number

0

2

4

6

8

1e6

1.5

1.4

1.3

1.2

Reacher Cumulative

Rewards

RCRL

FAC

CBF

Vanilla PPO

CRPO

P3O

PCPO

0

2

4

6

8

1e6

2.0

0.0

2.0

4.0

×10 3

HalfCheetah Cumulative

Rewards

0

2

4

6

8

0

5

10

15

Reacher Cumulative

Costs

0

2

4

6

8

0.0

0.2

0.4

0.6

0.8

1.0

×10 2

HalfCheetah Cumulative

Costs

0

2

4

6

8

1e6

10

0

10

20

CarGoal Cumulative

Rewards

RESPO w/ lr*0.1

RESPO

RESPO w/ lr*10

0

2

4

6

8

1e6

5

0

5

10

PointButton Cumulative

Rewards

0

2

4

6

8

1e6

0

20

40

60

80

CarGoal Cumulative

Costs

0

2

4

6

8

1e6

0

20

40

60

80

100

PointButton Cumulative

Costs

1e6

1e6

Drone makes way for the lower Drone to pass through but needs to make a drop to make a concave parabolic trajectory to the goal. Nonetheless, the hard constraints are almost always satisfied, thereby producing the behavior of allowing one drone through the tunnel at a time. The soft constraints are satisfied at the beginning and end but are violated, reasonably, in the middle of the episode since only one drone can pass through the tunnel at a time, thereby forcing the other drone into a standby mode.

6.3 Ablation Studies

We also perform ablation studies to experimentally confirm the design choices we made based on the theoretically established convergence and optimization framework. We particularly investigate the effects of changing the learning rate of our reachability function as well as changing the optimization framework. We present the results of changing the learning rate for REF in Figure 5 while our results for the ablation studies on our optimization framework can be seen in Figure 6.

In Figure 5, we show the effects of making the learning rate of REF slower and faster than the one we use in accordance with Assumption 1 . From these experiments, changing the learning rate in either direction produces poor reward performance. A fast learning rate makes the REF converge to the likelihood of infeasibility for the current policy, which can be suboptimal. But a very slow learning rate means the function takes too long to converge - the lagrange multiplier may meanwhile become very large, thus making it too difficult to optimize for reward returns. In both scenarios, the algorithm with modified learning rates produces conservative behavior that sacrifices reward performance.

In Figure 6, we compare RESPO with RCRL implemented with our REF and PPOLag in the CMDP framework with cost threshold χ = 0 to ensure hard constraint satisfaction. The difference between RESPO and the RCRL-based ablation approach is that the ablation still uses V π h instead of V π c . The ablation aproach's high cumulative cost can be attributed to the limitations of using V π h -particularly, the lower sensitivity of V π h to safety improvement and its lack of guarantees on feasible set (re)entrance. PPOLag with χ = 0 produces low violations but also very low reward performance that's close to zero. Naively using V π c in a hard constraints framework leads to very conservative

0

2

4

6

8

1e6

25

5

15

35

CarGoal Cumulative

Rewards

PPOLag,  =0

RCRL with REF

0

2

4

6

8

1e6

0

10

20

30

PointButton Cumulative

Rewards

0

2

4

6

8

1e6

1

0

1

2

3

×10 3

BallRun Cumulative

Rewards

0

2

4

6

8

1e6

0.0

2.0

4.0

6.0

8.0

×10 2

DroneCircle Cumulative

Rewards

0

2

4

6

8

0

20

40

60

80

CarGoal Cumulative

Costs

0

2

4

6

8

0.0

0.4

0.8

1.2

1.6

×10 2

PointButton Cumulative

Costs

0

2

4

6

8

0.0

1.0

2.0

×10 2

BallRun Cumulative

Costs

0

2

4

6

8

0

10

20

30

DroneCircle Cumulative

Costs

1e6

1e6

1e6

1e6

behavior that sacrifices reward performance. Ultimately, this ablation study experimentally highlights the importance of learning our REF and using value function V π c in our algorithm's design.

7 Discussion and Conclusion

In summary, we proposed a new optimization formulation and a class of algorithms for safetyconstrained reinforcement learning. Our framework optimizes reward performance for states in least-violation policy's feasible state space while maintaining persistent safety as well as providing the safest behavior in other states by ensuring entrance into the feasible set with minimal cumulative discounted costs. Using our proposed reachability estimation function, we prove our algorithm's class of actor-critic methods converge a locally optimal policy for our proposed optimization. We provide extensive experimental results on a diverse suite of environments in Safety Gym, PyBullet, and MuJoCo, and an environment with multiple hard and soft constraints, to demonstrate the effectiveness of our algorithm when compared with several SOTA baselines. We leave open various extensions to our work to enable real-world deployment of our algorithm. These include constraining violations during training, guaranteeing safety in single-lifetime reinforcement learning, and ensuring policies don't forget feasible sets as environment tasks change. Our approach of learning the optimal REF to reduce the state space into a low-dimensional likelihood representation to guide training for high-dimensional policies can have applications in other learning problems in answering binary classification or likelihood-based questions about dynamics in high-dimension feature spaces.

References

Javier García and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research , 16(1):1437-1480, 2015.

Shangding Gu, Long Yang, Yali Du, Guang Chen, Florian Walter, Jun Wang, Yaodong Yang, and Alois Knoll. A review of safe reinforcement learning: Methods, theory and applications. arXiv preprint arXiv:2205.10330 , 2022.

Tsung-Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection-based constrained policy optimization. In International Conference on Learning Representations , 2020.

Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems , 33:15338-15349, 2020.

Long Yang, Jiaming Ji, Juntao Dai, Yu Zhang, Pengfei Li, and Gang Pan. Cup: A conservative update policy algorithm for safe reinforcement learning. arXiv preprint arXiv:2202.07565 , 2022.

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning , pages 22-31. PMLR, 2017.

Chen Tessler, Daniel Mankowitz, and Shie Mannor. Reward constrained policy optimization. In International Conference on Learning Representations , 2019.

Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research , 18(1):6070-6120, 2017.

Haitong Ma, Yang Guan, Shegnbo Eben Li, Xiangteng Zhang, Sifa Zheng, and Jianyu Chen. Feasible actor-critic: Constrained reinforcement learning for ensuring statewise safety. arXiv preprint arXiv:2105.10682 , 2021.

Vivek S Borkar. Stochastic approximation: a dynamical systems viewpoint , volume 48. Springer, 2009.

A. D. Ames, S. Coogan, M. Egerstedt, G. Notomista, K. Sreenath, and P. Tabuada. Control barrier functions: Theory and applications. In European Control Conf. , 2019.

Jaime F Fisac, Mo Chen, Claire J Tomlin, and S Shankar Sastry. Reach-avoid problems with time-varying dynamics, targets and constraints. In Hybrid Systems: Computation and Control . ACM, 2015.

S. Bansal, M. Chen, S. Herbert, and C. J. Tomlin. Hamilton-Jacobi reachability: A brief overview and recent advances. In Conf. on Decision and Control , 2017.

Aaron D Ames, Kevin Galloway, Koushil Sreenath, and Jessy W Grizzle. Rapidly exponentially stabilizing control Lyapunov functions and hybrid zero dynamics. Trans. on Automatic Control , 2014.

Xiangru Xu, Paulo Tabuada, Jessy W. Grizzle, and Aaron D. Ames. Robustness of control barrier functions for safety critical control. Int. Federation of Automatic Control , 2015.

Ya-Chien Chang, Nima Roohi, and Sicun Gao. Neural lyapunov control. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Ya-Chien Chang and Sicun Gao. Stabilizing neural control using self-learned almost lyapunov critics. 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 1803-1809, 2021.

Zhizhen Qin, Tsui-Wei Weng, and Sicun Gao. Quantifying safety of learning-based selfdriving control using almost-barrier functions. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 12903-12910. IEEE, 2022.

Milan Ganai, Chiaki Hirayama, Ya-Chien Chang, and Sicun Gao. Learning stabilization control from observations by learning lyapunov-like proxy models. 2023 IEEE International Conference on Robotics and Automation (ICRA) , 2023.

Yujie Yang, Yuxuan Jiang, Yichen Liu, Jianyu Chen, and Shengbo Eben Li. Model-free safe reinforcement learning through neural barrier certificate. IEEE Robotics and Automation Letters , 2023.

Yixuan Wang, Simon Sinong Zhan, Ruochen Jiao, Zhilu Wang, Wanxin Jin, Zhuoran Yang, Zhaoran Wang, Chao Huang, and Qi Zhu. Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. In International Conference on Machine Learning , pages 36593-36604. PMLR, 2023.

Tamas G Molnar, Ryan K Cosner, Andrew W Singletary, Wyatt Ubellacker, and Aaron D Ames. Model-free safety-critical control for robotic systems. IEEE robotics and automation letters , 7(2):944-951, 2021.

Anayo K Akametalu, Shromona Ghosh, Jaime F Fisac, and Claire J Tomlin. A minimum discounted reward hamilton-jacobi formulation for computing reachable sets. arXiv preprint arXiv:1809.00706 , 2018.

Jaime F Fisac, Neil F Lugovoy, Vicenç Rubies-Royo, Shromona Ghosh, and Claire J Tomlin. Bridging hamilton-jacobi safety analysis and reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA) , pages 8550-8556. IEEE, 2019.

A. Ames, X. Xu, J. W. Grizzle, and P. Tabuada. Control barrier function based quadratic programs for safety critical systems. Trans. on Automatic Control , 2017.

Nuoya Xiong et al. Provably safe reinforcement learning with step-wise violation constraints. arXiv preprint arXiv:2302.06064 , 2023.

Dongjie Yu, Haitong Ma, Shengbo Li, and Jianyu Chen. Reachability constrained reinforcement learning. In International Conference on Machine Learning , pages 25636-25655. PMLR, 2022.

Eitan Altman. Constrained Markov decision processes: stochastic modeling . Routledge, 1999.

Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems , 5:411-444, 2022.

Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708 , 7(1):2, 2019.

Jingliang Duan, Zhengyu Liu, Shengbo Eben Li, Qi Sun, Zhenzhong Jia, and Bo Cheng. Adaptive dynamic programming for nonaffine nonlinear optimal control problem with state constraints. Neurocomputing , 484:128-141, 2022.

Linrui Zhang, Li Shen, Long Yang, Shi-Yong Chen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Penalized proximal policy optimization for safe reinforcement learning. In International Joint Conference on Artificial Intelligence , 2022.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

Tengyu Xu, Yingbin Liang, and Guanghui Lan. Crpo: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning , pages 11480-11491. PMLR, 2021.

Zengyi Qin, Yuxiao Chen, and Chuchu Fan. Density constrained reinforcement learning. In International Conference on Machine Learning , pages 8682-8692. PMLR, 2021.

Mo Chen, Sylvia L Herbert, Mahesh S Vashishtha, Somil Bansal, and Claire J Tomlin. Decomposition of reachable sets and tubes for a class of nonlinear systems. Trans. on Automatic Control , 2018.

Sylvia L Herbert, Somil Bansal, Shromona Ghosh, and Claire J Tomlin. Reachability-based safety guarantees using efficient initializations. In Conf. on Decision and Control , 2019.

T. Lew, L. Janson, R. Bonalli, and M. Pavone. A simple and efficient sampling-based algorithm for general reachability analysis. In Learning for Dynamics & Control Conference , March 2022.

Somil Bansal and Claire J Tomlin. Deepreach: A deep learning approach to high-dimensional reachability. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 1817-1824. IEEE, 2021.

Dongjie Yu, Wenjun Zou, Yujie Yang, Haitong Ma, Shengbo Eben Li, Jingliang Duan, and Jianyu Chen. Safe model-based reinforcement learning with an uncertainty-aware reachability certificate. arXiv preprint arXiv:2210.07553 , 2022.

Niklas Kochdumper, Hanna Krasowski, Xiao Wang, Stanley Bak, and Matthias Althoff. Provably safe reinforcement learning via action projection using reachability analysis and polynomial zonotopes. IEEE Open Journal of Control Systems , 2:79-92, 2023.

Shangding Gu, Guang Chen, Lijun Zhang, Jing Hou, Yingbai Hu, and Alois Knoll. Constrained reinforcement learning for vehicle motion planning with topological reachability analysis. Robotics , 11(4):81, 2022.

Mahmoud Selim, Amr Alanwar, Shreyas Kousik, Grace Gao, Marco Pavone, and Karl H Johansson. Safe reinforcement learning using black-box reachability analysis. IEEE Robotics and Automation Letters , 7(4):10665-10672, 2022.

Alessandro Abate, Maria Prandini, John Lygeros, and Shankar Sastry. Probabilistic reachability and safety for controlled discrete time stochastic hybrid systems. Automatica , 44(11):27242734, 2008.

Sean Summers and John Lygeros. Verification of discrete time stochastic hybrid systems: A stochastic reach-avoid decision problem. Automatica , 46(12):1951-1961, 2010.

Nikolaos Kariotoglou, Kostas Margellos, and John Lygeros. On the computational complexity and generalization properties of multi-stage and stage-wise coupled scenario programs. Systems &Control Letters , 94:63-69, 2016.

Abraham P Vinod and Meeko MK Oishi. Stochastic reachability of a target tube: Theory and computation. Automatica , 125:109458, 2021.

Margaret P Chapman, Riccardo Bonalli, Kevin M Smith, Insoon Yang, Marco Pavone, and Claire J Tomlin. Risk-sensitive safety analysis using conditional value-at-risk. IEEE Transactions on Automatic Control , 67(12):6521-6536, 2021.

Sven Gronauer. Bullet-safety-gym: A framework for constrained reinforcement learning. 2022.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033, 2012.

Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction . A Bradford Book, Cambridge, MA, USA, 2018.

Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov-based safe policy optimization for continuous control. arXiv preprint arXiv:1901.10031 , 2019.

Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society , 48(3):334-334, 1997.

H.K. Khalil. Nonlinear Systems . Pearson Education. Prentice Hall, 2002.

Charles Dawson, Sicun Gao, and Chuchu Fan. Safe control with learned certificates: A survey of neural lyapunov, barrier, and contraction methods for robotics and control. IEEE Transactions on Robotics , 2023.

Jason J Choi, Donggun Lee, Koushil Sreenath, Claire J Tomlin, and Sylvia L Herbert. Robust control barrier-value functions for safety-critical control. In 2021 60th IEEE Conference on Decision and Control (CDC) , pages 6814-6821. IEEE, 2021.

Eric Yang Yu, Zhizhen Qin, Min Kyung Lee, and Sicun Gao. Policy optimization with advantage regularization for long-term fairness in decision systems, 2022.

Ppo lagrangian pytorch. https://github.com/akjayant/PPO_Lagrangian_PyTorch , 2022.

Jiaming Ji, Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. arXiv preprint arXiv:2305.09304 , 2023.

A Notation

r

h

V

π

V

V

✶

c

π

π

h

s

∈S

ϕ

π

ϕ

∗

p

H

H

min

λ

R

S

s

S

v

S

f

′

E

s

E

reward function safety loss function cumulative discounted reward value function cumulative discounted cost value function reachability value function instantaneous violation indicator function reachability estimation function (REF) of a given policy reachability estimation function (REF) of safest policy predicted reachability estimation function (REF) of safest policy upper bound of function h minimum non-zero value of of function h maximum value to clip lagrange multiplier upper bound of function r safe set unsafe set expectation taken over possible trajectories

∼

τ

π,P

∼

π,P

E

feasible set (persistent safe set) expectation taken over possible next states expectation taken over initial distribution

s

∼

d

0

Table 1: Notation used in the paper.

B Gradient estimates

The Q value losses based on the MSE between the Q networks and the respective sampled returns result in the gradients:

Similarly the REF gradient update is:

From the policy gradient theorem in [52], we get the policy gradient loss as:

and the stochastic gradient of the multiplier is

and λ ω is clipped to be in range [0 , λ max ] (in particular, projection operator Γ Ω ( λ ω ) = arg min ˆ λ ω ∈ [0 ,λ max ] || λ ω -ˆ λ ω || 2 ).

C Proofs

C.1 Theorem 1 with Proof

Theorem 3. The REF can be reduced to the following recursive Bellman formulation:

v

where s ′ ∼ π, P ( s ) is a sample of the immediate successive state (i.e., s ′ ∼ P ( ·| s, a ∼ π ( ·| s )) ) and the expectation is taken over all possible successive states.

Proof.

Note that we use the notation τ ∼ π, P ( s ) to indicate a trajectory sampled from the MDP with transition probability P under policy π starting from state s , and use the notation s ′ ∼ π, P ( s ) to indicate the next immediate state from the MDP with transition probability P under policy π starting from state s . The third line holds because the indicator function is either 0 or 1 , so if it's 1 then ϕ π ( s ) = E τ ∼ π,P ( s ) 1 = 1 else ϕ π ( s ) = E τ ∼ π,P ( s ) max s t ∈ τ \{ s } ✶ s π t ∈ S v .

C.2 Proposition 1 with Proof

Proposition 3. The cost value function V π c ( s ) is zero for state s if and only if the persistent safety is guaranteed for that state under the policy π .

Proof. (IF) Assume for a given policy π , the persistent safety is guaranteed, i.e. h ( s t | s 0 = 0 , π ) = 0 holds for all s t ∈ τ for all possible trajectories τ sampled from the environment with control policy π . We then have:

(ONLY IF) Assume for a given policy π , V π c ( s ) = 0 . Since the image of the safety loss function h ( s ) is non-negative real, and V π c ( s ) is the expectation of the sum of non-negative real values, the only way V π c ( s ) = 0 is if h ( s t | s 0 = 0 , π ) = 0 , ∀ s t ∈ τ for all possible trajectories τ sampled from the environment with control policy π .

C.3 Proposition 2 with Proof

Proposition 4. If ∃ π that produces trajectory τ = { ( s i ) , i ∈ ◆ , s 1 = s } in deterministic MDP M starting from state s , and ∃ m ∈ ◆ , m < ∞ such that s m ∈ S π f , then ∃ ϵ > 0 where if discount factor γ ∈ (1 -ϵ, 1) , then the optimal policy π ∗ of Main paper Equation 3 will produce a trajectory τ ′ = { ( s ′ ) , j ∈ ◆ , s ′ = s } , such that ∃ n ∈ ◆ , n < ∞ , s ′ ∈ S π ∗ and V π ∗ ( s ) = min π ′ V π ′ ( s ) .

j 1 n f c c

In other words the proposition is stating for some state s , if there is a policy that enters its feasible set in a finite number ( m -1 ) of steps, then by ensuring discount factor γ is close to 1 we can guarantee that the optimal policy π ∗ of Main paper Equation 3 will also enter the feasible set in a finite number of steps with the minimum cumulative discounted sum of the costs. Note that π ∗ will always produce trajectories with the minimum discounted sum of costs whether the state is in the feasible or infeasible set of the policy by virtue of its optimization which constrains V π c .

Proof. We consider two cases: (Case 1) m = 1 and (Case 2) m> 1 .

Case 1 m = 1 : In this case, there exists a policy π in which the the current state s is in the feasible set of that policy. By definition, that means that in a trajectory τ sampled in the MDP using that

policy, starting from state s , there are no future violations incurred in τ . Thus V π c ( s ) = 0 . Since π ∗ incurs the minimum cumulative violation, V π ∗ c ( s ) = 0 trivially. Therefore, s , the first state of the trajectory, is in the feasible set of π ∗ .

Case 2 m> 1 : Since policy π ∗ produces the minimum cumulative discounted cost for a given state s , the core of this proof will be demonstrating that the minimum cumulative discounted cost of entering the feasible set (call this value H E ) is less than the minimum cumulative discounted cost of not entering the feasible set (call this value H N ), and therefore π ∗ will choose the route of entering the feasible set.

The proof will proceed by deriving a sufficient condition for H E < H N by establishing bounds on them.

We place an upper bound on the minimum cumulative discounted cost of entering the feasible set H E . Since ∃ π that enters the feasible set in m -1 steps, entering the feasible set can be at most the highest possible cost that π incurs. Since the maximum cost at any state is H max , the upper bound is the discounted sum of m -1 steps of violations H max , or

We place a lower bound on the minimum cumulative discounted cost of not entering the feasible set H N . In this case, say in the sampled trajectory, the maximum gap between any two non-zero violations is w . By definition, the trajectory cannot have an infinite sequence of violation-free states since the trajectory never enters the feasible set. Therefore w is finite. Now recall H min is the lower bound on the non-zero values of h . So the minimum cumulative discounted cost of not entering the feasible set must be at least the cost of the trajectory with a violation of H min at intervals of w steps. That is:

Now H E < H N will be true if the upper bound of H E is less than the lower bound of H N . In other words H E < H N is true if:

Let's define the RHS of the Inequality 6 as the function υ ( γ ) . Consider γ ∈ (0 , 1) . It is not difficult to demonstrate that υ ( γ ) in this domain range is a continuous function and that left directional limit lim γ → 1 -υ ( γ ) = ∞ . This suggests that there is an open interval of values for γ (whose supremum is 1 ) for which H max /H min < υ ( γ ) and so H E < H N . So we establish that ∃ ϵ > 0 such that for γ ∈ (1 -ϵ, 1) , we satisfy the sufficient condition H E < H N so that the optimal policy will enter its feasible set.

Thus, we prove that if there is a policy entering its feasible set from state s , then there is a range of values for γ that are close enough to 1 ensuring that the optimal policy of Main paper Equation 3 will enter its feasible set in a finite number of steps with minimum discounted sum of costs.

C.4 Theorem 2 with Proof

Theorem 4. Given Assumptions A1 -A3 in Main paper, the policy updates in Algorithm 1 will almost surely converge to a locally optimal policy for our proposed optimization in Equation RESPO.

We first provide an intuitive explanation behind why our REF learns to converge to the safest policy's REF, then a proof overview, and then the full proof.

Rearranging, we get:

W

戈

Z

Spl

Shl

S1

C.4.1 Intuition behind REF convergence

The approach can be explained by considering what happens in the individual regions of space. Consider a deterministic environment for simplicity. As seen in Figure 7, there are two subsets of the initial state space: a safest policy's "true" feasible set S hI and REF predicted feasible set S pI , and they create 4 regions in the initial state space S I : W = S hI ∩ S pI , X = S hI ∩ S pI , Y = S hI ∩ S pI , Z = S hI ∩ S pI . Consider a point during training when the lagrange multiplier λ is sufficiently large. For states in W , the set of correctly classified infeasible states, the algorithm will simply minimize cumulative violations V π θ c ( s ) , and thereby remain as safe as possible since the policy and critics learning rates are faster than that of REF. X , which is the set of infeasible states that are misclassified, is very small if we ensure the policy and REF are trained at much faster time scales than the multiplier and so when the agent starts in true infeasible states, it will by definition reach violations and therefore be labeled as infeasible. In Y , the set of truly feasible states that are misclassified, the algorithm also minimizes cumulative violations, which by the definition of feasibility should be 0 . It will then have no violations and enter the correctly predicted feasible set Z . And when starting in states in Z , the algorithm will optimize the lagrangian, and since the multiplier λ is sufficiently large, it will converge to a policy that optimizes for reward while ensuring safety, i.e. no future violations, and therefore the state will stay predictably feasible in Z . In this manner, REF's predicted feasible set will converge to the optimal feasible set, and the agent will be safe and have optimal performance in the feasible set and be the safest behavior outside the feasible set. Thereby, the algorithm finds a locally optimal solution to the proposed optimization formulation.

C.4.2 Proof Overview

We show our algorithm convergence to the optimal policy by utilizing the proof framework of multi-time scale presented in [8, 10, 53, 27]. Specifically, we have 4 time scales for (1) the critics, (2) policy, (3) REF function, and (4) lagrange multiplier, listed in order from fastest to slowest. The overview of each timescale proof step is as follows:

1 We demonstrate the almost sure convergence of the critics to the corresponding fixed point optimal critic functions of the policy.

2 Using multi-timescale theory, we demonstrate the policy almost surely converges to a stationary point of a continuous time system, which we show has a Lyapunov function certifying its locally asymptotic stability at the stationary point.

3 We demonstrate the almost sure convergence of the REF function to the REF of the policy that is safe insofar as the lagrange multiplier is sufficiently large.

4 We demonstrate the almost sure convergence of the lagrange multiplier to a stationary point similar to the proof in the policy timecale.

Finally, we demonstrate that the stationary points for the policy and lagrange multiplier form a saddle point, and so by local saddle point theorem we almost surely achieve the locally optimal policy of our proposed optimization.

C.4.3 Proof Details

Proof. Step 1 (convergence of the critics V η and V κ updates): From the multi-time scale assumption, we know that η and κ will convergence on a faster time scale than the other parameters θ , ξ , and ω . Therefore, we can leverage Lemma 1 of Chapter 6 of [10] to analyze the convergence properties while updating η k and κ k by treating θ , ξ , and ω as fixed parameters θ k , ξ k , and ω k . In other words, the policy, REF, and lagrange multiplier are fixed while computing Q π θ k ( s, a ) and Q π θ k c ( s, a ) . With the Finite MDP assumption and policy evaluation convergence results of [52], and assuming sufficiently expressive function approximator (i.e. wide enough neural networks) to ensure convergence to global mininum, we can use the fact that the bellman operators B and B c which are defined as

are γ -contraction mappings, and therefore as k approaches ∞ , we can be sure that Q ( s, a ; η k ) → Q ( s, a ; η ∗ ) = Q π θ k ( s, a ) and Q c ( s, a ; κ k ) → Q c ( s, a ; κ ∗ ) = Q π θ k c ( s, a ) . So since η k and κ k converge to η ∗ and κ ∗ , we prove convergence of the critics in Time scale 1.

Step 2 (convergence of the policy π θ update): Because ξ and ω updated on slower time scales than θ , we can again use Lemma 1 of Chapter 6 of [10] and treat these parameters are fixed at ξ k and ω k respectively when updating θ k . Additionally in Time scale 2, we have || Q ( s, a ; η k ) -Q ( s, a ; η ∗ ) ||→ 0 and || Q c ( s, a ; κ k ) -Q c ( s, a ; κ ∗ ) ||→ 0 almost surely. Now the update of the policy θ using the gradient from Equation 4 is:

where and

Lemma 1 : We can first demonstrate that δθ k +1 is square integrable. In particular,

Note that F θ,k = σ ( θ m , δθ m , m ≤ k ) is the filtration for θ k generated by different independent trajectories [8]. Also note that the indicator function is used because the expectation of || δθ k +1 || 2 is taken with respect to P π θ k and P π θ k ( s, a | s 0 ) = 0 if π ( a | s ; θ k ) = 0 . From the Assumptions on Lipschitz continuity and Finite MDPs reward and costs, we can bound the values of the functions and the gradients of functions. Specifically

where K 1 is a Lipschitz constant. Furthermore, note that because we are sampling, π ( a | s ; θ k ) will take on only a finite number of values, so its nonzero values will be bounded away from zero. Thus we can say

for some large enough K 2 . Thus using the bounds from these conditions, we can demonstrate

Therefore δθ k +1 is square integrable.

Lemma 2 : Secondly, we can demonstrate δθ ϵ → 0 .

And because we have || Q ( s, a ; η k ) -Q ( s, a ; η ∗ ) || → 0 and || Q c ( s, a ; κ k ) -Q c ( s, a ; κ ∗ ) || → 0 almost surely, we can therefore say δθ ϵ → 0 .

Lemma 3 : Finally, since ˆ ∇ θ J π ( θ ) | θ = θ k is a sample of ∇ θ L ( θ, ξ k , ω k ) | θ = θ k based on the history of sampled trajectories, we conclude that E [ δθ k +1 |F θ,k ] = 0 .

From the 3 above lemmas, the policy θ update is a stochastic approximation of a continuous system θ ( t ) defined by [10]

or in other words the left directional derivative of Γ Θ ( θ ) in the direction of M ( θ ) . Using the left directional derivative Υ Θ [ -∇ θ L ( θ, ξ, ω )] in the gradient descent algorithm for learning the policy π θ ensures the gradient will point in the descent direction along the boundary of Θ when the θ update hits its boundary. Using Step 2 in Appendix A.2 from [8], we have that dL ( θ, ξ, ω ) /dt = -∇ θ L ( θ, ξ, ω ) T · Υ Θ [ -∇ θ L ( θ, ξ, ω )] ≤ 0 and the value is non-zero if || Υ Θ [ -∇ θ L ( θ, ξ, ω )] || = 0 . Now consider the continuous system θ ( t ) . For some fixed ξ and ω , define a Lyapunov function

where θ ∗ is a local minimum point. Then there exists a ball centered at θ ∗ with a radius ρ such that ∀ θ ∈ B θ ∗ ( ρ ) = { θ ||| θ -θ ∗ || ≤ ρ } , L ξ,ω ( θ ) is a locally positive definite function, that is L ξ,ω ( θ ) ≥ 0 . Using Proposition 1.1.1 from [54], we can show that Υ Θ [ -∇ θ L ( θ, ξ, ω )] | θ = θ ∗ = 0 meaning θ ∗ is a stationary point. Since dL ( θ, ξ, ω ) /dt ≤ 0 , through Lyapunov theory for asymptotically stable systems presented in Chapter 4 of [55], we can use the above arguments to demonstrate that with any initial conditions of θ (0) ∈ B θ ∗ ( ρ ) , the continuous state trajectory of θ ( t ) converges to θ ∗ . Particularly, L ( θ ∗ , ξ, ω ) ≤ L ( θ ( t ) , ξ, ω ) ≤ L ( θ (0) , ξ, ω ) for all t > 0 .

Using these aforementioned properties, as well as the facts that 1) ∇ θ L ( θ, ξ, ω ) is a Lipschitz function (using Proposition 17 from [8]), 2) the step-sizes of Assumption on steps sizes, 3) δθ k +1 is a square integrable Martingale difference sequence and δθ ϵ is a vanishing error almost surely, and 4) θ k ∈ Θ , ∀ k implying that sup k || θ k || < ∞ almost surely, we can invoke Theorem 2 of chapter 6 in [10] to demonstrate the sequence { θ k } , θ k ∈ Θ converges almost surely to the solution of the ODE defined by Equation 7, which additionally converges almost surely to the local minimum θ ∗ ∈ Θ .

Step 3 (convergence of REF p ξ updates): Since ω is updated on a slower time scale that ξ , we can again treat ω as a fixed parameter at ω k when updating ξ . Furthermore, in Time scale 3 , we know that the policy has converged to a local minimum, particularly || θ k -θ ∗ ( ξ k , ω k ) || = 0 . Now the bellman operator for REF is defined by

We demonstrate this is a γ contraction mapping as follows:

So we can say that p ( s ; ξ k ) will converge to p ( s ; ξ ∗ ) as k → ∞ under the same assumptions of the Finite MDP and function approximator expressiveness in Step 1. Therefore, π θ k will also converge to π ⋄ = π θ ∗ ( ξ ∗ ,ω k ) as k →∞ . And because π θ is the sampling policy used to compute p , p ( s ; ξ ∗ ) = p π θ ∗ ( ξ ∗ ,ω k ) ( s ; ξ ∗ ) = p ⋄ ( s ) .

in which

̸

Notice that π ⋄ is a locally minimum optimal policy for the following optimization (recall λ ω is treated as constant in this timescale):

and therefore also locally minimum optimal policy for optimization:

Since p ⋄ ( s ) (1 -p ⋄ ( s )) ≥ 0 , and the Q functions are always nonnegative, we can know that π ⋄ is at least as safe as (i.e., its expected cumulative cost is at most that of) a locally optimal policy for the optimization:

As λ ω approaches λ max , which in turn approaches ∞ , the local minimum optimal policies of Equation 8 approach those of the optimization π △ = arg min π E s ∼ d 0 E a ∼ π ( ·| s ) Q π c ( s, a ) λ ω = arg min π E s ∼ d 0 E a ∼ π ( ·| s ) Q π c ( s, a ) . Therefore, the feasible set of the REF p ⋄ will approach that of the REF p π △ .

Step 4 (convergence of lagrange multiplier λ ω update): Since λ ω is on the slowest time scale, we have that || θ k -θ ∗ ( ω ) || = 0 , || ξ k -ξ ∗ ( ω ) || = 0 , and || Q c ( s, a ; κ k ) -Q π θ k c ( s, a ) || = 0 almost surely. Furthermore, due to the continuity of ∇ ω L ( θ, ξ, ω ) , we have that ||∇ ω L ( θ, ξ, ω ) | θ = θ k ,ξ = ξ k ,ω = ω k -∇ ω L ( θ, ξ, ω ) | θ = θ ∗ ( ω k ) ,ξ = ξ ∗ ( ω k ) ,ω = ω k || = 0 almost surely. The update of the multiplier using the gradient for Equation is:

where

Now, just as in the θ update convergence, we can demonstrate the following lemmas:

Lemma 4 : δω k +1 is square integrable since

for some large Lipschitz constant K 3 . Note that F ω,k = σ ( ω m , δω m , m ≤ k ) is the filtration for ω k generated by different independent trajectories [8].

Thus, the lagrange multiplier ω update is a stochastic approximation of a continuous system ω ( t ) defined by [10]

̸

with Martingale difference error of δω k and where Υ Ω is the left direction derivative defined similar to that in Time scale 2 of the convergence of θ update. Using Step 2 in Appendix A.2 from [8], we have that dL ( θ ∗ ( ω ) , ξ ∗ ( ω ) , ω ) /dt = ∇ ω L ( θ, ξ, ω ) | θ = θ ∗ ( ω ) ,ξ = ξ ∗ ( ω ) T · Υ Ω [ ∇ Ω L ( θ, ξ, ω ) | θ = θ ∗ ( ω ) ,ξ = ξ ∗ ( ω ) ] ≥ 0 and the value is non-zero if || Υ Ω [ ∇ ω L ( θ, ξ, ω ) | θ = θ ∗ ( ω ) ,ξ = ξ ∗ ( ω ) ] || = 0 .

For a local maximum point ω ∗ , define a Lyapunov function as

Then there exists a ball centered at ω ∗ with a radius ρ ′ such that ∀ ω ∈ B ω ∗ ( ρ ′ ) = { ω ||| ω -ω ∗ || ≤ ρ ′ } , L ( ω ) is a locally positive definite function, that is L ( ω ) ≥ 0 . Also, d L ( ω ( t )) /dt = -dL ( θ ∗ ( ω ) , ξ ∗ ( ω ) , ω ) /dt ≤ 0 and is equal only when Υ Ω [ ∇ ω L ( θ, ξ, ω ) | θ = θ ∗ ( ω ) ,ξ = ξ ∗ ( ω ) ] = 0 , so therefore ω ∗ is a stationary point. By leveraging Lyapunov theory for asymptotically stable systems presented in Chapter 4 of [55] we can demonstrate that for any initial conditions of ω ∈ B ω ∗ ( ρ ′ ) , the continuous state trajectory of ω ( t ) converges to the locally maximum point ω ∗ .

Using these aforementioned properties, as well as the facts that 1) ∇ ω L ( θ ∗ ( ω ) , ξ ∗ ( ω ) , ω ) is a Lipschitz function, 2) the step-sizes of Assumption on steps sizes, 3) { ω k +1 } is a stochastic approximation of ω ( t ) with a Martingale difference error, and 4) convex and compact properties in projections used, we can use Theorem 2 of chapter 6 in [10] to demonstrate the sequence { ω k } converges almost surely to a locally maximum point ω ∗ almost surely, that is L ( θ ∗ ( ω ) , ξ ∗ ( ω ) , ω ∗ ) ≥ L ( θ ∗ ( ω ) , ξ ∗ ( ω ) , ω ) .

From Time scales 2 and 3 we have that L ( θ ∗ ( ω ) , ξ ∗ ( ω ) , ω ) ≤ L ( θ, ξ, ω ) while from Time scale 4 we have that L ( θ ∗ ( ω ) , ξ ∗ ( ω ) , ω ∗ ) ≥ L ( θ ∗ ( ω ) , ξ ∗ ( ω ) , ω ) . Thus, L ( θ ∗ ( ω ) , ξ ∗ ( ω ) , ω ) ≤ L ( θ ∗ ( ω ) , ξ ∗ ( ω ) , ω ∗ ) ≤ L ( θ, ξ, ω ∗ ) . Therefore, ( θ ∗ , ξ ∗ , ω ∗ ) is a local saddle point of ( θ, ξ, ω ) . Invoking the saddle point theorem of Proposition 5.1.6 in [54], we can conclude that π ( ·|· ; θ ∗ ) is a locally optimal policy for our proposed optimization formulation.

C.4.4 Remark on Bounding Lagrange Multiplier

We can say our algorithm learns an REF that is closer to an optimally safe policy's REF as we take λ max →∞ . Nonetheless, we want to put a bound on the λ max . This λ max must be large enough so that choosing a policy that can reduce the expected cost returns by some non-zero amount is prioritized over increasing the reward returns. So any change in the reward critic terms must be less than any change in the cost critic term. If H ∆ is the minimum non-zero difference between any two cost values, and P min is the minimum sampled non-zero likelihood of reaching a particular state and a point in the sample trajectory, then we can bound the maximum change in the reward returns and the maximum change on the weighted cost returns:

So we can find the bound for λ max :

The second line holds since we are simply rearranging the comparative weightages of the reward and cost returns. Now -ϕ ( s ) (1 -ϕ ( s )) ≤ 0 (recall that if ϕ ( s ) = 1 , then λ is irrelevant in the lagrangian optimization). Thus, if λ > R max (1 -γ ) · γ T · H ∆ · P min then minimizing the cost returns is prioritized over maximizing reward returns.

D Complete Experiment Details and Analysis

D.1 Baselines

We compare our algorithm RESPO with 7 other safety RL baselines, which can be divided to CMDP class and hard constraints class, and unconstrained Vanilla PPO for reference.

CMDP Approaches

Proximal Policy Optimization-Lagrangian. PPOLag is a primal-dual method using Proximal Policy Optimization [33] based off of the implementation found in [7]. The lagrange multiplier is a scalar learnable parameter.

Constraint-Rectified Policy Optimization. CRPO [35] is a primal approach that switches between optimizing for rewards and minimizing constraint violations depending on whether the constraints are violated.

Penalized Proximal Policy Optimization. P3O [32] is another primal approach based on applying the technique of clipping the surrogate objectives found in PPO [33] to CMDPs.

Projection-Based Constrained Policy Optimization. PCPO [3] is a trust-region approach that takes a step in policy parameter space toward optimizing for reward and then projects this policy to the constraint set satisfying the CMDP expected cost constraints.

Hard Constraints Approaches

Reachability Constrained Reinforcement Learning. RCRL [27] is a primal-dual approach where the constraint is on the reachability value function and the lagrange multiplier is represented by a neural network parameterized by state.

Control Barrier Function. This CBF -based approach is inspired by the various energy-based certification approaches [56, 17, 18, 11, 57, 58]. This is implemented as a primal-dual approach where the control barrier-based constraint ˙ h ( s ) + ν · h ( s ) ≤ 0 is to ensure stabilization toward the safe set.

Feasible Actor Critic. FAC [9] is another primal-dual approach similar to RCRL (i.e. it uses the NN representation of the lagrange multiplier parameterized by state) except that the constraint in FAC is based on the cumulative discount sum of costs in lieu of the reachability value function. It is important to note that FAC is originally meant for the CMDP framework (with some positive cost threshold), but we adapt it to hard constraints by making the cost threshold χ = 0 . We do this to make a better comparison between an algorithm that relies on using the lagrange multiplier represented as a NN to learn feasibility with our approach of using our proposed REF function to learn the feasibility likelihood - both approaches enforce a hard constraint on the cumulative discounted costs.

D.2 Benchmarks

We compare the algorithms on a diverse suite of environments including those from the Safety Gym, PyBullet, and MuJoCo suites and a multi-constraint, multi-drone environment.

Safety Gym. In Safety Gym [30], we examine CarGoal and PointButton which have 72 D and 76 D observation spaces that include lidar, accelerometer, gyro, magnetometer, velocimeter, joint angles, and joint velocities sensors. In the CarGoal environment, the car agent has a 72 D observation space and is supposed to reach a goal region while avoiding both hazardous spaces and contact with fragile objects. In PointButton, the point agent has a 76 D observation space and must press a series of specified goal buttons while avoiding 1) quickly moving objects, 2) hazardous spaces, 3) hitting the wrong buttons.

Safety PyBullet. In Safety PyBullet [50], we evaluate in BallRun and DroneCircle environments. In the BallRun environment, the ball agent must move as fast as possible under the constraint of a speed limit, and it must be within some boundaries. In DroneCircle, the agent is based on the AscTec Hummingbird quadrotor and is rewarded for moving clockwise in a circle of a fixed radius with the constraint of remaining within a safety zone demarcated by two boundaries. Note that we use this environment to evaluate our algorithm and the baselines in a stochastic setting. We ensure the MDP is stochastic by adding a 5% gaussian noise to the transitions per step.

Safety MuJoCo. Furthermore, we compare the algorithms in with complex dynamics in MuJoCo. Specifically, we look at HalfCheetah and Reacher safety problems. In Safety HalfCheetah, the agent must move as quickly as possible in the forward direction without moving left of x = -3 . However, unlike the standard HalfCheetah environment, the reward is based on the absolute value of the distance traveled. In this paradigm, it is easier for the agent to learn to quickly run backward rather than forward without any directional constraints. In the Safety Reacher environment, the robotic arm must reach a certain point while avoiding an unsafe region.

Multi-Drone environment. We also compare in an environment with multiple hard and soft constraints. The environment requires controlling two drones to pass through a tunnel one at a time while respecting certain distance requirements. The reward is given for quickly reaching the goal positions. The two hard constraints involve (H1) ensuring neither drone collides into the wall and (H2) the distance between the two drones is more than 0.5 to ensure they do not collide. The soft constraint is that the two drones are within 0.8 of each other to ensure real-world communication. It is preferable to prioritize hard constraint H1 over hard constraint H2 , since colliding with the wall may have more serious consequences to the drones rather than violations of an overly precautious distance constraint - as we will show, our algorithm RESPO can perform this prioritization in its optimization.

D.3 Hyperparameters/Other Details

To ensure a fair comparison, the primal-dual based approaches and unconstrained Vanilla PPO were implemented based off of the same code base [59]. The other three approaches were implemented based on [60] with the similar corresponding hyperparameters as the primal-dual approaches. We run our experiments on Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz with 6 cores. For Safety Gym, PyBullet, MuJoCo, and the multi-drone environments, each algorithm, per seed, per environment, takes ∼ 4 hours to train.

D.4 Double Integrator

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

10.0

7.5

5.0

2.5

0.0

2.5

5.0

7.5

10.0

Start

Feasible Set

Infeasible Set

RCRL

Safe set boundry

We use the Double Integrator environment as a motivating example to demonstrate how performing constrained optimization using solely reachability-based value functions as in RCRL can produce nonoptimal behavior when the agent is outside the feasiblity set. Double Integrator has a 2 dimensional observation space [ x 1 , x 2 ] , 1 dimension action space a ∈ [ -0 . 5 , 0 . 5] , system dynamics is ˙ s = [ x 2 , a ] , and constraint as || s || ∞ ≤ 5 . Particularly, we make the cost as 1 if || s || ∞ > 5 , and 0 otherwise to emphasize the importance of capturing the frequency of violation during training.

Wetrain an RCRL controller and RESPO controller in this environment, and the results are visualized in Figure 8. The color scheme indicates the learned reachability value across the state space while the black line demarcates the border of the zero level set. We present the behavior of the trajectories of RCRL and RESPO . Because the RCRL optimizes for reachability value function when outside the feasible set, it simply minimizes the maximum violation, which as can be seen does not result in the agent reentering the feasible set since it is uniformly equal to or near 1 in the infeasible set. This is since it permits many violations of magnitude same or less than that of the maximum violation. On the other hand, RESPO optimizes for cumulative damage by considering total sum of costs, thereby re-entering the feasible set.

D.5 Safety Gym Environments

0

2

4

6

8

1e6

10

0

10

20

30

CarGoal Cumulative

Rewards

RCRL

FAC

CBF

Vanilla PPO

CRPO

P3O

PCPO

0

2

4

6

8

1e6

0

20

40

60

CarGoal Cumulative

Costs

0

2

4

6

8

0

10

20

PointButton Cumulative

Rewards

0

2

4

6

8

0.0

0.5

1.0

1.5

×10 2

PointButton Cumulative

Costs

1e6

1e6

RESPO : In the CarGoal Environment, our approach achieves the best performance among the safe RL algorithms while being within the acceptable range of cost violations. It is important to note that our algorithm has no access to information on the cost threshold. In PointButton, RESPO achieves very high reward performance among the safety baselines while maintaining among the lowest violations.

PPOLag : In both Safety Gym environments, PPOLag maintains relatively high performance, albeit less than our approach. Nonetheless, it always converges to the cost threshold amount of violations for the respective environments.

RCRL : RCRL has either high reward and high violations or low reward and low violations. It learns a very conservative behavior in CarGoal environment where the violations go down but the reward performance can also be seen to be sacrificed during training. For PointButton, RCRL achieves slightly higher reward performance but has over 3 × the number of violations as RESPO .

FAC : Using a NN to represent the lagrange multiplier in order to capture the feasible sets seems to produce very conservative behavior that sacrifices performance. In both CarGoal and PointButton the reward performance and cost violations are very low. This can be explained because the average observed lagrange multiplier across the states quickly grows, even becoming 9 × that of scalar learnable lagrange multiplier in RESPO .

CBF : The CBF approach has low reward performance in both the Safety Gym benchmarks and its cost violations are quite high.

CRPO , P3O ,& PCPO : These CMDP-based primal approaches have mediocre reward performance but, with the exception of CRPO, achieve violations within the cost threshold. CRPO, however, has high cost violations in both CarGoal and PointButton.

Vanilla PPO: This unconstrained algorithm consistently has high rewards and high costs, so maximizing rewards does not improve costs in these environments.

D.6 Safety PyBullet Environments

0

2

4

6

8

1e6

0.0

0.2

0.4

0.6

0.8

1.0

×10 3

DroneCircle Cumulative

Rewards

RCRL

FAC

CBF

Vanilla PPO

CRPO

P3O

PCPO

0

2

4

6

8

1e6

0

10

20

30

40

DroneCircle Cumulative

Costs

0

2

4

6

8

0.0

1.0

2.0

3.0

×10 3

BallRun Cumulative

Rewards

0

2

4

6

8

0.0

0.5

1.0

1.5

2.0

2.5

×10 2

BallRun Cumulative

Costs

1e6

1e6

RESPO : In the both BallRun and DroneCircle, our approach achieves the highest or among the highest reward performance compared to the other safety baselines. Furthermore, RESPO converges to almost 0 constraint violations for both environments.

PPOLag : In DroneCircle, PPOLag has the best reward performance. Although its costs violations are around the cost threshold, it is much higher than RESPO . On the other hand, in BallRun, PPOLag has very low reward performance.

RCRL : We again see RCRL take on behavior with extremes - it has low reward and cost violations in Drone Circle and has high reward and cost violations in BallRun. Constraining the maximum violation with the reachability value function as RCRL does seems to provide poor safety in an environment with non-tangible constraints (i.e. a speed limit in BallRun).

FAC : While in BallRun, we see FAC have the same low reward and low violations behavior, DroneCircle shows an instance where FAC can achieve decently high rewards while maintaining low violations.

CBF : In DroneCircle, the CBF approach has low rewards and relatively low violations; in BallRun, it has a bit higher rewards compared to all the low performance algorithms with very low violations.

CRPO , P3O ,& PCPO : These CMDP-based primal approaches have mediocre reward performance in DroneCircle and very low performance in BallRun. Nonetheless they achieve violations within the cost threshold.

Vanilla PPO: This unconstrained algorithm consistently has high rewards and high costs (sometimes out of the scope of the plots), so maximizing rewards does not improve costs in these environments.

D.7 Safety MuJoCo Environments

0

2

4

6

8

1e6

1.5

1.4

1.3

1.2

Reacher Cumulative

Rewards

RCRL

FAC

CBF

Vanilla PPO

CRPO

P3O

PCPO

0

2

4

6

8

1e6

0

5

10

15

Reacher Cumulative

Costs

0

2

4

6

8

1e6

2.0

0.0

2.0

4.0

×10 3

HalfCheetah Cumulative

Rewards

0

2

4

6

8

1e6

0.0

0.2

0.4

0.6

0.8

1.0

×10 2

HalfCheetah Cumulative

Costs

Note on HalfCheetah : The rewards for HalfCheetah are based on the absolute distance traveled in each step. Without the cost metric to constrain backward travel, it is easy to learn to run backward, as is the behavior learned in unconstrained PPO.

RESPO : Our approach achieves the highest reward performance among the safety baselines in HalfCheetah and decent reward performance in Reacher. Interestingly, RESPO also has 0 constraint violations in HalfCheetah and the second lowest constraint violations in Reacher.

PPOLag : The performance in Reacher for PPOLag is similar as in most of the previous environments: decently high rewrad, cost near the threshold. However, for HalfCheetah, interesting PPOLag learns to maintain the violations well below the cost threshold.

RCRL : We see yet again RCRL has high reward follow by very high constraint violations.

FAC : This approach has decent reward performance in Reacher and low reward performance in HalfCheetah. However, interestingly, FAC has high cost violations though below the cost threshold.

CBF : In Reacher, the CBF approach has conservative behavior with both low reward and low cost violations. But in HalfCheetah, it has very low reward performance and very high cost violations (not seen in the plot since its an order of magnitude larger than the visible range).

CRPO , P3O ,& PCPO : These CMDP-based primal approaches have decent reward performance in Reacher while maintaining violations within cost threshold. In HalfCheetah, however, they achieve low performance and low cost violations.

Vanilla PPO: This unconstrained algorithm consistently has high rewards and high costs (sometimes out of the scope of the plots), so maximizing rewards does not improve costs in these environments.

D.8 Hard and Soft Constraints

1.5

1.0

0.5

0.0

0.5

1.0

1.5

0.6

0.4

0.2

0.0

0.2

0.4

0.6

RESPO

(ours)

0

10

20

30

40

50

60

70

Trajectory Step Number

1.5

1.0

0.5

0.0

0.5

1.0

1.5

0.6

0.4

0.2

0.0

0.2

0.4

0.6

0

20

40

60

80

Trajectory Step Number

1.5

1.0

0.5

0.0

0.5

1.0

1.5

0.6

0.4

0.2

0.0

0.2

0.4

0.6

RCRL

0

10

20

30

40

50

60

70

80

Trajectory Step Number

1.5

1.0

0.5

0.0

0.5

1.0

1.5

0.6

0.4

0.2

0.0

0.2

0.4

0.6

FAC

0

20

40

60

80

100

120

Trajectory Step Number

RESPO : We manage multiple hard and soft Constraints by extending our framework to optimize the following Lagrangian:

The subscripts hc 1 indicates the first hard constraint (i.e. wall avoidance), hc 2 indicates the second hard constraint (i.e. drone cannot be too close), and sc indicates soft constraint (i.e. drone cannot be too far) - they are all based on discounted sum of costs. Recall V π ( s ) is reward returns. We color coded the corresponding parts of the optimization. Notice how we learn a different REF for each hard constraint. Also notice that the feasible set of the first hard constraint p hc 1 is placed in a manner so as to ensure prioritization of the first hard constraint. As can be seen in the top left plot of Figure 12, our approach successfully reaches the goals and avoids the walls. To enable mobility of the top drone to pass through the tunnel with wall collision, the second hard constraint is violated temporarily in the blue to cyan time period. Furthermore to allow the bottom drone to pass through the tunnel, the soft constraint is violated during the green to orange time period. Nonetheless, RESPO successfully manages the constraints and reward performance via Equation 10 optimization.

PPOLag, RCRL, FAC : The optimization formulation for these approaches is as follows:

For PPOLag and FAC , all the constraint value functions are discount sum of cost. For RCRL , V π sc ( s ) is based on discount sum of costs but V π hc 1 ( s ) and V π hc 2 ( s ) are based on the reachability value function. Furthermore, in PPOLag , all the lagrange multipliers are learnable scalar parameters. In FAC and RCRL all the lagrange multipliers are NN representations parameterized by state. These formulations are not able to provide a framework for the prioritization of the constraint satisfaction all the constraints are treated the same, weighted only on the learned lagrange multipliers. As can be seen in the other three images in Figure 12, the algorithms cannot manage the multiple constraints, and invariably collide with the wall.

D.9 Ablation - Learning Rate

0

2

4

6

8

1e6

10

0

10

20

CarGoal Cumulative

Rewards

RESPO w/ lr*0.1

RESPO

RESPO w/ lr*10

0

2

4

6

8

1e6

0

20

40

60

80

CarGoal Cumulative

Costs

0

2

4

6

8

1e6

5

0

5

10

PointButton Cumulative

Rewards

0

2

4

6

8

1e6

0

20

40

60

80

100

PointButton Cumulative

Costs

We performance Ablation study of varying the learning rate of the REF function to verify the importance of the multi-timescale assumption. Particular, we compare our algorithm's approach of placing the learning rate of the REF between the policy and lagrange multiplier with making the REF's learning rate in various orders of magnitudes slower and faster. Our approach with the learning rate satisfying the multi-timescale assumption experimentally appears to still have the best balance of reward optimization and constraint satisfaction. Particularly when we change the learning rate by one order of magnitude (i.e. × 10 or × 0 . 1 ), we see the reward performance reduce by around half and while the cost violations generally don't change. But when we change the learning rates by another order of magnitude, there reward performance effective becomes zero and the cost violations generally reduce further. By increasing the learning rate of the REF function, we can no longer guarantee that the REF convergences to near the optimally safe REF value. Instead, it becomes the REF of the policy in question. So instead, the optimization can learn to 'hack" the REF function to obtain a policy (and lagrange multiplier) that is not a local optimal for the optimization formulation. On the other hand, when the learning rate is too slow, the lagrange multiplier quickly explodes, thereby creating a very conservative solution - notice the similarity of the orange line in Figure 13 with learning rate 0 . 01 times that of standard in training behavior with PPOLag where χ = 0 in the ablation study on optimization.

D.10 Ablation - Optimization

0

2

4

6

8

1e6

25

5

15

35

CarGoal Cumulative

Rewards

PPOLag,  =0

RCRL with REF

0

2

4

6

8

1e6

0

20

40

60

80

CarGoal Cumulative

Costs

0

2

4

6

8

1e6

0

10

20

30

PointButton Cumulative

Rewards

0

2

4

6

8

1e6

0.0

0.4

0.8

1.2

1.6

×10 2

PointButton Cumulative

Costs

0

2

4

6

8

1e6

1

0

1

2

3

×10 3

BallRun Cumulative

Rewards

0

2

4

6

8

1e6

0.0

1.0

2.0

×10 2

BallRun Cumulative

Costs

0

2

4

6

8

1e6

0.0

2.0

4.0

6.0

8.0

×10 2

DroneCircle Cumulative

Rewards

0

2

4

6

8

1e6

0

10

20

30

DroneCircle Cumulative

Costs

In this ablation study, we examine the various optimization frameworks within the context of hard constraints. Particularly, we compare our RESPO framework with RCRL and CMDP . However, for RCRL we implement using our REF function method while still keeping the reachability value function. For CMDP , we make the cost threshold χ = 0 . These comparisons answer important questions about our design choices - specifically is it sufficient to simply to just use the REF component or to just learn the cost returns alone? From this ablation study, we propose that though we have provided theoretical support for adding each of these design components individually, in practice they are both required together in our algorithm. In RCRL implemented with our REF, we generally see decently high rewards but the cost violations are always very large. This highlights the problems of the reachability function again - if the agent starts or ever wanders into the infeasible set, there is no guarantee of (re)entrance into the feasible set. So the agent can indefinitely remain in the infeasible set, thereby incurring potentially an unlimited number of constraint violations. In PPOLag with χ = 0 , both the reward performance and constraint violations are very low. By using such hard constraints versions of these purely learning-based methods, even when using the cumulative discounted cost rather than reachability value function, the reward performance is very low because the lagrange multiplier becomes too large quickly and thereby overshadows the reward returns in the optimization. Ultimately, both the REF approach and the usage of the cumulative discounted costs are important components of our algorithm RESPO that encourage a good balance between the reward performance and safety constraint satisfaction in such stochastic settings.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed an iterative reachability estimation method for safe RL. The reachability is estimated by the probability of future trajectories entering unsafe state sets. Compare to previous reachability-based methods, the proposed method could handle stochastic dynamics and also improved the performance with deterministic dynamics. The proposed algorithm also leverages more information from data which could explain the performance improvement compared to existing methods. Theoretical convergence results are provided. Experimental results well supported the claimed performance improvement and safety guarantees. \n\nThis paper has removed one significant limitation, the deterministic dynamics assumption in previous studies. Removing this limitation should be very careful, and I think the authors did it very well. The author also connected HJ reachability with CMDP, which is two important definitions in the safe RL. Therefore, I strongly suggested the paper be accepted.\n\nSTRENGTHS:\nOriginality:\nVery good. As far as I now, no paper has considered stochastic reachability with the constrained RL setup. The authors also did a good job of connecting HJ reachability and CMDP, which are two important definitions in constrained RL that used to be separately considered. This combination also removed significant limitations, the deterministic dynamics of the previous study. \n\nQuality:\nExcellent. The paper very clearly explained the relation and improvement with respect to the previous paper in both deterministic and stochastic settings and did a comprehensive comparison in the experimental section. The authors also summarized the novelty and advantages of intuitions which is very easy to understand, i.e., previous methods only consider the maximum violation, which might lose information in the whole episode.\n\nClarity:\nThe presentation of this paper is good. \n\nSignificance:\nThe safety of stochastic systems is very important and challenging. There have been many theories and studies to formulate the problem, and reachability is undoubtedly one of the most powerful methods. Handling stochastic reachability should be very careful.\n\nWEAKNESSES:\nI feel good about most of the paper, I only have comments on some minor problems:\n1. The problem formulation, equation (4) should be emphasized better so that the reader will know this is the proposed problem formulation. \n2. The notation system is a bit messy. The readers might get lost easily, especially those not familiar with the previous paper.\n3. Algorithm 1 actually did not provide too much useful information. You should improve it to highlight the differences between your algorithm and the previous ones, like the REF update.\n\nQUESTIONS:\nI like most of the intuitions to explain the advantages of this paper. However, I have some questions:\n\n1. (line 179 - 181, about the claimed limitations of previous work RCRL) _**These improvements in costs can be crucial in guiding the optimization toward a safer policy.** And optimizing with $V_h(s) $ can result in accumulating an unlimited number of violations smaller than the maximum violation._\n\nI roughly got the intuition. I think you mean that some policies might be too far away from the safe policy. The optimization landscape looks like $V_h(s)$ should be non-decreasing in the first few steps but finally decrease to a low value. This is indeed interesting. Because from my intuition, a good policy update should always point to the direction that $V_h(s)$ decreases.\n\nDid you observe this in the experiment? If so, could you show me this phenomenon with some experimental results? It would be super helpful for enhancing the contribution of this paper.\n\n2. (Section 5.3, about the REF and multiplier.) The REF update actually relies on the distribution density function. It seems like you did not include this paper in the references,\n> Qin, Zengyi, Yuxiao Chen, and Chuchu Fan. \"Density constrained reinforcement learning.\" International Conference on Machine Learning. PMLR, 2021.\nI think it is a good paper to further understand the relationship between density function and Lagrange multipliers in safe RL problems. You should discuss the relationship with this paper.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents  Reachability Estimation for Safe Policy Optimization (RESPO), for safety-constrained RL in general stochastic settings. The authors extend the previous RCRL approach into stochastic settings and push the agent to (re)enter the feasible region. They formulate a safe RL problem with REF and further develop an adapted AC algorithm to solve it, with convergence analysis. They compare their approach against CMDP-based approaches with a soft constraint and RCRL with a hard constraint, showing the advantages.\n\nSTRENGTHS:\nThe paper is well-motivated and well-structured to follow. It studies a critical problem. \nBased on RCRL, this paper does have some novelties in problem formulation and proposed approach. Technically, this paper is sound to me, although I only checked part of the math proof in the appendix.  \nThe experiments are promising as they show mixed performance and safety violation improvements.\n\nWEAKNESSES:\n1. The writing could be further improved, especially the comparison with RCRL. The reviewer acknowledges that there is some explanation of the difference between the proposed approach and RCRL, still, it would be much better to add more and clarify it. For example, the reviewer is confused why RCRL cannot guarantee or optimize (re)entrance to the feasible set. Couldn't use the same proof of Proposition 2 to obtain the same (re)entrance proposition?  \n\n2. For the deterministic environments, my understanding is RCRL considers a harder constraint as it is per state constraint than the discounted additive constraints in your paper, why the harder constraint cannot optimize/guarantee (re)entrance to the feasible set?  \n\n3. How does the reentrance proved from the deterministic environment applied to stochastic systems? \n\n3. There are some recent safe RL papers considering hard constraints. For example, \n\n a. Wang, Y., Zhan, S. S., Jiao, R., Wang, Z., Jin, W., Yang, Z., ... & Zhu, Q. (2022). Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. ICML 2023.\n\n b.  Xiong, N. (2023). Provably Safe Reinforcement Learning with Step-wise Violation Constraints. arXiv preprint arXiv:2302.06064.\nThe authors may consider talking about these recent references in the paper revision. \n\n4. what do you mean by \"almost surely\" in the convergence analysis?\n\nQUESTIONS:\nPlease see the weakness.\n\nLIMITATIONS:\nWhat are the limitations of RESPO?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPrevious approaches to safe reinforcement learning used the constrained MDP formulation where there is a constraint imposed on the cumulative sum of costs to minimise violations. This framework is not applicable very easily where there is a need for hard constraint satisfaction. The previous approach (RCRL) which leverages reachability analysis to strictly satisfy hard constraints is limited to deterministic MDP and is not suited to bringing the state back to the feasible set when already outside the feasible set. In this work, the authors minimize the expected chance and frequency of violations under stochastic transition dynamics thus resolving the two problems with the previous work. The order in which to update the Q-networks, the policy networks, the Lagrangian dual factors and the reachability estimation function is studied using an empirical approach and a theoretical convergence guarantee to a local optimum is provided for this alternating optimization. Empirical comparisons are made extensively to a wide spectrum of existing approaches.\n\nSTRENGTHS:\nAdvantages of paper in relation to deterministic dynamics assumption of RCRL is clear and has merit. Empirical comparisons to previous work are quite extensive with additional explanations in the appendix. Convergence to local optimum is presented to establish rigor and soundness of the method.\n\nWEAKNESSES:\n1) In section 6, it was not very clear what to see in the figures and it looked like RESPO was achieving a different point in the trade-off curve compared to the other methods. More information was available in the appendix. More discussion on how to compare the different methods and why one method performs better in a specific metric can be written out in the main section. The motivation of the paper provides the twin advantages of getting back into the feasible set and accounting for stochasticity. The first advantage is seen in the double integrator example in the appendix. Do any of the previous methods suffer due to deterministic assumptions and is the actual MDP stochastic?\n\n2) Since there are many networks and parameters updated at the same time, the robustness and reproducibility of the training process for this method and similar previous methods seem suspect. The authors have discussed this aspect in the ablation studies and the best possible convergence is obtained in the way the authors are doing the training. This insight, though a mild weakness, could benefit the community as we are inferring new insights about alternating optimization between multiple networks tied to each other.\n\nQUESTIONS:\n1) While doing the comparison to CBF methods, it is not clearly what is being compared against. CBFs/Energy functions are usually handcrafted but there is CBVF (cited by the paper) and other related work that tries to construct the optimal CBF with maximum safety set (assuming known model). With the optimal CBF, the behaviour of CBF methods is not very conservative. If stochasticity is the source of trouble for the CBF methods, some extensions such as robust CBF and learned CBF are available. Here, the authors can provide more details about which exact CBF method they are using. \n\n2) In line 155, should $V_c$ be in the expression for optimal policy? $V_c$ is previously defined differently. “We define optimal REF based on an optimally safe policy π∗” is then misleading. The policy is optimizing a different loss in line 227 and it is not clear whether we actually get the optimal REF from this process. \n\n3) Assumption A3 (Lipschitz gradients) is a rather strong assumption. Are there previous cases where RL value functions are assumed to have Lipschitz gradients?\n\n4) In figure 3, why are RCRL and FAC violating the hard constraint if they are designed to respect hard constraints?\n\n5) The variance of the red RESPO learning curves seem to be high in certain figures indicating training instabilities\n\nLIMITATIONS:\nAuthors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new algorithm that may handle hard and soft constraints, in which the policy optimization and Hamilton-Jacobi reachability are leveraged to ensure safety. Moreover,  experiment results on safety gym, safety PyBullet, and safety MuJoCo also show the good performance of their algorithm.\n\nSTRENGTHS:\n1. The convergency analysis sounds good.\n2. Comprehensive experiments are provided.\n3. Hard constraints and soft constraints are investigated.\n\nWEAKNESSES:\n1. Paper writing quality needs to be improved a lot, I am confused about the paper notation, e.g., V_h and V_c.\n2. The experimental results are not correct regarding some baselines, especially for CRPO, in CRPO paper, the algorithm presents better performance than PPO-Lagrangian and CPO.\n\nQUESTIONS:\n1. Could you analyze the difference between RESPO and CRPO regarding the reward update and cost update?\n2. If the agent does not find the safe action, will it be stuck at a point?\n3. How do you define the reachable set when considering reward performance?\n\nLIMITATIONS:\n1. The balance between reward and cost is not addressed well, in the experiments, as shown in Figure 3, although RESPO can ensure safety, the trajectory is longer than other baselines, and also not smooth.\n2. Some related papers are not mentioned in the study, e.g., [Kochdumper, N., et al., 2023], [Gu, S., et al., 2022] and [Selim, M., et al., 2022].\n\n[Kochdumper, N., et al., 2023] Kochdumper, N., Krasowski, H., Wang, X., Bak, S., & Althoff, M. (2023). Provably safe reinforcement learning via action projection using reachability analysis and polynomial zonotopes. IEEE Open Journal of Control Systems, 2, 79-92.\n\n[Gu, S., et al., 2022]  Gu, S., Chen, G., Zhang, L., Hou, J., Hu, Y., & Knoll, A. (2022). Constrained reinforcement learning for vehicle motion planning with topological reachability analysis. Robotics, 11(4), 81.\n\n[Selim, M., et al., 2022] Selim, M., Alanwar, A., Kousik, S., Gao, G., Pavone, M., & Johansson, K. H. (2022). Safe reinforcement learning using black-box reachability analysis. IEEE Robotics and Automation Letters, 7(4), 10665-10672.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}