Title: Solving Minimum-Cost Reach Avoid using Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Reinforcement Learning

Optimal Control

Reachability Analysis
Abstract: Current reinforcement-learning methods are unable to directly learn policies that solve the minimum cost reach-avoid problem to minimize cumulative costs subject to the constraints of reaching the goal and avoiding unsafe states, as the structure of this new optimization problem is incompatible with current methods. Instead, a surrogate problem is solved where all objectives are combined with a weighted sum. However, this surrogate objective results in suboptimal policies that do not directly minimize the cumulative cost. In this work, we propose RC-PPO, a reinforcement-learning-based method for solving the minimum-cost reach-avoid problem by using connections to Hamilton-Jacobi reachability. Empirical results demonstrate that RC-PPO learns policies with comparable goal-reaching rates to while achieving up to 57% lower cumulative costs compared to existing methods on a suite of minimum-cost reach-avoid benchmarks on the Mujoco simulator. The project page can be found at https://oswinso.xyz/rcppo.

Solving Minimum-Cost Reach Avoid using Reinforcement Learning

Abstract

Current reinforcement-learning methods are unable to directly learn policies that solve the minimum cost reach-avoid problem to minimize cumulative costs subject to the constraints of reaching the goal and avoiding unsafe states, as the structure of this new optimization problem is incompatible with current methods. Instead, a surrogate problem is solved where all objectives are combined with a weighted sum. However, this surrogate objective results in suboptimal policies that do not directly minimize the cumulative cost. In this work, we propose RC-PPO , a reinforcement-learning-based method for solving the minimum-cost reach-avoid problem by using connections to Hamilton-Jacobi reachability. Empirical results demonstrate that RC-PPO learns policies with comparable goal-reaching rates to while achieving up to 57% lower cumulative costs compared to existing methods on a suite of minimum-cost reach-avoid benchmarks on the Mujoco simulator. The project page can be found at https://oswinso.xyz/rcppo/.

1 Introduction

Many real-world tasks can be framed as a constrained optimization problem where reaching a goal at the terminal state and ensuring safety (i.e., reach-avoid) is desired while minimizing some cumulative cost as an objective function, which we term the minimum-cost reach-avoid problem.

The cumulative cost, which differentiates this from the traditional reach-avoid problem, can be used to model desirable aspects of a task such as minimizing energy consumption, maximizing smoothness, or any other pseudo-energy function, and allows for choosing the most desirable policy among many policies that can satisfy the reach-avoid requirements. For example, energy-efficient autonomous driving [1, 2] can be seen as a task where the vehicle must reach a destination, follow traffic rules, and minimize fuel consumption. Minimizing fuel use is also a major concern for low-thrust or energy-limited systems such as spacecraft [3] and quadrotors [4]. Quadrotors often have to choose limited battery life to meet the payload capacity. Hence, minimizing their energy consumption, which can be done by taking advantage of wind patterns, is crucial for keeping them airborne to complete more tasks. Other use-cases important for climate change include plasma fusion (reach a desired current, minimize the total risk of plasma disruption) [5] and voltage control (reach a desired voltage level, minimize the load shedding amount) [6].

If only a single control trajectory is desired, this class of problems can be solved using numerical trajectory optimization by either optimizing the timestep between knot points [7] or a bilevel optimization approach that adjusts the number of knot points in an outer loop [8, 9, 10]. However, in this setting, the dynamics are assumed to be known, and only a single trajectory is obtained. Therefore, the computation will needs to be repeated when started from a different initial state. The computational complexity of trajectory optimization prevents it from being used in real time. Moreover, the use of nonlinear numerical optimization may result in poor solutions that lie in suboptimal local minim [11].

Alternatively, to obtain a control policy, reinforcement learning (RL) can be used. However, existing methods are unable to directly solve the minimum-cost reach-avoid problem. Although RL has been used to solve many tasks where reaching a goal is desired, goal-reaching is encouraged as a reward instead of as a constraint via the use of either a sparse reward at the goal [12, 13, 14], or a surrogate dense reward [14, 15] 1 . However, posing the reach constraint as a reward then makes it difficult to optimize for the cumulative cost at the same time. In many cases, this is done via a weighted sum of the two terms [18, 19, 20]. However, the optimal policy of this new surrogate objective may not necessarily be the optimal policy of the original problem. Another method of handling this is to treat the cumulative cost as a constraint and solve for a policy that maximizes the reward while keeping the cumulative cost under some fixed threshold, resulting in a new constrained optimization problem that can be solved as a constrained Markov decision process (CMDP) [21]. However, the choice of this fixed threshold becomes key: too small and the problem is not feasible, destabilizing the training process. Too large, and the resulting policy will simply ignore the cumulative cost.

To tackle this issue, we propose Reach Constrained Proximal Policy Optimization(RC-PPO) , a new algorithm that targets the minimum-cost reach-avoid problem. We first convert the reach-avoid problem to a reach problem on an augmented system and use the corresponding reach value function to compute the optimal policy. Next, we use a novel two-step PPO-based RL-based framework to learn this value function and the corresponding optimal policy. The first step uses a PPO-inspired algorithm to solve for the optimal value function and policy, conditioned on the cost upper bound. The second step fine-tunes the value function and solves for the least upper bound on the cumulative cost to obtain the final optimal policy. Our main contributions are summarized below:

We prove that the minimum-cost reach-avoid problem can be solved by defining a set of augmented dynamics and a simplified constrained optimization problem.

We propose RC-PPO, a novel algorithm based on PPO that targets the minimum-cost reach-avoid problem, and prove that our algorithm converges to a locally optimal policy.

Simulation experiments show that RC-PPO achieves reach rates comparable with the baseline method with the highest reach rate while achieving significantly lower cumulative costs.

2 Related Works

Terminal-horizon state-constrained optimization Terminal state constraints are quite common in the dynamic optimization literature. For the finite-horizon case, for example, one method of guaranteeing the stability of model predictive control (MPC) is with the use of a terminal state constraint [22]. Since MPC is implemented as a discrete-time finite-horizon numerical optimization problem, the terminal state constraints can be easily implemented in an optimization program as a normal state constraint. The case of a flexible-horizon constrained optimization is not as common but can still be found. For example, one method of time-optimal control is to treat the integration timestep as a control variable while imposing state constraints on the initial and final knot points [7]. Another method is to consider a bilevel optimization problem, where the number of knot points is optimized for in the outer loop [8, 9, 10].

Goal-conditioned Reinforcement Learning There have been many works on goal-conditioned reinforcement learning. These works mainly focus on the challenges of tackling sparse rewards [12, 14, 23, 15] or even learning without rewards completely, either via representation learning objectives [24, 25, 26, 27, 28, 29, 30, 31, 32, 33] or by using contrastive learning to learn reward functions [34, 35, 36, 37, 38, 39, 40, 41, 42, 43], often in imitation learning settings [44, 45]. However,

the manner in which these goals are reached is not considered, and it is difficult to extend these works to additionally minimize some cumulative cost.

Constrained Reinforcement Learning One way of using existing techniques to approximately tackle the minimum-cost reach-avoid problem is to flip the role of the cumulative-cost objective and the goal-reaching constraint by treating the goal-reaching constraint as an objective via a (sparse or dense) reward and the cumulative-cost objective as a constraint with a cost threshold, turning the problem into a CMDP [21]. In recent year, there has been significant interest in deep RL methods for solving CMDPs [46, 47, 48]. While these methods are effective at solving the transformed CMDP problem, the optimal policy to the CMDP may not be the optimal policy to the original minimum-cost reach-constrained problem, depending on the choice of the cost constraint.

State Augmentation in Constrained Reinforcement Learning To improve reward structures in constrained reinforcement learning, especially in safety-critical systems, one effective approach is state augmentation. This technique integrates constraints, such as safety or energy costs, into the augmented state representation, allowing for more effective constraint management through the reward mechanism [49, 50, 51]. While these methods enhance the reward structure for solving the transformed CMDP problems, they still face the inherent limitation of the CMDP framework: the optimal policy for the transformed CMDP may not always correspond to the optimal solution for the original problem.

Reachability Analysis Reachability analysis looks for solutions to the reach-avoid problem. That is, to solve for the set of initial conditions and an appropriate control policy to drive a system to a desired goal set while avoiding undesireable states. Hamilton-Jacobi (HJ) reachability analysis [52, 53, 54, 55, 56] provides a methodology for the case of dynamics in continuous-time via the solution of a partial differential equation (PDE) and is conventionally solved via numerical PDE techniques that use state-space discretization [54]. This has been extended recently to the case of discrete-time dynamics and solved using off-policy [57, 58] and on-policy [59, 60] reinforcement learning. While reachability analysis concerns itself with the reach-avoid problem, we are instead interested in solutions to the minimum-cost reach-avoid problem.

3 Problem Formulation

In this paper, we consider a class of minimum-cost reach-avoid problems defined by the tuple M := ⟨X , U , f, c, g, h ⟩ . Here, X ⊆ R n is the state space and U ⊆ R m is the action space. The system states x t ∈ X evolves under the deterministic discrete dynamics f : X × U → X as

The control objective for the system states x t is to reach the goal region G and avoid the unsafe set F while minimizing the cumulative cost ∑ T -1 t =0 c ( x t , π ( x t )) under control input u t = π ( x t ) for a designed control policy π : X → U . Here, T denotes the first timestep that the agent reaches the goal G . The sets G and F are given as the 0 -sublevel and strict 0 -superlevel sets g : X → R and h : X → R respectively, i.e.,

This can be formulated formally as finding a policy π that solves the following constrained flexible final-time optimization problem for a given initial state x 0 :

Note that as opposed to either traditional finite-horizon constrained optimization problems where T is fixed or infinite-horizon problems where T = ∞ , the time horizon T is also a decision variable. Moreover, the goal constraint (3b) is only enforced at the terminal timestep T . These two differences prevent the straightforward application of existing RL methods to solve (3).

3.1 Reachability Analysis for Reach-Avoid Problems

In discrete time, the set of initial states that can reach the goal G without entering the avoid set F can be represented by the 0 -sublevel set of a reach-avoid value function V π g,h [58]. Given functions g , h describing G and F and a policy π , the reach-avoid value function V π g,h : X → R is defined as

where x π t denote the system state at time t under a policy π starting from an initial state x π 0 = x 0 . In the rest of the paper, we suppress the argument x 0 for brevity whenever clear from the context. It can be shown that the reach-avoid value function satisfies the following recursive relationship via the reach-avoid Bellman equation (RABE) [58]

The Bellman equation (5) can then be used in a reinforcement learning framework (e.g., via a modification of soft actor-critic[61, 62]) as done in [58] to solve the reach-avoid problem.

Note that existing methods of solving reach-avoid problems through this formulation focus on minimizing the value function V π g,h . This is not necessary as any policy that results in V π g,h ≤ 0 solves the reach-avoid problem, albeit without any cost considerations. However, it is often the case that we wish to minimize a cumulative cost (e.g., (3a)) on top of the reach-avoid constraints (3b)-(3c) for a minimum-cost reach-avoid problem. To address this class of problems, we next present a modification to the reach-avoid framework that additionally enables the minimization of the cumulative cost.

3.2 Reachability Analysis for Minimum-cost Reach-Avoid Problems

We now provide a new framework to solve the minimum-cost reach-avoid by lifting the original system to a higher dimensional space and designing a set of augmented dynamics that allow us to convert the original problem into a reachability problem on the augmented system.

Let denote the shifted indicator function defined as

■

Define the augmented state as ˆ x = ( x, y, z ) ⊆ ˆ X := X×{1 , 1 }× R . Wenowdefine a corresponding augmented dynamics function f ′ : ˆ X × U → ˆ X as

ˆ f ( x t , y t , z t , u t ) = ( f ( x t ) , max { ■ f ( x t ) ∈F , y t } , z t -c ( x t , u t ) ) , (7) where y 0 = ■ x 0 ∈F . Note that y t = 1 if the state has entered the avoid set F at some timestep from 0 to t and is unsafe , and y t = 0 if the state has not entered the avoid set F at any timestep from 0 to t and is safe . Moreover, z t is equal to z 0 minus the cost-to-come, i.e., for state trajectory x 0: t and action trajectory u 0: t , i.e.,

Under the augmented dynamics, we now define the following augmented goal function ˆ g : ˆ X → R as

where C > 0 is an arbitrary constant. 2 With this definition of ˆ g , an augmented goal region ˆ G can be defined as

In other words, starting from initial condition ˆ x 0 = ( x 0 , y 0 , z 0 ) , reaching the goal on the augmented system ˆ x T ∈ ˆ g at timestep T implies that 1) the goal is reached at x T for the original system, 2) the state trajectory remains safe and does not enter the avoid set F , and 3) z 0 is an upper-bound on the total cost-to-come: ∑ T -1 t =0 c ( x t , u t ) ≤ z 0 . We call this the upper-bound property. The above intuition on the newly defined augmented system is formalized in the following theorem, whose proof is provided in Appendix D.1.

Theorem 1. For given initial conditions x 0 ∈ X , z 0 ∈ R and control policy π , consider the trajectory for the original system { x 0 , . . . x T } and its corresponding trajectory for the augmented system { ( x 0 , y 0 , z 0 ) , . . . ( x T , y T , z T ) } for some T > 0 . Then, the reach constraint x T ∈ G (3b), avoid constraint x t ̸∈ F ∀ t ∈ { 0 , 1 , . . . , T } (3c) and the upper-bound property z 0 ≥ ∑ T -1 k =0 c ( x k , π ( x k ) ) hold if and only if the augmented state reaches the augmented goal at time T , i.e., ( x T , y T , z T ) ∈ ˆ G .

With this construction, we have folded the avoid constraints x t ̸∈ F (3c) into the reach specification on the augmented system. In other words, solving the reach problem on the augmented system results in a reach-avoid solution of the original system. As a result, we can simplify the value function (4) and Bellman equation (5), resulting in the following definition of the reach value function ˜ V ˆ g : ˆ X → R

Similar to (4), the 0 -sublevel set of ˜ V ˆ g describes the set of augmented states ˆ x that can reach the augmented goal ˆ G . We can also similarly obtain a recursive definition of the reach value function ˜ V ˆ g given by the reachability Bellman equation (RBE)

We now solve the minimum-cost reach-avoid problem using this augmented system. By Theorem 1, the z 0 is an upper bound on the cumulative cost to reach the goal while avoiding the unsafe set if and only if the augmented state ˆ x reaches the augmented goal. Since this upper bound is tight, the least upper bound z 0 that still reaches the augmented goal thus corresponds to the minimum-cost policy that satisfies the reach-avoid constraints. In other words, the minimum-cost reach-avoid problem for a given initial state x 0 can be reformulated as the following optimization problem.

s . t . ˜ V π ˆ g ( x 0 , ■ x 0 ∈F , z 0 ) ≤ 0 . (13b) We refer to Appendix B for a detailed derivation of the equivalence between the transformed Problem 13 and the original minimum-cost reach-avoid Problem 3.

Remark 1 (Connections to the epigraph form in constrained optimization) . The resulting optimization problem (13) can be interpreted as an epigraph reformulation [63] of the minimum-cost reach-avoid problem (3). The epigraph reformulation results in a problem with linear objective but yields the same solution as the original problem [63]. The construction we propose in this work can be seen as a dynamic version of this epigraph reformulation technique originally developed for static problems and is similar to recent results that also make use of the epigraph form for solving infinite-horizon constrained optimization problems [59].

4 Solving with Reinforcement Learning

In the previous section, we reformulated the minimum-cost reach-avoid problem by constructing an augmented system and used its reach value function (11) in a new constrained optimization problem (13) over the cost upper-bound z 0 . In this section, we propose Reachability Constrained Proximal Policy Optimization (RC-PPO), a two-phase RL-based method for solving (13) (see Figure 1).

4.1 Phase 1: Learn z -conditioned policy and value function

In the first step, we learn the optimal policy π and value function ˜ V π ˆ g , as functions of the cost upper-bound z 0 , using RL. To do so, we consider the policy gradient framework [64]. However, since the policy gradient requires a stochastic policy in the case of deterministic dynamics [65], we consider an analog of the developments made in the previous section but for the case of a stochastic policy. To this end, we redefine the reach value function ˜ V π ˆ g using a similar Bellman equation under a stochastic policy as follows.

Phase one

Original System

Augmented Dynamic

System

Training

…

Minimize

Phase two

Fine-Tune

Calculate

Execution

VT

9

T Eg

20

企1

T

z*(x,y) = min{z | V(x,y,2) ≤0}

t+1=f(at,ut)

6

t≥0,π

CT E g'

π*(x,y) =π(x,y,2*(x,y)

Definition 1 (Stochastic Reachability Bellman Equation) . Given function ˆ g in (9), a stochastic policy π , and initial conditions x 0 ∈ X , z 0 ∈ R , the stochastic reach value function ˜ V π ˆ g is defined as the solution to the following stochastic reachability Bellman equation (SRBE):

where ˆ x 0 = ( x 0 , y 0 , z 0 ) with y 0 = ■ x 0 ∈F . For this stochastic Bellman equation, the Q function [66] is defined as

We next define the dynamics of our problem with stochastic policy below.

Definition 2 (Reachability Markov Decision Process) . The Reachability Markov Decision Process is defined on the augmented dynamic in Equation (7) with an added absorbing state s 0 . We define the transition function f ′ r with the absorbing state as

Denote by d ′ π (ˆ x ) the stationary distribution under stochastic policy π starting at ˆ x ∈ X ×{1 , 1 }× R .

We now derive a policy gradient theorem for the Reachability MDP in Definition 2 which yields an almost identical expression for the policy gradient.

Theorem 2. (Policy Gradient Theorem) For policy π θ parameterized by θ , the gradient of the policy value function ˜ V π θ ˆ g satisfies

under the stationary distribution d ′ π (ˆ x ) for Reachability MDP in Definition 2

The proof of this new policy gradient theorem (Theorem 2) follows the proof of the normal policy gradient theorem [66], differing only in the expression of the stationary distribution. We provide the proof in Appendix D.3.

Since the stationary distribution d ′ π (ˆ x ) in Definition 2 is hard to simulate during the learning process, we instead consider the stationary distribution under the original augmented dynamic system. Note that Definition 1 does not induce a contraction map, which harms performance. To fix this, we apply the same trick as [58] by introducing an additional discount factor γ into the Bellman equation (12):

This provides us with a contraction map (proved in [58]) and we leave the discussion of choosing γ in Appendix C. The Q-function corresponding to (18) is then given as

Following proximal policy optimization (PPO) [67], we use generalized advantage estimation (GAE) [68] to compute a variance-reduced advantage function ˜ A π ˆ g = ˜ Q π ˆ g -˜ V π ˆ g for the policy gradient (Theorem 2) using the λ -return [66]. We refer to Appendix A for the definition of ˆ A π (GAE) ˆ g and denote the loss function when θ = θ l as

We wish to obtain the optimal policy π and the value function ˜ V π θ ˆ g conditioned on z 0 . Hence, at the beginning of each rollout, we uniformly sample z 0 within a user-specified range [ z min , z max ] . Since the optimal z 0 is the cumulative cost of the policy that solves the minimum-cost reach-avoid problem, z min and z max are user-specified bounds on the optimal cost. In particular, when the cost-function is bounded and the optimal cost is non-negative, we can choose z min to be some negative number and z max to be the maximum possible discounted cost.

4.2 Phase 2: Solving for the optimal z

In the second phase, we first compute a deterministic version π ∗ of the stochastic policy π from phase 1 by taking the mode. Next, we fine-tune V π ˆ g based on the now deterministic π ∗ to obtain ˜ V ∗ ˆ g .

Given any state x , the final policy is then obtained by solving for the optimal cost upper-bound z ∗ from Equation (13), which is a 1D root-finding problem and can be easily solved using bisection. Note that Equation (13) must be solved online for z ∗ at each state x . Alternatively, to avoid performing bisection online, we can instead learn the map ( x, y ) ↦→ z ∗ offline using regression with randomly sampled ( x, y ) pairs and z ∗ labels obtained from bisection offline.

We provide a convergence proof of an actor-critic version of our method without the GAE estimator in Appendix E.

5 Experiments

Baselines We consider two categories of RL baselines. The first is goal-conditioned reinforcement learning which focuses on goal-reaching but does not consider minimization of the cost. For this category, we consider the Contrastive Reinforcement Learning (CRL) [33] method. We also compare against safe RL methods that solve CMDPs. As the minimum-cost reach-avoid problem (3) cannot be posed as a CMDP, we reformulate (3) into the following surrogate CMDP:

where the reward r incentivies goal-reaching, C fail is a term balancing two constraint terms, and X threshold is a hyperparameter on the cumulative cost. For this category, we consider the CPPO [48] and RESPO [60]. Note that RESPO also incorporates reachability analysis to adapt the Lagrange multipliers for each constraint term. We implement the above CMDP-based baselines with three different choices of X thresholds : X L , X M and X H . For RESPO, we found X M to outperform both X L and X H and thus only report results for X M .

We also consider the static Lagrangian multiplier case. In this setting, the reward function becomes r ( x t ) -β ( ✶ x t ∈F × C fail + c ( x t , u t )) for a constant Lagrange multiplier β . We consider two different levels of β ( β L , β H ) in our experiments, resulting in the baselines PPO_ β L , PPO_ β H , SAC_ β L , SAC_ β H . More details are provided in Appendix F.

Pendulum

Safety Hopper

WindField

PointGoal

Safety HalfCheetah

FixedWing

0.0

0.2

0.4

0.6

0.8

1.0

RC-PPO

RESPO

PPO_

β

L

PPO_

β

H

SAC_

β

L

SAC_

β

H

CPPO_



L

CPPO_



M

CPPO_



H

CRL

Reach Rate ↑

Pendulum

0.0

0.2

0.4

0.6

0.8

RC-PPO

RESPO

PPO_

β

L

PPO_

β

H

SAC_

β

L

SAC_

β

H

CPPO_



L

CPPO_



M

CPPO_



H

CRL

Reach Rate ↑

Safety

Hopper

0.0

0.2

0.4

0.6

0.8

RC-PPO

RESPO

PPO_

β

L

PPO_

β

H

SAC_

β

L

SAC_

β

H

CPPO_



L

CPPO_



M

CPPO_



H

CRL

Reach Rate ↑

WindField

0.0

0.2

0.4

0.6

0.8

RC-PPO

RESPO

PPO_

β

L

PPO_

β

H

SAC_

β

L

SAC_

β

H

CPPO_



L

CPPO_



M

CPPO_



H

CRL

Reach Rate ↑

PointGoal

0.0

0.2

0.4

0.6

RC-PPO

RESPO

PPO_

β

L

PPO_

β

H

SAC_

β

L

SAC_

β

H

CPPO_



L

CPPO_



M

CPPO_



H

CRL

Reach Rate ↑

Safety

HalfCheetah

0.0

0.2

0.4

0.6

0.8

RC-PPO

RESPO

PPO_

β

L

PPO_

β

H

SAC_

β

L

SAC_

β

H

CPPO_



L

CPPO_



M

CPPO_



H

CRL

Reach Rate ↑

FixedWing

Benchmarks We compare RC-PPO with baseline methods on several minimum-cost reach-avoid environments. We consider an inverted pendulum ( Pendulum ), an environment from Safety Gym [69] ( PointGoal ) and two custom environments from MuJoCo [70], ( Safety Hopper , Safety HalfCheetah ) with added hazard regions and goal regions. We also consider a 3D quadrotor navigation task in a simulated wind field for an urban environment [71, 72] ( WindField ) and an Fixed-Wing avoid task from [59] with an additional goal region ( FixedWing ). More details on the benchmark can be found in Appendix G.

Evaluation Metrics Since the goal of RC-PPO is minimizing cost consumption while reaching goal without entering the unsafe region F . We evaluate algorithm performance based on (i) reach rate, (ii) cost. The reach rate is the ratio of trajectories that enter goal region G without violating safety along the trajectory. The cost denotes the cumulative cost over the trajectory ∑ T k =0 c ( x k , π ( x k )) .

5.1 Sparse Reward Setting

Wefirst compare our algorithm with other baseline algorithms under a sparse reward setting (Figure 3). In all environments, the reach rate for the baseline algorithms is very low. Also, there is a general trend between the reach rate and the Lagrangian coefficient. CPPO_ X L , PPO_ β H and SAC_ β H have higher Lagrangian coefficients which lead to a lower reach rate.

5.2 Comparison under Reward Shaping

Reward shaping is a common method that can be used to improve the performance of RL algorithms, especially in the sparse reward setting [73, 74]. To see whether the same conclusions still hold even in the presence of reward shaping, we retrain the baseline methods but with reward shaping using a distance function-based potential function (see Appendix F for more details).

The results in Figure 4 demonstrate that RC-PPO remains competitive against the best baseline algorithms in reach rate while achieving significantly lower cumulative costs. The baseline methods (PPO_ β H , SAC_ β H , CPPO_ X L ) fail to achieve a high reach rate due to the large weights placed on minimizing the cumulative cost. CRL can reach the goal for simpler environments ( Pendulum ) but struggles with more complex environments. However, since goal-conditioned methods do not consider minimize cumulative cost, it achieves a higher cumulative cost relative to other methods.

80

160

240

320

CRL

CPPO_



H

CPPO_



M

SAC_

β

L

PPO_

β

L

RESPO

RC-PPO

Additional Cumulative Cost ↓

0.0

0.2

0.4

0.6

0.8

1.0

RC-PPO

RESPO

PPO_

β

L

PPO_

β

H

SAC_

β

L

SAC_

β

H

CPPO_



L

CPPO_



M

CPPO_



H

CRL

Reach Rate ↑

Pendulum

150

200

250

CPPO_



H

CPPO_



M

SAC_

β

L

PPO_

β

L

RESPO

RC-PPO

Additional Cumulative Cost ↓

0.0

0.2

0.4

0.6

0.8

RC-PPO

RESPO

PPO_

β

L

PPO_

β

H

SAC_

β

L

SAC_

β

H

CPPO_



L

CPPO_



M

CPPO_



H

CRL

Reach Rate ↑

PointGoal

50

75

100

125

CPPO_



H

PPO_

β

L

RESPO

RC-PPO

Additional Cumulative Cost ↓

0.0

0.2

0.4

0.6

RC-PPO

RESPO

PPO_

β

L

PPO_

β

H

SAC_

β

L

SAC_

β

H

CPPO_



L

CPPO_



M

CPPO_



H

CRL

Reach Rate ↑

Safety

HalfCheetah

180

240

300

360

CPPO_



H

CPPO_



M

SAC_

β

L

PPO_

β

L

RESPO

RC-PPO

Additional Cumulative Cost ↓

0.0

0.2

0.4

0.6

0.8

1.0

RC-PPO

RESPO

PPO_

β

L

PPO_

β

H

SAC_

β

L

SAC_

β

H

CPPO_



L

CPPO_



M

CPPO_



H

CRL

Reach Rate ↑

WindField

0

π

2

π

θ

-7.5

-5.0

-2.5

0.0

2.5

5.0

ω

Goal

CPPO_



H

0

π

2

π

θ

-5

0

5

ω

RESPO

0

π

2

π

θ

-5

0

5

ω

RC-PPO

0

100

200

300

400

Timestep

0

200

400

600

Cumulative

Cost

RC-PPO

RESPO

CPPO_



H

(a)

Pendulum

0

50

100

150

200

250

Timestep

0

100

200

300

Cumulative

Cost

RC-PPO

RESPO

CPPO_



H

Goal

(b)

WindField

Other baselines focus more on goal-reaching tasks while putting less emphasis on the cost part. As a result, they suffer from higher costs than RC-PPO. We can also observe that RESPO achieves lower cumulative cost compared to CPPO_ X M which shares the same X threshold . This is due to RESPO making use of reachability analysis to better satisfy constraints.

To see how RC-PPO achieves lower cumulative costs, we visualize the resulting trajectories for Pendulum and WindField in Figure 5. For Pendulum , we see that RC-PPO learns to perform energy pumping to reach the goal in more time but with a smaller cumulative cost. The optimal behavior is opposite in the case of WindField , which contains an additional constant term in the cost to model the energy draw of quadcopters (see Appendix G). Here, we see that RC-PPO takes advantage of the wind at the beginning by moving downwind , arriving at the goal faster and with less cumulative cost.

We also visualize the learned RC-PPO policy for different values of z on the Pendulum benchmark (see Appendix H.2). For small values of z , the policy learns to minimize the cost, but at the expense of not reaching the goal. For large values of z , the policy reaches the goal quickly but at the expense of a large cost. The optimal z opt found using the learned value function ˜ V π θ ˆ g finds the z that minimizes the cumulative cost but is still able to reach the goal.

5.3 Optimal solution of minimum-cost reach-avoid cannot be obtained using CMDP

Though the previous subsections show the performance benefits of RC-PPO over existing methods, this may be due to badly chosen hyperparameters for the baseline methods, particularly in the formulation of the surrogate CMDP (22). We thus pose the following question: Can CMDP methods perform well under the right parameters of the surrogate CMDP problem (22) ? .

200

300

0.0

0.2

0.4

0.6

0.8

1.0

Reach Rate ↑

0

20

40

60

80

100

Additional Cumulative Cost ↓

RC-PPO

PPO across different

reward coefficients

Empirical Study. To answer this question, we first perform an extensive grid search over both the different coefficients in (22) and the static Lagrange multiplier for PPO (see Appendix H.3) and plot the result in Figure 6. RC-PPO outperforms the entire Pareto front formed from this grid search, providing experimental evidence that the performance improvements of RC-PPO stem from having a better problem formulation as opposed to badly chosen hyperparameters for the baselines.

Theoretical Analysis on Simple Example. To complement the empirical study, we provide an example of a simple minimum-cost reach-avoid problem where we prove that no choice of hyperparameter leads to the optimal solution in Appendix I.

5.4 Robustness to Noise

Finally, we investigate the robustness to varying levels of control noise in Appendix H.4. Even with the added noise, RC-PPO achieves the lowest cumulative cost while maintaining a comparable reach rate to other methods.

6 Conclusion and Limitations

We have proposed RC-PPO, a novel reinforcement learning algorithm for solving minimum-cost reach-avoid problems. We have demonstrated the strong capabilities of RC-PPO over prior methods in solving a multitude of challenging benchmark problems, where RC-PPO learns policies that match the reach rates of existing methods while achieving significantly lower cumulative costs.

However, it should be noted that RC-PPO is not without limitations. First, the use of augmented dynamics enables folding the safety constraints within the goal specifications through an additional binary state variable. While this reduces the complexity of the resulting algorithm, it also means that two policies that are both unable to reach the goal can have the same value ˜ V π g ′ even if one is unsafe, which can be undesirable. Next, the theoretical developments of RC-PPO are dependent on the assumptions of deterministic dynamics, which can be quite restrictive as it precludes the use of commonly used techniques for real-world deployment such as domain randomization. We acknowledge these limitations and leave resolving these challenges as future work.

Acknowledgments and Disclosure of Funding

This work was partly supported by the National Science Foundation (NSF) CAREER Award #CCF2238030, the MIT Lincoln Lab, and the MIT-DSTA program. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the authors and don't necessarily reflect the views of the sponsors.

References

Xuewei Qi, Yadan Luo, Guoyuan Wu, Kanok Boriboonsomsin, and Matthew Barth. Deep reinforcement learning enabled self-learning control for energy efficient driving. Transportation Research Part C: Emerging Technologies , 99:67-81, 2019.

Ying Zhang, Tao You, Jinchao Chen, Chenglie Du, Zhaoyang Ai, and Xiaobo Qu. Safe and energy-saving vehicle-following driving decision-making framework of autonomous vehicles. IEEE Transactions on Industrial Electronics , 69(12):13859-13871, 2021.

Mirco Rasotto, Roberto Armellin, and Pierluigi Di Lizia. Multi-step optimization strategy for fuel-optimal orbital transfer of low-thrust spacecraft. Engineering Optimization , 48(3):519-542, 2016.

Jack Langelaan. Long distance/duration trajectory optimization for small uavs. In AIAA guidance, navigation and control conference and exhibit , page 6737, 2007.

Allen M Wang, Oswin So, Charles Dawson, Darren T Garnier, Cristina Rea, and Chuchu Fan. Active disruption avoidance and trajectory design for tokamak ramp-downs with neural differential equations and reinforcement learning. arXiv preprint arXiv:2402.09387 , 2024.

Renke Huang, Yujiao Chen, Tianzhixi Yin, Xinya Li, Ang Li, Jie Tan, Wenhao Yu, Yuan Liu, and Qiuhua Huang. Accelerated deep reinforcement learning based load shedding for emergency voltage control. arXiv preprint arXiv:2006.12667 , 2020.

Christoph Rösmann, Frank Hoffmann, and Torsten Bertram. Timed-elastic-bands for timeoptimal point-to-point nonlinear model predictive control. In 2015 european control conference (ECC) , pages 3352-3357. IEEE, 2015.

Robin M Pinson and Ping Lu. Trajectory design employing convex optimization for landing on irregularly shaped asteroids. Journal of Guidance, Control, and Dynamics , 41(6):1243-1256, 2018.

Haichao Hong, Arnab Maity, and Florian Holzapfel. Free final-time constrained sequential quadratic programming-based flight vehicle guidance. Journal of Guidance, Control, and Dynamics , 44(1):181-189, 2021.

Kyle Stachowicz and Evangelos A Theodorou. Optimal-horizon model predictive control with differential dynamic programming. In 2022 International Conference on Robotics and Automation (ICRA) , pages 1440-1446. IEEE, 2022.

Damien Ernst, Mevludin Glavic, Florin Capitanescu, and Louis Wehenkel. Reinforcement learning versus model predictive control: a comparison on a power system problem. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) , 39(2):517-529, 2008.

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems , 30, 2017.

Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.09464 , 2018.

Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Keeping your distance: Solving sparse reward tasks using self-balancing shaped rewards. Advances in Neural Information Processing Systems , 32, 2019.

Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Problems and solutions. arXiv preprint arXiv:2201.08299 , 2022.

Jack Clark and Dario Amodei. Faulty reward functions in the wild, 2016.

Pulkit Agrawal. The task specification problem. In Conference on Robot Learning , pages 1745-1751. PMLR, 2022.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Gabriel B Margolis, Tao Chen, Kartik Paigwar, Xiang Fu, Donghyun Kim, Sangbae Kim, and Pulkit Agrawal. Learning to jump from pixels. arXiv preprint arXiv:2110.15344 , 2021.

Dhawal Gupta, Yash Chandak, Scott Jordan, Philip S Thomas, and Bruno C da Silva. Behavior alignment via reward function optimization. Advances in Neural Information Processing Systems , 36, 2024.

Eitan Altman. Constrained Markov decision processes . Routledge, 2004.

Lorenzo Fagiano and Andrew R Teel. Generalized terminal state constraint for model predictive control. Automatica , 49(9):2622-2631, 2013.

Yifan Wu, George Tucker, and Ofir Nachum. The laplacian in rl: Learning representations with efficient approximations. arXiv preprint arXiv:1810.04586 , 2018.

Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 6292-6299. IEEE, 2018.

Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv:1912.06088 , 2019.

Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in neural information processing systems , 31, 2018.

David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. arXiv preprint arXiv:1811.11359 , 2018.

Hao Sun, Zhizhong Li, Xiaotong Liu, Bolei Zhou, and Dahua Lin. Policy continuation with hindsight inverse dynamics. Advances in Neural Information Processing Systems , 32, 2019.

Suraj Nair and Chelsea Finn. Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. arXiv preprint arXiv:1909.05829 , 2019.

Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua B Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. arXiv preprint arXiv:2006.12122 , 2020.

Suraj Nair, Silvio Savarese, and Chelsea Finn. Goal-aware prediction: Learning to model what matters. In International Conference on Machine Learning , pages 7207-7219. PMLR, 2020.

Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Discovering and achieving goals via world models. Advances in Neural Information Processing Systems , 34:24379-24391, 2021.

Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems , 35:35603-35620, 2022.

David Fischinger, Markus Vincze, and Yun Jiang. Learning grasps for unknown objects in cluttered scenes. In 2013 IEEE international conference on robotics and automation , pages 609-616. IEEE, 2013.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Annie Xie, Avi Singh, Sergey Levine, and Chelsea Finn. Few-shot goal inference for visuomotor learning and planning. In Conference on Robot Learning , pages 40-52. PMLR, 2018.

Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. Advances in neural information processing systems , 31, 2018.

Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International conference on machine learning , pages 783-792. PMLR, 2019.

Ksenia Konyushkova, Konrad Zolna, Yusuf Aytar, Alexander Novikov, Scott Reed, Serkan Cabi, and Nando de Freitas. Semi-supervised reward learning for offline reinforcement learning. arXiv preprint arXiv:2012.06899 , 2020.

Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.08212 , 2021.

Danfei Xu and Misha Denil. Positive-unlabeled reward learning. In Conference on Robot Learning , pages 205-219. PMLR, 2021.

Konrad Zolna, Scott Reed, Alexander Novikov, Sergio Gomez Colmenarejo, David Budden, Serkan Cabi, Misha Denil, Nando de Freitas, and Ziyu Wang. Task-relevant adversarial imitation learning. In Conference on Robot Learning , pages 247-263. PMLR, 2021.

Suraj Nair, Eric Mitchell, Kevin Chen, Silvio Savarese, Chelsea Finn, et al. Learning languageconditioned robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning , pages 1303-1315. PMLR, 2022.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems , 29, 2016.

Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprint arXiv:1710.11248 , 2017.

Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning , pages 22-31. PMLR, 2017.

Chen Tessler, Daniel J Mankowitz, and Shie Mannor. Reward constrained policy optimization. arXiv preprint arXiv:1805.11074 , 2018.

Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning , pages 9133-9143. PMLR, 2020.

Aivar Sootla, Alexander I Cowen-Rivers, Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmentation. In International Conference on Machine Learning , pages 20423-20443. PMLR, 2022.

Hao Jiang, Tien Mai, Pradeep Varakantham, and Minh Huy Hoang. Solving richly constrained reinforcement learning through state augmentation and reward penalties. arXiv preprint arXiv:2301.11592 , 2023.

Hao Jiang, Tien Mai, Pradeep Varakantham, and Huy Hoang. Reward penalties on augmented states for solving richly constrained rl effectively. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 19867-19875, 2024.

Claire J Tomlin, John Lygeros, and S Shankar Sastry. A game theoretic approach to controller design for hybrid systems. Proceedings of the IEEE , 88(7):949-970, 2000.

John Lygeros. On reachability and minimum cost optimal control. Automatica , 40(6):917-927, 2004.

Ian M Mitchell, Alexandre M Bayen, and Claire J Tomlin. A time-dependent hamilton-jacobi formulation of reachable sets for continuous dynamic games. IEEE Transactions on automatic control , 50(7):947-957, 2005.

Kostas Margellos and John Lygeros. Hamilton-jacobi formulation for reach-avoid differential games. IEEE Transactions on automatic control , 56(8):1849-1861, 2011.

Somil Bansal, Mo Chen, Sylvia Herbert, and Claire J Tomlin. Hamilton-jacobi reachability: A brief overview and recent advances. In 2017 IEEE 56th Annual Conference on Decision and Control (CDC) , pages 2242-2253. IEEE, 2017.

Jaime F Fisac, Neil F Lugovoy, Vicenç Rubies-Royo, Shromona Ghosh, and Claire J Tomlin. Bridging hamilton-jacobi safety analysis and reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA) , pages 8550-8556. IEEE, 2019.

Kai Chieh Hsu, Vicenç Rubies-Royo, Claire J Tomlin, and Jaime F Fisac. Safety and liveness guarantees through reach-avoid reinforcement learning. In 17th Robotics: Science and Systems, RSS 2021 . MIT Press Journals, 2021.

Oswin So and Chuchu Fan. Solving stabilize-avoid optimal control via epigraph form and deep reinforcement learning. arXiv preprint arXiv:2305.14154 , 2023.

Milan Ganai, Zheng Gong, Chenning Yu, Sylvia Herbert, and Sicun Gao. Iterative reachability estimation for safe reinforcement learning. Advances in Neural Information Processing Systems , 36, 2024.

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Stephen P Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems , 12, 1999.

David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning , pages 387-395. Pmlr, 2014.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015.

Alex Ray, Joshua Achiam, and Dario Amodei. Benchmarking safe exploration in deep reinforcement learning. arXiv preprint arXiv:1910.01708 , 7(1):2, 2019.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012.

Steven Waslander and Carlos Wang. Wind disturbance estimation and rejection for quadrotor position control. In AIAA Infotech@ Aerospace conference and AIAA unmanned... Unlimited conference , page 1983, 2009.

Sanjeeb T Bose and George Ilhwan Park. Wall-modeled large-eddy simulation for complex turbulent flows. Annual review of fluid mechanics , 50:535-561, 2018.

Maja J Mataric. Reward functions for accelerated learning. In Machine learning proceedings 1994 , pages 181-189. Elsevier, 1994.

Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml , volume 99, pages 278-287, 1999.

Dongjie Yu, Haitong Ma, Shengbo Li, and Jianyu Chen. Reachability constrained reinforcement learning. In International Conference on Machine Learning , pages 25636-25655. PMLR, 2022.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.

A GAE estimator Definition

Note, however, that the definition of return (19) is different from the original definition and hence will result in a different equation for the GAE.

To simplify the form of the GAE, we first define a 'reduction' function ϕ ( n ) : R n → R that applies itself recursively to its n arguments, i.e.,

where

The k -step advantage function ˆ A π ( k ) ˆ g can then be written as

We can then construct the GAE ˆ A π (GAE) ˆ g as the λ k -weighted sum over the k -step advantage functions ˆ A π ( k ) ˆ g : Overall, the GAE estimator can be described as

B Equivalence of Problem 3 and Problem 13

The equivalence between the transformed Problem 13 and the original minimum-cost reach-avoid Problem 3 can be shown in the following sequence of optimization problems that yield the exact same solution if feasible:

This shows that the minimum-cost reach-avoid Problem 3 is equivalent to the formulation we solve in this work (29), which is Problem 13 in the paper. The formulation of RC-PPO solves (29) and thus also solves Problem 3 because they are equivalent.

C Optimal Reach Value Function

As shown in (18), we introduce an additional discount factor into the estimation of ˜ V π ˆ g . It will incur imprecision on the calculation of ˜ V π ˆ g defined in Definition 1. In this section, we show that for a large enough discount factor γ < 1 , we could reach unbiased ˜ z in phase two of RC-PPO .

Theorem 3. We denote max ˆ x ∈ ˆ X { ˆ g (ˆ x ) } = G max and maximal episode length T max . If there exists a positive value ϵ where

Then for any γ Tmax 1 -γ Tmax > G max ϵ , for any deterministic policy π satisfies 18. If there exists a trajectory under given policy π leading to the extended goal region ˆ G . We have

The proof for Theorem 3 is provided in Appendix D.4.

D Proofs

D.1 Proof for Theorem 1

Proof. We separately consider three elements in augmented state ( x T , y T , z T ) . First, note that 3b holds if and only if x T ∈ G . For the second element y , from the definition of the augmented dynamics 7, it holds that y T = max i ∈{ 0 ,...,T } ■ x i ∈F (30) As a result 3c holds if and only if y T = -1 . For the third element z , note that z T = z 0 -∑ T -1 k =0 c ( x k , u ( x k )) . Hence, z T ≥ 0 if and only if z 0 ≥ ∑ T -1 k =0 c ( x k , u ( x k )) .

D.2 Proof for Property 12

Proof. From Definition 12, we know

D.3 Proof for Theorem 2

We first derive the state value function in a recursive form similar as [66]

Proof.

Next, we consider unrolling ˜ V π θ g (ˆ x ′ ) under Reachability MDP in Definition 2. We define Pr(ˆ x → ˆ x † , k, π θ ) as the probability of transitioning from state ˆ x to ˆ x † in k steps under policy π θ in 2. Note that ✶ ˆ g (ˆ x ) > ˜ V π θ g (ˆ x ′ ) is absorbed using the absorbing state in 2. Then we can get

D.4 Proof for Theorem 3

Proof. Consider trajectory { ˆ x 0 , . . . , ˆ x T } where ˆ x T ∈ ˆ G . We consider the worst-case scenario where ˆ g (ˆ x t ) = g max for t ∈ { 0 , . . . , T -1 } . Then

E Convergence Guarantee on an Actor-Critic Version of Our Method

In this section, we provide the convergence proof of phase one of our method under the actor-critic framework. Notice that similar to Bellman equation (18) for ˜ V π ˆ g . We could also derive the Bellman equation for ˜ Q π ˆ g as

Next, we show our method under the actor-critic framework without GAE estimator in Algorithm 1

Algorithm 1 RC-PPO (Actor Critic)

Require: Initial policy parameter θ 0 , Q function parameter ω 0 , horizon T, convex projection operator Γ Θ , and value function learning rate β 1 ( k ) , policy learning rate β 2 ( k )

In this algorithm, the Γ Θ ( θ ) operator projects a vector θ ∈ R k to the closest point in a compact and convex set Θ ⊂ R k , i.e., Γ Θ ( θ ) = arg min θ ′ ∈ Θ ∥ θ ′ -θ ∥ 2 .

Next, we provide the convergence analysis for Algorithm 1 under the following assumptions.

Assumption 1. (Step Sizes) The step size schedules { β 1 ( k ) } and { β 2 ( k ) } have below properties:

Assumption 2. (Differentiability and and Lipschitz Continuity) For any state and action pair (ˆ x, u ) , ˜ Q ˆ g (ˆ x, u ; ω ) and π (ˆ x ; θ ) are continuously differentiable in ω and θ . Furthermore, for any state and action pair (ˆ x, u ) , ∇ ω ˜ Q ˆ g (ˆ x, u ; ω ) and π (ˆ x ; θ ) are Lipschitz function in ω and θ .

Also, we assume that X and U are finite and bounded and the horizon T is also bounded by T max , then the cost function c can be bounded by C max and g can be bounded within G max . We can limit the space of cost upper bound z ∈ [ -G max , T · C max ] instead of R . This is due to ˆ g (ˆ x ) = -z for z ≤ -G max . Next, we could do discretization on [ -G max , T · C max ] and cost function c to make the augmented state set ˆ X finite and bounded.

With the above assumptions, we can provide a convergence guarantee for Algocrithm 1.

Theorem 4. Under Assumptions 1 and 2, the policy update in Algorithm 1 converge almost surely to a locally optimal policy.

Proof. The proof follows from the proof of Theorem 2 in [75], differing only in whether an update exists for the Lagrangian multiplier. We provide a proof sketch as

First, we prove that the critic parameter almost surely converges to a fixed point ω ∗ .

This step is guaranteed by the assumption of finite and bounded state and action set and Assumption 1. The γ -contraction property of the following operator

is also proved in Lemma B.1 in [75] to make sure the convergence of the first step.

Second, due to the fast convergence of ω ∗ , we can show policy paramter θ converge almost surely to a stationary point θ ∗ which can be further proved to be a locally optimal solution.

We refer to [75] for proof details.

F Implementation Details of Algorithms

In this section, we will provide more details about CMDP-based baselines (different between optimization goal with multiple constraints) and other hyperparameter settings like X threshold .

F.1 CMDP-based Baselines

In this section, we will clarify the optimization target for CPPO and RESPO under CMDP formulation of both hard and soft constraints. Recall that our formulation of CMDP is

We then denote

The optimization goal formulation for CPPO is as follows:

In this formulation, the soft constraint V π c has the same priority as the hard constraint V π f . This leads to a potential imbalance between soft constraints and hard constraints. Instead, the optimization goal for RESPO is as follows:

where p ( x ) denotes the probability of entering the unsafe region F start from state x . It is called the reachability estimation function (REF). This formulation prioritizes the satisfaction of hard constraints but still suffers from balancing soft constraints and reward terms.

F.2 Hyperparameters

We first clarify how we set proper X threshold for each environment. First, we will run our method RC-PPO and calculate the average cost, we denote it as c average . Weset X low = c average 10 , X medium = c average 3 and X high = c average . For static lagrangian multiplier β , we set β lo = 0 . 1 and β hi = 10 . Also, we set C fail = 20 in every environment.

Note that CRL is an off-policy algorithm, while RC-PPO and other baselines are on-policy algorithms. We provide Table 1 showing hyperparameters for on-policy algorithms and Table 2 showing hyperparameters for off-policy algorithm (CRL).

F.3 Implementation of the baselines

The implementation of the baseline follows their original implementations:

RESPO: https://github.com/milanganai/milanganai.github.io/tree/main/NeurIPS2023/code (No license)

CRL: https://github.com/google-research/google-research/tree/master/ contrastive_rl (No License)

G Experiment Details

In this section, we provide more details about the benchmarks and the choice of reward function r , g , cost function c and C cost in each environment. Under the sparse reward setting, we apply the following structure of reward design r ( x t , u t , x t +1 ) = R goal × ✶ x t +1 ∈G where R goal is an constant. After doing reward shaping, we add an extra term γϕ ( x t +1 ) -ϕ ( x t ) and the reward becomes

where γ denotes the discount factor.

Note that we set R goal = C cost = 20 in all the environments. Note that if there is a gap between max { g ( x ) | g ( x ) < 0 } , we could get unbiased ˜ z during phase two of RC-PPO guaranteed by Theorem 3. To achieve better performance in phase two of RC-PPO, we set

for all x ∈ G to maintain such a gap. Also, we implement all the environments in Jax [76] for better scalability and parallelization.

G.1 Pendulum

The Pendulum environment is taken from Gym [18] and the torque limit is set to be 1 . The state space is given by x = [ θ, ˙ θ ] where θ ∈ [ -π, π ] , ˙ θ ∈ [ -8 , 8] . In this task, we do not consider unsafe regions and set

where dt = 0 . 05 is the time interval during environment simulation. This is for preventing environment overshooting during simulation.

In the Pendulum environment, cost function c is given by

for better visualization of policies with different energy consumption. g is given by

G.2 Safety Hopper

The Safety Hopper environment is taken from Safety Mujoco, we add static obstacles in the environment to increase the difficulty of the task. We use x to denote the x-axis position of the head of Hopper, y to be the y-axis position of the head of Hopper. Then the goal region can be described as

The unsafe set is described as

We use ˜ x thigh , ˜ x leg , ˜ x foot to denote the angular velocity of the thigh, leg, foot hinge. The cost function is described as

where g is given by

G.3 Safety HalfCheetah

The Safety HalfCheetah environment is taken from Safety Mujoco, we add static obstacles in the environment to increase the difficulty of the task. We use x front to denote the x-axis position of the front foot of Halfcheetah, y front to be the y-axis position of the back foot of Halfcheetah, x back to denote the x-axis position of the back foot of Halfcheetah, y back to be the y-axis position of the back foot of Halfcheetah, x head to denote the x-axis position of the head of Halfcheetah, y head to be the y-axis position of the head of Halfcheetah. Then the goal region can be described as

The unsafe set is described as

The cost function is described as

g is given by

G.4 FixedWing

FixedWing environment is taken from [59] and we follow the same design of F as [59]. We denote the x PE as the eastward displacement of F16 with given state x . Then the goal region G is given by

G.5 Quadrotor in Wind Field

We take quadrotor dynamics from crazyflies and wind field environments in the urban area from [71]. The wind field will disturb the quadrotor with extra movement on both x -axis and y -axis. There are static building obstacles in the environment and we treat them as the unsafe region F . The goal for the quadrotor is to reach the mid-point of the city. We divide the whole city into four sections and train single policy on each of the sections. We use x ∈ [ -30 , 30] to denote the x-axis position of quadrotor, y ∈ [ -30 , 30] to be the y-axis position of quadrotor.

The cost c is given by and g is given by

The cost c is given by

g is given by

G.6 PointGoal

The PointGoal environment is taken from Safety Gym [69] We implement PointGoal environments in Jax. In Safety Gym environment, we do not perform reward-shaping and use the original reward defined in Safety Gym environments. In this case, the distance reward is set also to be 20 in order to align* with C goal and C cost . Different from sampling outside the hazard region which is implemented in Safety Gym, we allow Point to be initialized within the hazard region. We use x to denote the x-axis position of Point, y to be the y-axis position of Point, x goal to denote the x-axis position of Goal, and y goal to denote the y-axis position of Goal. The goal region is given by

The cost c is given by

g is given by

G.7 Experiment Harware

We run all our experiments on a computer with CPU AMD Ryzen Threadripper 3970X 32-Core Processor and with 4 GPUs of RTX3090. It takes at most 4 hours to train on every environment.

H Additional Experiment Results

We put additional experiment results in this section.

H.1 Additional Cumulative Cost and Reach Rates

We show the cumulative cost and reach rates of the final converged policies for additional environments ( F16 and Safety Hopper ) in Figure 7.

80

160

240

CPPO_



H

CPPO_



M

SAC_

β

L

PPO_

β

L

RESPO

RC-PPO

Additional Cumulative Cost ↓

0.0

0.2

0.4

0.6

0.8

RC-PPO

RESPO

PPO_

β

L

PPO_

β

H

SAC_

β

L

SAC_

β

H

CPPO_



L

CPPO_



M

CPPO_



H

CRL

Reach Rate ↑

(a)

FixedWing

100

200

300

400

CRL

CPPO_



H

CPPO_



M

SAC_

β

L

PPO_

β

L

RESPO

RC-PPO

Additional Cumulative Cost ↓

0.0

0.2

0.4

0.6

0.8

1.0

RC-PPO

RESPO

PPO_

β

L

PPO_

β

H

SAC_

β

L

SAC_

β

H

CPPO_



L

CPPO_



M

CPPO_



H

CRL

Reach Rate ↑

(b)

Safety

Hopper

H.2 Visualization of learned policy for different z

To obtain better intuition for how the learned policy depends on z , we rollout the policy choices of z 0 in the Pendulum environment and visualize the results in Figure 8.

0

π

2

π

θ

-5.0

-2.5

0.0

2.5

5.0

ω

z

opt-300

0

π

2

π

θ

-7.5

-5.0

-2.5

0.0

2.5

5.0

ω

z

opt-200

0

π

2

π

θ

-5

0

5

ω

z

opt-100

0

π

2

π

θ

-5

0

5

ω

z

opt

0

π

2

π

θ

-5

0

5

ω

z

opt+100

0

π

2

π

θ

-5

0

5

ω

z

opt+200

0

π

2

π

θ

-5

0

5

ω

z

opt+300

0

100

200

300

400

500

Timestep

0

100

200

300

Cumulative

Cost

z

opt-300

z

opt-200

z

opt-100

z

opt

z

opt+100

z

opt+200

z

opt+300

H.3 Grid search

We perform an extensive grid search over different reward coefficients for the baseline PPO method and plotted the Pareto front across the reach rate and cost in Figure 6. The reward we use is r = R goal × ✶ x ∈G -P goal × ✶ x ̸∈G -βc ( x, u ) . and we search over the Cartesian product of R goal = { 2 , 20 , 200 , 2000 , 20000 } , P goal = { 1 , 10 , 100 , 1000 , 10000 } , β = { 0 . 1 , 1 , 10 } .

H.4 Performance with external noise

We also performed additional experiments to illustrate the performance of RC-PPO under a changing environment. Specifically, we add uniform noise to the output of the learned policy and see what happens in the Pendulum environment.

We first compare the reach rates of the different methods in Table 3. In this environment, we see that noise does not affect the reach rate too much.

Next, we look at how the cumulative cost changes with noise by comparing methods with a near 100% reach rate in Table 4. Unsurprisingly, larger amounts of noise reduce the performance of almost all policies. Even with the added noise, RC-PPO uses the least cumulative cost compared to all other methods.

I Discussion on Limitation of CMDP-Based Algorithms

In this section, we will use an example to illustrate further why CMDP-based algorithms won't solve the minimum-cost reach-avoid problem optimally compared with our method. We focus on two parts of CMDP formulation:

Weight coefficient assigned to different objectives

Threshold assigned to each constraint

A

G2

B

G3

I

𝑝𝑝

𝐴𝐴

1

-𝑝𝑝𝐴𝐴

𝑝𝑝

𝐵𝐵

1

-𝑝𝑝𝐵𝐵

𝐶𝐶

= 10

𝐶𝐶

= 20

𝐶𝐶

= 30

𝐶𝐶

= 0

Consider the minimum-cost reach-avoid problem shown in Figure 9, where we use C to denote the cost. States A and B are two initial states with the same initial distribution probability. State G 1 , G 2 , and G 3 are three goal states. State I is the absorbing state (non-goal). We use p A and p B to denote the policy parameter, which represents the probability of choosing left action on state A and B separately.

The optimal policy for this minimum-cost reach-avoid problem is to take the left action from both A and B , i.e., p A = p B = 1 , which gives an expected cost of

To convert this into a multi-objective problem, we introduce a reward that incentivizes reaching the goal as follows (we use R to denote reward):

This results in the following multi-objective optimization problem:

I.1 Weight assignment

We first consider solving multi-objective optimization Problem 34 by assigning weights on different objectives. We introduce w ≥ 0 , giving

Solving the scalarized Problem 34 gives us the following solution as a function of w :

p A = ✶ ( w ≥ 1) , p B = ✶ ( w ≤ 2 3 ) (36) Notice that the *true* optimal solution of p A = p B = 1 is NOT an optimal solution to the original minimum-cost reach-avoid problem shown in Figure 9 under any w .

Hence, the optimal solution of the surrogate multi-objective problem can be suboptimal for the original minimum-cost reach-avoid problem under any weight coefficients.

Of course, this is just one choice of reward function where the optimal solution of the minimum-cost reach-avoid problem cannot be recovered. Given knowledge of the optimal policy, we can construct the reward such that the multi-objective optimization problem does include the optimal policy as a solution. However, this is impossible to do if we do not have prior knowledge of the optimal policy, as is typically the case.

I.2 Threshold assignment

Next, we consider solving multi-objective optimization Problem 34 by assigning a threshold X _thresh on the cost constraint:

The optimal solution to this CMDP can be solved to be

However, the true optimal solution of p A = p B = 1 is NOT an optimal solution to the CMDP. To see this, taking X _thresh = 20 , the real optimal solution p A = p B = 1 gives a reward of R = 15 , but the CMDP solution p A = 0 , p B = 20 -10 15 = 2 3 gives R = 23 . 33 > 15 . Moreover, any uniform scaling of the rewards or costs does not change the solution.

We can "fix" this problem if we choose the rewards to be high only along the optimal solution p A = p B = 1 , but this requires knowledge of the optimal solution beforehand and is not feasible for all problems.

Another way to "fix" this problem is if we consider a "per-state" cost threshold, e.g.,

Choosing exactly the cost of the optimal policy, i.e., X A = 10 and X B ≥ 30 , also recovers the optimal solution of p A = p B = 1 . This now requires knowing the smallest cost to reach the goal for every state, which is difficult to do beforehand and not feasible. On the other hand, RC-PPO does exactly this in the second phase when optimizing for z 0 . We can thus interpret RC-PPO as automatically solving for the best cost threshold to use as a constraint for every initial state.

J Broader impact

Our proposed algorithm solves an important problem that is widely applicable to many different real-world tasks including robotics, autonomous driving, and drone delivery. Solving this brings us one step closer to more feasible deployment of these robots in real life. However, the proposed algorithm requires GPU training resources, which could contribute to increased energy usage.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main contributions of the paper are summarized in the Introduction section as well as in the abstract of the paper, describing the proposed method briefly as well as its advantages compared to the existing methods.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, we have discussed about the limitation of the work in Section 6

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes. The completed proofs are provided in the Appendix (see Appendix C for example)

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes. We provide details on all experiments and baselines used. Moreover, we provide the code used to reproduce our results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Yes, the code used for generating the results in the paper has been provided.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes. The full details are provided in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Yes, we report confidence intervals in our plots. See Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Yes, all the required computational resources used for generating the results are provided with the experiment details.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, the research conducted in this paper adheres to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The license of our baselines are mentioned in Appendix F.3.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: we include the code of the algorithm in the supplementary materials. Implementation details are introduced in Section 4 and Appendix F.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe papers coinsiders a reach-avoid problem under cost minimization objective, where a CMDP has to be solved under the goal and avoid constraints (the final state has to be in the certain set, and the trajectories must avoid an unsafe region). The authors derive an optimal control problem under these constraints and show how to solve it in the RL setting. The results show an improvement in terms of the optimized objective in comparison with method that include the goal objective in the optimization cost.\n\nSTRENGTHS:\nThe paper delivers well the proposed solution and the state augmentation technique that the authors propose seem to be crucial to the solution. The algorithm does outperform the competitors, where the goal is included in the objective.\n\nWEAKNESSES:\n1. The state augmentation is not a novel technique in constrained RL, (see references below) albeit the application here is different.\n1. I am not fully convinced that this is actually an important sub-problem in RL to solve. Can the authors elaborate why we should consider this problem in more detail? For instance, can we formulate safety gym benchmarks as this?\n1. Some of the design choices are not fully clear, why the authors minimize the initial state z_0 with the constraint z_t >=0. Wouldn't it be easier to track the accumulated cost and minimize the terminal state?\n1. I am not fully convinced that there are no simpler methods to solve this problem. For example, for the baselines, the authors reshape the reward by giving a modest reward for goal reaching. I’d like to see experiments with heavy penalization of not reaching the goal (for example, a scale from -10 to -10000) and different rewards for reaching the goal (not just one 20). \n\n\n* [Sootla et al 22] Sootla, Aivar, et al. \"Sauté rl: Almost surely safe reinforcement learning using state augmentation.\" International Conference on Machine Learning. PMLR, 2022.\n\n* [Jiang et al 23] Jiang, Hao, et al. \"Solving Richly Constrained Reinforcement Learning through State Augmentation and Reward Penalties.\" arXiv preprint arXiv:2301.11592 (2023).\n\n* [Jiang et al 24] Jiang, Hao, et al. \"Reward Penalties on Augmented States for Solving Richly Constrained RL Effectively.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 18. 2024.\n\nQUESTIONS:\n1. I appreciate it’s a personal preference, but I don’t see the need to formulate the optimal control problem as cost minimization. Furthermore, in safe RL the costs typically refer to constraints.\n2. Line 33 “Moreover, the use of nonlinear numerical optimization may result in poor solutions that lie in suboptimal local minima [14].” How does the paper address this issue? Wouldn’t any hard problem have the same problem?\n3. Line 38 “However, posing the reach constraint as a reward then makes it difficult to  optimize for the cumulative cost at the same time.”. I think adding a penalty for not reaching the goal will address this issue. Recall that in RL the rewards need not to be differentiable. \n4. Line 45. “However, the choice of  this fixed threshold becomes key: too small and the problem is not feasible, destabilizing the training  process. Too large, and the resulting policy will simply ignore the cumulative cost.” - I am not sure this statement is necessary. Problem statement is a bit of an art and the threshold is chosen on the problem design level. Sometimes it is simply given.\n\nLIMITATIONS:\nlimitations are discussed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces RC-PPO, an RL algorithm designed to solve the minimum-cost reach-avoid problem by reformulating the optimization problem on an augmented system. The paper addresses the limitations of current RL that mostly solve surrogate problems to fit to the problem setting. Furthermore, a comprehensive analysis, including theoretical foundations, algorithmic details, and empirical validation is presented. Experimental comparison with respect to existing methods is also provided.\n\nSTRENGTHS:\nThe paper (including the appendix) provides solid theoretical foundation, including proofs and detailed explanations of the key concepts and assumptions.\nThe paper is overall well-written, definitions and theorems are formulated clearly. The main thread of the paper can be followed.\nThe presented experimental results are well illustrated and generally the claims based on the results are comprehensible. \nIn addition to solid theoretical arguments, the appendix also provides vast implementational details and further experimental results.\nThe presented idea seems to offer an elegant solution to minimum-cost reach-avoid problems. \n\nOverall a nice and insightful read.\n\nWEAKNESSES:\n(minor)\nThe presented experimental results are solid and illustrative, but could be extended. \nThe extra title of Figure 3 is slightly irritating.\nFigure 3 and Figure 4 use different axis labeling logic.\nAppendix G.2 is empty (- a formatting problem?).\nThere are typos, missing spaces, missing punctuation, repeated words and repeated phrases in the paper, e.g., L99, L186, L187, L193, L200, L206, L210+L225, L215, L232, L570.\n\nQUESTIONS:\nIn Section 5, the paper claims that RC-PPO remains competitive which is based on Figure 6. Can you either elaborate how this claim can be made from Figure 6, or maybe it should be Figure 4?\n\nLIMITATIONS:\nLimitations are briefly discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Reach Constrained Proximal Policy Optimization which targets to solve the minimum-cost reach-avoid problem. The authors first convert the reach-avoid problem to a reach problem on an augmented system and use the corresponding reach value function to compute the optimal policy. Next, The authors use a novel two-step PPO-based RL-based framework to learn this value function and the corresponding optimal policy.\n\nSTRENGTHS:\n1.The studied problem is interesting\n2.This paper is easy to follow\n\nWEAKNESSES:\n1.What is the motivation for using two-phase PPO to solve this problem? Providing performance comparisons with other RL algorithms, such as TD3 and SAC, will significantly strengthen the rationale behind proposing this approach.\n\n2.To verify the statement in the abstract, \"which leads to suboptimal policies that do not directly minimize the cumulative cost,\" the authors should compare the performance of RC-PPO with other multi-objective optimization algorithms.\n\n\n3. Please reconsider whether it is reasonable to reformulate the minimum-cost reach-avoid problem by constructing an augmented system, as the limitation that \"two policies that are both unable to reach the goal can have the same value even if one is unsafe\" is undesirable.\n\nQUESTIONS:\n1.What is the motivation for using two-phase PPO to solve this problem? Providing performance comparisons with other RL algorithms, such as TD3 and SAC, will significantly strengthen the rationale behind proposing this approach.\n\n2.To verify the statement in the abstract, \"which leads to suboptimal policies that do not directly minimize the cumulative cost,\" the authors should compare the performance of RC-PPO with other multi-objective optimization algorithms.\n\nLIMITATIONS:\n3. Please reconsider whether it is reasonable to reformulate the minimum-cost reach-avoid problem by constructing an augmented system, as the limitation that \"two policies that are both unable to reach the goal can have the same value even if one is unsafe\" is undesirable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}