SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Step 3 (convergence of REF  $p_{\xi}$  updates): Since  $\omega$  is updated on a slower time scale than  $\xi$ , we can again treat  $\omega$  as a fixed parameter at  $\omega_{k}$  when updating  $\xi$ . Furthermore, in Time scale 3, we know that the policy has converged to a local minimum, particularly  $||\theta_{k} - \theta^{*}(\xi_{k},\omega_{k})|| = 0$ . Now the bellman operator for REF is defined by
$$
\mathcal{B}_{p}[p](s) = \max \bigl\{\mathbb{1}_{s\in S_{v}},\gamma \underset {s^{\prime}\sim \pi ,P(s)}{\mathbb{E}}[p(s^{\prime})]\bigr \} .
$$
We demonstrate this is a  $\gamma$  contraction mapping as follows:
$$
\begin{array}{l} \left| \mathcal {B} _ {p} [ p ] (s) - \mathcal {B} _ {p} [ \hat {p} ] (s) \right| \\ = \left| \max  \left\{\mathbb {1} _ {s \in S _ {v}}, \gamma \underset {s ^ {\prime} \sim \pi , P (s)} {\mathbb {E}} \left[ p \left(s ^ {\prime}\right) \right] \right\} - \max  \left\{\mathbb {1} _ {s \in S _ {v}}, \gamma \underset {s ^ {\prime} \sim \pi , P (s)} {\mathbb {E}} \left[ \hat {p} \left(s ^ {\prime}\right) \right] \right\} \right| \\ \leq \left| \gamma \underset {s ^ {\prime} \sim \pi , P (s)} {\mathbb {E}} \left[ p \left(s ^ {\prime}\right) \right] - \gamma \underset {s ^ {\prime} \sim \pi , P (s)} {\mathbb {E}} \left[ \hat {p} \left(s ^ {\prime}\right) \right] \right| \\ = \gamma | _ {s ^ {\prime} \sim \pi , P (s)} \mathbb {E} [ p (s ^ {\prime}) - \hat {p} (s ^ {\prime}) ] | \\ \leq \gamma \sup  _ {s} | p (s) - \hat {p} (s) | = \gamma | | p - \hat {p} | | _ {\infty} \\ \end{array}
$$
So we can say that  $p(s; \xi_k)$  will converge to  $p(s; \xi^*)$  as  $k \to \infty$  under the same assumptions of the Finite MDP and function approximator expressiveness in Step 1. Therefore,  $\pi_{\theta_k}$  will also converge to  $\pi^\diamond = \pi_{\theta^*(\xi^*, \omega_k)}$  as  $k \to \infty$ . And because  $\pi_\theta$  is the sampling policy used to compute  $p$ ,  $p(s; \xi^*) = p^{\pi_{\theta^*}(\xi^*, \omega_k)}(s; \xi^*) = p^\diamond(s)$ .
Notice that  $\pi^{\diamond}$  is a locally minimum optimal policy for the following optimization (recall  $\lambda_{\omega}$  is treated as constant in this timescale):
$$
\min _ {\pi} \underset {s \sim d _ {0}} {\mathbb {E}} \underset {a \sim \pi (\cdot | s)} {\mathbb {E}} \left[ - Q ^ {\pi} (s, a) \cdot [ 1 - p ^ {\diamond} (s) ] + Q _ {c} ^ {\pi} (s, a) \cdot [ (1 - p ^ {\diamond} (s)) \lambda_ {\omega} + p ^ {\diamond} (s) ] \right]
$$
and therefore also locally minimum optimal policy for optimization:
$$
\begin{array}{l} \min  _ {\pi} \mathbb {E} _ {s \sim d _ {0}} \mathbb {E} _ {a \sim \pi (\cdot | s)} \left[ - Q ^ {\pi} (s, a) + Q _ {c} ^ {\pi} (s, a) \cdot \left[ \lambda_ {\omega} + \frac {p ^ {\diamond} (s)}{(1 - p ^ {\diamond} (s))} \right] \right], \text {i f} p ^ {\diamond} (s) > 0 \\ \underset {s \sim d _ {0}} {\mathbb {E}} \min _ {\pi} \underset {a \sim \pi (\cdot | s)} {\mathbb {E}} \left[ Q _ {c} ^ {\pi} (s, a) \right], \mathrm {i f} p ^ {\diamond} (s) = 0 \\ \end{array}
$$
Since  $\frac{p^{\diamond}(s)}{(1 - p^{\diamond}(s))} \geq 0$ , and the  $Q$  functions are always nonnegative, we can know that  $\pi^{\diamond}$  is at least as safe as (i.e., its expected cumulative cost is at most that of) a locally optimal policy for the optimization:
$$
\min  _ {\pi} \mathbb {E} _ {s \sim d _ {0}} \mathbb {E} _ {a \sim \pi (\cdot | s)} \left[ - Q ^ {\pi} (s, a) + Q _ {c} ^ {\pi} (s, a) \lambda_ {\omega} \right] \tag {8}
$$
As  $\lambda_{\omega}$  approaches  $\lambda_{\mathrm{max}}$ , which in turn approaches  $\infty$ , the local minimum optimal policies of Equation 8 approach those of the optimization  $\pi^{\triangle} = \arg \min_{\pi} \mathbb{E}_{s \sim d_0} \mathbb{E}_{a \sim \pi(\cdot|s)} Q_c^\pi(s, a) \lambda_{\omega} = \arg \min_{\pi} \mathbb{E}_{s \sim d_0} \mathbb{E}_{a \sim \pi(\cdot|s)} Q_c^\pi(s, a)$ . Therefore, the feasible set of the REF  $p^\diamond$  will approach that of the REF  $p^{\pi^{\triangle}}$ .
Step 4 (convergence of lagrange multiplier  $\lambda_{\omega}$  update): Since  $\lambda_{\omega}$  is on the slowest time scale, we have that  $||\theta_k - \theta^* (\omega)|| = 0$ $||\xi_k - \xi^* (\omega)|| = 0$  , and  $||Q_c(s,a;\kappa_k) - Q_c^{\pi_\theta_k}(s,a)|| = 0$  almost surely. Furthermore, due to the continuity of  $\nabla_{\omega}L(\theta ,\xi ,\omega)$  , we have that  $||\nabla_{\omega}L(\theta ,\xi ,\omega)|_{\theta = \theta_k,\xi = \xi_k,\omega = \omega_k} -$ $\nabla_{\omega}L(\theta ,\xi ,\omega)|_{\theta = \theta^{*}(\omega_{k}),\xi = \xi^{*}(\omega_{k}),\omega = \omega_{k}}| = 0$  almost surely. The update of the multiplier using the gradient for Equation is:
$$
\begin{array}{l} \omega_ {k + 1} = \Gamma_ {\Omega} [ \omega_ {k} + \zeta_ {4} (k) (\nabla_ {\omega} L (\theta , \xi , \omega) | _ {\theta = \theta_ {k}, \xi = \xi_ {k}, \omega = \omega_ {k}}) ] \\ = \Gamma_ {\Omega} \left[ \omega_ {k} + \zeta_ {4} (k) \left(Q _ {c} \left(s _ {t}, a _ {t}; \kappa_ {k}\right) [ 1 - p \left(s _ {t}; \xi_ {k}\right) ] \nabla_ {\omega} \lambda_ {\omega} | _ {\omega = \omega_ {k}} \right] \right. \\ = \Gamma_ {\Omega} \left[ \omega_ {k} + \zeta_ {4} (k) \left(\nabla_ {\omega} L (\theta , \xi , \omega) \mid_ {\theta = \theta^ {*} \left(\omega_ {k}\right), \xi = \xi^ {*} \left(\omega_ {k}\right), \omega = \omega_ {k}} + \delta \omega_ {k + 1}\right) \right] \\ \end{array}
$$
$$
\begin{array}{l} \delta \omega_ {k + 1} = - \nabla_ {\omega} L (\theta , \xi , \omega) | _ {\theta = \theta^ {*} (\omega_ {k}), \xi = \xi^ {*} (\omega_ {k}), \omega = \omega_ {k}} + Q _ {c} \left(s _ {t}, a _ {t}; \kappa_ {k}\right) [ 1 - p \left(s _ {t}; \xi_ {k}\right) ] \nabla_ {\omega} \lambda_ {\omega} | _ {\omega = \omega_ {k}} \\ = - \sum_ {s _ {i}, a _ {i}} d _ {0} (s _ {0}) P ^ {\pi_ {\theta_ {k}}} (s _ {i}, a _ {i} | s _ {0}) [ Q _ {c} ^ {\pi_ {\theta^ {*}}} (s _ {i}, a _ {i}) [ 1 - p _ {\xi^ {*}} (s _ {i}) ] \nabla_ {\omega} \lambda_ {\omega} | _ {\omega = \omega_ {k}} ] \\ + Q _ {c} \left(s _ {t}, a _ {t}; \kappa_ {k}\right) \left[ 1 - p \left(s _ {t}; \xi_ {k}\right) \right] \nabla_ {\omega} \lambda_ {\omega} | _ {\omega = \omega_ {k}} \\ = - \sum_ {s _ {i}, a _ {i}} d _ {0} (s _ {0}) P ^ {\pi_ {\theta_ {k}}} (s _ {i}, a _ {i} | s _ {0}) [ Q _ {c} ^ {\pi_ {\theta^ {*}}} (s _ {i}, a _ {i}) [ 1 - p _ {\xi^ {*}} (s _ {i}) ] \nabla_ {\omega} \lambda_ {\omega} | _ {\omega = \omega_ {k}} ] \\ + \left[ Q _ {c} \left(s _ {t}, a _ {t}; \kappa_ {k}\right) \left[ 1 - p \left(s _ {t}; \xi_ {k}\right) \right] - Q _ {c} ^ {\pi_ {\theta_ {k}}} \left(s _ {t}, a _ {t}\right) \left[ 1 - p \left(s _ {t}; \xi_ {k}\right) \right] + \right. \\ Q _ {c} ^ {\pi_ {\theta_ {k}}} \left(s _ {t}, a _ {t}\right) \left[ 1 - p \left(s _ {t}; \xi_ {k}\right) \right] - Q _ {c} ^ {\pi_ {\theta_ {k}}} \left(s _ {t}, a _ {t}\right) \left[ 1 - p ^ {\diamond} \left(s _ {t}\right) \right] + \\ Q _ {c} ^ {\pi_ {\theta_ {k}}} \left(s _ {t}, a _ {t}\right) \left[ 1 - p ^ {\diamond} \left(s _ {t}\right) \right] ] \nabla_ {\omega} \lambda_ {\omega} | _ {\omega = \omega_ {k}} \\ = - \sum_ {s _ {i}, a _ {i}} d _ {0} (s _ {0}) P ^ {\pi_ {\theta_ {k}}} (s _ {i}, a _ {i} | s _ {0}) [ Q _ {c} ^ {\pi_ {\theta^ {*}}} (s _ {i}, a _ {i}) [ 1 - p _ {\xi^ {*}} (s _ {i}) ] \nabla_ {\omega} \lambda_ {\omega} | _ {\omega = \omega_ {k}} ] \\ + \left[ \left(Q _ {c} \left(s _ {t}, a _ {t}; \kappa_ {k}\right) - Q _ {c} ^ {\pi_ {\theta_ {k}}} \left(s _ {t}, a _ {t}\right)\right) \left[ 1 - p \left(s _ {t}; \xi_ {k}\right) \right] + \right. \\ Q _ {c} ^ {\pi_ {\theta_ {k}}} (s _ {t}, a _ {t}) [ p ^ {\diamond} (s _ {t}) - p (s _ {t}; \xi_ {k}) ] + \\ Q _ {c} ^ {\pi \theta_ {k}} \left(s _ {t}, a _ {t}\right) \left[ 1 - p ^ {\diamond} \left(s _ {t}\right) \right] ] \nabla_ {\omega} \lambda_ {\omega} | _ {\omega = \omega_ {k}} ] \\ \end{array}
$$
Now, just as in the  $\theta$  update convergence, we can demonstrate the following lemmas:
Lemma 4:  $\delta \omega_{k + 1}$  is square integrable since
$$
\mathbb {E} [ | | \delta \omega_ {k + 1} | | ^ {2} | \mathcal {F} _ {\omega , k} ] \leq 2 \cdot \frac {H _ {\max}}{1 - \gamma} \cdot 1 \cdot K _ {3} (1 + | | \omega_ {k} | | _ {\infty} ^ {2}) <   \infty
$$
for some large Lipschitz constant  $K_{3}$ . Note that  $\mathcal{F}_{\omega ,k} = \sigma (\omega_m,\delta \omega_m,m\leq k)$  is the filtration for  $\omega_{k}$  generated by different independent trajectories [8].
Lemma 5: Because  $||Q_{c}(s_{t},a_{t};\kappa_{k}) - Q_{c}^{\pi_{\theta_{k}}}(s_{t},a_{t})||_{\infty}\to 0$  and  $||p^{\diamond}(s_t) - p(s_t;\xi_k)||_{\infty}\to 0$  and  $Q_{c}^{\pi_{\theta_{k}}}(s_{t},a_{t})[1 - p_{\xi^{*}}(s_{t})]\nabla_{\omega}\lambda_{\omega}|_{\omega = \omega_{k}}$  is a sample of  $Q_{c}^{\pi_{\theta^{*}}}(\boldsymbol {s}_i,\boldsymbol {a}_i)[1 - p_{\xi^*}(\boldsymbol {s}_i)]\nabla_\omega \lambda_\omega |_\omega = \omega_k$ , we conclude that  $\mathbb{E}[\delta \omega_{k + 1}|\mathcal{F}_{\omega ,k}] = 0$  almost surely.
Thus, the lagrange multiplier  $\omega$  update is a stochastic approximation of a continuous system  $\omega(t)$  defined by [10]
$$
\dot {\omega} = \Upsilon_ {\Omega} \left[ - \nabla_ {\omega} L (\theta , \xi , \omega) | _ {\theta = \theta^ {*} (\omega), \xi = \xi^ {*} (\omega)} \right] \tag {9}
$$
with Martingale difference error of  $\delta \omega_{k}$  and where  $\Upsilon_{\Omega}$  is the left direction derivative defined similar to that in Time scale 2 of the convergence of  $\theta$  update. Using Step 2 in Appendix A.2 from [8], we have that  $dL(\theta^{*}(\omega),\xi^{*}(\omega),\omega) / dt = \nabla_{\omega}L(\theta ,\xi ,\omega)|_{\theta = \theta^{*}(\omega),\xi = \xi^{*}(\omega)}^{T}\cdot \Upsilon_{\Omega}[\nabla_{\Omega}L(\theta ,\xi ,\omega)|_{\theta = \theta^{*}(\omega),\xi = \xi^{*}(\omega)}]\geq 0$  and the value is non-zero if  $||\Upsilon_{\Omega}[\nabla_{\omega}L(\theta ,\xi ,\omega)|_{\theta = \theta^{*}(\omega),\xi = \xi^{*}(\omega)}||\neq 0.$
For a local maximum point  $\omega^{*}$ , define a Lyapunov function as
$$
\mathcal {L} (\omega) = L \left(\theta^ {*} (\omega), \xi^ {*} (\omega), \omega^ {*}\right) - L \left(\theta^ {*} (\omega), \xi^ {*} (\omega), \omega\right)
$$
Then there exists a ball centered at  $\omega^{*}$  with a radius  $\rho^{\prime}$  such that  $\forall \omega \in \mathfrak{B}_{\omega^{*}}(\rho^{\prime}) = \{\omega ||\omega -$ $\omega^{*}||\leq \rho^{\prime}\}$ ,  $\mathcal{L}(\omega)$  is a locally positive definite function, that is  $\mathcal{L}(\omega)\geq 0$ . Also,  $d\mathcal{L}(\omega (t)) / dt = -dL(\theta^{*}(\omega),\xi^{*}(\omega),\omega) / dt\leq 0$  and is equal only when  $\Upsilon_{\Omega}[\nabla_{\omega}L(\theta ,\xi ,\omega)|_{\theta = \theta^{*}(\omega)},\xi = \xi^{*}(\omega)] = 0$ , so therefore  $\omega^{*}$  is a stationary point. By leveraging Lyapunov theory for asymptotically stable systems presented in Chapter 4 of [55] we can demonstrate that for any initial conditions of  $\omega \in \mathfrak{B}_{\omega^{*}}(\rho^{\prime})$ , the continuous state trajectory of  $\omega (t)$  converges to the locally maximum point  $\omega^{*}$ .
Using these aforementioned properties, as well as the facts that 1)  $\nabla_{\omega}L(\theta^{*}(\omega),\xi^{*}(\omega),\omega)$  is a Lipschitz function, 2) the step-sizes of Assumption on steps sizes, 3)  $\{\omega_{k + 1}\}$  is a stochastic approximation of  $\omega (t)$  with a Martingale difference error, and 4) convex and compact properties in projections used, we can use Theorem 2 of chapter 6 in [10] to demonstrate the sequence  $\{\omega_k\}$  converges almost surely to a locally maximum point  $\omega^{*}$  almost surely, that is  $L(\theta^{*}(\omega),\xi^{*}(\omega),\omega^{*})\geq L(\theta^{*}(\omega),\xi^{*}(\omega),\omega)$ .
From Time scales 2 and 3 we have that  $L(\theta^{*}(\omega), \xi^{*}(\omega), \omega) \leq L(\theta, \xi, \omega)$  while from Time scale 4 we have that  $L(\theta^{*}(\omega), \xi^{*}(\omega), \omega^{*}) \geq L(\theta^{*}(\omega), \xi^{*}(\omega), \omega)$ . Thus,  $L(\theta^{*}(\omega), \xi^{*}(\omega), \omega) \leq L(\theta^{*}(\omega), \xi^{*}(\omega), \omega^{*}) \leq L(\theta, \xi, \omega^{*})$ . Therefore,  $(\theta^{*}, \xi^{*}, \omega^{*})$  is a local saddle point of  $(\theta, \xi, \omega)$ . Invoking the saddle point theorem of Proposition 5.1.6 in [54], we can conclude that  $\pi(\cdot | \cdot; \theta^{*})$  is a locally optimal policy for our proposed optimization formulation.
C.4.4 Remark on Bounding Lagrange Multiplier
We can say our algorithm learns an REF that is closer to an optimally safe policy's REF as we take  $\lambda_{\mathrm{max}} \to \infty$ . Nonetheless, we want to put a bound on the  $\lambda_{\mathrm{max}}$ . This  $\lambda_{\mathrm{max}}$  must be large enough so that choosing a policy that can reduce the expected cost returns by some non-zero amount is prioritized over increasing the reward returns. So any change in the reward critic terms must be less than any change in the cost critic term. If  $H_{\Delta}$  is the minimum non-zero difference between any two cost values, and  $P_{\mathrm{min}}$  is the minimum sampled non-zero likelihood of reaching a particular state and a point in the sample trajectory, then we can bound the maximum change in the reward returns and the maximum change on the weighted cost returns:
$$
\Delta \mathop{\mathbb{E}}_{s\sim d_{0}}[V(s)]\leq \frac{R_{\max}}{1 - \gamma}
$$
$$
\gamma^ {T} \cdot H _ {\Delta} \cdot P _ {\mathrm {m i n}} \cdot (\lambda + \frac {\phi (s)}{(1 - \phi (s))}) \leq \Delta \underset {s \sim d _ {0}} {\mathbb {E}} [ V _ {c} (s) \cdot (\lambda + \frac {\phi (s)}{(1 - \phi (s))}) ]
$$
So we can find the bound for  $\lambda_{\mathrm{max}}$ :
$$
\Delta \underset {s \sim d _ {0}} {\mathbb {E}} [ V (s) \cdot (1 - \phi (s)) ] <   \Delta \underset {s \sim d _ {0}} {\mathbb {E}} [ V _ {c} (s) \cdot (\lambda \cdot (1 - \phi (s)) + \phi (s)) ]
$$
$$
\Delta \underset {s \sim d _ {0}} {\mathbb {E}} [ V (s) ] <   \Delta \underset {s \sim d _ {0}} {\mathbb {E}} [ V _ {c} (s) \cdot (\lambda + \frac {\phi (s)}{(1 - \phi (s))}) ]
$$
$$
\frac {R _ {\operatorname* {m a x}}}{1 - \gamma} <   \gamma^ {T} \cdot H _ {\Delta} \cdot P _ {\min } \cdot (\lambda + \frac {\phi (s)}{(1 - \phi (s))})
$$
$$
\frac {R _ {\operatorname* {m a x}}}{(1 - \gamma) \cdot \gamma^ {T} \cdot H _ {\Delta} \cdot P _ {\min }} <   \lambda + \frac {\phi (s)}{(1 - \phi (s))}
$$
$$
\frac {R _ {\mathrm {m a x}}}{(1 - \gamma) \cdot \gamma^ {T} \cdot H _ {\Delta} \cdot P _ {\mathrm {m i n}}} - \frac {\phi (s)}{(1 - \phi (s))} <   \lambda
$$
The second line holds since we are simply rearranging the comparative weightages of the reward and cost returns. Now  $-\frac{\phi(s)}{(1 - \phi(s))} \leq 0$  (recall that if  $\phi(s) = 1$ , then  $\lambda$  is irrelevant in the lagrangian optimization). Thus, if  $\lambda > \frac{R_{\max}}{(1 - \gamma)\cdot\gamma^T\cdot H_\Delta\cdot P_{\min}}$  then minimizing the cost returns is prioritized over maximizing reward returns.
D Complete Experiment Details and Analysis
We compare our algorithm RESPO with 7 other safety RL baselines, which can be divided to CMDP class and hard constraints class, and unconstrained Vanilla PPO for reference.
CMDP Approaches
Proximal Policy Optimization-Lagrangian. PPOLag is a primal-dual method using Proximal Policy Optimization [33] based off of the implementation found in [7]. The lagrange multiplier is a scalar learnable parameter.
Constraint-Rectified Policy Optimization. CRPO [35] is a primal approach that switches between optimizing for rewards and minimizing constraint violations depending on whether the constraints are violated.
Penalized Proximal Policy Optimization. P30 [32] is another primal approach based on applying the technique of clipping the surrogate objectives found in PPO [33] to CMDPs.
Projection-Based Constrained Policy Optimization. PCPO [3] is a trust-region approach that takes a step in policy parameter space toward optimizing for reward and then projects this policy to the constraint set satisfying the CMDP expected cost constraints.
Hard Constraints Approaches
Reachability Constrained Reinforcement Learning. RCRL [27] is a primal-dual approach where the constraint is on the reachability value function and the lagrange multiplier is represented by a neural network parameterized by state.
Control Barrier Function. This CBF-based approach is inspired by the various energy-based certification approaches [56, 17, 18, 11, 57, 58]. This is implemented as a primal-dual approach where the control barrier-based constraint  $\dot{h}(s) + \nu \cdot h(s) \leq 0$  is to ensure stabilization toward the safe set.
Feasible Actor Critic. FAC [9] is another primal-dual approach similar to RCRL (i.e. it uses the NN representation of the lagrange multiplier parameterized by state) except that the constraint in FAC is based on the cumulative discount sum of costs in lieu of the reachability value function. It is important to note that FAC is originally meant for the CMDP framework (with some positive cost threshold), but we adapt it to hard constraints by making the cost threshold  $\chi = 0$ . We do this to make a better comparison between an algorithm that relies on using the lagrange multiplier represented as a NN to learn feasibility with our approach of using our proposed REF function to learn the feasibility likelihood - both approaches enforce a hard constraint on the cumulative discounted costs.
D.2 Benchmarks
We compare the algorithms on a diverse suite of environments including those from the Safety Gym, PyBullet, and MuJoCo suites and a multi-constraint, multi-drone environment.
Safety Gym. In Safety Gym [30], we examine CarGoal and PointButton which have 72D and 76D observation spaces that include lidar, accelerometer, gyro, magnetometer, velocimeter, joint angles, and joint velocities sensors. In the CarGoal environment, the car agent has a 72D observation space and is supposed to reach a goal region while avoiding both hazardous spaces and contact with fragile objects. In PointButton, the point agent has a 76D observation space and must press a series of specified goal buttons while avoiding 1) quickly moving objects, 2) hazardous spaces, 3) hitting the wrong buttons.
Safety PyBullet. In Safety PyBullet [50], we evaluate in BallRun and DroneCircle environments. In the BallRun environment, the ball agent must move as fast as possible under the constraint of a speed limit, and it must be within some boundaries. In DroneCircle, the agent is based on the AscTec Hummingbird quadrotor and is rewarded for moving clockwise in a circle of a fixed radius with the constraint of remaining within a safety zone demarcated by two boundaries. Note that we use this environment to evaluate our algorithm and the baselines in a stochastic setting. We ensure the MDP is stochastic by adding a  $5\%$  gaussian noise to the transitions per step.
Safety MuJoCo. Furthermore, we compare the algorithms in with complex dynamics in MuJoCo. Specifically, we look at HalfCheetah and Reacher safety problems. In Safety HalfCheetah, the agent must move as quickly as possible in the forward direction without moving left of  $x = -3$ . However, unlike the standard HalfCheetah environment, the reward is based on the absolute value of the distance traveled. In this paradigm, it is easier for the agent to learn to quickly run backward rather than forward without any directional constraints. In the Safety Reacher environment, the robotic arm must reach a certain point while avoiding an unsafe region.
Multi-Drone environment. We also compare in an environment with multiple hard and soft constraints. The environment requires controlling two drones to pass through a tunnel one at a time while respecting certain distance requirements. The reward is given for quickly reaching the goal positions. The two hard constraints involve (H1) ensuring neither drone collides into the wall and (H2) the distance between the two drones is more than 0.5 to ensure they do not collide. The soft constraint is that the two drones are within 0.8 of each other to ensure real-world communication. It is preferable to prioritize hard constraint H1 over hard constraint H2, since colliding with the wall may have more serious consequences to the drones rather than violations of an overly precautious distance constraint – as we will show, our algorithm RESPO can perform this prioritization in its optimization.
D.3 Hyperparameters/Other Details
Hyperparameters for Safe RL Algorithms	Values
On-policy parameters
Network Architecture	MLP
Units per Hidden Layer	256
Numbers of Hidden Layers	2
Hidden Layer Activation Function	tanh
Actor/Critic Output Layer Activation Function	linear
Lagrange multiplier Output Layer Activation Function	softplus
Optimizer	Adam
Discount factor γ	0.99
GAE lambda parameter	0.97
Clip Ratio	0.2
Target KL divergence	0.1
Total Env Interactions	9e6
Reward/Cost Critic Learning rate	Linear Decay 1e-3→0
Actor Learning rate	Linear Decay 3e-4→0
Lagrange Multiplier Learning rate	Linear Decay 5e-5→0
Number Seeds per algorithm per experiment	5
RESPO specific parameters
REF Output Layer Activation Function	sigmoid
REF Learning rate	1e-4→0
CBF specific parameters
ν	0.2
RCRL/FAC Note
Lagrange Multiplier	2-Layer, MLP(other algs just use scalar parameter)
Table 2: Hyperparameter Settings Details
To ensure a fair comparison, the primal-dual based approaches and unconstrained Vanilla PPO were implemented based off of the same code base [59]. The other three approaches were implemented based on [60] with the similar corresponding hyperparameters as the primal-dual approaches. We run our experiments on Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz with 6 cores. For Safety Gym, PyBullet, MuJoCo, and the multi-drone environments, each algorithm, per seed, per environment, takes  $\sim$  4 hours to train.
D.4 Double Integrator
Figure 8: Comparison of the trajectories in the Double Integrator Environment of an agent controlled by policies obtained by RCRL (in red) and our proposed algorithm RESPO (in green) when starting from the infeasible set (but still within the safe set). Notice how our approach actively enters the feasible set (blue region), while RCRL fails to do so. The level set demarcating the feasible/infeasible set boundary is in black. The safe set (i.e. the set of states that have no violations) is the region within the dashed purple square. The infeasible set is in yellow.
We use the Double Integrator environment as a motivating example to demonstrate how performing constrained optimization using solely reachability-based value functions as in RCRL can produce nonoptimal behavior when the agent is outside the feasibility set. Double Integrator has a 2 dimensional observation space  $[x_1, x_2]$ , 1 dimension action space  $a \in [-0.5, 0.5]$ , system dynamics is  $\dot{s} = [x_2, a]$  and constraint as  $\|s\|_{\infty} \leq 5$ . Particularly, we make the cost as 1 if  $\|s\|_{\infty} > 5$ , and 0 otherwise to emphasize the importance of capturing the frequency of violation during training.
We train an RCRL controller and RESPO controller in this environment, and the results are visualized in Figure 8. The color scheme indicates the learned reachability value across the state space while the black line demarcates the border of the zero level set. We present the behavior of the trajectories of RCRL and RESPO. Because the RCRL optimizes for reachability value function when outside the feasible set, it simply minimizes the maximum violation, which as can be seen does not result in the agent reentering the feasible set since it is uniformly equal to or near 1 in the infeasible set. This is since it permits many violations of magnitude same or less than that of the maximum violation. On the other hand, RESPO optimizes for cumulative damage by considering total sum of costs, thereby re-entering the feasible set.
D.5 Safety Gym Environments
Figure 9: Closer look at comparison of algorithms in Safety Gym CarGoal and PointButton Environments.
RESPO: In the CarGoal Environment, our approach achieves the best performance among the safe RL algorithms while being within the acceptable range of cost violations. It is important to note that our algorithm has no access to information on the cost threshold. In PointButton, RESPO achieves very high reward performance among the safety baselines while maintaining among the lowest violations.