SlowGuess/NeurIPS_2023_OCR huggingface dataset:
PPOLag: In both Safety Gym environments, PPOLag maintains relatively high performance, albeit less than our approach. Nonetheless, it always converges to the cost threshold amount of violations for the respective environments.
RCRL: RCRL has either high reward and high violations or low reward and low violations. It learns a very conservative behavior in CarGoal environment where the violations go down but the reward performance can also be seen to be sacrificed during training. For PointButton, RCRL achieves slightly higher reward performance but has over  $3 \times$  the number of violations as RESPO.
FAC: Using a NN to represent the lagrange multiplier in order to capture the feasible sets seems to produce very conservative behavior that sacrifices performance. In both CarGoal and PointButton the reward performance and cost violations are very low. This can be explained because the average observed lagrange multiplier across the states quickly grows, even becoming  $9 \times$  that of scalar learnable lagrange multiplier in RESPO.
CBF: The CBF approach has low reward performance in both the Safety Gym benchmarks and its cost violations are quite high.
CRPO, P3O, & PCPO: These CMDP-based primal approaches have mediocre reward performance but, with the exception of CRPO, achieve violations within the cost threshold. CRPO, however, has high cost violations in both CarGoal and PointButton.
Vanilla PPO: This unconstrained algorithm consistently has high rewards and high costs, so maximizing rewards does not improve costs in these environments.
D.6 Safety PyBullet Environments
Figure 10: Closer look at comparison of algorithms in Safety PyBullet DroneCircle and BallRun Environments.
RESPO: In the both BallRun and DroneCircle, our approach achieves the highest or among the highest reward performance compared to the other safety baselines. Furthermore, RESPO converges to almost 0 constraint violations for both environments.
PPOLag: In DroneCircle, PPOLag has the best reward performance. Although its costs violations are around the cost threshold, it is much higher than RESPO. On the other hand, in BallRun, PPOLag has very low reward performance.
RCRL: We again see RCRL take on behavior with extremes - it has low reward and cost violations in Drone Circle and has high reward and cost violations in BallRun. Constraining the maximum violation with the reachability value function as RCRL does seems to provide poor safety in an environment with non-tangible constraints (i.e. a speed limit in BallRun).
FAC: While in BallRun, we see FAC have the same low reward and low violations behavior, DroneCircle shows an instance where FAC can achieve decently high rewards while maintaining low violations.
CBF: In DroneCircle, the CBF approach has low rewards and relatively low violations; in BallRun, it has a bit higher rewards compared to all the low performance algorithms with very low violations.
CRPO, P3O, & PCPO: These CMDP-based primal approaches have mediocre reward performance in DroneCircle and very low performance in BallRun. Nonetheless they achieve violations within the cost threshold.
Vanilla PPO: This unconstrained algorithm consistently has high rewards and high costs (sometimes out of the scope of the plots), so maximizing rewards does not improve costs in these environments.
D.7 Safety MuJoCo Environments
Figure 11: Closer look at comparison of algorithms in Safety MuJoCo Reacher and HalfCheetah Environments.
Note on HalfCheetah: The rewards for HalfCheetah are based on the absolute distance traveled in each step. Without the cost metric to constrain backward travel, it is easy to learn to run backward, as is the behavior learned in unconstrained PPO.
RESPO: Our approach achieves the highest reward performance among the safety baselines in HalfCheetah and decent reward performance in Reacher. Interestingly, RESPO also has 0 constraint violations in HalfCheetah and the second lowest constraint violations in Reacher.
PPOLag: The performance in Reacher for PPOLag is similar as in most of the previous environments: decently high rewrad, cost near the threshold. However, for HalfCheetah, interesting PPOLag learns to maintain the violations well below the cost threshold.
RCRL: We see yet again RCRL has high reward follow by very high constraint violations.
FAC: This approach has decent reward performance in Reacher and low reward performance in HalfCheetah. However, interestingly, FAC has high cost violations though below the cost threshold.
CBF: In Reacher, the CBF approach has conservative behavior with both low reward and low cost violations. But in HalfCheetah, it has very low reward performance and very high cost violations (not seen in the plot since its an order of magnitude larger than the visible range).
CRPO, P30, & PCPO: These CMDP-based primal approaches have decent reward performance in Reacher while maintaining violations within cost threshold. In HalfCheetah, however, they achieve low performance and low cost violations.
D.8 Hard and Soft Constraints
Figure 12: Closer look at comparison of RESPO with baselines trajectories in hard & soft Constraints multi-Drone control. Starting at gold circles, drones must enter the tunnel one at a time and reach green star while avoiding the wall and satisfying distance constraints. The colors indicate time along the trajectories.
RESPO: We manage multiple hard and soft Constraints by extending our framework to optimize the following Lagrangian:
$$
\begin{array}{l} \min _ {\pi} \max _ {\lambda} \left(L (\pi , \lambda) = \underset {s \sim d _ {0}} {\mathbb {E}} \left[ \left[ - V ^ {\pi} (s) + \lambda_ {s c} \cdot (V _ {s c} ^ {\pi} (s) - \chi) + \lambda_ {h c 2} \cdot V _ {h c 2} ^ {\pi} (s) \right] \cdot (1 - p _ {h c 2} (s)) \right]\right) \\ \left. \left. + V _ {h c 2} ^ {\pi} (s) \cdot p _ {h c 2} (s) + \lambda_ {h c 1} \cdot V _ {h c 1} ^ {\pi} (s) \right] \cdot \left(1 - p _ {h c 1} (s)\right) + V _ {h c 1} ^ {\pi} (s) \cdot p _ {h c 1} (s) \right] \Bigg) \tag {10} \\ \end{array}
$$
The subscripts  $hc1$  indicates the first hard constraint (i.e. wall avoidance),  $hc2$  indicates the second hard constraint (i.e. drone cannot be too close), and  $sc$  indicates soft constraint (i.e. drone cannot be too far) – they are all based on discounted sum of costs. Recall  $V^{\pi}(s)$  is reward returns. We color coded the corresponding parts of the optimization. Notice how we learn a different REF for each hard constraint. Also notice that the feasible set of the first hard constraint  $p_{hc1}$  is placed in a manner so as to ensure prioritization of the first hard constraint. As can be seen in the top left plot of Figure 12, our approach successfully reaches the goals and avoids the walls. To enable mobility of the top drone to pass through the tunnel with wall collision, the second hard constraint is violated temporarily in the blue to cyan time period. Furthermore to allow the bottom drone to pass through the tunnel, the soft constraint is violated during the green to orange time period. Nonetheless, RESPO successfully manages the constraints and reward performance via Equation 10 optimization.
PPOLag, RCRL, FAC: The optimization formulation for these approaches is as follows:
$$
\min  _ {\pi} \max  _ {\lambda} \left(L (\pi , \lambda) = - V ^ {\pi} (s) + \lambda_ {s c} \cdot \left(V _ {s c} ^ {\pi} (s) - \chi\right) + \lambda_ {h c 2} \cdot V _ {h c 2} ^ {\pi} (s) + \lambda_ {h c 1} \cdot V _ {h c 1} ^ {\pi} (s)\right) \tag {11}
$$
For PPOLag and FAC, all the constraint value functions are discount sum of cost. For RCRL,  $V_{sc}^{\pi}(s)$  is based on discount sum of costs but  $V_{hc1}^{\pi}(s)$  and  $V_{hc2}^{\pi}(s)$  are based on the reachability value function. Furthermore, in PPOLag, all the lagrange multipliers are learnable scalar parameters. In FAC and RCRL all the lagrange multipliers are NN representations parameterized by state. These formulations are not able to provide a framework for the prioritization of the constraint satisfaction - all the constraints are treated the same, weighted only on the learned lagrange multipliers. As can be seen in the other three images in Figure 12, the algorithms cannot manage the multiple constraints, and invariably collide with the wall.
D.9 Ablation - Learning Rate
Figure 13: Closer look at Ablation study on the learning rate of REF.
We performance Ablation study of varying the learning rate of the REF function to verify the importance of the multi-timescale assumption. Particular, we compare our algorithm's approach of placing the learning rate of the REF between the policy and lagrange multiplier with making the REF's learning rate in various orders of magnitudes slower and faster. Our approach with the learning rate satisfying the multi-timescale assumption experimentally appears to still have the best balance of reward optimization and constraint satisfaction. Particularly when we change the learning rate by one order of magnitude (i.e.  $\times 10$  or  $\times 0.1$ ), we see the reward performance reduce by around half and while the cost violations generally don't change. But when we change the learning rates by another order of magnitude, there reward performance effective becomes zero and the cost violations generally reduce further. By increasing the learning rate of the REF function, we can no longer guarantee that the REF convergences to near the optimally safe REF value. Instead, it becomes the REF of the policy in question. So instead, the optimization can learn to "hack" the REF function to obtain a policy (and lagrange multiplier) that is not a local optimal for the optimization formulation. On the other hand, when the learning rate is too slow, the lagrange multiplier quickly explodes, thereby creating a very conservative solution - notice the similarity of the orange line in Figure 13 with learning rate 0.01 times that of standard in training behavior with PPOLag where  $\chi = 0$  in the ablation study on optimization.
Figure 14: Closer look at Ablation study on hard constraints optimization frameworks.
D.10 Ablation - Optimization
In this ablation study, we examine the various optimization frameworks within the context of hard constraints. Particularly, we compare our RESPo framework with RCRL and CMDP. However, for RCRL we implement using our REF function method while still keeping the reachability value function. For CMDP, we make the cost threshold  $\chi = 0$ . These comparisons answer important questions about our design choices - specifically is it sufficient to simply to just use the REF component or to just learn the cost returns alone? From this ablation study, we propose that though we have provided theoretical support for adding each of these design components individually, in practice they are both required together in our algorithm. In RCRL implemented with our REF, we generally see decently high rewards but the cost violations are always very large. This highlights the problems of the reachability function again - if the agent starts or ever wanders into the infeasible set, there is no guarantee of (re)entrance into the feasible set. So the agent can indefinitely remain in the infeasible set, thereby incurring potentially an unlimited number of constraint violations. In PPOLag with  $\chi = 0$ , both the reward performance and constraint violations are very low. By using such hard constraints versions of these purely learning-based methods, even when using the cumulative discounted cost rather than reachability value function, the reward performance is very low because the lagrange multiplier becomes too large quickly and thereby overshadows the reward returns in the optimization. Ultimately, both the REF approach and the usage of the cumulative discounted costs are important components of our algorithm RESPO that encourage a good balance between the reward performance and safety constraint satisfaction in such stochastic settings.