SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The environmental settings for the button task are the same as in Liu et al. [2022]. There are five hazard regions, four dynamic obstacles, and four buttons, and all components are fixed throughout the training. The initial position of a robot and an activated button are randomly placed at the beginning of each episode. The reward function is the same as in (44), but the cost is different since there is no dense signal for contacts. We define the cost function for the button task as an indicator function that outputs one if the robot makes contact with an obstacle or an inactive button or enters a hazardous region. We add LiDAR values of buttons and obstacles to the state of the goal task, and actions are the same as the goal task. The length of the episode is 1000 steps without early termination.
Locomotion Tasks. We use three different legged robots, Mini-Cheetah, Laikago, and Cassie, for the locomotion tasks, as shown in Figure 6e, 6d, and 6c. The tasks aim to control robots to follow a velocity command on flat terrain. A velocity command is given by  $(v_{x}^{\mathrm{cmd}}, v_{y}^{\mathrm{cmd}}, \omega_{z}^{\mathrm{cmd}})$ , where  $v_{x}^{\mathrm{cmd}} \sim \mathcal{U}(-1.0, 1.0)$  for Cassie and  $\mathcal{U}(-1.0, 2.0)$  otherwise,  $v_{y}^{\mathrm{cmd}} = 0$ , and  $\omega_{z}^{\mathrm{cmd}} \sim \mathcal{U}(-0.5, 0.5)$ . To lower the task complexity, we set the  $y$ -directional linear velocity to zero but can scale to any non-zero value. As in other locomotion studies [Lee et al., 2020, Miki et al., 2022], central phases are introduced to produce periodic motion, which are defined as  $\phi_{i}(t) = \phi_{i,0} + f \cdot t$  for  $\forall i \in \{1, \dots, n_{\mathrm{legs}}\}$ , where  $f$  is a frequency coefficient and is set to 10, and  $\phi_{i,0}$  is an initial phase. Actuators of robots are controlled by PD control towards target positions given by actions. The state consists of velocity command, orientation of the robot frame, linear and angular velocities of the robot, positions and speeds of the actuators, central phases, history of positions and speeds of the actuators (past two steps), and history of actions (past two steps). A foot contact timing  $\xi$  can be defined as follows:
$$
\left. \xi_ {i} (s) = - 1 + 2 \cdot \mathbf {1} _ {\sin \left(\phi_ {i}\right) \leq 0} \forall i \in \{1, \dots , n _ {\text {l e g s}} \}, \right. \tag {45}
$$
where a value of -1 means that the  $i$ th foot is on the ground; otherwise, the foot is in the air. For the quadrupedal robots, Mini-Cheetah and Laikago, we use the initial phases as  $\phi_0 = \{0,\pi ,\pi ,0\}$ , which generates trot gaits. For the bipedal robot, Cassie, the initial phases are defined as  $\phi_0 = \{0,\pi \}$ , which generates walk gaits. Then, the reward and cost functions are defined as follows:
$$
R (s, a, s ^ {\prime}) = - 0. 1 \cdot \left(\left\| v _ {x, y} ^ {\text {b a s e}} - v _ {x, y} ^ {\text {c m d}} \right\| _ {2} ^ {2} + \left\| \omega_ {z} ^ {\text {b a s e}} - \omega_ {z} ^ {\text {c m d}} \right\| _ {2} ^ {2} + 1 0 ^ {- 3} \cdot R _ {\text {p o w e r}}\right),
$$
$$
C _ {1} (s, a, s ^ {\prime}) = \mathbf {1} _ {\text {a n g l e} \geq a}, C _ {2} (s, a, s ^ {\prime}) = \mathbf {1} _ {\text {h e i g h t} \leq b}, C _ {3} (s, a, s ^ {\prime}) = \sum_ {i = 1} ^ {n _ {\text {l e g s}}} \left(1 - \xi_ {i} \cdot \hat {\xi} _ {i}\right) / \left(2 \cdot n _ {\text {l e g s}}\right), \tag {46}
$$
where the power consumption  $R_{\mathrm{power}} = \sum_{i}|\tau_{i}v_{i}|$ , the sum of the torque times the actuator speed, is added to the reward as a regularization term,  $v_{x,y}^{\mathrm{base}}$  is the  $xy$ -directional linear velocity of the base
frame of robots,  $\omega_{z}^{\mathrm{base}}$  is the  $z$ -directional angular velocity of the base frame, and  $\hat{\xi} \in \{-1, 1\}^{n_{\mathrm{legs}}}$  is the current feet contact vector. For balancing, the first cost indicates whether the angle between the  $z$ -axis vector of the robot base and the world is greater than a threshold ( $a = 15^\circ$  for all robots). For standing, the second cost indicates the height of CoM is less than a threshold ( $b = 0.3, 0.35, 0.7$  for Mini-Cheetah, Laikago, and Cassie, respectively), and the last cost is to check that the current feet contact vector  $\hat{\xi}$  matches the pre-defined timing  $\xi$ . The length of the episode is 500 steps. There is no early termination, but if a robot falls to the ground, the state is frozen until the end of the episode.
Hyperparameter Settings. The structure of neural networks consists of two hidden layers with (512, 512) nodes and ReLU activation for all baselines and the proposed method. The input of value networks is state-action pairs, and the output is the positions of atoms. The input of policy networks is the state, the output is mean  $\mu$  and std  $\sigma$ , and actions are squashed into  $\tanh (\mu + \epsilon \sigma)$ ,  $\epsilon \sim \mathcal{N}(0,1)$ . We use a fixed entropy coefficient  $\beta$ . The trust region size  $\epsilon$  is set to 0.001 for all trust region-based methods. The overall hyperparameters for the proposed method can be summarized in Table 3.
Table 3: Hyperparameter settings for the Safety Gym and locomotion tasks.
Parameter	Safety Gym	Locomotion
Discount factor γ	0.99	0.99
Trust region size ε	0.001	0.001
Length of replay buffer	105	105
Critic learning rate	0.0003	0.0003
Trace-decay λ	0.97	0.97
Entropy coefficient β	0.0	0.001
The number of critic atoms M	25	25
The number of target atoms M'	50	50
Constraint risk level α	0.25, 0.5, and 1.0	1.0
threshold dk	0.025/(1-γ)	[0.025, 0.025, 0.4]/(1-γ)
Slack coefficient ζ	-	minkdk=0.025/(1-γ)
Since the range of the cost is  $[0,1]$ , the maximum discounted cost sum is  $1 / (1 - \gamma)$ . Thus, the threshold is set by target cost rate times  $1 / (1 - \gamma)$ . For the locomotion tasks, the third cost in (46) is designed for foot stamping, which is not essential to safety. Hence, we set the threshold to near the maximum (if a robot does not stamp, the cost rate becomes 0.5). In addition, baseline safe RL methods use multiple critic networks for the cost function, such as target [Yang et al., 2021] or square value networks [Kim and Oh, 2022a]. To match the number of network parameters, we use two critics as an ensemble, as in Kuznetsov et al. [2020].
Tips for Hyperparameter Tuning.
- Discount factor  $\gamma$ , Critic learning rate: Since these are commonly used hyperparameters, we do not discuss these.
- Trace-decay  $\lambda$ , Trust region size  $\epsilon$ : The ablation studies on these hyperparameters are presented in Appendix C.3. From the results, we recommend setting the trace-decay to  $0.95 \sim 0.99$  as in other TD( $\lambda$ -based methods [Precup et al., 2000]. Also, the results show that the performance is not sensitive to the trust region size. However, if the trust region size is too large, the approximation error increases, so it is better to set it below 0.003.
- Entropy coefficient  $\beta$ : This value is fixed in our experiments, but it can be adjusted automatically as done in SAC [Haarnoja et al., 2018].
- The number of atoms  $M, M'$ : Although experiments on the number of atoms did not performed, performance is expected to increase as the number of atoms increases, as in other distributional RL methods [Dabney et al., 2018a].
- Length of replay buffer: The effect of the length of the replay buffer can be confirmed through the experimental results from an off policy-based safe RL method [Kim and Oh, 2022a]. According to that, the length does not impact performance unless it is too short. We recommend setting it to 10 to 100 times the collected trajectory length.
- Constraint risk level  $\alpha$ , threshold  $d_k$ : If the cost sum follows a Gaussian distribution, the mean-standard constraint is identical to the CVaR constraint. Then, the probability of the
worst case can be controlled by adjusting  $\alpha$ . For example, if we set  $\alpha = 0.125$  and  $d = 0.03 / (1 - \gamma)$ , the mean-standard constraint enforces the probability that the average cost is less than 0.03 during an episode greater than  $95\% = \Phi (\phi (\Phi^{-1}(\alpha)) / \alpha)$ . Through this meaning, proper  $\alpha$  and  $d_{k}$  can be found.
- Slack coefficient  $\zeta$ : As mentioned at the end of Section 3.1, it is recommended to set this coefficient as large as possible. Since  $d_k - \zeta$  should be positive, we recommend setting  $\zeta$  to  $\min_k d_k$ .
In conclusion, most hyperparameters are not sensitive, so few need to be optimized. It seems that  $\alpha$  and  $d_{k}$  need to be set based on the meaning described above. Additionally, if the approximation error of critics is significant, the trust region size should be set smaller.
C.1 Safety Gym
In this section, we present the training curves of the Safety Gym tasks separately according to the risk level of constraints for better readability. Figure 7 shows The training results of the risk-neutral constrained algorithms and risk-averse constrained algorithms with  $\alpha = 1.0$ . Figures 8 and 9 show the training results of the risk-averse constrained algorithms with  $\alpha = 0.25$  and 0.5, respectively.
Figure 7: Training curves of risk-neutral constrained algorithms for the Safety Gym tasks. The solid line and shaded area represent the average and std values, respectively. The black dashed lines in the second row indicate thresholds. All methods are trained with five random seeds.
Figure 8: Training curves of risk-averse constrained algorithms with  $\alpha = 0.5$  for the Safety Gym.
Figure 9: Training curves of risk-averse constrained algorithms with  $\alpha = 0.25$  for the Safety Gym.
C.2 Ablation Study on Components of SDAC
There are three main differences between SDAC and the existing trust region-based safe RL algorithm for mean-standard constraints [Kim and Oh, 2022a], called OffTRC: 1) feasibility handling methods in multi-constraint settings, 2) the use of distributional critics, and 3) the use of Q-functions instead of advantage functions, as explained in Appendix A.8 and A.10. Since the ablation study for feasibility handling is conducted in Section 5.3, we perform ablation studies for the distributional critic and Q-function in this section. We call SDAC with only distributional critics as SDAC-Dist and SDAC with only Q-functions as  $SDAC-Q$ . If all components are absent, SDAC is identical to OffTRC [Kim and Oh, 2022a]. The variants are trained with the point goal task of the Safety Gym, and the training results are shown in Figure 10. SDAC-Q lowers the cost rate quickly but shows the lowest score. SDAC-Dist shows scores similar to SDAC, but the cost rate converges above the threshold 0.025. In conclusion, SDAC can efficiently satisfy the safety constraints through the use of Q-functions and improve score performance through the distributional critics.
Figure 10: Training curves of variants of SDAC for the point goal task.
C.3 Ablation Study on Hyperparameters
To check the effects of the hyperparameters, we conduct ablation studies on the trust region size  $\epsilon$  and entropy coefficient  $\beta$ . The results on the entropy coefficient are presented in Figure 11a, showing that the score significantly decreases when  $\beta$  is 0.01. This indicates that policies with high entropy fail to improve score performance since they focus on satisfying the constraints. Thus, the entropy coefficient should be adjusted cautiously, or it can be better to set the coefficient to zero. The results on the trust region size are shown in Figure 11b, which shows that the results do not change significantly regardless of the trust region size. However, the score convergence rate for  $\epsilon = 0.01$  is the slowest because the estimation error of the surrogate increases as the trust region size increases according to Theorem A.8.
(a) Entropy coefficient  $\beta$
(b) Trust region size  $\epsilon$
Figure 11: Training curves of SDAC with different hyperparameters for the point goal task.
D Comparison with RL Algorithms
In this section, we compare the proposed safe RL algorithm with traditional RL algorithms in the locomotion tasks and show that safe RL has the advantage of not requiring reward tuning. We use the truncated quantile critic (TQC) [Kuznetsov et al., 2020], a state-of-the-art algorithm in existing RL benchmarks [Todorov et al., 2012], as traditional RL baselines. To apply the same experiment to traditional RL, it is necessary to design a reward reflecting safety. We construct the reward through a weighted sum as  $\bar{R} = (R - \sum_{i=1}^{3} w_i C_i) / (1 + \sum_{i=1}^{3} w_i)$ , where  $R$  and  $C_{\{1,2,3\}}$  are used to train safe RL methods and are defined in Appendix B, and  $R$  is called the true reward. The weights of the reward function  $w_{\{1,2,3\}}$  are searched by a Bayesian optimization tool<sup>1</sup> to maximize the true reward of TQC in the Mini-Cheetah task. Among the 63 weights searched through Bayesian optimization, the top five weights are listed in Table 4.
Table 4: Weights of the reward function for the Mini-Cheetah task.
Reward weights	w1	w2	w3
#1	1.588	0.299	0.174
#2	1.340	0.284	0.148
#3	1.841	0.545	0.951
#4	6.560	0.187	4.920
#5	1.603	0.448	0.564
Figure 12 shows the training curves of the Mini-Cheetah task experiments where TQC is trained using the weight pairs listed in Table 4. The graph shows that it is difficult for TQC to lower the second cost below the threshold while all costs of SDAC are below the threshold. In particular, TQC with the fifth weight pairs shows the lowest second cost rate, but the true reward sum is the lowest. This shows that it is challenging to obtain good task performance while satisfying the constraints through reward tuning.
Figure 12: Training curves of the Mini-Cheetah task. The black dashed lines show the thresholds used for the safe RL method. The solid line represents the average value, and the shaded area shows one-fifth of the std value. The number after TQC in the legend indicates which of the reward weights in Table 4 is used. All methods are trained with five different random seeds.
<sup>1</sup>We use Sweeps from Weights & Biases Biewald [2020].
E Computational Cost Analysis
E.1 Complexity of Gradient Integration Method
In this section, we analyze the computational cost of the gradient integration method. The proposed gradient integration method has three subparts. First, it is required to calculate policy gradients of each cost surrogate,  $g_{k}$ , and  $H^{-1}g_{k}$  for  $\forall k\in \{1,2,\dots,K\}$ , where  $H$  is the Hessian matrix of the KL divergence.  $H^{-1}g_{k}$  can be computed using the conjugate gradient method, which requires only a constant number of back-propagation on the cost surrogate, so the computational cost can be expressed as  $K\cdot O(\mathrm{BackProp})$ .
Second, the quadratic problem in Section 3.1 is transformed to a dual problem, where the transformation process requires inner products between  $g_{k}$  and  $H^{-1}g_{m}$  for  $\forall k,m\in \{1,2,\dots,K\}$ . The computational cost can be expressed as  $K^2\cdot O(\mathrm{InnerProd})$ .
Finally, the transformed quadratic problem is solved in the dual space  $\in \mathbb{R}^K$  using a quadratic programming solver. Since  $K$  is usually much smaller than the number of policy parameters, the computational cost almost negligible compared to the others. Then, the cost of the gradient integration is  $K\cdot O(\mathrm{BackProp}) + K^2\cdot O(\mathrm{InnerProd}) + C$ . Since the back-propagation and the inner products are proportional to the number of policy parameters  $|\psi|$ , the computational cost can be simplified as  $O(K^2\cdot |\psi|)$ .
E.2 Quantitative Analysis
Table 5: Training time of Safe RL algorithms (in hours). The training time of each algorithm is measured as the average time required for training with five random seeds. The total training steps are  $5 \cdot 10^{6}$  and  $3 \cdot 10^{6}$  for the point goal task and the Mini-Cheetah task, respectively.
Task	SDAC (proposed)	OffTRC	WCSAC	CPO	CVPO
Point goal (Safety Gym)	7.96	4.86	19.07	2.61	47.43
Mini-Cheetah (Locomotion)	8.36	6.54	16.41	1.99	-
We analyze the computational cost of the proposed method quantitatively. To do this, we measure the training time of the proposed method, SDAC, and the safe RL baselines. We use a workstation whose CPU is the Intel Xeon e5-2650 v3, and GPU is the NVIDIA GeForce GTX TITAN X. The results are presented in Table 5. While CPO is the fastest algorithm, its performance, such as the sum of rewards, is relatively poor compared to other algorithms. The main reason why CPO shows the fastest computation time is that CPO is an on-policy algorithm, hence, it does not require an insertion to (and deletion from) a replay memory, and batch sampling. SDAC shows the third fastest computation time in all algorithms and the second best one among off-policy algorithms. Especially, SDAC is slightly slower than OffTRC, which is the fastest one among off-policy algorithms. This result shows the benefit of SDAC since SDAC outperforms OffTRC in terms of the returns and CV, but the training time is not significantly increased over OffTRC. WCSAC, which is based on SAC, has a slower training time because it updates networks more frequently than other algorithms. CVPO, an EM-based safe RL algorithm, has the slowest training time. In the E-step of CVPO, a non-parametric policy is optimized to solve a local subproblem, and the optimization process requires discretizing the action space and solving a non-linear convex optimization for all batch states. Because of this, CVPO takes the longest to train an RL agent.