SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Besides, We conduct experiments to confirm the performance of the adaptive penalty factor with dual update compared to that of the fixed penalty factor as introduced in Section 4. For fairness, we check the converged values of the adaptive penalty factor with learning rate  $\eta = 0.02$  in RPOSAC, and we find that the converged penalty factors are ranging around 100. Therefore, we chose  $\nu^{j} = 100$  for all penalty terms in the setting of RPOSAC
with the fixed penalty factor. Results in Figure 6 show the advantage of the adaptive penalty factors in terms of episodic reward.
Figure 6: Comparison between RPO with fixed penalty factor and adaptive penalty factor
E.3 Necessity of Backpropagation with Generalized Reduced Gradient
As we illustrate in the construction stage, the integrated actions are actually determined once the basic actions are given. Therefore, it is possible for the value network to approximate the mapping from the basic actions to the nonbasic actions. This seems that there is no need for the backpropagation with generalized reduced gradient which requires extra gradient flow from nonbasic actions if only the hard equality constraints need to be addressed or the hard inequality constraints are only related to the basic actions.
To explore the necessity of the backpropagation with generalized reduced gradient, we also contrast the performance in RPOSAC trained with/without the complete gradient. Concretely, for RPOSAC trained without the complete gradient, we only input the basic actions into the value networks and expect it can approximate the complete gradient, which is known as generalized reduced gradient. The results are shown in Figure 7. It reflects that
the value network cannot backpropagate an accurate gradient directly, and indicates the necessity to explicitly construct the gradient flow from nonbasic actions to basic actions.
Figure 7: Comparison between RPO with partial gradient and complete gradient.
E.4 Comparison in Training Time
The training time comparison in our three benchmarks is shown in Tables 3. It can be observed that the training time of RPO-DDPG and RPO-SAC grows linearly with the number of constraints to deal with. However, RPO-DDPG and RPO-SAC still have significant advantages on some of Safe RL algorithms like Safety Layer.
Task	CPO	CUP	Safety Layer	DDPG-L	SAC-L	RPO-DDPG(*)	RPO-SAC(*)
Safe CartPole	68.7	75.2	1067.0	150.2	268.5	190.2	375.5
Spring Pendulum	98.6	81.4	3859.7	174.9	309.5	242.6	419.0
OPF with Battery Energy Storage	288.8	239.4	6395.0	498.9	892.7	4462.1	4828.4
Table 3: Training time(s) of different algorithms in the three benchmarks. Each item in the table is averaged across 5 runs.
F Hyper-parameters
We implemented our experiments on a GPU of NVIDIA GeForce RTX 3090 with 24GB. Each experiment on Safe CartPole and Spring Pendulum takes about 0.5 hours. Each experiment on OPF with Battery Energy Storage takes about 4 hours. Moreover, we adopt similar neural network architectures for policy and value networks except for the input and output dimensions in all experiments. The policy and value networks both have two hidden layers with 256 hidden units and only differ in input and output layers. Besides, we also show the detailed hyper-parameters used in our experiments. Table 4, Table 5 and Table 6 respectively present the parameters used in Safe CartPole, Spring Pendulum, and OPF with Battery Energy Storage. Additionally, the implementation of three safe RL algorithms in our experiments are based on omnisafe  $^{2}$  and safe-explorer  $^{3}$ , and recommended values are adopted for hyper-parameters not mentioned in the following tables.
Parameter	CPO	CUP	Safety Layer	DDPG-L	SAC-L	RPODDPG	RPOSAC
Batch Size B	256	256	256	256	256	256	256
Discount Factor γ	0.95	0.95	0.95	0.95	0.95	0.95	0.95
Target Smoothing Coefficient τ	N/A	N/A	0.005	0.005	0.005	0.005	0.005
Frequency of Policy Update	N/A	N/A	N/A	0.25	0.25	0.25	0.25
Total Epochs T	2E4	2E4	2E4	2E4	2E4	2E4	2E4
Capacity of Replay Buffer D	N/A	N/A	2E4	2E4	2E4	2E4	2E4
Random Noise in Exploration ε	N/A	N/A	1E-2	1.0	N/A	1.0	N/A
LR for Policy Network μ	1E-4	1E-4	1E-4	1E-4	1E-4	1E-4	1E-4
LR for Value Network Q	3E-4	3E-4	3E-4	3E-4	3E-4	3E-4	3E-4
LR for Penalty Factor ν	N/A	N/A	N/A	0.2	0.2	0.2	0.2
Temperature α	N/A	N/A	N/A	N/A	0.1	N/A	0.1
Projection Step ηa	N/A	N/A	N/A	N/A	N/A	2E-2	2E-2
Max GRG Updates K	N/A	N/A	N/A	N/A	N/A	10	10
Projection Step in Evaluation ηa e	N/A	N/A	N/A	N/A	N/A	2E-2	2E-2
Max GRG Updates in Evaluation Ke	N/A	N/A	N/A	N/A	N/A	50	50
Table 4: Hyper-parameters for experiments in Safe CartPole.
Parameter	CPO	CUP	Safety Layer	DDPG-L	SAC-L	RPODDPG	RPOSAC
Batch Size B	256	256	256	256	256	256	256
Discount Factor γ	0.95	0.95	0.95	0.95	0.95	0.95	0.95
Target Smoothing Coefficient τ	N/A	N/A	0.005	0.005	0.005	0.005	0.005
Frequency of Policy Update	N/A	N/A	N/A	0.25	0.25	0.25	0.25
Total Epochs T	2E4	2E4	2E4	2E4	2E4	2E4	2E4
Capacity of Replay Buffer D	N/A	N/A	2E4	2E4	2E4	2E4	2E4
Random Noise in Exploration ε	N/A	N/A	1E-2	0.5	N/A	0.5	N/A
LR for Policy Network μ	1E-4	1E-4	1E-4	1E-4	1E-4	1E-4	1E-4
LR for Value Network Q	3E-4	3E-4	3E-4	3E-4	3E-4	3E-4	3E-4
LR for Penalty Factor ν	N/A	N/A	N/A	0.01	0.01	0.01	0.01
Temperature α	N/A	N/A	N/A	N/A	0.01	N/A	0.01
Projection Step ηa	N/A	N/A	N/A	N/A	N/A	2E-3	2E-3
Max GRG Updates K	N/A	N/A	N/A	N/A	N/A	10	10
Projection Step in Evaluation ηa e	N/A	N/A	N/A	N/A	N/A	2E-3	2E-3
Max GRG Updates in Evaluation Ke	N/A	N/A	N/A	N/A	N/A	50	50
Table 5: Hyper-parameters for experiments in Spring Pendulum.
G Systematical Action Division Method
While it is usually not difficult to divide actions into basic and nonbasic actions, we present the systematical action division method below to further show the generality of our method. Firstly, we
$^{2}$ https://github.com/PKU-Alignment/omnisafe
<sup>3</sup>https://github.com/AgrawalAmey/safe-explorer
Parameter	CPO	CUP	Safety Layer	DDPG-L	SAC-L	RPODDPG	RPOSAC
Batch Size B	256	256	256	256	256	256	256
Discount Factor γ	0.95	0.95	0.95	0.95	0.95	0.95	0.95
Target Smoothing Coefficient τ	N/A	N/A	0.005	0.005	0.005	0.005	0.005
Frequency of Policy Update	N/A	N/A	N/A	0.25	0.25	0.25	0.25
Total Epochs T	4E4	4E4	4E4	4E4	4E4	4E4	4E4
Capacity of Replay Buffer D	N/A	N/A	2E4	2E4	2E4	2E4	2E4
Random Noise in Exploration ε	N/A	N/A	1E-2	1E-4	N/A	1E-4	N/A
LR for Policy Network μ	1E-4	1E-4	1E-4	1E-4	1E-4	1E-4	1E-4
LR for Value Network Q	3E-4	3E-4	3E-4	3E-4	3E-4	3E-4	3E-4
LR for Penalty Factor ν	N/A	N/A	N/A	0.02	0.02	0.02	0.02
Temperature α	N/A	N/A	N/A	N/A	0.001	N/A	0.001
Projection Step ηa	N/A	N/A	N/A	N/A	N/A	1E-4	1E-4
Max GRG Updates K	N/A	N/A	N/A	N/A	N/A	10	10
Projection Step in Evaluation ηa e	N/A	N/A	N/A	N/A	N/A	1E-4	1E-4
Max GRG Updates in Evaluation Ke	N/A	N/A	N/A	N/A	N/A	50	50
Table 6: Hyper-parameters for experiments in OPF with Battery Energy Storage.
would like to clarify that the principal purpose of the systematical action division method is to cope with the problem that  $J_{:,m+1:n}^{F}$  is not invertible under some division or the equations defined by basic actions are not solvable. In that case, we will separate the situation into situations with linear equality constraint and nonlinear equality constraints, and illustrate different problems and corresponding systematical action division methods in both of them.
Linear. In the linear case, if  $J_{\cdot ;m + 1:n}^{F}$  is not invertible, this may result from two possible problems. Firstly, it indicates there exist redundant equality constraints, then we can directly delete these redundant equality constraints (i.e., choose the maximal linearly independent subset of row vectors of the coefficient matrix) and redefine the dimension of basic actions to make  $J_{\cdot ;m + 1:n}^{F}$  invertible. For example, if we have 3 linear equality constraints  $Ax + b = 0$  on 4 actions, where
$$
[ A   | b ] = \left( \begin{array}{c c c c c} 1 & 0 & - 2 & 3 & 2 \\ 5 & - 3 & 1 & 4 & - 1 \\ 4 & - 3 & 3 & 1 & - 3 \end{array} \right).
$$
Here we choose  $a_1$  as the basic action, and then  $J_{\cdot;m+1:n}^{F} = A_{\cdot;2:4}$  is not invertible, since the third equality constraint can be represented by another two constraints. Hence, we can delete this redundant constraint and the new equality constraints are  $\tilde{A} + \tilde{b} = 0$ , where
$$
\left[ \tilde {A}   \big |   \tilde {b} \right] = \left( \begin{array}{c c c c c} 1 & 0 & - 2 & 3 & 2 \\ 5 & - 3 & 1 & 4 & - 1 \end{array} \right).
$$
Then, the new basic actions will be  $(a_{1},a_{2})$  and new  $J_{:,m:n}^{F} = \tilde{A}_{:,2:4}$  is invertible. Another problem is with respect to the division of basic and nonbasic actions. For example, let us consider other 2 linear equality constraints  $Ax + b = 0$  without redundant equality constraints on 3 actions as below:
$$
[ A   | b ] = \left( \begin{array}{c c c c} 1 & - 1 & - 2 & 2 \\ 5 & - 1 & - 2 & - 1 \end{array} \right).
$$
Although there is no redundant equality constraint,  $J_{\therefore m + 1:n}^{F}$  will be still not invertible if  $(a_2, a_3)$  are divided as nonbasic actions. Therefore, after removing the redundant equality constraints, we need to choose actions that correspond to the maximal linearly independent column vectors of the coefficient matrix as the nonbasic actions to ensure the invertibility of  $J_{\therefore m + 1:n}^{F}$ .
Nonlinear. In the nonlinear case, the systematical way that divides the full action into the basic and nonbasic actions is principally to ensure the solvability of the equality equation.
Firstly, assume there are  $n - m$  equality constraints and the full action  $a \in \mathbb{R}^n$ . We can construct a  $0 - 1$  relationship matrix  $E$  with the shape of  $(n - m) \times n$ , and  $E_{ij}$  is to describe whether the equality constraint  $f_i$  is related to  $a_j$ .
For example, if we have 3 equality constraints on action  $a \in \mathbb{R}^4$  like that
$$
f _ {1} \left(a _ {1}, a _ {2}, a _ {4}\right) = 0, f _ {2} \left(a _ {2}, a _ {3}\right) = 0, f _ {3} \left(a _ {1}\right) = 0,
$$
Then, the relationship matrix will be
$$
\left( \begin{array}{c c c c} 1 & 1 & 0 & 1 \\ 0 & 1 & 1 & 0 \\ 1 & 0 & 0 & 0 \end{array} \right)
$$
Now, we need to choose the nonbasic actions that cover the equality constraints as many as possible. This is the maximal matching problem in the bipartite graph, where constraints and actions are nodes and there exists an edge between constraint  $f_{i}$  are related to action  $a_{j}$  if  $E_{ij} = 1$ . Hence,  $(a_{1}, a_{2}, a_{3}), (a_{1}, a_{2}, a_{4}), (a_{1}, a_{3}, a_{4})$  are valid choices of nonbasic actions here, and then the equations can be solved with such divisions. In contrast, if we choose  $(a_{2}, a_{3}, a_{4})$  as the nonbasic actions, the equations cannot be solved. It is because if basic action  $a_{1}$  is determined, the equations will be
$$
f _ {1} \left(a _ {2}, a _ {4}; a _ {1}\right) = 0, f _ {2} \left(a _ {2}, a _ {3}\right) = 0, f _ {3} (\emptyset ; a _ {1}) = 0,
$$
which is unsolvable.
Besides,  $J_{:,m+1:n}^{F}$  may not be invertible in some specific action points. In such situation, we can add a small perturbation on the current action point or  $J^{F}$  to make  $J^{F}$  invertible. This is a useful trick in classical control methods. However, this situation is seldom seen in practice.