SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\nu_ {s} ^ {j} \triangleq \underset {\nu_ {s} ^ {j}} {\arg \max } \left\{\underset {\theta} {\min } - J _ {R} \left(\tilde {\pi} _ {\theta}\right) + \mathbb {E} _ {s \sim \pi} \left[ \sum_ {j} \sum_ {s} \nu_ {s} ^ {j} \max  \{0, g _ {j} (\tilde {\pi} (s); s) \} \right] \right\}. \tag {18}
$$
Therefore, the unconstrained problem  $\min_{\theta} - J_R(\tilde{\pi}_{\theta}) + \mathbb{E}_{s\sim \pi}\left[\sum_j\sum_s\nu_s^j\max \left\{0,g_j(\tilde{\pi} (s);s)\right\} \right]$  is equivalent to the constrained problem (7). Let  $\theta^{\star}$  be the optimal solution of the unconstrained problem. Then, we have
$$
\begin{array}{l} \tilde {\mathcal {L}} (\boldsymbol {\theta} ^ {\star}) = - J _ {R} (\tilde {\pi} _ {\boldsymbol {\theta} ^ {\star}}) + \mathbb {E} _ {s \sim \pi} \left[ \sum_ {j} \sum_ {s} \nu_ {s} ^ {j} \max \{0, g _ {j} (\tilde {\pi} _ {\boldsymbol {\theta} ^ {\star}} (s); s) \} \right] \\ \leq - J _ {R} \left(\tilde {\pi} _ {\theta}\right) + \mathbb {E} _ {s \sim \pi} \left[ \sum_ {j} \sum_ {s} \nu_ {s} ^ {j} \max  \{0, g _ {j} (\tilde {\pi} (s); s) \} \right] \tag {19} \\ \leq \tilde {\mathcal {L}} (\theta) \\ \end{array}
$$
Hence, the unconstrained problem (8) is equivalent to the constrained problem (7).
A.4 Limitations of  $\ell_2$  Norm Objectives
Projection with  $\ell_2$  norm objectives will never come into the feasible region defined by inequality constraints with a sufficiently small projection step.
Assume that the inequality constraint that needs to be satisfied is  $g(a;s) \leq 0$ . For a given action  $a_{k}$  that does not satisfy the inequality constraint after  $k$  updates, i.e.,  $g(a_{k}) > 0$ . If  $a_{k}$  comes close enough to the feasible region by using the  $\ell_2$  norm objective, we apply a linear approximation on  $g(a_{k}) \approx c^{T}a_{k} - b$  and then obtain
$$
\begin{array}{l} g \left(a _ {k + 1}\right) = g \left(a _ {k} - \Delta a\right) \\ = c ^ {T} \left(a _ {k} - \eta_ {a} \nabla_ {a} \| c ^ {T} a _ {k} - b \| _ {2} ^ {2}\right) - b \\ = c ^ {T} \left(a _ {k} - 2 \eta_ {a} c \left(c ^ {T} a _ {k} - b\right)\right) - b \tag {20} \\ = c ^ {T} \left(a _ {k} - 2 \eta_ {a} g (a _ {k}) c\right) - b \\ = g \left(a _ {k}\right) \left(1 - 2 \eta_ {a} c ^ {T} c\right). \\ \end{array}
$$
Obviously, to ensure the satisfaction of the inequality constraint, i.e.,  $g(a_{k + 1}) \leq 0$ , we must limit the projection step with  $\eta_{a} \geq 1 / (2c^{T}c)$ . However,  $\eta_{a}$  is often a sufficiently small number for the satisfaction of nonlinear equality constraints. In this case, the inequality constraints will never be satisfied with a sufficiently small projection step.
B GRG algorithm
The GRG algorithm is shown as Algorithm 2.
Algorithm 2 Generalized Reduced Gradient Algorithm
Input: Optimization problem  $\min_{\mathbf{x} \in \mathbb{R}^n} f(\mathbf{x})$ , s.t.  $\mathbf{h}(\mathbf{x}) = \mathbf{0} \in \mathbb{R}^m$ ,  $\mathbf{a} \leq \mathbf{x} \leq \mathbf{b}$ .
Output: Optimal solution x.
Assumptions:
1. Divide  $\mathbf{x}$  into  $\mathbf{x} = (\mathbf{x}^N, \mathbf{x}^B)$ , where  $\mathbf{x}^N \in \mathbb{R}^m$ ,  $\mathbf{x}^B \in \mathbb{R}^{n - m}$ .
2. If  $(\mathbf{a}^N,\mathbf{a}^B)$  and  $(\mathbf{b}^N,\mathbf{b}^B)$  are the corresponding partitions of  $\mathbf{a},\mathbf{b}$ , then  $\mathbf{a}^N\leq \mathbf{x}^N\leq \mathbf{b}^N$
3. The  $m \times m$  matrix  $\nabla_{\mathbf{x}^N}\mathbf{h}\left(\mathbf{x}^N,\mathbf{x}^B\right)$  is nonsingular at  $\mathbf{x} = (\mathbf{x}^N,\mathbf{x}^B)$ .
1: Define the reduced gradient (with respect to  $\mathbf{x}^N$ ) as
$$
\mathbf {r} ^ {T} = \nabla_ {\mathbf {x} ^ {B}} f (\mathbf {x} ^ {N}, \mathbf {x} ^ {B}) + \lambda^ {T} \nabla_ {\mathbf {x} ^ {B}} \mathbf {h} (\mathbf {x} ^ {N}, \mathbf {x} ^ {B}), \text {w h e r e} \nabla_ {\mathbf {x} ^ {N}} f (\mathbf {x} ^ {N}, \mathbf {x} ^ {B}) + \lambda^ {T} \nabla_ {\mathbf {x} ^ {N}} \mathbf {h} (\mathbf {x} ^ {N}, \mathbf {x} ^ {B}) = 0.
$$
2: Obtain  $\mathbf{r}^T = \nabla_{\mathbf{x}^B}f(\mathbf{x}^N,\mathbf{x}^B) - \nabla_{\mathbf{x}^N}f(\mathbf{x}^N,\mathbf{x}^B)\left[\nabla_{\mathbf{x}^N}\mathbf{h}(\mathbf{x}^N,\mathbf{x}^B)\right]^{-1}\nabla_{\mathbf{x}^B}\mathbf{h}(\mathbf{x}^N,\mathbf{x}^B).$
3: Let
$$
\Delta x _ {i} ^ {B} = \left\{ \begin{array}{c c} - r _ {i} & \text {i f} r _ {i} <   0, x _ {i} ^ {B} <   B ^ {B} \text {o r} r _ {i} > 0, x _ {i} ^ {B} > a ^ {B} \\ 0 & \text {o t h e r w i s e} \end{array} \right.,
$$
$$
\Delta \mathbf {x} ^ {N} = - \left[ \nabla_ {\mathbf {x} ^ {N}} \mathbf {h} \right] ^ {- 1} \nabla_ {\mathbf {x} ^ {B}} \mathbf {h} \Delta \mathbf {x} ^ {B}.
$$
4: Find  $\alpha_{1},\alpha_{2},\alpha_{3}$  that respectively satisfy
$$
\max  \left\{\alpha : \mathbf {a} ^ {N} \leq \mathbf {x} ^ {N} + \alpha \Delta \mathbf {x} ^ {N} \leq \mathbf {b} ^ {N} \right\},
$$
$$
\max  \left\{\alpha : \mathbf {a} ^ {B} \leq \mathbf {x} ^ {B} + \alpha \Delta \mathbf {x} ^ {B} \leq \mathbf {b} ^ {B} \right\},
$$
$$
\min  \left\{f (\mathbf {x} + \alpha \Delta \mathbf {x}): 0 \leq \alpha \leq \alpha_ {1}, 0 \leq \alpha \leq \alpha_ {2} \right\}.
$$
5: Update
$$
\mathbf {x} ^ {B} \gets \mathbf {x} ^ {B} + \alpha_ {3} \Delta \mathbf {x} ^ {B}
$$
$$
\mathbf {x} ^ {N} \leftarrow \mathbf {x} ^ {N} + \alpha_ {3} \Delta \mathbf {x} ^ {N}
$$
6: Perform (1)-(5) until convergence.
C Algorithm Details
We present the concrete algorithm about how RPODDPG and RPOSAC work in Algorithm 3 and Algorithm 4.
Algorithm 3 RPO-DDPG
Input: Initial parameters  $\theta, \omega$  of policy network  $\mu$  and value network  $Q$ .
1: Initialize the corresponding target network  $\theta^{\prime}\gets \theta ,\omega^{\prime}\gets \omega$
2: Initialize replay buffer  $\mathcal{D}$
3: Initialize penalty factor for inequality constraints  $\nu \gets 0$ .
4: for each episode do
5: for each environment epoch do
6: Select partial action  $a_{t}^{B} = \mu_{\theta}(s_{t}) + \mathcal{N}(0,\epsilon_{t})$
7: Employ construction stage by solving the equations defined equality constraints
$$
\tilde {a} _ {t} = \left[ \begin{array}{c c} a _ {t} ^ {B} & \phi_ {N} (a _ {t} ^ {B}) \end{array} \right].
$$
8: Employ projection stage on the concatenated action  $\tilde{a}_t$  to obtain feasible action  $a_t$
9: for  $k = 1,\dots ,K$  do
$$
\Delta a _ {t, k} ^ {B} = \nabla_ {a ^ {B}} \mathcal {G} (a ^ {B}, a ^ {N}), \quad \Delta a _ {t, k} ^ {N} = \frac {\partial \phi_ {N} (a ^ {B})}{\partial a ^ {B}} \Delta a _ {t, k} ^ {B},
$$
$$
a _ {t, k + 1} ^ {B} = a _ {t, k} ^ {B} - \eta_ {a} \Delta a _ {t, k} ^ {B}, \quad a _ {t, k + 1} ^ {N} = a _ {t, k} ^ {N} - \eta_ {a} \Delta a _ {t, k} ^ {N}.
$$
11: Execute action  $a_{t}$  and observe reward  $r_{t}$  and next state  $s_{t+1}$ .
12: Store the transition  $< s_t, a_t, r_t, s_{t+1} >$  in  $\mathcal{D}$ .
13: Sample a mini-batch of transitions in  $\mathcal{D}$
14: Update actor and reward critic networks
$$
\theta \leftarrow \theta + \eta_ {\mu} \hat {\nabla} _ {\theta} \mathbb {E} _ {\mathcal {D}} \left[ Q _ {\omega} \left(s _ {t}, \tilde {\pi} _ {\theta} \left(s _ {t}\right)\right) - \sum_ {j} \nu^ {j} \max  \left\{0, g _ {j} \left(\tilde {\pi} _ {\theta} \left(s _ {t}\right); s _ {t}\right) \right\} \right],
$$
$$
\omega \gets \omega - \eta_ {Q} \hat {\nabla} _ {\omega} \mathbb {E} _ {\mathcal {D}} \left[ Q _ {\omega} (s _ {t}, a _ {t}) - (r _ {t} + \gamma Q _ {\omega^ {\prime}} (s _ {t + 1}, \pi_ {\theta^ {\prime}} (s _ {t + 1}))) \right] ^ {2}.
$$
15: Perform dual update on  $\nu$
$$
\nu_ {k + 1} ^ {j} = \nu_ {k} ^ {j} + \eta_ {\nu} ^ {j} \mathbb {E} _ {s \sim \pi} [ \max  \{0, g _ {j} (\tilde {\pi} (s _ {t}); s _ {t}) \} ] \quad \forall j.
$$
16: Soft update target networks:
$$
\theta^ {\prime} \leftarrow \tau \theta + (1 - \tau) \theta^ {\prime},
$$
$$
\omega^ {\prime} \leftarrow \tau \omega + (1 - \tau) \omega^ {\prime}.
$$
17: end for
18: end for
In this section, we will illustrate the dynamics of our three benchmarks in detail.
D.1 Safe CartPole
Different from the CartPole environment in Gym [12], two forces from different directions are controlled in the Safe CartPole environment.
State. The state space  $|S| \in \mathbb{R}^6$  of Safe CartPole includes the position, velocity, and acceleration of the cart; and the angle, angular velocity, and angular acceleration of the pole.
Algorithm 4 RPO-SAC
Input: Initial parameters  $\theta, \omega_1, \omega_2$  of policy network  $\mu$  and value network  $Q_1, Q_2$ , Temperature parameter  $\alpha$ .
1: Initialize the corresponding target network  $\omega_1^{\prime}\gets \omega_1,\omega_2^{\prime}\gets \omega_2$
2: Initialize replay buffer  $\mathcal{D}$
3: Initialize penalty factor for inequality constraints  $\nu \gets 0$ .
4: for each episode do
5: for each environment epoch do
6: Select partial action  $a_{t}^{B}\sim \mu_{\theta}(a_{t}^{B}|s_{t})$
7: Employ construction stage by solving the equations defined equality constraints
8: Employ projection stage on the concatenated action  $\tilde{a}_t$  to obtain feasible action  $a_t$
9: for  $k = 1, \dots, K$  do
$$
\theta \leftarrow \theta + \eta_ {\mu} \hat {\nabla} _ {\theta} \mathbb {E} _ {\mathcal {D}} \left[ - \alpha \log \left(\mu_ {\theta} (a _ {t} | s _ {t})\right) + Q _ {\omega} (s _ {t}, \tilde {\pi} _ {\theta} (s _ {t})) - \sum_ {j} \nu^ {j} \max  \left\{0, g _ {j} (\tilde {\pi} _ {\theta} (s _ {t}); s _ {t}) \right\} \right],
$$
$$
\omega_ {i} \leftarrow \omega_ {i} - \eta_ {Q} \hat {\nabla} _ {\theta Q} \mathbb {E} _ {\mathcal {D}} \left[ Q _ {\omega_ {i}} (s _ {t}, a _ {t}) - \left(r _ {t} + \gamma Q _ {\omega_ {i} ^ {\prime}} (s _ {t + 1}, \pi_ {\theta} (s _ {t + 1})) - \alpha \log (\mu_ {\theta} (a _ {t + 1} | s _ {t + 1}))\right) \right] ^ {2}.
$$
$$
\omega_ {i} ^ {\prime} \leftarrow \tau \omega_ {i} + (1 - \tau) \omega_ {i} ^ {\prime}.
$$
17: end for
Action. The action space  $|A| \in \mathbb{R}^2$  of Safe CartPole is the sign and magnitude of two forces  $f_1, f_2$  from different directions. Specifically, one force is inclined 30 degrees below the  $x$  axis, and the other is inclined 60 degrees above the  $x$  axis.
Reward. The goal of Safe CartPole is similar to the CartPole in Gym [12], which requires the pole to keep upright as long as possible. Therefore, we employ the same reward policy that returns 1 if the pole keeps upright. Otherwise, this episode will end since the pole falls.
Equality Constraints. The equality constraint of Safe CartPole is that the summation of two forces in  $y$  axis should be zero. That is, we desire to avoid extra friction on the cart, i.e.,
$$
f _ {y} := f _ {1} \sin \theta_ {1} + f _ {2} \sin \theta_ {2} = 0. \tag {21}
$$
Inequality Constraints. The inequality constraint is that the summation of two forces in the  $x$  axis should be bounded by a box constraint, which indicates the physical limitation of the magnitude of the summation force in the  $x$  axis.
$$
\underline {{f}} _ {x} \leq f _ {x} := f _ {1} \cos \theta_ {1} + f _ {2} \cos \theta_ {2} \leq \bar {f} _ {x}. \tag {22}
$$
Furthermore, according to [19], we derive the dynamics of Safe CartPole when there exists a force in the vertical direction as shown in (23). Notably, we can always derive the position or angle, velocity or angular velocity in the next step, using the semi-implicit Euler method.
$$
\begin{array}{l} N _ {c} = f _ {y} + \left(m _ {c} + m _ {p}\right) g - m _ {p} l \left(\ddot {\theta} \sin \theta + \dot {\theta} ^ {2} \cos \theta\right), \\ \ddot {\theta} = \frac {g \sin \theta - \frac {\mu_ {p} \theta}{m _ {p} l}}{l \left\{\frac {4}{3} - \frac {m _ {p} \cos \theta}{m _ {c} + m _ {p}} [ \cos \theta - \mu_ {c} \operatorname {s g n} (N _ {c} \dot {x}) ] \right\}} + \\ \frac {\cos \theta \left\{\frac {- f _ {x} - m _ {p} l \dot {\theta} ^ {2} [ \sin \theta + \mu_ {c} \operatorname {s g n} \left(N _ {c} \dot {x}\right) \cos \theta ]}{m _ {c} + m _ {p}} + \mu_ {c} g \operatorname {s g n} \left(N _ {c} \dot {x}\right) \right\}}{l \left\{\frac {4}{3} - \frac {m _ {p} \cos \theta}{m _ {c} + m _ {p}} [ \cos \theta - \mu_ {c} \operatorname {s g n} \left(N _ {c} \dot {x}\right) ] \right\}}, \tag {23} \\ \ddot {x} = \frac {f _ {x} + m _ {p} l \left(\dot {\theta} ^ {2} \sin \theta - \ddot {\theta} \cos \theta\right) - \mu_ {c} N _ {c} \operatorname {s g n} \left(N _ {c} \dot {x}\right)}{m _ {c} + m _ {p}}, \\ \end{array}
$$
where  $N_{c}$  is the pressure on the cart,  $m_c, m_p$  are the mass of the cart and pole,  $l$  is the length of the pole,  $x, \theta$  are the position of the cart and the angle of the pole respectively, and  $\mu_c$  is the dynamic friction coefficient of the cart.
D.2 Spring Pendulum
Motivated by the Pendulum environment [12], Spring Pendulum environment replaces the pendulum with a light spring, which connects the fixed point and the ball in the end of the spring.
State. The state space  $|S| \in \mathbb{R}^5$  of Spring Pendulum contains the cosine and sine of the angle, angular velocity, length of the spring, and the change rate in length.
Action. The action space  $|A| \in \mathbb{R}^2$  of Spring Pendulum is the sign and magnitude of two forces  $f_x, f_y$  in the  $x, y$  axes. Notably, since the spring pendulum is rotating, the angle between the  $x$  or  $y$  axis and the spring is changing as well. Thus, this environment is more difficult than Safe CartPole in some sense.
Reward. The goal of Spring Pendulum is to keep the spring pendulum in an upright position. The episode will never be done until the maximum time step. Specifically, the reward function is  $\frac{1}{1 + 100|\theta|}$ , where  $\theta$  is the angle between the spring pendulum and the  $y$  axis. That means the agent will achieve a reward of nearly 1 when the spring pendulum is close enough to the upright position. Otherwise, a reward of almost 0 will be returned to the agent.
Equality Constraints. The equality constraint of the Spring Pendulum is to limit the change rate of length to zero since the spring pendulum is expected to perform like a normal pendulum without changing the length of the pendulum.
$$
\dot {l} _ {t} = \dot {l} _ {t - 1} + \ddot {l} _ {t} d t = 0. \tag {24}
$$
Notably, when  $\dot{l}_{t - 1} = 0$ , the equality constraint will be  $\ddot{l} = 0$
Inequality Constraints. The inequality constraint is the magnitude constraint on the summation of these two forces.
$$
f _ {x} ^ {2} + f _ {y} ^ {2} \leq \bar {f} ^ {2}. \tag {25}
$$
Furthermore, to connect each component mentioned above, we derive the dynamics of the Spring Pendulum. Applying Euler-Lagrange Equation  $\mathcal{L} = T - V$  to spring pendulum, then we obtain
$$
\begin{array}{l} V = m g y + \frac {1}{2} k (l - l _ {0}) ^ {2} \\ = m g l \cos \theta + \frac {1}{2} k \left(l - l _ {0}\right) ^ {2}, \tag {26} \\ \end{array}
$$
$$
\begin{array}{l} T = \frac {1}{2} m v ^ {2} = \frac {1}{2} m \left(\dot {x} ^ {2} + \dot {y} ^ {2}\right) \\ = \frac {1}{2} m \left(\dot {l} ^ {2} + \dot {l} ^ {2} \dot {\theta} ^ {2}\right). \\ \end{array}
$$
Applying the Euler-Lagrange equations for  $\theta$  and  $l$ , we have
$$
\frac {d}{d t} \frac {\partial \mathcal {L}}{\partial \dot {\theta}} = 2 m \dot {l} \dot {\theta} + m l \ddot {\theta},
$$
$$
\frac {\partial \mathcal {L}}{\partial \theta} = - m g \sin \theta ,
$$
$$
\frac {d}{d t} \frac {\partial \mathcal {L}}{\partial \dot {l}} = m \ddot {l} - m l \dot {\theta} ^ {2} + k (l - l _ {0}), \tag {27}
$$
$$
\frac {\partial \mathcal {L}}{\partial l} = - m g \cos \theta .
$$
Finally, we will obtain the dynamics of the spring pendulum
$$
\ddot {\theta} = \frac {f _ {r} - 2 m \dot {l} \dot {\theta} - m g \sin \theta}{m l},
$$
$$
\ddot {l} = \frac {f _ {s} + m l \dot {\theta} ^ {2} - k (l - l _ {0}) - m g \cos \theta}{m}, \tag {28}
$$
where  $f_{r} = -f_{y}\sin \theta +f_{x}\cos \theta$  and  $f_{s} = f_{y}\cos \theta +f_{x}\sin \theta$  are the force perpendicular to and along the spring pendulum.
D.3 Optimal Power Flow with Battery Energy Storage
In smart grid operation controlling, Optimal Power Flow (OPF) is defined as
$$
\min  _ {p _ {g}, q _ {g}, v} \quad p _ {g} ^ {T} A p _ {g} + b ^ {T} p _ {g}
$$
subject to  $\underline{p}_g\leq p_g\leq \overline{p}_g,$
$$
\underline {{q}} _ {g} \leq q _ {g} \leq \bar {q} _ {g}, \tag {29}
$$
$$
\underline {{v}} \leq | v | \leq \bar {v},
$$
$$
\left(p _ {g} - p _ {d}\right) + \left(q _ {g} - q _ {d}\right) i = \operatorname {d i a g} (v) Y v,
$$
where  $p_g, q_g \in \mathbb{R}^n$  are the active and reactive power generation of the buses, and  $v \in \mathbb{C}^n$  are the voltage of the buses in the grid.  $Y \in \mathbb{C}^{n \times n}$  denotes the admittance matrix.  $p_d, q_d \in \mathbb{R}^n$  are active and reactive power demand of all buses. Notably, some buses in the electricity grid are not generator buses, and  $p_g, q_g$  of these buses will be zero and one of the generator buses is the reference (slack) bus which has a fixed voltage argument  $\angle v$ . Therefore, the actual dimension of  $p_g, q_g$  to determine is the number of generator buses  $n^g$ . Based on the OPF problem, OPF with Battery Energy Storage is defined as
$$
\min  _ {p _ {g}, q _ {g}, v, p _ {b}} \sum_ {t = 0} ^ {T} p _ {g} ^ {T} (t) A p _ {g} (t) + b ^ {T} p _ {g} (t) + c ^ {T} (t) p _ {b} (t)
$$
subject to  $(p_{g}(t) - p_{d}(t) - p_{b}(t)) + (q_{g}(t) - q_{d}(t))i = \mathrm{diag}(v(t))Yv(t),$
$$
\underline {{p}} _ {g} \leq p _ {g} (t) \leq \bar {p} _ {g}, \tag {30}
$$
$$
\underline {{q}} _ {g} \leq q _ {g} (t) \leq \overline {{q}} _ {g},
$$
$$
\underline {{v}} \leq | v (t) | \leq \bar {v},
$$
$$
\underline {{p}} _ {b} (t) \leq p _ {b} (t) \leq \overline {{p}} _ {b} (t).
$$
The additional variables  $p_b \in \mathbb{R}^n$  is the charging (positive) or discharging (negative) power of the battery groups, and  $c(t)$  represents the cost or the income in the time step  $t$ . Exactly, we only connect the batteries with the generator buses. Therefore, the actual number of  $p_b$  to determine is  $n^g$  in this benchmark as well.
Specifically, this benchmark is based on a 14-node power system, which is available in the PYPOWER package. We adopt the same topology of the electricity grid with 5 generator nodes in this benchmark. The data on power demand and day-ahead electricity prices are from [1, 2]. We refer to the distribution of the real-world data in one day, which is shown in Figure 4, and normalize
Figure 4: Distribution of power demand (left) and electricity price (right) in 24 hours.
the magnitude of the concrete data to incorporate them into the 14-node power system.
State. The state space  $|S| \in \mathbb{R}^{57}$  of OPF with Battery Energy Storage contains active and reactive power demand of 14 buses, the state of 5 batteries connected to 5 generator nodes, and the 24-hour day-ahead electricity price.
Action. The action space  $|A| \in \mathbb{R}^{43}$  of OPF with Battery Energy Storage involves the active and reactive power generation of 5 generator buses, voltage magnitude and argument of 14 buses, and the power generation or demand of 5 batteries. Notably, since the voltage argument in the slack bus is fixed, the actual action space is  $|\tilde{A}| \in \mathbb{R}^{42}$ . However, for convenience, we include it in the action space and set the fixed value for it in the practical implementation.
Reward. The goal of OPF with Battery Energy Storage is to minimize the total cost. Therefore, we regard the negative cost in the current time step  $-p_{g}^{T}(t)Ap_{g}(t) - b^{T}p_{g}(t) + c^{T}(t)p_{b}(t)$  as the reward. Actually, there exist efficiency parameters  $\eta_{ch},\eta_{dis}$  in the procedure of charging and discharging. Moreover, each part of the cost may have a different magnitude, which needs us to consider a tradeoff between different parts. For convenience, we model all of these factors into  $c(t)$ .
Equality Constraints. The 28 equality constraints of OPF with Battery Energy Storage are the equations of power flow, i.e.,
$$
\left(p _ {g} (t) - p _ {d} (t) - p _ {b} (t)\right) + \left(q _ {g} (t) - q _ {d} (t)\right) i = \operatorname {d i a g} (v (t)) Y v (t). \tag {31}
$$
Inequality Constraints. The inequality constraint is 58 box constraints on the decision variables.
$$
\underline {{p}} _ {g} \leq p _ {g} (t) \leq \overline {{p}} _ {g},
$$
$$
\underline {{q}} _ {g} \leq q _ {g} (t) \leq \bar {q} _ {g}, \tag {32}
$$
The dynamics of OPF with Battery Energy Storage are much simpler than the former two benchmarks, since the active and reactive power demand  $p_d$ ,  $q_d$  in each time step, is irrelevant to the last state and action. Thus, we only need to care about the change in the electrical power of batteries, i.e.,  $soc(t) = soc(t - 1) + [\eta_{ch}p_{ch}(t) + p_{dis}(t) / \eta_{dis}]$ , where  $p_{ch}(t) = \max \{0, p_b(t)\}$ ,  $p_{dis}(t) = \min \{0, p_b(t)\}$ .
Implementation of Construction Stage. Unlike previous robotic manipulation cases, OPF with Battery Energy Storage environment has nonlinear equality constraints. This means the analytical solution of the corresponding equations cannot be obtained. In addition, while Newton's methods can be applied directly here, it is sometimes unstable empirically. Hence, we followed the idea in
[18], which divides the equation-solving procedure into two steps. The first step is to compute the amplitude of voltage  $|v|$  in buses without the generator and voltage argument  $\angle v$  in all buses via Newton's methods. Then, we can directly compute the solution of reactive power generation  $q_{g}$  in all buses and  $p_{g}$  in the slack bus in the second step. Moreover, the gradient flow from nonbasic actions to power generation or demand of battery groups  $p_{b}$  can be calculated similarly to the gradient flow from chosen nonbasic actions to active power demand  $p_{d}$ .
D.4 More Explanation on Box Inequality Constraints
Notably, while our three benchmarks only contain box inequality constraints, they are sufficient to evaluate the performance of our RPO algorithm in scenarios with nonlinear inequality constraints. As mentioned in Section 3, any nonlinear inequality constraints can always be transformed into equality constraints plus inequality box constraints by adding slack variables. Hence, evaluating RPO with box inequality constraints and nonlinear equality constraints is sufficient to show RPO's generality. For example, if we have a complex inequality constraint  $g(s,a) \leq 0$ . Then, we can always transform it into
$$
g (s, a) + a ^ {\mathrm {a u g}} = 0, \quad a ^ {\mathrm {a u g}} \leq 0.
$$
where  $a^{\mathrm{aug}}$  is the slack variable, which can be viewed as the augmented actions in RL.
To further discuss the efficiency of RPO, we do some extra experiments in our hardest benchmark, OPF with Battery Energy Storage. The results and related analysis are presented below.
E.1 Sensitivity Analysis
The projection stage performs the GRG updates until all inequality constraints are satisfied. It is valuable to investigate the impact of the number of maximum GRG updates  $K$  on model performance. Here, we conduct experiments under different  $K$  values in OPF with the Battery Energy Storage task. As shown in Figure 5, we compare the performance of RPOSAC under  $K = 0,10,50,200$  in the projection stage. The result indicates that the
Figure 5: Comparison on RPO under different  $K$ .
choice of  $K$  does not have an obvious impact on the episodic reward when  $K$  is not too large. The slight improvement in constraint violation between  $K = 10$  and  $K = 50$  illustrates that  $K = 10$  is actually the most appropriate since it needs much less computation compared with  $K = 50$  during the projection stage. Besides, the case of  $K = 200$  shows that too large modifications on actions during the training process will lead to poor performance in some situations. The principal reason we believe is that the value network cannot back-propagate an accurate gradient to the policy network when  $K$  is too large. Concretely, a large  $K$  leads to the samples being too far from the policy  $\tilde{\pi}_{\theta}$ , which further results in the inaccurate estimation of the policy  $\tilde{\pi}_{\theta}$  in the value network. Notably, a practical trick is to set a small  $K$  during the training period and a large  $K$  during the evaluation period.
E.2 Adaptive v.s. Fixed Penalty Factor