SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Policy Loss with Modified Lagrangian Relaxation. While the above two-stage decision procedure can cope with the satisfaction of equality and inequality constraints, the policy network is also required to guarantee certain feasibility of inequality constraints for better initial actions  $\tilde{a}$ . Otherwise, we may need hundreds of GRG updates in the projection stage to satisfy all the inequality constraints. Here, we use an augment loss with Lagrangian relaxation for the policy network to obtain initial actions for the inequality projection stage. Common approaches such as [50] use fixed Lagrangian multipliers in their loss function. However, such fixed Lagrangian multipliers are not easy to obtain and may require extra information and computation to tune. In this paper, we perform a dual update on Lagrangian multipliers to adaptively tune them during the training period. As illustrated in Section 3, after the equality construction stage the constrained MDP problem is formulated as
$$
\max _ {\pi} J _ {R} \big (\tilde {\pi} _ {\theta} \big)
$$
$$
\begin{array}{l} \pi \\ \text {s u b j e c t t o} \quad g _ {j} \left(\tilde {\pi} _ {\theta} \left(s _ {t}\right); s _ {t}\right) \leq 0, \forall j, t. \end{array} \tag {7}
$$
This means we need to deal with instantaneous inequality constraints in all states. Hence, we cannot apply the primal-dual update method directly like PDO [13]. Otherwise, we need to compute the dual variables on all states, which is obviously unrealistic. Fortunately, we can maintain only one Lagrangian multiplier  $\nu^{j}$  for each inequality constraint in all states with the exact penalty term  $\max \left\{0,g_{j}(\tilde{\pi}(s_{t});s_{t})\right\}$  [27]. Accordingly, the new objective with the exact penalty term is
$$
\min  _ {\theta} \tilde {\mathcal {L}} (\theta) \triangleq - J _ {R} (\tilde {\pi} _ {\theta}) + \mathbb {E} _ {s \sim \pi} \left[ \sum_ {j} \nu^ {j} \max  \{0, g _ {j} (\tilde {\pi} (s _ {t}); s _ {t}) \} \right] \tag {8}
$$
The following theorem establishes the equivalence between the unconstrained problem (8) and the constrained problem (7).
Theorem 2. (Exact Penalty Theorem) Assume  $\nu_{s}^{j}$  is the Lagrangian multiplier vector corresponding to  $j$ th constraints in state  $s$ . If the penalty factor  $\nu^{j} \geq \| \nu_{s}^{j} \|_{\infty}$ , the unconstrained problem (8) is equivalent to the constrained problem (7).
The proof is referred to in the supplementary materials.
$$
\nu_ {k + 1} ^ {j} = \nu_ {k} ^ {j} + \eta_ {\nu} ^ {j} \mathbb {E} _ {s \sim \pi} [ \max  \{0, g _ {j} (\tilde {\pi} (s _ {t}); s _ {t}) \} ] \tag {9}
$$
where  $\eta_{\nu}^{j}$  is the learning rate of  $j$ -th penalty factors,  $\nu_{k}^{j}$  is value of  $\nu^{j}$  in the  $k$ -th step and  $\nu_{0}^{j} = 0$ . Since the exact penalty term,  $\mathbb{E}_{s\sim \pi}\left[\max \left\{0,g_{j}(\tilde{\pi}(s_{t});s_{t})\right\}\right]$ , is always non-negative, the penalty factors are monotonically increasing during the training procedure. Hence, we can always obtain sufficiently large  $\nu^{j}$  that satisfies the condition in Theorem 2, i.e.,  $\nu^{j} \geq \| \nu_{s}^{j}\|_{\infty}$ . Besides, we also find that the adaptive penalty term does not prevent the exploration for higher rewards of the RL agent at the beginning of the training procedure in Section 5.
Off-policy RL Training. Since we augment the policy loss function with the exact penalty term, the actual objective of our algorithm is two-fold. One is to obtain the optimal policy with the satisfaction of hard constraints. Another is to reduce the number of GRG updates performed during the projection stage. This indicates there exists a gap between the behavioral policy and the target policy in RPO, which results from the changing times of GRG updates performed in the projection stage.
Hence, RPO should be trained like off-policy RL methods as Algorithm 1. Specifically, we regard the initial actions  $\tilde{a}$  output by the construction stage as the optimization object rather than the actions  $a$  post-processed by the projection stage. Otherwise, the training process will be unstable and even collapse due to the changing times of GRG updates in the projection stage, which can also be viewed as a changing network architecture. Besides, we use the  $y_{t} = r_{t} + \gamma Q_{\omega}(s_{t+1}, \pi_{\theta}(s_{t+1}))$  to construct the TD target since  $\pi_{\theta}$  is the actual policy we deploy in the environment.
To validate our method and further facilitate research for MDP with hard constraints, we develop three benchmarks with visualization according to the dynamics in the real world, ranging from classical robotic control to smart grid operation. They involve Safe CartPole, Spring Pendulum, and Optimal Power Flow with Battery Energy Storage. Then, we incorporate RPO into two classical off-policy RL algorithms, DDPG [23] and SAC [21], which we call RPO-DDPG and RPO-SAC respectively.
RPO is compared with three representative Safe RL algorithms, including CPO [4], CUP [48], and Safety Layer [16]. Notably, we transform the hard equality constraints into two inequality constraints since existing Safe RL methods cannot handle both general equality and inequality constraints. Furthermore, we also contrast the RPO-DDPG and RPO-SAC with DDPG-L and SAC-L, where DDPG-L and SAC-L represent DDPG and SAC only modified with the Lagrangian relaxation method we mentioned in Section 4.3 and without the two-stage decision process in RPO respectively. Besides, DDPG-L and SAC-L deal with the equality constraints as we mentioned in Safe RL algorithms. More details related to the concrete RPO-DDPG and RPO-SAC algorithms are illustrated in the supplementary materials. Our code is available at: https://github.com/wadx2019/rpo.
5.1 RL Benchmarks with Hard Constraints
Specifically, our benchmarks are designed based on [12], with extra interfaces to return the information of the hard constraints. To the best of our knowledge, it is the first evaluation platform in RL that considers both equality constraints and inequality constraints. Figure 2 shows the visualization of these three benchmarks, and the simple descriptions for them are presented in the contexts below. More details about them are provided in the supplementary materials.
1) Safe CartPole. Different from that standard CartPole environment in Gym [12], we control two forces from different directions in the Safe CartPole environment. The goal is to keep the pole upright as long as possible while the summation of the two forces should be zero in the vertical direction and be bounded by a box constraint in the horizontal direction. That is, the former is the hard equality constraint, while the latter is the hard inequality constraint.
2) Spring Pendulum. Motivated by the Pendulum environment [12], we construct a Spring Pendulum environment that replaces the pendulum with a light spring, which connects the fixed point and the ball. In order to keep the spring pendulum in the upright position, two torques are required to apply in both vertical and horizontal directions. Meanwhile, the spring should be maintained at a fixed length, which introduces a hard equality constraint. Unlike that in Safe CartPole, here equality constraint is
(a) Safe CartPole
(b) Spring Pendulum
(c) OPF with Battery Energy Storage
Figure 2: Visualization of RL benchmarks with hard constraints. (a) Safe CartPole. The indicator lamp in the cart will be green if the given actions are feasible, and will be red otherwise. (b) Spring Pendulum. The length of the spring will be changed if the equality constraint is violated. (c) OPF with Battery Energy Storage. The circle nodes represent the buses in the electricity grid, and the square nodes represent the batteries connected to the generator buses. We use light and shade to reflect the state of batteries and the power generation and demand of buses. In addition, the edge between the generator bus and the battery will be red if the battery is charging, and green if the battery is discharging.
Figure 3: Learning Curves of different algorithms on our three benchmarks across 5 runs. The x-axis is the number of training epochs. The y-axis is the episodic reward (the first line), and max instantaneous constraint violation (the second line) respectively.
state-dependent since the position of the spring is changed during the dynamical process. Besides, the summation of the two torques is also bounded by introducing a quadratic inequality constraint.
3) Optimal Power Flow with Battery Energy Storage. Optimal Power Flow (OPF) is a classical problem in smart grid operation to minimize the total cost of power generation with the satisfaction of grid demand. However, battery energy storage systems or electric vehicles have been integrated into smart grids to alleviate the fluctuations of renewable energy generation recently. As illustrated in [30, 36], we need to jointly optimize the charging or discharging of batteries with OPF for the long-term effect of the smart grid operation. According to this real-world application, we design this environment, which is possessed of nonlinear power flow balance equality constraints and inequality box constraints on the actions for feasibility.
We use three metrics to evaluate the algorithms as follows:
Episodic Reward: cumulative reward in a whole episode.
Task	Metrics	CPO	CUP	Safety Layer	DDPG-L	SAC-L	RPO-DDPG(*)	RPO-SAC(*)
Safe CartPole	Ep. reward	20.1000(2.9138)	63.9000(15.0629)	53.8000(25.7364)	15.5000(1.7464)	40.5000(15.5772)	200.0000(0.0000)	200.0000(0.0000)
Max In. ineq	0.0000(0.0000)	0.0000(0.0000)	0.0086(0.0153)	0.0000(0.0000)	0.0000(0.0000)	0.0000(0.0000)	0.0000(0.0000)
Max In. eq	0.0389(0.0155)	0.5344(0.0148)	1.7099(1.8248)	0.0408(0.0055)	0.0487(0.0126)	0.0000(0.0000)	0.0000(0.0000)
Max Ep. ineq	0.0000(0.0000)	0.0000(0.0000)	0.7904(1.6355)	0.0000(0.0000)	0.0000(0.0000)	0.0000(0.0000)	0.0000(0.0000)
Max Ep. eq	0.1096(0.0217)	0.0000(0.0000)	13.6603(0.0000)	0.1180(0.0131)	0.1038(0.0645)	0.0000(0.0000)	0.0000(0.0000)
Spring Pendulum	Ep. reward	15.4687(12.7866)	0.8599(0.5242)	1.1155(0.6959)	76.4383(27.4531)	149.6762(4.2608)	175.1558(15.1087)	182.4221(2.8414)
Max In. ineq	0.0000(0.0000)	0.0000(0.0000)	6.0975(6.8860)	0.0000(0.0000)	0.0000(0.0000)	0.0000(0.0000)	0.0000(0.0000)
Max In. eq	0.7274(0.4259)	9.5898(2.6630)	3.9134(2.8721)	0.3805(0.0441)	0.0078(0.0013)	0.0000(0.0000)	0.0000(0.0000)
Max Ep. ineq	0.0000(0.0000)	0.0000(0.0000)	36.9545(9.1365)	0.0000(0.0000)	0.0000(0.0000)	0.0000(0.0000)	0.0000(0.0000)
Max Ep. eq	1.9064(0.7869)	16.3114(4.9027)	25.2101(10.1734)	1.0905(0.5024)	0.0837(0.0809)	0.0000(0.0000)	0.0000(0.0000)
OPF with Battery Energy Storage	Ep. reward	-10.3469(1.3825)	-5.8377(0.9521)	-24.1147(8.9916)	-22.0649(0.9199)	-19.4738(2.0778)	42.9710(17.8438)	42.0764(10.1048)
Max In. ineq	0.0000(0.0000)	0.0000(0.0000)	0.0000(0.0000)	0.0000(0.0000)	0.0000(0.0000)	0.0002(0.0004)	0.0003(0.0005)
Max In. eq	0.4413(0.0215)	0.4866(0.0136)	11.4698(10.7497)	0.1348(0.0136)	0.1908(0.0203)	0.0001(0.0000)	0.0000(0.0000)
Max Ep. ineq	0.0000(0.0000)	0.0000(0.0000)	0.0000(0.0000)	0.0000(0.0000)	0.0000(0.0000)	0.0042(0.0094)	0.0046(0.0060)
Max Ep. eq	0.6821(0.0523)	0.6827(0.0612)	69.9568(6.0647)	0.2845(0.0484)	0.3994(0.1243)	0.0003(0.0001)	0.0002(0.0001)
Table 2: Mean evaluation performance of different algorithms in the three benchmarks. We compare the performance of RPO (RPODDPG, RPOSAC) with the abridged version of our algorithm (DDPG-L, SAC-L) and other Safe RL algorithms (CPO, CUP, Safety Layer) according to episodic reward and max instantaneous & episodic constraint violation for equality and inequality constraints. Each item in the table is averaged across 10 runs with the standard deviations shown in the parentheses.
Max Instantaneous Constraint Violation: maximum instantaneous violation of all constraints. It denotes the feasibility of actions in a state because the feasibility of actions relies on the constraint that is the most difficult to be satisfied.
Max Episodic Constraint Violation: maximum episodic violation of all constraints. Similar to Mean Constraint Violation, it denotes the feasibility of actions in a whole episode.
5.3 Performance of RPO on Reward and Constraints
We plot the learning curves of CPO, CUP, Safety Layer, DDPG-L, SAC-L, RPO-DDPG, and RPO-SAC in Figure 3. Notably, to validate the importance of the two-stage decision procedure, here SAC-L and DDPG-L are actually the versions of RPO-SAC and RPO-DDPG without this procedure. Given the fairness of our experiments, we apply the same shared hyper-parameters for all the algorithms. This empirical result reflects that existing Safe RL algorithms cannot handle the MDP problems with hard constraints, and our approach outperforms other algorithms in terms of both episodic reward and the max constraint violation. Moreover, the learning curves confirm that RPO can also guarantee certain feasibility during the training period.
Besides the learning curves, Table 2 shows the performance of different algorithms after convergence. To present more details on the constraint violations, here the equality and inequality constraint violations are shown separately. Notably, since the tanh and state-dependent tanh activation function are added to limit the output of the neural network for the box constraints, there is no inequality constraint that needs to be satisfied for DDPG-L, SAC-L, and three Safe RL algorithms. That's why these two algorithms achieve zero violation in the inequality constraints in OPF with Battery Energy Storage. However, it is hard for them to satisfy both equality and inequality constraints exactly in OPF with Battery Energy Storage. This is because this environment is very difficult and involves many complex equality and inequality constraints. Practically, a tolerance 1e-3 is introduced here to evaluate the satisfaction of constraints.
6 Limitation and Future Work
We acknowledge that there still exist some limitations in RPO. One is that RPO is time-consuming compared to standard neural networks due to the projection stage, where the GRG updates may need to be performed several times. Another is that the equality equation solver required in the construction stage may need to be either chosen or specially designed with domain knowledge. Hence, accelerating RPO and developing more powerful RL algorithms that can handle hard constraints are under consideration in our future works. Besides, while we only validate RPO in RL benchmarks with hard constraints, our method can also be easily extended to cases with both hard constraints and soft constraints as long as a neural network is utilized to fit the mapping between the state-action pair and the cost. Moreover, RPO can be viewed as a supplement to Safe RL algorithms. In scenarios with both hard instantaneous constraints and soft cumulative constraints, we can always use RPO to handle the hard instantaneous constraints and apply some existing Safe RL methods to handle soft cumulative constraints.
In this paper, we outlined a novel algorithm called RPO to handle general hard constraints under the off-policy reinforcement learning framework. RPO consists of two stages, the construction stage for equality constraints and the projection stage for inequality constraints. Specifically, the construction stage first predicts the basic actions, then calculates the nonbasic action through an equation-solving procedure, and finally concatenates them as the output of the construction stage. The projection stage applies GRG updates to the concatenated actions until all the inequality constraints are satisfied. Furthermore, we also design a special augmented loss function with the exact penalty term and illustrate how to fuse RPO with the off-policy RL training process. Finally, to validate our method and facilitate the research in RL with hard constraints, we have also designed three benchmarks according to the physical nature of the real-world applications, including Safe CartPole, Spring Pendulum, and Optimal Power Flow with Battery Energy Storage. Experimental results in these benchmarks demonstrate the superiority of RPO in terms of both episodic reward and constraint violation.
This work was supported by NSFC (No.62303319), Shanghai Sailing Program (22YF1428800, 21YF1429400), Shanghai Local College Capacity Building Program (23010503100), Shanghai Frontiers Science Center of Human-centered Artificial Intelligence (ShangHAI), MoE Key Laboratory of Intelligent Perception and Human-Machine Collaboration (ShanghaiTech University), and Shanghai Engineering Research Center of Intelligent Vision and Imaging.
[1] Electricity - u.s. energy information administration (eia). https://www.eia.gov/electricity/. Accessed November 10, 2022 [online].
[2] Nord pool. https://www.nordpoolgroup.com/. Accessed November 10, 2022 [online].
[3] Jean Abadie. Generalization of the wolfe reduced gradient method to the case of nonlinear constraints. Optimization, pages 37-47, 1969.
[4] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 2017.
[5] A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and Z. Kolter. Differentiable convex optimization layers. In Advances in Neural Information Processing Systems, 2019.
[6] Dima Waleed Hanna Alrabadi. Portfolio optimization using the generalized reduced gradient nonlinear algorithm: An application to amman stock exchange. International Journal of Islamic and Middle Eastern Finance and Management, 9(4):570-582, 2016.
[7] Eitan Altman. Constrained Markov decision processes: stochastic modeling. Routledge, 1999.
[8] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, pages 136-145. PMLR, 2017.
[9] Osbert Bastani, Shuo Li, and Anton Xu. Safe reinforcement learning via statistical model predictive shielding. In Robotics: Science and Systems, pages 1-13, 2021.
[10] Abhinav Bhatia, Pradeep Varakantham, and Akshit Kumar. Resource constrained deep reinforcement learning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 29, pages 610-620, 2019.
[11] Steven Bohez, Abbas Abdelmaleki, Michael Neunert, Jonas Buchli, Nicolas Heess, and Raia Hadsell. Value constrained model-free continuous control. arXiv preprint arXiv:1902.04623, 2019.
[12] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
[13] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1):6070-6120, 2017.
[52] Yuhao Zhou, Bei Zhang, Chunlei Xu, Tu Lan, Ruisheng Diao, Di Shi, Zhiwei Wang, and Wei-Jen Lee. A data-driven method for fast ac optimal power flow solutions via deep reinforcement learning. Journal of Modern Power Systems and Clean Energy, 8(6):1128-1139, 2020.
Proposition 1. (Gradient Flow in Construction Stage) Assume that we have  $(n - m)$  equality constraints denoted as  $F(a;s) = 0$  in each state  $s$ . Let  $a^B \in \mathbb{R}^m$  and  $a \in \mathbb{R}^n$  be the basic actions and integrated actions, respectively. Let  $a^N = \phi_N(a^B) \in \mathbb{R}^{n - m}$  denote the nonbasic actions where  $\phi_N$  is the implicit function that determined by the  $n - m$  equality constraints. Then, we have
$$
\frac {\partial \phi_ {N} \left(a ^ {B}\right)}{\partial a ^ {B}} = - \left(J _ {:, m + 1: n} ^ {F}\right) ^ {- 1} J _ {:, 1: m} ^ {F} \tag {10}
$$
where  $J^{F} = \frac{\partial F(a;s)}{\partial a} \in \mathbb{R}^{(m - n)\times n}$ ,  $J_{:,1:m}^{F}$  is the first  $m$  columns of  $J^{F}$  and  $J_{:,m + 1:n}^{F}$  is the last  $(n - m)$  columns of  $J^{F}$ .
Proof. Considering an equality-constrained optimization problem, the reduced gradient [27] is defined as
$$
\mathbf {r} ^ {T} = \nabla_ {\mathbf {x} ^ {B}} f (\mathbf {x} ^ {N}, \mathbf {x} ^ {B}) + \lambda^ {T} \nabla_ {\mathbf {x} ^ {B}} \mathbf {h} (\mathbf {x} ^ {N}, \mathbf {x} ^ {B}), \tag {11}
$$
where  $\lambda^T$  should satisfy  $\nabla_{\mathbf{x}^N}f(\mathbf{x}^N,\mathbf{x}^B) + \lambda^T\nabla_{\mathbf{x}^N}\mathbf{h}(\mathbf{x}^N,\mathbf{x}^B) = 0$ .  $\mathbf{f},\mathbf{h}$  denote the objective function and the equality constraints, and  $x^{B},x^{N}$  are the basic variables and nonbasic variables respectively.
Finally, we will obtain
$$
\mathbf {r} ^ {T} = \nabla_ {\mathbf {x} ^ {B}} f (\mathbf {x} ^ {N}, \mathbf {x} ^ {B}) - \nabla_ {\mathbf {x} ^ {N}} f (\mathbf {x} ^ {N}, \mathbf {x} ^ {B}) [ \nabla_ {\mathbf {x} ^ {N}} \mathbf {h} (\mathbf {x} ^ {N}, \mathbf {x} ^ {B}) ] ^ {- 1} \nabla_ {\mathbf {x} ^ {B}} \mathbf {h} (\mathbf {x} ^ {N}, \mathbf {x} ^ {B}). \tag {12}
$$
Similarly, the gradient flow from the nonbasic actions to basic actions is
$$
\frac {\partial \phi_ {N} \left(a ^ {B}\right)}{\partial a ^ {B}} = - \left(J _ {:, m: n} ^ {F}\right) ^ {- 1} J _ {:, 1: m} ^ {F}. \tag {13}
$$
Proof. Firstly, assume the tangent space of the actions in the manifold defined by the equality constraints is  $J^{F}a = d$  where  $a = [a^{B},a^{N}] = [a^{B},\phi_{N}(a^{B})]$ ,  $a^{B}\in \mathbb{R}^{m}$ ,  $\phi_N(a^N)\in \mathbb{R}^{n - m}$ .
According to Proposition 1, we have
$$
\frac {\partial \phi_ {N} \left(a ^ {B}\right)}{\partial a ^ {B}} = - \left(J _ {:, m + 1: n} ^ {F}\right) ^ {- 1} J _ {:, 1: m} ^ {F}. \tag {14}
$$
Then, we define the projection objective as  $\mathcal{G}(a^B,a^N)\triangleq \sum_j\max \left\{0,g_j(a;s)\right\}$  and obtain
$$
\nabla_ {a ^ {B}} \mathcal {G} \left(a ^ {B}, a ^ {N}\right) \triangleq \frac {\partial \mathcal {G} \left(a ^ {B} , a ^ {N}\right)}{\partial a ^ {B}} + \frac {\partial \phi_ {N} \left(a ^ {B}\right)}{\partial a ^ {B}} \frac {\partial \mathcal {G} \left(a ^ {B} , a ^ {N}\right)}{\partial a ^ {N}} \tag {15}
$$
$$
\begin{array}{l} J ^ {F} \Delta a = \left[ J _ {:, 1: m} ^ {F}, J _ {:, m + 1: n} ^ {F} \right] \left[ \begin{array}{c} \Delta a ^ {B} \\ \Delta a ^ {N} \end{array} \right] \\ = \left[ J _ {,: 1: m} ^ {F}, J _ {,: m + 1: n} ^ {F} \right] \left[ \begin{array}{c} \Delta a ^ {B} \\ \Delta \phi_ {N} (a ^ {B}) \end{array} \right] \\ = J _ {: 1: m} ^ {F} \Delta a ^ {B} + J _ {:, m + 1: n} ^ {F} \Delta \phi_ {N} (a ^ {B}) \\ = J _ {:, 1: m} ^ {F} \Delta a ^ {B} + J _ {:, m + 1: n} ^ {F} \frac {\partial \phi_ {r} (a ^ {B})}{\partial a ^ {B}} \Delta a ^ {B} \\ = J _ {:, 1: m} ^ {F} \nabla_ {a ^ {B}} \mathcal {G} (a ^ {B}, a ^ {N}) + J _ {:, m + 1: n} ^ {F} \frac {\partial \phi_ {r} (a ^ {B})}{\partial a ^ {B}} \nabla_ {a ^ {B}} \mathcal {G} (a ^ {B}, a ^ {N}) \\ = J _ {: 1: m} ^ {F} \left(\frac {\partial \mathcal {G} \left(a ^ {B} , a ^ {N}\right)}{\partial a ^ {B}} + \frac {\partial \mathcal {G} \left(a ^ {B} , a ^ {N}\right)}{\partial \phi_ {N} \left(a ^ {B}\right)} \frac {\partial \phi_ {N} \left(a ^ {B}\right)}{\partial a ^ {B}}\right) \tag {16} \\ + J _ {:, m + 1: n} ^ {F} \frac {\partial \phi_ {r} \left(a ^ {B}\right)}{\partial a ^ {B}} \left(\frac {\partial \mathcal {G} \left(a ^ {B} , a ^ {N}\right)}{\partial a ^ {B}} + \frac {\partial \mathcal {G} \left(a ^ {B} , a ^ {N}\right)}{\partial \phi_ {N} \left(a ^ {B}\right)} \frac {\partial \phi_ {N} \left(a ^ {B}\right)}{\partial a ^ {B}}\right) \\ = J _ {:, 1: m} ^ {F} \frac {\partial \mathcal {G} \left(a ^ {B} , a ^ {N}\right)}{\partial a ^ {B}} - J _ {:, 1: m} ^ {F} \frac {\partial \mathcal {G} \left(a ^ {B} , a ^ {N}\right)}{\partial \phi_ {N} \left(a ^ {B}\right)} \left(J _ {:, m + 1: n} ^ {F}\right) ^ {- 1} J _ {:, 1: m} ^ {F} \\ - J _ {:, 1: m} ^ {F} \frac {\partial \mathcal {G} (a ^ {B} , a ^ {N})}{\partial a ^ {B}} + J _ {:, m + 1: n} ^ {F} \frac {\partial \mathcal {G} (a ^ {B} , a ^ {N})}{\partial \phi_ {N} (a ^ {B})} \left(J _ {:, m + 1: n} ^ {F}\right) ^ {- 1} J _ {:, 1: m} ^ {F} \\ = 0. \\ \end{array}
$$
Proof. Since  $\nu^j \geq \|\nu_s^j\|_\infty$ , we have
$$
\tilde {\mathcal {L}} (\theta) \geq - J _ {R} (\tilde {\pi} _ {\theta}) + \mathbb {E} _ {s \sim \pi} \left[ \sum_ {j} \sum_ {s} \nu_ {s} ^ {j} \max  \{0, g _ {j} (\tilde {\pi} (s); s) \} \right]. \tag {17}
$$
The equality holds when all the inequality constraints are satisfied. Here