Reduced Policy Optimization for Continuous Control with Hard Constraints
Authors: Shutong Ding
Jingya Wang
Yali Du
Ye Shi
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Hard Constraint
Generalized Reduced Gradient
TL;DR: We propose a reduced policy optimization algorithm that combines RL with generalized reduced gradient for continuos control tasks with hard constraints, and develop three new benchmarks for RL with hard constraints.

Abstract: Recent advances in constrained reinforcement learning (RL) have endowed reinforcement learning with certain safety guarantees. However, deploying existing constrained RL algorithms in continuous control tasks with general hard constraints remains challenging, particularly in those situations with non-convex hard constraints. Inspired by the generalized reduced gradient (GRG) algorithm, a classical constrained optimization technique, we propose a reduced policy optimization (RPO) algorithm that combines RL with GRG to address general hard constraints. RPO partitions actions into basic actions and nonbasic actions following the GRG method and outputs the basic actions via a policy network. Subsequently, RPO calculates the nonbasic actions by solving equations based on equality constraints using the obtained basic actions. The policy network is then updated by implicitly differentiating nonbasic actions with respect to basic actions. Additionally, we introduce an action projection procedure based on the reduced gradient and apply a modified Lagrangian relaxation technique to ensure inequality constraints are satisfied. To the best of our knowledge, RPO is the first attempt that introduces GRG to RL as a way of efficiently handling both equality and inequality hard constraints. It is worth noting that there is currently a lack of RL environments with complex hard constraints, which motivates us to develop three new benchmarks: two robotics manipulation tasks and a smart grid operation control task. With these benchmarks, RPO achieves better performance than previous constrained RL algorithms in terms of both cumulative reward and constraint violation. We believe RPO, along with the new benchmarks, will open up new opportunities for applying RL to real-world problems with complex constraints.

Reduced Policy Optimization for Continuous Control with Hard Constraints
Shutong Ding'
Jingya Wang'
Yali Du
Ye Shi
SShanghaiTech University
"King's College London
fdingsht, wangjingya, shinye/oshagghaitech.eee yali. www.cl. ac.
Abstract
Recent advances in constrained reinforcement learning (RL) have endowed rein- forcement learning with certain safety guarantees. However, deploying existing constrained RL algorithms in continuous control tasks with general hard constraints remains challenging, particularly in those situations with non-convex hard con- straints. Inspired by the generalized reduced gradient (GRG) algorithm, classical constrained optimization technique, we propose reduced policy optimization (RPO) algorithm that combines RL with GRG to address general hard constraints. RPO partitions actions into basic actions and nonbasic actions following the GRG method and output the basic actions via a policy network. Subsequently. RPO calculates the nonbasic actions by solving equations based on equality constraints using the obtained basic actions. The policy network is then updated by implicitly differentiating nonbasic actions with respect to basic actions. Additionally, we introduce action projection procedure based on the reduced gradient and apply a modified Lagrangian relaxation technique to ensure inequality constraints are satisfied. To the best of our knowledge, RPO is the first attempt that introduces GRG to RL as a way of efficiently handling both equality and incquality hard constraints. It is worth noting that there is currently lack of RI environments with complex hard constraints, which motivates us to develop three new benchmarks: roboties manipulation tasks and smart grid operation control task. With these benchmarks, RPO achieves better performance than previous constrained RL algorithms terms of both cumulative reward and constraint violation. We believe RPO, along with the new benchmarks, will open new opportunities for applying RL real-world problems with complex constraints.
1
Introduction
The past few years have witnessed the significant success of reinforcement learning (RL) 4 in
various fields such mastering GO (38), robotic manipulations (20 24), autonomous driving [34) and smart grid controlling [52] 47 33), etc. However, it is still challenging to deploy RL algorithms in real-world control tasks, such as operating robots on a specific surface, controlling power generation to fulfill the demand, etc. The principal reason here is that hard constraints must be taken into account in these control problems. Concretely, such constraints come the form of both equality and inequality constraints and can be nonlinear or even nonconvex. which makes it difficult to handle them in RL. Moreover, unlike soft constraints, hard constraints take explicit form and require strict compliance, which poses additional challenges.
33),
*Coresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Existing work on constrained RL can be divided into two categories. The first category involves treating constraints as implicit or soft constraints and using Safe RL algorithms 4 14 49
25
51
26 48). These algorithms approximate the cumulative costs associated with the constraints and optimize the policy network to balance the trade-off between the cumulative rewards and costs. While Safe RL algorithms have provided certain guarantees on soft constraints, they cannot handle equality constraints since the constraints may not be satisfied due to approximation errors. Moreover, handling multiple constraints using Safe RL algorithms can be computationally expensive, and the existing benchmarks generally only involve simple soft constraints that do not reflect the complexity of real-world applications.
In the second category, the approaches treat the output of the policy network as a set of sub-optimal actions, correcting them to satisfy the constraints by adding an extra constrained optimization procedure. This technique has been explored in several works, such as [6 29 Compared to Safe RI. algorithms, these algorithms can guarantee the satisfaction of hard constraints but are mostly designed for specific applications and hard constraints of particular form. For instance, OptLayer [29] employs OptNet [ into RL to ensure the satisfaction of linear constraints in robotic manipulation. As a result, these approaches to RL with hard constraints are limited in their ability to generalize and lack a generalized formulation.
10
33
32 24].
To address the limitations of existing RL algorithms in handling hard constraints, we propose a constrained off- off-cy reinforcement learning algorithm called Reduced Policy Optimization (RPO). Our approach is inspired by Generalized Reduced Gradient (GRG), classical optimization method. RPO partitions actions into basic actions and nonbasic actions following the GRG method and uses a policy network to output the basic actions. The nonbasic actions are then calculated by solving equations based on equality constraints using the obtained basic actions. Lastly, the policy network is updated by the reduced gradient with respect to basic actions, ensuring the satisfaction of the equality constraints. Moreover, we also incorporate modified Lagrangian relaxation method with an exact penalty term into the loss function of the policy network to improve the initial actions. Our approach provides more confidence when deploying off-policy RL algorithms in real- sallw applications, it ensures that the algorithms behave in feasible and predictable manner. is worth noting that there is currently lack of RL environments with complex hard constraints. This motivates us to develop three new benchmarks to validate the performance of our proposed method. To summarize, our main contributions are as follows:
1) Reduced Policy Optimization. We present RPO, an innovative approach that introduces the GRG algorithm into off-policy RL algorithms. RPO treats the output of the policy network good initial solution and enforces the satisfaction of equality and inequality constraints via solving corresponding equations and applying reduced gradient projections respectively. To the best of our knowledge, this the first attempt to fuse RL algorithms with the GRG method, providing a novel and effective solution to address the limitations of existing RL algorithms in handling hard constraints.
2) RL Benchmarks with Hard Constraints. We develop three benchmarks with hard constraints to validate the performance of our method, involving Safe CartPole, Spring Pendulum and Optimal Power Flow (OPF) with Battery Energy Storage. Comprehensive experiments on these benchmarks demonstrate the superiority of RPO in terms of both cumulative reward and constraint violation. We believe that these benchmarks will be valuable resources for the research community to evaluate and compare the performance of RL algorithm in environments with complex hard constraints.
2 Related Works
In this section, we first review the existing methods in the field of constrained RL and divide them into different types according different constraints. We summarize the differences between method and previous works in Table Besides, we also introduce the literature on the GRG method as the motivation and background of our method.
Cott-Constrainted RL. RL with soft constraints is well-studies and also known as safe reinforcement learning (Safe RL). One of the principal branches in Safe RL methods is based on the Lagrangian relaxation such as [3 39 22], where the primal-dual update used to enforce the satisfaction
of constraints. Besides, different penalty terms
(25 35 42 are designed to maintain the tradeoff
between the optimality in reward and safety guarantees. Moreover, Notable classical methods of
safe reinforcement learning include CPO [4] based on the local policy search; crapunvv-base

---

approaches 14;; PCPO 49). FOCOPS [5T, CUP (26] based on two-step optimization, EHCSB 46 based on bi-level optimization formulation, and SMPS based on recovery and shielding. Besides, RL with soft instantaneous constraints was first studied separately in [6)]. This approach adds safety layer to the policy with a pre-trained constraint-violation classifier but can only handle linear constraints. Other approaches include [43 based on the Recovery Policy and A5
44 based on
the Gaussian Process. Very recently, Unrolling Safety Layer [50] was proposed to handle the soft instantaneous constraints in RL. However, these approaches in soft-constrained RL tackle constraints implicitly and cannot ensure strict compliance with the constraints, especially the equality ones. By contrast, method can handle both hard equality and inequality constraints effectively under the RL framework.
RL. Compared to RL with soft constraints, RL with hard constraints is rarely studied, and most schemes are designed for some specific application. Pham et al. [29] proposed a plug-in architecture called Layer based on OptNet [8. which belongs to special neural network called optimization layer 5 40, to avoid infeasible actions in robotics manipulation. In studying resource allocation problems, Bhatia et al. [10] developed further Optlayer techniques to deal with hierarchical linear constraints. Liu et al. [24] investigated robotics manipulation tasks based on RL and used manifold optimization to handle the hard constraints with the inverse dynamic model of robots. Other researchers such as [47 33 incorporated special optimization techniques into RL to handle power operation tasks in smart grids. However, these methods are designed solely for specific applications or constraints of special types. For example, OptLayer can only handle linear constraints in robotics manipulation. By contrast, RPO is not designed for one specific application and can handle general hard constraints in decision-making problems.
Generalized Reduced Gradient Method. GRG 3 is a classical constrained optimization technique and has the powerful capability to handle optimization with nonlinear hard constraints [27]. The basic idea of GRG is closely related to the simplex method in linear programming which divides variables into basic and nonbasic groups and then utilizes the reduced gradient to perform the update on the basic variables and nonbasic variables respectively. In the past decades, the GRG method has been applied to stock exchange (6), optimal control in very-large-see
robotic systems [II, optimal power flow models [17), and many other fields. Additionally, more recently, several works such as [37] also fuse the genetic algorithms with the GRG method. Besides, recent research like DC3 [T8] in deep learning is also based on the idea of the GRG method. DC3 was proposed for learning solve constrained optimization problems and has demonstrated its good capability to obtain near-optimal decisions with the satisfaction of nonlinear constraints.
3 Preliminaries
Markov Decision Process. A classical Markov decision process (MDP) 41 can be represented as tuple (S, A, R, P. p).. where is the state space, A is the action space, R : x A x S + R is the reward function, P S x A x S [0, 1] is the transition probability function (where P s, a) is the transition probability from the previous state to the state s' when the agent took action a in s), and f 5 + [0, 1] is the distribution of the initial state. A stationary policy π : - P(A) is a map from states to probability distributions over actions, and π [ 8) denotes the probability of taking action in state 8. The set of all stationary policies is denoted by II. The goal of RL is to find an optimal I²" that maximizes the expectation of the discounted cumulative reward, which is Jn(T) Ex-- EE 7 R (st, dt2 814+)) )). Here T (sq, ao 81, ...) denotes a trajectory, and T ~ is the distribution of trajectories when the policy æ is employed. Then, The value function of
Constraint
Method
Multiple
Insquality
Equality
Genemlity
Model Aguostic
Soft, Cumulative
RCPO PCPO
Cumulatiie
Lyngunav
15
Solt, Cumulative
FOCOPS [1]
x
Soft Cumulative
IPO [5]
35
/
Sont, Iimutaiwefnsaattnooss
1 Lagrangian [1: 50 FAC
X
Soft, Instamtaneves
Safety Layer
16
7
/
X (Lineur)
Sonl, Instamianevss
Recovery RL
[k
x
Hand, Instantancnus Instantaneuus
OptLayer 2
ReCO-.11 10
(Specificc Lincar) (Specific Nonssnwes)
[
(Robotics) (Power Grid)
Instintaneous
DDPG RPOO*)
(Specitic Nonconves)
Hard, Instuntaneuus
Table 1: Comparison among constrained RI. algorithms of different categories
3

---

state s is (s(()) = Errr -E== / (st, ae, S141) 80 s), the action-value function of state and action a is (8, Byyxx Ei=o? (8t, or: 8841) ) | 80 = 8, 0g = a).
Generalized Reduced Gradient Method. GRG considers the following nonlinear optimization problem:
min f(x,, s.t. h(x) 0 E R" x b
(1)
The optimization problem (T) is general formulation of nonlinear optimization since any nonlinear inequality constraints can always be transformed into equality constraints with inequality box con- straints by adding slack variables. GRG first partitions the variable x into the basic variable x and nonbasic variable x. Then the reduced gradient with respect to xB is derived as follows (27
r-= V==><
Sx x²²) | h(x
]]
V Vx h(xx x
Finally, GRG defines the update step as Ax = -r and Ax = V 111 Vx hAx to ensure the equality constraints still hold during the iterations. More details of the GRG method can be referred to in supplementary materials.
4 Reduced Policy Optimization
Although simple explicit constraints in neural networks can be easily handled by some specific activation functions (e.g., the Softmax operator for probability Simplex constraints and the ReLU operators for positive orthant constraints), it hard make the output of the policy network satisfy general hard constraints, especially for nonlinear and nonconvex constraints. In this section, we propose RPO to handle MDP with hard constraints formulated as follows:
Problem Formulation.
max Jn(Fo)
subject to
fi (To (s:): 8t) Vi,t, 9j (To(s:); S 0 Vj.t.
where Si and fj are the hard deterministic constraints that are only related to 8; and a, in the current step, and they are required be satisfied in all states for the policy. Notably, while this formulation is actually a special case of CMDP 17.. it focuses more on the hard instantaneous constraints in constrained RL and is different from the cases 4 considered by previous works in Safe RL, where the constraints only involve implicit inequality ones. Besides, it is also possible to transform the action- independent constraints like g; (5t41) s 0 into gj (TT(s:) < 0 in some real-world applications if' the transition function is deterministic.
RPO consists of policy network combined with equation solver to predict the initial actions and a post-plugged GRG update procedure to generate feasible actions under a differentiable framework from end end. Specifically, the decision process of RPO can be decomposed into construction stage and a projection stage to deal with equality and inequality constraints respectively. In addition, also developed practical implementation tricks combined with modified Lagrangian relaxation method in order to further enforce the satisfaction of hard constraints and fuse the GRG method into RL algorithms appropriately. The pipeline of RPO is shown in Figure
4.1 Construction Stage to Handle Equality Constraints
Recalling the nature of equality constraints, it can viewed as the reduction of the freedom degree in actions. Hence, we follow the formulation in GRG method and divide actions E R" into basic actions B e R and nonbasic actions ' E pn-m where n - m is defined as the number of equality constraints. Hence, the actual actions that we need to determine are the basic actions. Given that, we utilize the policy network to output this part of actions and then calculate the nonbasic actions, via solving set of equations defined by equality constraints and the predicted basic actions, to guarantee the satisfaction of equality constraints. Additionally, we also present a correct gradient flow based on GRG method, which makes it possible train the policy network in end-to-end way. As shown in Proposition 1, we illustrate how to backpropagate from the nonbasic actions N to the basic actions B
Proposition 1. (Gradient Flow in Construction Stage) Assume that we have (n m) equality constraints denoted F(a; s) 0 in each state Let a E 18th and Rn be the basic actions

---

Construction Stage for Equality Constraints
Projection Stage for Inequality Constraints
Equation Solver
State
Policy Network
action
Generalized Reduced Gradient Update
Action
=0)
{a .04
Constraints not satisfied
Backpropagation with Generalized Reduced Gradient Flow (()) Mx(āa(s))
Σ max((1, By (s.Fo()))
Figure 1: The training and decision procedure of RPO.
and integrated actions, respectively. Let a ON (a Rn- denote the nonbasic actions where NN the implicit function that determined by the n m equality constraints. Then, we have
(a
(J. (J.m+1mm)
J.P J.
0a
where J (F(a:s) E m)) J. 1m is the first columns of J and J. J.m+lon is the last (n- m) columns of JF.
The proof is provided in supplementary materials. Additionally, we denote the decision process of the construction stage, including the policy network pi and the equation solving procedure div as
to. and initial actions â as the concatenation a and a i.e., à = (a aa Furthermore, with always apply the modified Newton's 's method [27]
respect the equation-solving procedure, in the construction stage. However, for some special equality constraints like linear constraints, we can obtain the analytical solutions directly instead of using the modified Newton's method.
4.2 Projection Stage Handle Inequality Constraints
After the construction stage, the initial actions denoted as ā may still fall out of feasible region determined by inequality constraints. Hence, the principal difficulty here is to project the action into the feasible region without violating the equality constraints that have been done in the construction stage. To address this issue, the proposed projection stage is to correct the action in the null space defined by equality constraints. Specifically, this projection is iterative procedure similar to GRG method, which regards the summation of violation in inequality constraints as the objective (a aa)) A Σ, max (0, (; +)]- Since the update may destroy the satisfaction of equality constraints, we also need to consider equality constraints in the optimization problem. Then, we can employ GRG updates the action until all the inequality constraints are satisfied, which means find the optimal solution to this optimization problem illustrated above. Here the GRG update is defined as
VasG(a NN 09(a",a" .N
daB + dox(a (ay(a)) 00((8,,"
dnB
OnN
(5)
Aa ( N) Aa 0
DaB
da
Then, the GRG updates are conducted as follows,
B #
Aa 0++1 ax
Ba Aa
(6)
where 7, called projection step can be viewed the learning rate of GRG update to control the step of update and dg denotes the actions at the k-th update where ag A à. After the projection stage, we obtain the feasible actions that can be deployed in the environment and denote the whole process of the construction stage and projection stage as ne. which represents the complete policy performed.
5

---

Theorem (GRG update in Tangent Space) If DoN inequality constraints is in the tangent space of the manifold determined by linear equality constraints.
Bég (a
Da the GRG update for
da
The proof is referred to in supplementary materials. Theorem indicates that the projection stage will not violate the linear equality constraints. With regard to nonlinear equality constraints, we can approximate this nonlinear equality manifold with the tangent space at each GRG update. In that case, we need
the projection step 1/a with sufficiently small value, practically, smaller than 10-$, to avoid destroying the satisfaction of equality constraints. In this way, we can ensure that the GRG update is conducted in the manifold defined by the nonlinear equality constraints. Additionally, a similar projection method was proposed in [18] with lg norm objective, which is likely to fail in the nonlinear situation as we analyze in the supplementary materials.
4.3 Practical Implementation
Here we present details about the implementation of our RPO model and some training designs. They mainly include two aspects: 1) how to achieve better initial actions from the policy network and 2) how to incorporate such a decision process into the training of poflcc RL algorithms.
Algorithm 1 Training Procedure of RPO
Input: Policy network Ho (s), value network Qw, a), penalty factors v, replay buffer D.
1: for in 1, ..... TT do
2: Sample the basic actions at" with the output of ( and some random process.
3: Calculate the action a according to the construction stage 4.1 and projection stage 4.2
4: Take the action 0, in the environment and store the returned transition in D.
5: Sample a mini-batch
transitions in D.
6: Update the parameters of the policy network using 8 and the penalty factors using 9
7: Construct TD target as Yt rt (& No (8++1)).
8: Update the parameters of the value network using MSE loss.
9: end for
Policy Loss with Modified Lagrangian Relaxation. While the above two-stage decision procedure cope with the satisfaction of equality and inequality constraints, the policy network is also required to guarantee certain feasibility of inequality constraints for better initial actions a. Otherwise, we may need hundreds of GRG updates in the projection stage satisfy all the inequality constraints.
Here, we use an augment loss with Lagrangian relaxation for the policy network to obtain initial actions for the inequality projection stage. Common approaches such (50) use fixed Lagrangian multipliers in their loss function. However, such fixed Lagrangian multipliers are not easy to obtain and may require extra information and computation tune. In this paper, we perform a dual update on Lagrangian multipliers to adaptively tune them during the training period. As illustrated in Section B. after the equality construction stage the constrained MDP problem is formulated as
JR(#t)
subject to 9j (fo(s)) st) S Vj,t.
This means need to deal with instantaneous inequality constraints in all states. Hence, we cannot apply the primal-dual update method directly like PDO [3). Otherwise, we need to compute the dual variables on all states, which is obviously unrealistic. Fortunately, Lagrangian multiplier 2 for each inequality constraint in all states with the exact penalty term max {0. 9, (ā(s:); st}} [27] Accordingly, the new objective with the exact penalty term is
maintain only one
min 2(0 A ( + Es-x Σ vn max (0, gy (f(ss):::))
(8)
The following theorem establishes the equivalence between the unconstrained problem 8 and the constrained problem (7).
Theorem 2. (Exact Penalty Theorem) Assume vš is the Lagrangian multiplier vector corresponding to jth constraints in state 8. If the penalty factor vå 2 || boo the unconstrained problem is equivalent to the constrained problem 7

---

The proof is referred to in the supplementary materials. According to the
u/++1 = 1/ + nrll [max (0, gy (T(())
where ng is the learning rate of j-th penalty factors, vif is value of v in the k-th step and vyj 0. Since the exact penalty term, E,~x [max {0. gy (5t); st))l. always non-negative, the penalty factors are monotonically increasing during the training procedure. Hence, we can always obtain sufficiently large v that satisfies the condition in Theorem 2, i.e., vf 2 || Besides, we also find that the adaptive penalty term does not prevent the exploration for higher rewards of the RL agent at the beginning of the training procedure in Section [
Off-policy RL Training. Since we augment the policy loss function with the exact penalty term, the actual objective of our algorithm is two-fold. One is obtaín the optimal policy with the satisfaction of hard constraints. Another is to reduce the number of GRG updates performed during the projection stage. This indicates there exists a gap between the behavioral policy and the target policy in RPO, which results from the changing times of GRG updates performed in the projection stage.
Hence, RPO should be trained like off-polic RL methods Algorithm I Specifically, regard the initial actions à output by the construction stage as the optimization object rather than the actions post-processed by the projection stage. Otherwise, the training process will be unstable and even collapse due to the changing times of GRG updates in the projection stage, which can also be viewed as changing network architecture. Besides, use the y/+ rt + Qu Te (8t44)) to construct the TD target since Te is the actual policy we deploy in the environment.
Experiments
To validate our method and further facilitate research for MDP with hard constraints, we develop three benchmarks with visualization according to the dynamics in the real world, ranging from classical robotic control to smart grid operation. They involve Safe CartPole, Spring Pendulum, and Optimal Power Flow with Battery Energy Storage. Then, we incorporate RPO into two classical off off-policy RL algorithms, DDPG (23] and SAC (21), which we call RPO-DDPG and RPO-SAC respectively.
RPO is compared with three representative Safe RI. algorithms, including CPO 4 CUP 48 and Safety Layer [16]. Notably, we transform the hard equality constraints into two inequality constraints since existing Safe RL methods cannot handle both general equality and inequality constraints. Furthermore, we also contrast the RPO-DDPG and RPO-SAC with DDPG-L and SAC-L, where DDPG-L and SAC-L represent DDPG and SAC only modified with the Lagrangian relaxation method we mentioned in Section 4.3 and without the two-stage decision process in RPO respectively. Besides, DDPG-L and SAC-L deal with the equality constraints as we mentioned in Safe RL algorithms. More details related the concrete RPO-DDPG and RPO-SAC algorithms are illustrated in the supplementary materials. Our code is available at: https://github wwwpwwwwwwww////eeeoo
5.1 RL Benchmarks with Hard Constraints
Specifically, our benchmarks are designed based 12) with extra interfaces to return the informa- tion of the hard constraints. To the best of our knowledge, it is the first evaluation platform in RL that considers both equality constraints and inequality constraints. Figure 2 shows the visualization of these three benchmarks, and the simple descriptions for them are presented in the contexts below. More details about them are provided in the supplementary materials.
1) Safe CartPole. Different from that standard CartPole environment in Gym 12.. we control two forces from different directions in the Safe CartPole environment. The goal is to keep the pole upright as long as possible while the summation of the two forces should be zero in the vertical direction and be bounded by box constraint in the horizontal direction. That is, the former is the hard equality constraint, while the latter is the hard inequality constraint.
Motivated by the Pendulum environment 12 we construct Spring Pendulum environment that replaces the pendulum with light spring, which connects the fixed point and the ball. In order to keep the spring pendulum in the upright position, two torques are required to apply in both vertical and horizontal directions. Meanwhile, the spring should be maintained at fixed length, which introduces a hard equality constraint. Unlike that in Safe CartPole, here equality constraint is
2) Spring Pendulum.
7

---

2 20
[ =
Jz
Power
Power
Generation
(a) Safe CartPole
(b) Spring Pendulum
(c) OPF with Battery Energy Storage
Figure 2: Visualization of RL benchmarks with hard constraints. (a) Safe CartPole. The indicator lamp in the cart will be green if the given actions are feasible, and will be red otherwise. (b) Spring Pendulum. The length of the spring will be changed if the equality constraint violated. (c) OPF with Battery Energy Storage. The circle nodes represent the buses in the electricity grid, and the square nodes represent the batteries connected to the generator buses. We use light and shade to reflect the state of batteries and the power generation and demand of buses. In addition, the edge between the generator bus and the battery will be red the battery is charging, and green the battery is discharging.
(AISafe CartPole
(B)Spring Pendulum
(C/OPF with Battery Energy System
200
40
230
100
75
50
-20
50
25
-10
10000
15000
20000
10300 Epoch
15000
20203
10000
20000 Epsch
30000
40020
Epoch
3.00
101
Contraant Voad
10a
10-1
20
20-4
10-4
10-1
20-1
5000
10000
15000
20000
5000
10000
15000
20003
10.00
20000
30000
43030
Figure Learning Curves of different algorithms on our three benchmarks across 5 runs. The x-axis is the number of training epochs. The y-axis is the episodic reward (the first line), and max instantaneous constraint violation (the second line) respectively.
-eaeedeeendent since the position of the spring is changed during the dynamical process. Besides, the summation of the two torques is also bounded by introducing quadratic inequality constraint.
3) Optimal Power Flow with Battery Energy Storage. Optimal Power Flow (OPF) is classical problem in smart grid operation to minimize the total cost of power generation with the satisfaction of grid demand. However, battery energy storage systems or electric vehicles have been integrated into smart grids to alleviate the fluctuations of renewable energy generation recently. As illustrated in 30 36), we need to jointly optimize the charging or discharging of batteries with OPF for the long-term effect of the smart grid operation. According to this real-world application, we design this environment, which possessed of nonlinear power flow balance equality constraints and inequality box constraints on the actions for fcasibility.
5.2 Evaluation Metrics
We use three metrics to evaluate the algorithms as follows: Episodic Reward: cumulative reward in whole episode.

---

Meuics
CPO
CUP
DDPG-L
SPOGED
SACO
20:100022913))
65,0000
S3.800125
17550))))444)
(55777))
D.00000
00000
ineq
0000030000000
00086000115))
Cardrsle
Mas In. eq
0.0890000 55)
0.534001458
1.0091
0.040500 (OSS)
BIDIR00010 261
(00050000000)
BLDROOO 2O0M))
Sis Ep. incq
000000000000
0.00000000000
0,7504/15885
0,000000.0000
0000000000000
0L00000000000
0LD0CON0-0000)
Max Ep:
0.1096000217)
0.00000000000
0.118CX1DL3)
0L102990/055)
0.005000.0000)
BLINCON0.0000)
Ep. reward
15.468712.7666
0.59900.2-211
76-4583(274555
149/671214 2005)
175, (O67)
182 A82 (2.5-4)
000000000000
0.00000000000
6,995516 8960)
0.000200
BLIGEONO (0000)
BLOGSONG
Spring Pendulam
Mas eq
0:722400-259)
0.58982
39/54/228721)
0.30050000441
(OOM))
inct
0,0030000000
0.0020000000
3695459.1365
0.00KON DOOD)
BLDROONO DO000
Max Ep:
1.90640.7669
(6:311444-007)
25.2101/001724
1,000500 5024)
0L00000000000)
Ep reward
10,340113825)
5337770-95211
24 26.11473.99166
91,96-91191999
L9,4738422 783
42,07GAL1O, (D4E)
Bantry Frengy Storage
(19 with
Max In maq
Mas Io.
0:44130002115)
000030000000
00000000000)
0:986600156)
11 in (497)
01000009000000
0.134610 D:36)
D. (199
0000010000000
BLDOGOOO (OO0))
BLIRKOO (OOENS)
0,0002000000
0.00000000007
0.00000000000
0,00020 00000
000000000000
000042000004)
BL004400.0000
Max Fp. 4q
08821000075
(.8370000177)
Table 2: Mean evaluation performance of different algorithms in the three benchmarks. We compare the performance of RPO (RPODDPG, RPOSAC) with the abridged version of our algorithm (DDPG- I. SAC-L) and other Safe RL algorithms (CPO, CUP, Safety Layer) according to episodic reward and max instantaneous & episodic constraint violation for equality and inequality constraints. Each item in the table is averaged across 10 runs with the standard deviations shown in the parentheses.
Max Instantaneous Constraint Violation: maximum instantaneous violation of all constraints. denotes the feasibility of actions in a state because the feasibility of actions relies on the constraint that is the most difficult to be satisfied.
Max Episodic Constraint Violation: maximum episodic violation of all constraints. Similar to Mean Constraint Violation, it denotes the feasibility of actions in whole episode.
5.3 Performance of RPO on Reward and Constraints
We plot the learning curves of CPO, CUP, Safety Layer, DDPG-L., SAC-L,, RPO-DDPG, and RPO- SAC in F Notably, to validate the importance of the two-stage decision procedure, here SAC-L and DDPG-L are actually the versions of RPO-SAC and RPO-DDPG without this procedure. Given the fairness of our experiments, we apply the same shared This empirical result reflects that existing Safe RL algorithms cannot handle the MDP problems with hard constraints, and our approach outperforms other algorithms in terms of both episodic reward and the max constraint violation. Moreover, the learning curves confirm that RPO can also guarantee certain feasibility during the training period.
for all the algorithms.
Besides the learning curves, Table|sshows the performance of different algorithms after convergence. To present more details the constraint violations, here the equality and inequality constraint violations are shown separately. Notably, since the tanh and state-dependent tanh activation function are added limit the output of the neural network for the box constraints, there is no inequality constraint that needs to be satisfied for DDPG-L, SAC-L, and three Safe RL algorithms. That's why these two algorithms achieve zero violation in the inequality constraints in OPF with Battery Energy Storage. However, it is hard for them to satisfy both equality and inequality constraints exactly in OPF with Battery Energy Storage. This is because this environment is very difficult and involves many complex equality and inequality constraints. Practically, a tolerance 1e-3 is introduced here to evaluate the satisfaction of constraints.
Limitation and Future Work
We acknowledge that there still exist some limitations in RPO. One is that RPO is time-consuming compared to standard neural networks due to the projection stage, where the GRG updates may need to be performed several times. Another is that the equality equation solver required in the construction stage may need to be either chosen or specially designed with domain knowledge. Hence, accelerating RPO and developing more powerful RL algorithms that can handle hard constraints are under consideration in our future works. Besides, while we only validate RPO in RL benchmarks with hard constraints, our method can also easily extended cases with both hard constraints and soft constraints as long as neural network is utilized to fit the mapping between the state-action pair and the cost. Moreover, RPO can be viewed as a supplement to Safe RL algorithms. In scenarios with both hard instantaneous constraints and soft cumulative constraints, we can always use RPO to handle the hard instantaneous constraints and apply some existing Safe RI, methods to handle soft cumulative constraints.
9

---

7 Conclusion
In this paper, we outlined novel algorithm called RPO handle general hard constraints under the off-policy reinforcement learning framework. RPO consists of two stages, the construction stage for equality constraints and the projection stage for inequality constraints. Specifically, the construction stage first predicts the basic actions, then calculates the nonbasic action through an equation- solving procedure, and finally concatenates them as the output of the construction stage. The projection stage applies GRG updates to the concatenated actions until all the inequality constraints are satisfied. Furthermore, we also design special augmented loss function with the exact penalty term and illustrate how to fuse RPO with the off-policy RI training process. Finally, to validate our method and facilitate the rescarch in RL with hard constraints, we have also designed three benchmarks according to the physical nature of the real-world applications, including Safe CartPole, Spring Pendulum, and Optimal Power Flow with Battery Energy Storage. Experimental results in these benchmarks demonstrate the superiority of RPO in terms of both episodic reward and constraint violation.
Acknowledgement
This work was supported by NSFC (No.62303319), Shanghai Sailing Program (22YF1428800, 21YF1429400), Shanghai Local College Capacity Building Program (23010503 100), Shanghai Fron- tiers Science Center of Human-centered Artificial Intelligence (ShangHAD), MoE Key Laboratory of Intelligent Perception and Human-Machine Collaboration (ShanghaiTech University), and Shanghai Engineering Research Center of Intelligent Vision and Imaging.
References
[1] Electricity u.s. energy information administration (eia).
gov.
electricity/ Accessed November 10, 2022 [online].
[2] Nord pool.
com/ Accessed November 10, 2022 [online].
[3] Jean Abadic. Generalization of the wolfe reduced gradient method to the case of nonlinear constraints. Optimization, pages 37-47, 1969.
[4] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference on machine learning, pages 22-31. PMLR, 2017.
[5] A. Agrawal, B. Amos, Barratt, S. Boyd, S. Diamond, and Z. Kolter. Differentiable convex optimization layers. In Advances in Neural Information Processing Systems, 2019.
[6] Dima Waleed Hanna Alrabadi. Portfolio optimization using the generalized reduced gradient nonlinear algorithm: An application to amman stock exchange. International Journal of Islamic and Middle Eastern Finance and Management, 9(4):570-582 2016.
[7] Eitan Altman. Constrained Markov decision processes: stochastic modeling. Routledge, 1999.
[8] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, pages 136-145. PMLR, 2017.
[9] Osbert Bastani, Shuo Li, and Anton Xu. Safe reinforcement learning via statistical model predictive shielding. In Robotics: Science and Systems, pages 1-13, 2021.
[10] Abhinav Bhatia, Pradeep Varakantham, and Akshat Kumar. Resource constrained deep rein- forcement learning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 29, pages 610-620, 2019.
[11] Steven Bohez, Abbas Abdolmaleki, Michael Neunert, Jonas Buchli, Nicolas Heess, and Raia Hadsell. Value constrained model-free continuous control. arXiv preprint arXiv:1902.0468 2019.
[12] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 1oXXvv:666.005444 2016.
[13] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. The Journal of Machine Learning Research, 18(1):6070-612) 2017.
10

---

[14] Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzmal Guzman, and Mohammad Ghavamzadeh. A lyapunov-based approach to safe reinforcement learning. Advances in neural information processing systems, 31, 2018.
[15] Yinlam Chow, Ofir Nachum, Aleksandra Faust, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. Lyapunov-hased safe policy optimization for continuous control. arXiv preprint arXiv: 2019.
[16] Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:/80I.08s 08757, 2018.
[17] C Yu David, John E Fagan, Bobbie Foote, and Adel A Aly. An optimal load flow study by the generalized reduced gradient approach. Electric Power Systems Research, 10(1):47-53) 1986.
[18] Priya L Donti, David Rolnick, and Zico Kolter. Dc3: A learning method for optimization with hard constraints. In International Conference on Learning Representations, 2020.
[19] Razvan Florian. Correct equations for the dynamics of the cart-pole system. Center for Cognitive and Neural Studies (Coneural), Romania, 2007.
[20] Sehoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, and Jie Tan. Learning to walk in the real world with minimal human effort. In Conference on Robot Learning, pages 1110-1120. PMLR, 2021.
[21] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on muchine learning, pages 1861-1870. PMLR, 2018.
[22] Qingkai Liang, Fanyu Que, and Eytan Modiano. Accelerated primal-dual policy optimization for safe reinforcement learning. arXiv preprint arXiv: 1802.06400, 2018.
[23] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster), 2016.
[24] Puze Liu, Davide Tateo, Haitham Bou Ammar, and Jan Peters. Robot reinforcement learning on the constraint manifold. In Conference Robot Learning, pages 1357-1366. PMLR, 2022.
[25] Yongshuai Liu, Jiaxin Ding, and Xin Liu. Ipo: Interior periior-pointtto policy optimization under constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4940 4947, 2020.
[26] Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pages 13644-13668. PMLR, 2022.
[27] David G Luenberger, Yinyu Ye, et al. Linear and nonlinear programming, volume 2. Springer, 1984.
[28] Haitong Ma, Yang Guan, Shegnbo Eben Li, Xiangteng Zhang, Sifa Zheng, and Jianyu Chen. Feasible actor-critic: Constrained reinforcement learning for ensuring statewise safety, arXiv preprint arXiv:2105. 10682, 2021.
[29] Tu-Hoa Pham, Giovanni De Magistris, and Ryuki Tachibana. Optlayer-practica constrained optimization for deep reinforcement learning in the real world. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6236-6243. IEEE, 2018
[30] Yann Riffonneau, Seddik Bacha, Franck Barruel, and Stephane Ploix. Optimal power flow management for grid connected pv systems with batterics. IEEE Transactions on sustainable energy, 203):309-320, 2011.
[31] Keith Rudd, Greg Foderaro, Pingping Zhu, and Silvia Ferrari. A generalized reduced gradient method for the optimal control of very-large-scale robotic systems. IEEE transactions on robotics, 33(5): 1232, 2017.
[32] Shah Sanket, Arunesh Sinha, Pradeep Varakantham, Perrault Andrew, and Milind Tambe. Solving online threat screening games using constrained action space reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2226-2235,
2020.
11

---

[33] Ahmed Rabee Sayed, Cheng Wang, Hussein Anis, and Tianshu Bi. Feasibility constrained online calculation for real-time optimal power flow: A convex constrained deep reinforcement learning approach. IEFE Trunsactions on Power Systems, 2022.
[34] Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv: 1610.03295, 2016.
[35] Li Shen, Long Yang. Shixiang Chen, Bo Yuan, Xueqian Wang, Dacheng Tao, al. Penalized proximal policy optimization for safe reinforcement learning. arXiv preprint arXiv:2205. 11814, 2022.
[36] Ye Shi, Hoang Duong Tuan, Andrey V Savkin, Trung Q Duong. and H Vincent Poor. Model predictive control for smart grids with multiple electric-vehicle charging stations. IEEE Trans- actions on Smart Grid, 10(2):127-2136, 2018.
[37] M Shivashankar, Manish Pandey, and Mohammad Zakwan. Estimation of settling velocity using generalized reduced gradient (grg) and hybrid generalized reduced madient-ggenete algorithm (hybrid grg-ga). Acta Geophysica, 70K5):247722977, 2022.
[38] David Silver, Julian Schrittwicser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. nature, 550(7676):354-35) 2017.
[39] Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pages 9133-9145 PMLR, 2020.
[40] Haixiang Sun, Ye Shi, Jingya Wang, Hoang Duong Tuan, H Vincent Poor, and Dacheng Tao. Alternating differentiation for optimization layers. International Conference on Learning Representations, 2023.
[41] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[42] Chen Tessler, Daniel Mankowitz, and Shie Mannor. Reward constrained policy optimization. In International Conference on Learning Representations, 2018.
[43] Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair. Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg, Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 6(3):4915-4922, 2021.
[44] Akifumi Wachi and Yanan Sui. Safe reinforcement learning in constrained markov decision processes. In International Conference on Machine Learning, pages 9797-9808 PMLR, 2020.
[45] Akifumi Wachi, Yanan Sui, Yisong Yue, and Masahiro Ono. Safe exploration and optimization of constrained mdps using gaussian processes. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018
[46] Yixuan Wang, Simon Sinong Zhan, Ruochen Jiao, Zhilu Wang, Wanxin Jin, Zhuoran Yang, Zhaoran Wang, Chao Huang, and Qi Zhu. Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. In International Conference on Machine Learning, pages 36593-36604, PMLR, 2023.
[47] Ziming Yan and Yan Xu. A hybrid data-driven method for fast solution of constry-- optimal power flow. IEEE Transactions on Power Systems, 2022.
[48] Long Yang. Jiaming Ji, Juntao Dai, Linrui Zhang, Binbin Zhou, Pengfei Li, Yaodong Yang, and Gang Pan. Constrained update projection approach to safe policy optimization. arXiv preprint 209.002209.07089, 2022.
[49] Tsung- Yen Yang, Justinian Rosca, Karthik Narasimhan, and Peter J Ramadge. Projection- constrained policy optimization. In International Conference on Learning Representations, 2019.
[50] Linrui Zhang, Qin Zhang, Li Shen, Bo Yuan, Xueqian Wang, and Dacheng Tao. Evaluating
mrdel-free reinforcement learning toward safety-critical tasks. arXiv preprini arXiv:2212.00572, 2022.
[51] Yiming Zhang, Quan Vuong, and Keith Ross. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 111115338-53499 2020.
12

---

[52] Yuhao Zhou, Bei Zhang, Chunlei Xu, Tu Lan, Ruisheng Diao, Di Shi, Zhiwei Wang, and Wei- Jen Lee. A data-driven method for fast ac optimal power flow solutions via deep reinforcement learning. Journal of Modern Power Systems and Clean Energy, 866):122--11333 2020.
13

---

A Proofs
A.1 Proof of Proposition
Proposition 1. (Gradient Flow in Construction Stage) Assume that we have (n m) equality constraints denoted as F(a;s)= = in each state Let a E T and a E Rn be the basic actions and integrated actions, respectively. Let a ON (a E TR- denote the nonbasic actions where &N is the implicit function that determined by the n m. equality constraints. Then, we have
d
(J.
J J.1m
(10)
Ba
where J aF(a:s) E R(m-n)xn, 1m is the first m columns of J and J. iim+lon is the last 8a
(n- m. columns of JF.
Proof. Considering an equality-constrained optimization problem, the reduced gradient 27
is
defined as
TT Vx" x) + A2 xx h(x", x²),
(11)
where X should satisfy Vx* fx xB) x x h(x xB) h function and the equality constraints, and B x are the basic variables and nonbasic variables respectively.
0. f, h denote the objective
Finally, we will obtain
a f(x", xB
f(x" xB) W x²)] Vxah(x" x8)
(12)
Similarly, the gradient flow from the nonbasic actions to basic actions is
dāv(a (J.m.n) J.F 1.
(13)
On
A.2 Proof of Theorem
Theorem (GRG update in Tangent Space) If Aa
(a
Da the GRG update for inequal-
Ou
ity constraints is in the tangent space the manifold determined by linear equality constraints.
Proof. Firstly, assume the tangent space of the actions in the manifold defined by the equality constraints is J a d where a = [a aN = [a (x(a aB E RM (a E "
According to Proposition 1, we have
(a daB
(J.F (1m++)n)) J.p.m.
(14)
Then, we define the projection objective as G(a a, 2 E, max {0, (0, s)} and obtain
VanG(aB, N) A 0g(a N
Oa 0g(a NN
dnB
OnB
OnN
(15)
Aa =V veeg(a .N), Aa da
DaB
da
14

---

Finally, we have
JF Aa Aa
Jo J.
mm++im
Da
AaB
[J Jimm
AΦN(aB)
= - 1,lmm Da + +m Iin ApN(a
j.P
0dr(a daB
-J. - Da
J. J.m+in
Du
00,(a
-JP i.am
Vaog(a ) + Jmm+iin
8 V Va=g(a ,ay)
0g(aB,a") .
09(a",a" )dN((
(16)
=f
1,1m da
daB
00,(a²)
@0((a" )
20(a N
doN(a.)
J
mm+lm da
da
(pN(aB)
8a
0g(a9,a")
0g(a .aa) (J
dnB
Obb (aB)
0g(a ,,
09(a , (J.m+iim) J.P1.m
J
+ mm+ (5)(a
JFF
0a
=0.
A.3 Proof of Theorem 2
Theorem (Exact Penalty Theorem) Assume v is the Lagrangian multiplier vector corresponding to jth constraints in state s. If the penalty factor vd 2 || ll the unconstrained problem is equivalent to the constrained problem
Proof. Since vd 2 ||221bos we have
E(0) 2 Jn(#o)) + EgNn
ΣΣ vs max (((s))
(17)
The equality holds when all the inequality constraints satisfied. Here
vg A argmax min (n( + E 0
ΣΣ vg max {0, (0,gg(((ss):s))
(18)
Therefore, the unconstrained problem ming - - + Exrx Σ, E. us max (0, gs (+();;))) is equivalent to the constrained problem problem. Then, we have
Let 0* be the optimal solution of the unconstrained
2(00) = -J(((()) + Eswr
ΣΣ vy max (0, (5))()))))
(19)
S -nn((oo) + E
ΣΣ is max (0,g;(((s);:))
< [(()
Hence, the unconstrained problem g is equivalent to the constrained problem 7
15

---

A.4 Limitations of &g Norm Objectives
Projection with & norm objectives will never come into the feasible region defined by inequality constraints with sufficiently small projection step.
Assume that the inequality constraint that needs to be satisfied is g(a; s) S 0. For given action ag that does not satisfy the inequality constraint after &o updates, i.e., (ax) > 0. If ag comes close enough to the feasible region by using the l2 norm objective, we apply a linear approximation on g(ax) = ct ax and then obtain
g(a++1) [aan Aa)
=c (ax 1/a Vallc* ak bl(2)
=c (ax 2na ax b)) b
(20)
=c- (ax (ax)(a))0) b
=1(ax) 2n c).
Obviously, to ensure the satisfaction of the inequality constraint, i.e., (((++1) S O, we must limit the projection step with %e 2 1/(2c"'c) c). However, no is often sufficiently small number for the satisfaction of nonlinear equality constraints. In this case, the inequality constraints will never be satisfied with a sufficiently small projection step.
B
GRG algorithm
The GRG algorithm is shown Algorithm
Algorithm 2 Generalized Reduced Gradient Algorithm
Input: Optimization problem minxer" f(x), h(x) 0 R < S b. Output: Optimal solution x.
Assumptions:
(xx,x"), where x E R", Rn-m 2. If (a", a²) and (b bd are the corresponding partitions of a, b, then N xxx bba.
1. Divide into x
3. The
matrix Vxsh (x-V, xx is nonsingular at (x xB).
1: Define the reduced gradient (with respect to x- as
I" = V vx* fxx", x XT V hxx" xx x), where V f(x", xx8) ( + X x Vxxh(xX (x 0.
2: Obtain -- Vxn fx xB
Vxv f(x BB) V h(x xB)] Vxnh(x>,>>) (BB)
3: Let
Ar! r; ifr; 0 B or 0, x. > a
otherwise
Ax - V
4: Find an 02, 0g that respectively satisfy
max/a N N + aAx <b"}, maxfa: : a B < + oAx <b²),
min{f(x a 0 a 0 s a az).
5: Update
B ト- B + Ax
x -- N + Ax
6: Perform (1)-(5) until convergence.
16

---

C
Algorithm Details
We present the concrete algorithm about how RPODDPG and RPOSAC work in Algorithm 3 and Algorithm
Algorithm 3 RPO-DDPG
Input:
Initial parameters 0, of policy network μ and value network Q.
1: Initialize the corresponding target network 0' 0, w'
2: Initialize replay buffer 'D.
Initialize penalty factor for inequality constraints < O.
4: for each episode do
5: for each environment epoch do
6: Select partial action at = (o(ss) + (0(
7:
Employ construction stage by solving the equations defined equality constraints
ã = [a
ΦN((,))
8:
Employ projection stage the concatenated action â, obtain feasible action oa for k = 1.. K do
9:
Adi, a N aikt1
Dot v.ag(a )a
B
Da Sag,
B
B Oi.k
noDa
oi,k na Dar
a1,41
end for
Execute action and observe reward Tt and next state S1+1- 11:
Store the transition < St. Tt, St+1 in D. Sample a mini-batch of transitions in D. Update actor and reward critic networks
13:
0 + 0 n1 VOED
Qu (st, mo(ss))
Σ max {0, وو ((
w + w nQ ED [Qw ar) - (re + 7Qw ($1+1, ne (8 (8(+1))))
Perform dual update on
/ = u + Eswx [max (f).)(((()):)))) fj.
16:
Soft update target networks:
0' +- r0 + (1 (1-7)9 7)0''
w/ +- Tw + (1 - T)w'.
17:
end for
18:
end for
D Benchmark Details
In this section, we will illustrate the dynamics of our three benchmarks in detail.
D.1 Safe CartPole
Different from the CartPole environment in Gym two forces from different directions are controlled in the Safe CartPole environment.
State. The state space |S| TR. Safe CartPole includes the position, velocity, and acceleration of the cart; and the angle, angular velocity, and angular accelcration of the pole.
17

---

Algorithm 4 RPO-SAC
Input: Initial parameters 0, W1, u2 of policy network μ and value network Q1, Q2, Temperature parameter a.
1: Initialize the corresponding target network w1 + W1, w/ c- w2.
2: Initialize replay buffer D.
3: Initialize penalty factor for inequality constraints // < 0.
4: for each episode do
5: for each environment epoch do
6: Select partial action a ~ po sst).
7:
Employ construction stage by solving the equations defined equality constraints
ã = [a
(x(a
8:
Employ projection stage on the concatenated action â to obtain feasible action for 1: =1,1 1,... K do
9:
AayA Va ,a DaiE dDN(a daB Aa N no 2a
B 11,4+ Ot,k na
abxt1
10:
end for
Execute action a, and observe reward Ft and next state 81+1- Store the transition < St: Qt, Tt. St+1 > in D.
12:
Sample mini-batch of transitions in D.
14:
Update actor and reward critic networks
0 - 0 + Tl v D
a log (ne(ar)s)) + Qw (50, mo(ss)) Σ vs max 1axx(0,g, (fo((s)))
wi - wi n v 00 EDD
[Qw, (se, at) - (r+ + 7@w (sc+1, no (8(+1))) a log (1o(at+11)4411
15:
Perform dual update on
w = v + Esn [max (f.))(((s)):s))) Vj.
16:
Soft update target networks:
w/ + TW; + (1 (1-r)
17:
end for
18:
end for
Action. The action space |A| E R² of Safe CartPole is the sign and magnitude of two forces fis f2 from different directions. Specifically, one force inclined 30 degrees below the æ axis, and the other is inclined 60 degrees above the axis.
Reward. The goal of Safe CartPole is similar to the CartPole in Gym [12] which requires the pole to keep upright as long possible. Therefore, we employ the same reward policy that returns 1 if the pole keeps upright. Otherwise, this episode will end since the pole falls.
Equality Constraints. The equality constraint of Safe CartPole is that the summation of two forces in axis should be zero. That is, we desire avoid extra friction the cart, i.e.,
fy sin 01 + sin
(21)
Inequality Constraints. The inequality constraint is that the summation of two forces in the axis should bounded by box constraint, which indicates the physical limitation magnitude of the summation force in the axis.
f, Sx == fi cos 01 + cos 02 < f
(22)
18

---

Furthermore, according to 19. we derive the dynamics of Safe CartPole when there exists a force in the vertical direction as shown 23). Notably, we can always derive the position angle, velocity or angular velocity in the next step, using the semi-implicit Fuler method.
No Ja + (me + mp) g m,p (ōsin 0 + 0² cos
ng8 myd
g ssin
0
1 113
Wip cos
(cos 0
Me 8gu (Nai))
m. mp
cos { f my 16
(sin the sgms No coed) + Megsgn (Not)
(23)
mo+
mp cos
[cos 0 - Me sgn (Not)]
m. +rp
fa + mpl (02 sin 0 cos 0) - M (Nci)
m + m&p
where N₂ is the pressure on the cart, me, my are the mass of the cart and pole, 1 is the length of the pole, x; 0 are the position of the cart and the angle of the pole respectively, and Pec is the dynamic friction coefficient the cart.
D.2 Spring Pendulum
Motivated by the Pendulum environment 12). with light spring, which connects the fixed point and the ball the end of the spring.
Spring Pendulum environment replaces the pendulum
State. The state space [S] E 12 of Spring Pendulum contains the cosine and sine of the angle, angular velocity, length of the spring, and the change rate in length.
Action. The action space |A| E R2 of Spring Pendulum is the sign and magnitude of two forces fz; fy in the æ, y axes. Notably, since the spring pendulum is rotating, the angle between the y axis and the spring is changing as well. Thus, this environment is more difficult than Safe CartPole in some sense.
Reward. The goal of Spring Pendulum is to keep the spring pendulum in an upright position. The episode will never be done until the maximum time step. Specifically, the reward function is 1+100 where 0 is the angle between the spring pendulum and the y axis. That means the agent will achieve a reward of nearly 1 when the spring pendulum is close enough to the upright position. Otherwise, a reward of almost 0 will be returned to the agent.
Equality Constraints. The equality constraint of the Spring Pendulum is to limit the change rate of length to zero since the spring pendulum is expected to perform like normal pendulum without changing the length of the pendulum.
-
T, 0.
(24)
Notably, when 4-1 0, the equality constraint will be i =
Inequality Constraints. The inequality constraint is the magnitude constraint on the summation of these two forces.
1 f
(25)
Furthermore, to connect each component mentioned above, we derive the dynamics of the Spring Pendulum. Applying Euler-Lagrange Equation L = T V to spring pendulum, then obtain
1
V myy
k 10)²
2
1
-mgl cos 0 +
k - lo)?
2
(26)
T 2 mn² = 2
m (i² + is2)
m (22 1202))
19

---

Applying the Euler-Lagrange equations for and 1, we have
d 0C
2ml0 + ml0,
dt 00
QC
mgsin& 8
00
(27)
0C dt ai
=mi ml6² + k(1 lo),
0C
a1 mg cos 0.
Finally, will obtain the dynamics of the spring pendulum
J 2mld mg sin
ml
(28)
i + ml02 - 1o) mg cos 0
m
where f. -f sin fx cos 9 and fs fy cos fo sin are the force perpendicular and along the spring pendulum.
D.3 Optimal Power Flow with Battery Energy Storage
In smart grid operation controlling, Optimal Power Flow (OPF) is defined
min Po-
Pg Apg + 6% Pg
subject
Pg S Pg p.
(29)
2, I S 99
2 |b| S
(p, Pa) + (g.- qa) diag(v)Yu,
where Pg: 9g E R" are the active and reactive power generation of the buses, and v E C" are the voltage of the buses in the grid. Y C" denotes the admittance matrix. pd, qd E R are active and reactive power demand all buses. Notably, some buses in the electricity grid are not generator buses, and Pg: 9g of these buses will be and one of the generator buses is the reference (slack) bus which has a fixed voltage argument Zv. Therefore, the actual dimension of Pa: lg to determine is the number of generator buses n Based the OPF problem, OPF with Battery Energy Storage is defined as
min P.PP
+ P3 (p,())A,00
(1))((1)
subject to
(t)(t) pa(t) - pp(tt) (gg(t) qa(t))
2g S (t())= S Epg
(30)
9, S (t) S Ag
2 (t) ST,
(t) (t(t) < (t).
The additional variables ps E R" is the charging (positive) or discharging (negative) power of the battery groups, and c(t) represents the cost or the income in the time step t. Exactly, only connect the batteries with the generator buses. Therefore, the actual number of ps to determine is n in this benchmark as well.
Specifically, this benchmark based on 14-node power system, which available the PYPOWER package. We adopt the same topology of the electricity grid with 5 generator nodes in this benchmark. The data on power demand and day-ahead electricity prices are from 2 We refer the distribution of the real-world data in one day, which is shown in Figure and normalize
20

---

Demand
Electricity Price
0.05
0.04
0.04
0.03
0.03
0.02
0.02
0.01
0.01
0.00
22
22
Time
Time
Figure 4: Distribution of power demand (left) and electricity price (right) in 24 hours.
the magnitude of the concrete data to incorporate them into the 14-node power system.
State. The state space |S| E 1257 of OPF with Battery Energy Storage contains active and reactive power demand of 14 buses, the state of batteries connected to 5 generator nodes, and the 24-hour day-ahead electricity price.
Action. The action space |A|| E RR43 of OPF with Battery Energy Storage involves the active and reactive power generation of 5 generator buses, voltage magnitude and argument of 14 buses, and the power generation or demand of 5 batteries. Notably, since the voltage argument the slack bus fixed, the actual action space |Á| E R However, for convenience, we include it in the action space and the fixed value for in the practical implementation.
Reward. The goal of OPF with Battery Energy Storage is to minimize the total cost. Therefore, we regard the negative cost in the current time step -P' (t) (p, (t) 5" (t + c (t) as the reward. Actually, there exist efficiency parameters Meh 7dds the procedure of charging and discharging. Moreover, each part of the cost may have a different magnitude, which needs us consider tradeoff between different parts. For convenience, model all of these factors into c(t).
The 28 equality constraints of OPF with Battery Energy Storage are the equations of power flow, i.e..
Equality Constraints.
(ty pa(t) - pt(t) + (g)(t) qa(t)) = (iag(((())) (t)).
(31)
Inequality Constraints. The inequality constraint is 58 box constraints on the decision variables.
Pg S pg (t) S D
9, S (t) S 4
(32)
1 S e(t) <
2, (4) < (1) S (t).
The dynamics of OPF with Battery Energy Storage are much simpler than the former two benchmarks, since the active and reactive power demand Pas qa in each time step, is irrelevant to the last state and action. Thus, we only need to care about the change in the electrical power of batteries, i.e., soc(t) = soc(t 1) + Pes (t) Pais (t)) where Pek
max{0. pt))), Pdis (t) min Pb (t)).
Implementation of Construction Stage. Unlike previous robotic manipulation cases, OPF with Battery Energy Storage environment has nonlinear equality constraints. This means the analytical solution of the corresponding equations cannot obtained. In addition, while Newton's ''s methods can be applied directly here, is sometimes unstable empirically. Hence, followed the idea in
21

---

18 which divides the amplitude of voltage |o| buses without the generator and voltage argument Le in all buses via Newton's s methods. Then, we can directly compute the solution of reactive power generation Z in all buses and Pg in the slack bus in the second step. Moreover, the gradient flow from nonbasic actions to power generation or demand of battery groups Ps can be calculated similarly to the gradient flow from chosen nonbasic actions to active power demand pa-
procedure into two steps. The first step is to compute the
D.4
More Explanation on Box Inequality Constraints
Notably, while our three benchmarks only contain box inequality constraints, they are sufficient to evaluate the performance of our RPO algorithm in scenarios with nonlinear inequality constraints. As mentioned in Section [] any nonlinear inequality constraints can always be transformed into equality constraints plus inequality box constraints by adding slack variables. Hence, evaluating RPO with box inequality constraints and nonlinear equality constraints is sufficient to show RPO's generality. For example, if we have complex inequality constraint g(s, s, a) S Then, we can always transform it into
g(s, a) + tee O,
aug < 0.
where an is the slack variable, which can be viewed as the augmented actions in RL.
E
Additional Experiments
To further discuss the efficiency of RPO, we do some extra experiments in our hardest benchmark, OPF with Battery Energy Storage. The results and related analysis are presented below.
E.1 Sensitivity Analysis
The projection stage performs the GRG updates until all inequality constraints are satisfied. It is valu- able to investigate the impact of the number of maximum GRG updates K on model performance. Here, we conduct experiments under dif- ferent K values in OPF with the Battery Energy Storage task. As shown in Figure the performance of RPOSAC under K = 0, 10, 50, 200 in the projection stage. The result indicates that the choice of K does not have an obvious impact on the episodic reward when K is not too large. The slight improvement in constraint violation between K 10 and K 50 illustrates that K = is actually the most appropriate since needs much less computation compared with K = 50 during the projection stage. Besides, the case of K 200 shows that too large modifications during the training process will lead to poor performance in some situations. The principal reason we believe that the value network cannot back-propagate an accurate gradient to the policy network when K is too large. Concretely, large K leads to the samples being too far from the policy Te. which further results in the inaccurate estimation of the policy मo in the value network. Notably, practical trick is to set small K during the training period and a large K during the evaluation period.
Max Constraine Vioiztton
20-1
Episodic Roward
22
we compare
Figure 5: Comparison RPO under different K.
actions
22

---

E.2 Adaptive v.s. Fixed Penalty Factor
Besides, We conduct experiments to confirm the performance of the adaptive penalty factor with dual up- date compared to that of the fixed penalty factor as introduced Sec- tion For fairness, check the converged values the adaptive penalty factor with learning rate 0.02 in RPOSAC, and we find that the converged penalty factors are ranging around 100. Therefore, chose v 100 for all penalty terms the setting of RPOSAC with the fixed penalty factor. Results in Figure show the advantage of the adaptive penalty factors in terms of episodic reward.
3.024
Max Constraint Vicleliin
Vicledion
Epicodic Roword
9,20
3,006
9,003
10800
20068
20000
40300
19500
20083
39000
40000
Epoch
Epoch
Figure 6: Comparison between RPO with fixed penalty factor and adaptive penalty factor
E.3 Necessity of Backpropagation with Generalized Reduced Gradient
As illustrate in the construction stage, the integrated actions are actually determined once the basic actions are given. Therefore, it is possible for the value network to approximate the mapping from the basic actions to the nonbasic actions. This seems that there is no need for the backpropagation with generalized reduced gradient which requires extra gradient flow from nonbasic actions if only the hard equality constraints need to be addressed the hard inequality constraints are only related to the basic actions.
To explore the necessity of the back- propagation with generalized re- duced gradient, also contrast the performance in RPOSAC trained with/without the complete gradient. Concretely, for RPOSAC trained without the complete gradient, we only input the basic actions into the value networks and expect it can approximate the complete gradi- ent, which is known as generalized reduced gradient. The results are shown in Figure 7 reflects that the value network cannot backpropagate accurate gradient directly, and indicates the necessity to explicitly construct the gradient flow from nonbasic actions to basic actions.
Max Comtrriitt Violatin
3.008
Episadic Reward
3.035
3,00
3,0s
3,000
10000
20006
20000
40300
Figure 7: Comparison between RPO with partial gradient and complete gradient.
E.4
Comparison in Training Time
The training time comparison in our three benchmarks is shown in Tables the training time RPO-DDPG and RPO-SAC grows linearly with the number of constraints to deal with. However, RPO-DDPG and RPO-SAC still have significant advantages algorithms like Safety Layer.
3 It can be observed that
of Safe RI.
Task
CPO
CUP
Safety Layer
DDPG-L
SAC-L
RPO-DDPG(+)
RPO-SAC(+)
Safe CartPole
68.7
75.2
1067.0
150.2
268.5
190.2
375.5
Spring Pendulum
98.6
81.4
3859.7
174.9
309.5
242.6
419.0
OPF with Battery Energy Storage
288.8
239.4
6395.0
498.9
892.7
4462.1
4828.4
Table Training time(s) of different algorithms in the three benchmarks, Each item in the table is averaged across runs.
23

---

Hyper-parameters
We implemented experiments GPU of NVIDIA GeForce RTX 3090 with 24GB. Each exper- iment on Safe CartPole and Spring Pendulum takes about 0.5 hours. Each experiment on OPF with Battery Energy Storage takes about hours. Moreover, for policy and value networks except for the input and output dimensions in all experiments. The policy and value networks both have two hidden layers with 256 hidden units and only differ in input and output layers. Besides, we also show the detailed hyper-parametee used in our experiments. Table 4 Table [5] and Table respectively present the parameters used in Safe CartPole, Spring Pendulum, and OPF with Battery Energy Storage. Additionally, the implementation of three safe RL algorithms in our experiments are based on omnisafe [ and safc-explorer and recommended values are adopted for hhper--pramenn parameters not mentioned in the following tables.
adopt similar neural network architectures
Parameter
CPO
CUP
Safety Layer
DDPG-L
SAC-L
RPODDPG
RPOSAC
Batch Size B
256
256
256
256
256
256
256
Discount Factor y
0.95
0.95
0.95
0.95
0.95
0.95
0.95
Target Smoothing Coefficient
N/A
N/A
0,005
0.005
0,005
0.005
0,005
Frequency of Policy Update
N/A
N/A
N/A
0.25
0.25
0.25
0.25
Total Epochs T
2E4
2E4
2E4
2E4
2E4
2E4
2E4
Capacity of Replay Buffer D
N/A
N/A
2E4
2E4
264
2E4
2B4
Random Noise Exploration
N/A
N/A
1E-2
1.0
N/A
1.0
N/A
LR for Policy Network
IE-4
IE-4
1E44
1E4
IE-4
IE4
IE-4
Value Network Q LR for Penalty Factor
3E-4
3E-4
3E4
3E-4
3E-4
3E-4
N/A
N/A
N/A
0.2
0.2
0.2
0.2
Temperature
N/A
N/A
N/A
N/A
0.1
N/A
0.1
Projection Step Vie
N/A
N/A
N/A
N/A
N/A
2E-2
2B-2
Max GRG Updates K
N/A
NA
NIA
N/A
N/A
10
10
Projection Step Evaluation vi
N/A
N/A
N/A
2E-2
2B-2
Max GRG Updates in Evaluation K
N/A
N/A
N/A
N/A
N/A
50
50
Table 4: Hyper-parameters for experiments in Safe CartPole.
Parameter
CPO
CUP
Safety Layer
DDPG-L
SAC-L
RPODDPG
RPOSAC
Batch Size B
256
256
256
256
256
256
256
Discount Factor
0.95
0.95
0.95
0.95
0.95
0.95
0.95
Target Smoothing Coefficient
N/A
N/A
0.005
0.005
0.005
0.005
0.005
Frequency of Policy Update
N/A
N/A
N/A
0.25
0.25
0.25
Total Epochs 7'
2E4
2E4
2E4
2E4
2B4
2E4
2B4
Capacity of Replay Buffer D
N/A
N/A
2F4
2F4
2F4
2FA
2F4
Random Noise in Exploration
N/A
N/A
1E-2
0.5
N/A
0.5
N/A
I.R for Policy Network
IF-4
IF-4
IF-4
IR-4
IR-4
IF-4
IR-4
Value Network Q
3F-4
3F-4
3F-4
3F-4
3R-4
3F-4
3R-4
LR for Penalty Factor
N/A
N/A
NIA
0.01
0.01
0.01
0,01
Temperature
N/A
N/A
NIA
N/A
0.01
N/A
0.01
Projection Step %
N/A
N/A
N/A
N/A
N/A
2F-3
2F-3
Max GRG Updates K
N/A
N/A
N/A
N/A
N/A
10
Projection Step Evaluation vg
N/A
N/A
N/A
N/A
N/A
2F-3
2F-3
Max GRG Updates in Evaluation K
N/A
N/A
N/A
N/A
N/A
50
50
Table 5: Syyper-praaeteet for experiments Spring Pendulum.
G Systematical Action Division Method
While it is usually not difficult to divide actions into basic and nonbasic actions, we present the systematical action division method below to further show the generality of our method. Firstly, we

---

Parameter
CPO
CUP
Safety Layer
DDPG-L
SAC-L
RPODDPG
RPOSAC
Batch Size B
256
256
256
256
256
256
256
Discount Factor ?
0.95
0.95
0.95
0.95
0.95
Target Smoothing Coefficient
N/A
N/A
0.005
0.005
0.005
0.005
0.005
Frequency of Policy Update
N/A
N/A
N/A
0.25
0.25
0.25
0.25
Total Epochs 7
4E4
4E4
4E4
4E4
4E4
4E4
Capacity of Replay Buffer D
N/A
N/
2E4
2E4
2B4
2E4
2B4
Random Noise Exploration
N/A
IE-2
IB-4
N/A
1F-4
N/A
LR for Policy Network μ
IE-4
IE-4
1E-4
1E4
1E-4
1E4
1E-4
Value Network Q
3E-4
AЕ-4
3E-
3E-4
3E-4
3E-4
3E-4
L.R for Penalty Factor
N/A
N/A
NIA
0.02
0.02
0.02
0.02
Temperature Projection Step % Max GRG Updates K
N/A
NAA
NIA
N/A
0.001
N/A
0.001
N/A
N/A
N/A
N/A
N/A
IF-4
IE-4
N/A
N/A
N/A
N/A
N/A
10
10
Projection Step in Evaluation
N/A
N/A
N/A
N/A
NIA
IE4
1E-4
Max GRG Updates Evaluation
N/A
N/A
N/A
N/A
N/A
50
50
Table 6: Hyper-parameetess for experiments in OPF with Battery Energy Storage.
would like to clarify that the principal purpose of the systematical action division method is to cope with the problem that J.F 1,mm+1um I:m is not invertible under some division or the equations defined by basic actions are not solvable. In that case, we will separate the situation into situations with linear equality constraint and nonlinear equality constraints, and illustrate different problems and corresponding systematical action division methods in both of them.
Linear. In the linear case, if J mm is not invertible, this may result from two possible problems. Firstly, it indicates there exist redundant equality constraints, then we can directly delete these redundant equality constraints (i.e., choose the maximal linearly independent subset of row vectors of the coefficient matrix) and redefine the dimension of basic actions to make J.F J. invertible, For example, if we have 3 linear equality constraints Ax + = 0 on 4 actions, where
1
0
-2
[A [6]
5
3
-1 3
4
3
Here we choose ai the basic action, and then J J 2.2 is not invertible, since the third equality constraint can be represented by another two constraints. Hence, we can delete this redundant constraint and the new equality constraints are Á + b 0, where
A b = 5 3
0
2
3
1
4
-1
Then, the new basic actions will be (a1, 42) and new mmm. A. is invertible. Another problem is with respect to the division of basic and nonbasic actions. For example, let consider other linear equality constraints Ar + b = 0 without redundant equality constraints on 3 actions as below:
-1 - -1 -2 -11
[A bb]
Although there is no redundant equality constraint, J.F J. will be still not invertible if (a2, ag) are divided as nonbasic actions. Therefore, after removing the redundant equality constraints, we need to choose actions that correspond the maximal linearly independent column vectors of the coefficient matrix as the nonbasic actions to ensure the invertibility of J.
Nonlinear. In the nonlinear case, the systematical way that divides the full action into the basic and nonbasic actions is principally to ensure the solvability of the equality equation.
Firstly, assume there are a relationship matrix with the shape of (n m) n, and Fig to describe whether the equality constraint is related to aj.
equality constraints and the full action E R We can construct
25

---

For example, if we have 3 equality constraints on action a E R like that
fi a2, a4) = O, f2(a2, 03) = 0, fa fa(a)) = O,
Then, the relationship matrix will be
1 1 0
1 0 0
Now, we need to choose the nonbasic actions that cover the equality constraints as many as possible. This is the maximal matching problem in the bipartite graph, where constraints and actions nodes and there exists an edge between constraint fi are related to action if Fij = 1. Hence, (a1, 02, a3); (#1, 022 aa), (a1, 03, 04) are valid choices of nonbasic actions here, and then the equa- tions can be solved with such divisions. In contrast, we choose (02, 0z, aa) as the nonbasic actions, the equations cannot be solved. It is because if basic action a is determined, the equations will be
Ji (02, a1; 0, f2(22, az) = 0, (0; a1) = 0,
which is unsolvable.
Besides, Jf 1:m may not be invertible in some specific action points. In such situation, we can add small perturbation on the current action point or J make J invertible. This useful trick in classical control methods. However, this situation is seldom seen in practice.
26


Reviews:
[{"review_id": "kxFqFNj7rB", "reviewer": "Reviewer_RHA4", "summary": "The authors introduce a policy optimization methodology suitable for continuous control problems with hard constraints. The optimization framework, named Reduced Policy Optimization (RPO), utilizes mathematical tools such as Generalized Reduced Gradient (GRG) and Lagrangian relaxation to address hard (equality) and soft (inequality) constraints respectively. Through GRG basic and non-basic actions are extracted. The policy network produces the basic actions and by solving the equality equations non-basic actions are calculated. The Lagrangian relaxation is implemented in the policy’s cost function to fulfill the inequality constraints. The evaluation of RPO occurred in three different environments the Safe Cartpole, Spring Pendulum, and OPF with battery energy storage.", "questions": "- How do you think the RPO is probable to perform in complex tasks with hard constraints such as locomotion tasks or tasks requiring a robotic manipulator?\n- What is the average training time using the RPO in comparison to vanilla policy optimization (e.g. SAC, DDPG)?\n-  How can we get define this equation solver for the non-basic actions in a generic way?\n", "limitations": "The authors adequately discuss the limitations of their proposed method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed methodology is innovative since it introduces the GRG to the RL setting. The goal of RPO is to create a policy optimization with hard constraints in an agnostic manner, which can be very beneficial in the field of RL and set the agent capable to solve a variety of complex tasks. \n- The manuscript is well-written and thorough. Related work and relative background are on point.\n- The experiment results present significant improvements in comparison with current policy optimization methods. \n", "weaknesses": "- The number of experiments and the complexity is insufficient to back up the algorithm's robustness and versatility for complex tasks.\n- To obtain the non-basic actions one should implement an equation solver depending on the task, which makes the implementation more difficult. How can we get define this solver in a generic way?\n- In general, off-policy algorithms tend to be more time-consuming than on-policy algorithms. GRG adds more time complexity to the whole optimization process. The authors are aware of that. Hence a training time comparison should have been included.\n- The included code contains only the cart pole benchmark.\n- The code does not run as is: there is no version defined and a lot of deprecated numpy methods (not present on my machine for example) are used.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your valuable feedback and constructive comments! We itemize the weaknesses or comments you mentioned and answers to them.\n\n> **Q1**: The number of experiments and the complexity is insufficient to back up the algorithm's robustness and versatility for complex tasks.\n\n**A1**: Thanks for your comments. However, we cannot agree with you that our experiments are insufficient. We have tested our algorithm in three environments, and we noticed that similar papers like [R1, R2] tested 3-4 environments as well. Besides, the complexity of our three test environments has been demonstrated in **Global Response A1**.\n\n[R1] Liu P, Tateo D, Ammar H B, et al. Robot reinforcement learning on the constraint manifold[C]//Conference on Robot Learning. PMLR, 2022: 1357-1366.\n\n[R2] Wang, Y., Zhan, S. S., Jiao, R., Wang, Z., Jin, W., Yang, Z., … & Zhu, Q. (2023, July). Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. In International Conference on Machine Learning (pp. 36593-36604). PMLR.\n\n> **Q2**: To obtain the non-basic actions one should implement an equation solver depending on the task, which makes the implementation more difficult. How can we get define this solver in a generic way?\n\n**A2**: This is a good question. Different from nonconvex optimization, there exist many efficient methods like modified Newton's method that can solve general nonlinear equations that possess global convergence. More theoretical analysis of the convergence of modified Newton's method can be seen in Chapter 10 of [R3]. Actually, in OPF with battery energy storage case, we apply Newton's method in the construction stage. Therefore, we can define the modified Newton's method as a general solver for this problem.   Notably, for some special equality constraints, we can use analytical solutions directly instead of the modified Newton's method. That's why we do not define a fixed solver.\n\n[R3] Luenberger D G, Ye Y. Linear and nonlinear programming[M]. Reading, MA: Addison-wesley, 1984.\n\n> **Q3**: The comparison of training time between GRG and off-policy algorithms should be included. \n\n**A3**: Thank you for your valuable suggestion. The training time comparison in the environment Safe CartPole and Spring Pendulum is shown in the following tables. Each result is averaged in 5 runs. \n\nFor the time limitation, the training time comparison on OPF with battery energy storage is still in progress, and we will include all results in our final version.\n\nThe training time comparison in Safe Cartpole.\n\n|  Method            |    RPO-DDPG   |     RPO-SAC   |    DDPG-L     |      SAC-L    |     CPO       |      CUP      | Safety Layer  |\n|:------------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|\n| **Training Time (s)**  |   190.2    |    375.5  |          150.2     |        268.5      |      68.7        |   75.2  |      1067.0         |\n\nThe training time comparison in Spring Pendulum.\n|  Method            |    RPO-DDPG   |     RPO-SAC   |    DDPG-L     |      SAC-L    |     CPO       |      CUP      | Safety Layer  |\n|:------------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|\n| **Training Time (s)**  |   242.6    |    419.0  |      174.9    |        309.5      |     98.6      |   81.4  |    3859.7           |\n\n\n> **Q4 & Q5**: The included code contains only the cart pole benchmark. The code does not run as is: there is no version defined and a lot of deprecated numpy methods (not present on my machine for example) are used.\n\n**A4 & A5**: Thank you for raising the concern. According to the principle of NIPS 2023, we cannot present an anonymous repository during the rebuttal phase. Hence, we will release all codes in our three benchmarks later. \n\nTo help you solve the code-running problems, we also present the versions of Python packages used in our work as below.\n\n```\nnumpy                         1.21.6\ntorch                         1.13.1\ngym                           0.19.0\nmatplotlib                    3.5.2\nscipy                         1.9.1\nscikit-learn                  1.0.2\npandas                        1.4.4\n```\n\nIn addition, our experimental operating system is Ubuntu 20.04.6 LTS, the experimental GPU is NVIDIA GeForce RTX 3090 and the CUDA Version is 11.7 for your reference. If you still have any problems, please let us know and we will help you solve them to the experimental results.\n\n> **Q6**: How do you think the RPO is probable to perform in complex tasks with hard constraints such as locomotion tasks or tasks requiring a robotic manipulator?\n\n**A6**: Thank you for raising the concern. We believe that RPO can work in complex tasks with hard constraints. On one hand, as we illustrated in **A1**, our test environment --- OPF with battery energy storage, is actually a complex task with 28 equality constraints and 58 inequality constraints. In contrast, constraints in locomotion tasks or tasks requiring a robotic manipulator like [R4] are generally simpler than our OPF with energy storage. For example, [R4] only contains several constraints. On the other hand, our experiments in classical robot control such as cartpole can also confirm certain transferability of our method in robot control tasks.\n\n[R4] Liu, P., Tateo, D., Ammar, H. B., & Peters, J. (2022, January). Robot reinforcement learning on the constraint manifold. In Conference on Robot Learning (pp. 1357-1366). PMLR. "}, {"review_id": "wPBTQ7zyp1", "reviewer": "Reviewer_ixcf", "summary": "The paper solves the RL problem with equality and inequality hard constraints with a reduced policy optimization (RPO) algorithm, which combines RL with the generalized reduced gradient (GRG) algorithm. RPO partitions actions into basic actions and nonbasic actions following the GRG method, outputs the basic actions using a policy network, and calculates the nonbasic actions using domain knowledge to satisfy equality constraints. Then, the actions go through a projection stage to handle inequality constraints. Experimental results show that the proposed algorithm behaves better than the baselines for MDP with hard constraints. ", "questions": "1. Do you assume that  $J_{:,m:n}^F$ is invertible in Equation 4? How can we handle the case when $F$ is only a function of states?\n\n2. Why do we need to satisfy the equality constraints using 4.1 instead of just satisfying them by construction since the domain knowledge required by 4.1 and by construction are the same (the second point in Weaknesses)?\n\n3. Why is there a mismatch between Figure 3 and Table 2 (the third point in Weaknesses)?\n\n4. Can the authors give an analysis of the reason why constraint violation still exists in the experiments?\n", "limitations": "The authors address the limitations well in Section 6. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Considering both equality and inequality constraints in RL is an interesting problem. The proposed algorithm can handle the problem well. Also, the paper introduced some new RL benchmarks with hard constraints. \n", "weaknesses": "1. In Equation (4), it seems that there is an implicit assumption that $J^F_{\\:,m\\:n}$ is invertible. However, this may not always be true. I think in some environments, the equality constraints may even do not depend on the actions but only the states. For example, if we want a car to always stay in a pre-defined path, then $F$ is only a function of states, and $\\frac{\\partial F}{\\partial a}=0$. The proposed approach seems cannot solve this problem. \n\n2. In the experiments, the equality constraints can be solved by constructions, e.g., in Figure 2(a), let $f_2=\\frac{f_1 \\cos\\theta_1}{\\cos\\theta_2}$, where $\\theta_1$/$\\theta_2$ are the angles between $f_1$/$f_2$ and the vertical direction. It is unclear to me why we need to satisfy the equality constraints using 4.1 instead of just satisfying them by construction since the domain knowledge required by 4.1 and by construction are the same. \n\n3. It seems that there is a mismatch between Figure 3 and Table 2, where the constraint violation of RPO methods are less than $10^{-4}$ in the first two environments in Table 2, but can be larger than $10^{-4}$ in Figure 3. It is also not clear about the meaning of “max violation”.\n\n4. There are typos even in the mathematical parts of the paper. For example, line 161 $a^N\\in\\mathbb{R}^m$ -> $a^N\\in\\mathbb{R}^{n-m}$; line 173 $J^F\\in\\mathbb{R}^{(m-n)\\times n}$ -> $J^F\\in\\mathbb{R}^{(n-m)\\times n}$. \n\n5. Constraint violation still exists in the experiments, and there is a lack of discussion. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback on our paper. Here we address your detailed questions as below:\n\n> **Q1**: In Equation (4), it seems that there is an implicit assumption that $J^F_{:, m:n}$ is invertible. However, this may not always be true.\n\n**A1**: Thank you for raising the concern. We will analyze the invertibility of $J^F_{:, m:n}$ in two situations: linear equality constraints and nonlinear equality constraints. \n\nIn the linear case, if $J^F$ is not invertible, it means there exist redundant equality constraints, then we can directly delete these redundant equality constraints and redefine the dimension of basic actions to make $J^F$ invertible. For example, if we have 3 linear equality constraints $Ax+b=0$ on 4 actions, where \n$$\n\\left[A\\left|b\\right.\\right]=\\left(\n\\begin{array}{cccc|c}\n    1 & 0 & -2 & 3 & 2\\\\\\\\\n    5 & -3 & 1 & 4 & -1\\\\\\\\\n    4 & -3 & 3 & 1 & -3\n\\end{array}\\right).\n$$\n\nHere we choose $a_1$ as the basic action, and then $J^F_{:, m:n}=A_{:, 1:4}$ is not invertible, since the third equality constraint can be represented by another two constraints. Hence, we can delete this redundant constraint and the new equality constraints are $\\tilde{A} + \\tilde{b} = 0$, where\n$$\n\\left[\\tilde{A}\\left|\\tilde{b}\\right.\\right]=\\left(\n\\begin{array}{cccc|c}\n    1 & 0 & -2 & 3 & 2\\\\\\\\\n    5 & -3 & 1 & 4 & -1\n\\end{array}\\right).\n$$\n\nThen, the new basic actions will be $(a_1, a_2)$ and new $J^F_{:, m:n}=\\tilde{A}_{:, 2:4}$ is invertible.\n\nIn the nonlinear case, $J^F$ may not be invertible in some specific action points. In such situation, we can add a small perturbation on the current action point or $J^F$ to make $J^F$ invertible. This is a useful trick in classical control methods. Actually, this situation is seldom seen in practice. \n\n> **Q2**:  I think in some environments, the equality constraints may even do not depend on the actions but only the states. For example, if we want a car to always stay in a pre-defined path, then $F$ is only a function of states, and $\\frac{\\partial F}{\\partial a}=0$. The proposed approach seems cannot solve this problem.\n\n**A2**: We believe our method can still handle the above problem by modifying the model formulation. Note that the current state $s_t$ is inherently defined by the last state $s_{t-1}$ and last action $a_{t-1}$, i.e. $s_t = f(s_{t-1},a_{t-1})$. Therefore, we can transform action-independent equality constraints into action-dependent ones. For example, if we want a car to always stay in pre-defined path, we can formulate the equality constraints on current action by predicting the next position (state).  \n\n> **Q3**: It is unclear to me why we need to satisfy the equality constraints using 4.1 instead of just satisfying them by construction since the domain knowledge required by 4.1 and by construction are the same.\n\n**A3**: Thank you for raising the concern. Section 4.1 actually introduces how to backpropagate the gradient in the construction stage. If we satisfy equality constraints by direct construction and ignore the gradient $\\frac{\\partial\\mathcal{L}}{\\partial a^N}  \\frac{\\partial a^N}{\\partial a^B}$ from the Construction Stage in Section 4.1, we will only obtain the partial gradient $\\frac{\\partial \\mathcal{L}}{\\partial a^B}$ on the basic actions as shown below. This will ruin the training of the policy network. The complete gradient flow is as follows,\n$$\n\\nabla_{a^B} \\mathcal{L} = \\frac{\\partial \\mathcal{L}}{\\partial a^B} + \\frac{\\partial\\mathcal{L}}{\\partial a^N}  \\frac{\\partial a^N}{\\partial a^B} \n$$ \n\nIn Appendix E.3, we have also analyzed your concern with experiments. You can find in Figure 7 that RPO with only a partial gradient (without the gradient from construction) performs much worse than RPO with a gradient from construction in cumulative reward. In other words, if we construct the nonbasic actions directly and do not consider them in backpropagation, the performance of RPO will be very poor. \n\n> **Q4**: It seems that there is a mismatch between Figure 3 and Table 2, where the constraint violation of RPO methods are less than $10^{-4}$ in the first two environments in Table 2 but can be larger than $10^{-4}$ in Figure 3. It is also not clear about the meaning of “max violation”. \n\n**A4**: The mismatch between Figure 3 and Table 2 is because the former is for the training process while the latter is for the evaluation process. Here are two reasons why the evaluation performance is better than the training one. Generally, randomness is introduced during the training process for exploration while it is removed in the evaluation process. This trick is commonly seen in RL methods. Besides, we use more maximum GRG updates in the evaluation procedure. This is a practical trick that we have mentioned in Appendix E.1 (lines 625-627). \n\nThe \"max violation\" in Figure 3 means the maximum violation of all constraints in one state, as we illustrated in lines 292-294.\n\n> **Q5**: There are typos even in the mathematical parts of the paper. For example, line 161 $a^N\\in\\mathbb{R}^m$ -> $a^N\\in\\mathbb{R}^{n-m}$; line 173 $J^F\\in\\mathbb{R}^{(m-n)\\times n}$ -> $J^F\\in \\mathbb{R}^{(n-m)\\times n}$.\n\n**A5**: Thank you for pointing out our typos. We will correct them in the final version.\n\n> **Q6**: Constraint violation still exists in the experiments, and there is a lack of discussion.\n\n**A6**: Thank you for your constructive suggestion. Our method involves multiple hard equality and inequality constraints. In the practical computation, the tolerance of the constraint satisfaction is usually set as 1e-3 or 1e-4. This is also achieved by our method. The constraint violation you see in the experiments is within this tolerance. we will add the discussion in Section 5.3 of the final version."}, {"review_id": "kxPcQE7HVE", "reviewer": "Reviewer_u5yD", "summary": "Inspired by the GRG algorithm, this paper proposed a new reduced policy optimization (RPO) algorithm to handle hard equality and inequality constraints that must be satisfied by any learned policies for continuous control. The algorithm consists of two separate phases. Phase 1 involves the training of the policy network and the equation solver for tackling the equality constraints. Phase 2 further utilizes GRG to tackle the remaining inequality constraints without violating any equality constraints. Experiments conducted on three benchmark problems show that RPO allows the trained policies to achieve high cumulative rewards while minimizing the potential chances of constraint violation, compared to several baseline algorithms.", "questions": "What is the real technical contribution of RPO for handling multiple time independent equality and inequality constraints?\n\nHow to systematically divide the full action space into the basic and nonbasic subspaces? Will any arbitrary division work? What will be the impact of any division on the performance of the learned policy?\n\nWhen is it important and necessary to use Lagrangian penalties in the loss function for training policy networks?", "limitations": "I do not have any concerns regarding this question.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It is important to develop new learning algorithms that can handle a variety of hard constraints requested for solving a reinforcement learning problem. This paper proposed an interesting new algorithm design towards achieving this goal. The validity of some aspects of the new algorithm design is also supported by some theoretical results. Meanwhile, the experiment results look promising.", "weaknesses": "I have some concerns regarding the novelty of the constrained reinforcement learning problems studied in this paper. Past research works often studied constraints regarding the expected behaviors of the trained policy across a full episode. In other words, the constraints are defined over a long sequence of states and actions where all state-action pairs in the sequence are interdependent. In contrast, this paper considered constraints that can be independently defined for every state-action pair. While this consideration certainly has its practical value, however the RPO algorithm developed in this paper may not be directly comparable to those algorithms proposed for solving the time correlated constraints rather than the time independent constraints. Hence, the real technical contribution of RPO for handling multiple equality and inequality constraints may need to be more precisely described and clearly justified.\n\nThe experimental evaluation was performed on three relatively simple reinforcement learning problems. To fully understand the technical advantages of the new RPO algorithm over existing approaches, it may be important to conduct further evaluations of the RPO algorithm on more (and perhaps more challenging) benchmark problems. For example, the review of existing hard-constrained reinforcement learning algorithms in Section 2 seems to suggest a few additional benchmark problems that may need to be considered in this paper too.\n\nOther than GRG, there are a variety of ways to handle time independent constraints on state-action pairs. While this paper introduced the basic idea of GRG clearly, the importance of using GRG over other competing constraint handling or constrained optimization methods may need to be further clarified and justified. Additional theoretical and empirical evaluations may be helpful to clearly reveal the technical novelty of using GRG for constrained policy optimization.\n\nThis paper seems to assume that equality constraints can be easily satisfied by separating all dimensions of the action space into basic dimensions and nonbasic dimensions. Upon fixing the basic dimensions, it is feasible to find suitable values for the nonbasic dimensions to satisfy all equality constraints. This assumption may be debatable and may need to be further investigated. Particularly, it is not clear to me how to systematically divide the full action space into the basic and nonbasic subspaces. Will any arbitrary division work? What will be the impact of any division on the performance of the learned policy? Furthermore, given that the equality constraints can be highly sophisticated and nonlinear, it may not always be possible for the equation solver to find feasible action that can satisfy all equality constraints, subject to the division used.\n\nWhat seems also questionable regarding the new algorithm design is the necessity of using Lagrangian penalties in the loss function for training policy networks. I understand that this may make the projection stage more efficient and reliable. However, the penalties do not seem compulsory and are subject to the complexity of the inequality constraints. For the benchmark problems studied in the experiments, it seems that the inequality constraints are not hard to satisfy. The authors mentioned some existing approaches that can satisfy the inequality constraints directly. In view of this, the necessity of using Lagrangian penalties in the loss function should be further studied, both theoretically and experimentally. Meanwhile, the authors may need to evaluate their RPO algorithm on benchmark problems with more challenging inequality constraints, in order to better demonstrate its strong constraint handling capabilities.\n\nBesides the above, according to my understanding, the projection stage is treated as part of the learning environment. Does this stage depend fully on the action output from the construction stage? If the projection stage is random or time-varying, the stability of the policy network training process may be affected. Furthermore, the reparameterization trick requires propagating the gradients back from the action space to the trainable parameters in the policy network. The effectiveness of this trick may be affected since the projection stage may not be easily supported by the gradient calculation map. On the other hand, if the learning environment is re-defined such that the projection stage is treated as part of the environment, this trick may also affect the effectiveness and reliability of the policy network training process and may need to be further examined.\n\nSince the benchmark problems studied in this paper are new, please give a more detailed explanation of each problem in this paper. For example, the authors mentioned that the equality constraint of the Spring Pendulum problem is state-dependent. However, the actual dependence relationship is not presented, making it hard to understand how the equality constraint is defined and whether it is difficult to satisfy the equality constraint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thorough reviews to help us improve the quality of our work. We will answer all the questions that you concern about.\n\n> **Q1**: What is the real technical contribution of RPO? \n\n**A1**: Thank you for your constructive suggestion. The real technical contribution of RPO is to handle both hard instantaneous (time-independent) equality and inequality constraints in a general RL paradigm. It's worth noting that past research works in constrained RL cannot handle such hard instantaneous constraints. The properties of RPO and the differences between RPO and other methods have been summarized in Table 1. In the final version, we will present the technical contribution of RPO more precisely.\n\n> **Q2**: The experimental evaluation was performed on three relatively simple problems ... additional benchmark problems may be needed.\n\n**A2**: We respectively disagree with this comment that our test problems are relatively simple. Please refer to the **Global Response A1**. \n\n> **Q3**: The importance of using GRG over other competing constraint handling or constrained optimization methods may need to be further clarified and justified. Additional theoretical and empirical evaluations may be added for GRG.\n \n**A3**: Thank you for your constructive suggestion. As we illustrated in **A1**, RPO utilizing the GRG method can handle both general hard instantaneous equality and inequality constraints, while previous research approaches cannot handle this problem. \n\nWith respect to additional empirical evaluation, although there are various RL methods for time-independent constraints, most of them only consider inequality constraints. Hence, they will fail in our benchmarks with both equality constraints and inequality constraints like Safety Layer [14]. This has been shown in our experiments in Table 2 and Figure 3. \n\nWith respect to additional theoretical evaluation, we think the constraint satisfaction of RPO is from GRG updates post-plugged in the policy network, while previous constrained RL methods directly use actions from the policy network and enforce the constraint satisfaction via optimization tricks during training. Hence, the theoretical guarantee of RPO on constraint satisfaction is obvious since the convergence of GRG and Newton's method has been proved in many classical optimization books [25]. Instead, we have presented two theorems to show the property of our RPO, including the GRG update executed in the tangent space and the exact penalty theorem for inequality constraints.\n\n> **Q4**: How to divide the full action space into the basic and nonbasic subspaces? What will be the impact of division? \n\n**A4**: Thank you for raising the concern. If you meet up with cases where there exists an equality constraint that is not related to the full action space. The systematical way can be referred to as our **Global Response A2**. \n\n> **Q5(1)**: Some existing approaches can satisfy the inequality constraints directly; the necessity of using Lagrangian penalties in the loss function should be studied. When is it important and necessary to use this penalties? \n\n**A5(1)**: Thank you for your valuable suggestion. However, we cannot agree with you that the Lagrangian penalties are unnecessary. Here are two reasons. Firstly, if we remove the exact penalty term in the loss function, the constraint violation of $\\tilde{a}$ will be very large, which will lead to thousands of iterations in the projection stage. It is obviously unacceptable in real-world applications. Secondly, if there exists a large gap between the initial action $\\tilde{a}$ and the final action $a$, training without the Lagrangian penalties will be unstable. \n\nIn addition, while we mentioned some existing approaches that can satisfy the inequality constraints directly, most of them cannot deal with equality constraints, like Safety Layer [14], as we showed in Table 2 and Figure 3. \n\n> **Q5(2)**: It seems that the inequality constraints are not hard to satisfy for the benchmark problems. Require challenging inequality constraints ...\n\n**A5(2)**: Please refer to our **Global Response A1(2)**.\n\n> **Q6(1)**: Does the projection stage depend fully on the action output from the construction stage? Is the projection stage random or time-varying? \n\n**A6(1)**: Yes, the projection stage depends fully on the action output from the construction stage without any extra randomness. However, during the training process, this output action of the given state changes, which may lead to a changing number of iterations in the projection stage. Hence, we do not let the projection stage participate in the backpropagation, as shown in Equation 8. \n\n> **Q6(2)**: Treating the projection stage as part of learning may affect the effectiveness and reliability of the policy network. \n\n**A6(2)**: The projection stage is not used as a part of the learning environment. Instead, as shown in Algorithm 1, it is used in the TD target since the action output by the projection stage is the actual behaviour policy. As we analyzed in lines 237-247, that's why we chose off-policy RL methods as the base of RPO to ensure the stability of the training. \n\n> **Q7**: A more detailed explanation of each problem. Why the equality constraint of the Spring Pendulum is state-dependent? \n\n**A7**: Thank you for your constructive suggestion. We will add a more detailed explanation of our benchmark problems in the final version. Below we will explain why the equality constraints in Spring Pendulum are state-dependent. As shown in Appendix D.2, the equality constraint of the spring pendulum is $\\dot{l}+\\ddot{l}dt=0$, where $\\dot{l}$ is the stretching velocity of the spring and $\\ddot{l}$ is the acceleration of the spring. According to Euler-Lagrange equations, we have \n$$\n\\ddot{l}=\\frac{f_{s}+m l \\theta^{2}-k\\left(l-l_{0}\\right)-mg\\cos\\theta}{m}.\n$$\nThis means the equality constraint $\\dot{l}+\\ddot{l}dt=0$ is related to the state variables: the cosine and sine of the angle."}, {"review_id": "dEaZRr3b09", "reviewer": "Reviewer_DYt8", "summary": "This paper proposes RPO to handle general hard constraints (equality and/or inequality constraints per step) for RL. The framework consists of construction and projection stages with a penalty loss for end-to-end training. Finally, the authors validate the effectiveness of their approach on 3 test benchmarks with hard constraints and compare them to previous CMDP-based approaches. ", "questions": "1. What are the soft constraints mentioned in the introduction? \n\n2. The reviewer cannot agree that \"it is worth noting that there is currently a lack of RL environments with complex hard constraints\". The reviewer feels you can always define your own constraints either hard or soft as long as you have the environment model and full observability. The reviewer acknowledges the efforts to develop test examples but cannot agree those three can be claimed as \"benchmarks\"\n\n3. Even if the final action satisfies the constraints, how should I know it is because of the two-stage procedure or the training objective with penalty loss?\n\n4. How to enforce the policy network output basic actions? ", "limitations": "The authors have clearly presented the limitations in the paper. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper studies safe RL with hard constraints which is an important problem. It is well-motivated and well-organized. The experiment results are promising as the proposed approach shows advantages against the CMDP baselines.  This paper is novel as the first attempt to introduce GRG to RL to solve the hard equality and inequality constraints. ", "weaknesses": "There are several weak points of this paper for the reviewer. \n\n1. This paper misses the recent literature addressing RL with hard safety constraints, such as \n\n Wang, Y., Zhan, S. S., Jiao, R., Wang, Z., Jin, W., Yang, Z., ... & Zhu, Q. (2022). Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments. arXiv preprint arXiv:2209.15090.\n\nThis recent ICML paper also claims they are dealing with hard safety constraints for RL. \n\n2. For the reviewer, the two stages proposed by the authors work like a shielding to modify the action generated by the policy network to satisfy the hard constraints. The reviewer would like to know how the authors compare their approach to the literature on RL + shielding, such as \n\nBastani, Osbert, Shuo Li, and Anton Xu. \"Safe Reinforcement Learning via Statistical Model Predictive Shielding.\" Robotics: Science and Systems. 2021.\n\n3. The MDP formulated in this paper assumes a general stochastic environment, this could raise a problem - \"how could you let a random variable (s_t, a_t) be less than 0\" for Eq(3), I feel it should be within the probability format, e.g., \"Pr(g <= 0) = 1\". The tested examples are all deterministic environments as continuous tasks. Therefore, the authors may want to reconsider how to formulate their safe RL problem. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his/her careful reading and valuable suggestions. Below we will answer your concerns point-by-point.\n\n> **Q1**: This paper misses the recent literature [R5], which also deals with hard safety constraints for RL. \n\n**A1**: Thank you for your valuable suggestion. We have carefully studied [R5] and will cite it in our final version. However, the studied problem in [R5] is different from that in our works. As the authors illustrated in this paper, they attempt to solve the chance-constrained RL problem, which has the form: \n$$\n\\begin{array}{c}\n\\max_{\\theta} J\\left(\\pi_{\\theta}\\right), \\\\\\\\\n\\text{ s.t. }P\\left(s(t) \\notin S_{u}\\mid\\pi_{\\theta}, s(0)\\right)\\geq 1-\\eta,\\forall t \\in[0, T],\\forall s(0)\\in S_{0},\n\\end{array}\n$$\nwhile the problem we studied is:\n$$\n\\begin{aligned}\n        \\max_\\theta \\ &J_R(\\pi_\\theta) \\\\\\\\ \n        \\text{ s.t. }\\ &f_{i}(\\pi_\\theta(s_t);s_t)=0\\quad\\forall i, t, \\\\\\\\\n       &g_{j}(\\pi_\\theta(s_t); s_t)\\leq 0\\quad\\forall j, t,\n\\end{aligned}\n$$\nAlthough [R5] can satisfy the safety chance constraints, the safety chance constraints themselves may still provide a soft guarantee for the satisfaction of the actual constraints. \n\n[R5] Wang, Yixuan, et al. \"Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments.\" ICML 2023.\n\n> **Q2**: The reviewer would like to know how the authors compare their approach to the literature on RL + shielding, such as [R6].\n\n**A2**: Thank you for raising the concern. We think the RL + shielding method described in [R6] also differs from our RPO. There are at least two different points. Firstly, the RL + shielding method in [R6] does not perform direct modification on the action generated by the policy network like our RPO but maintains two policies. One is for maximizing the cumulative reward, and another backup policy will be switched to when the potential violation exists. We think such methods should be classified as the methods based on Recovery like [40], which we have mentioned in line 90.\n\nSecondly, the problem studied by [R6], is also different from our RPO. RPO aims to solve hard instantaneous constraints, which are commonly very tight and exist in all states. For example, we must always satisfy the power balance equations in smart grid control. However, the RL + shielding method in this paper can only deal with the long-term soft constraints, which are not very tight, and violations may only exist in a few states. Hence, it is hard for this kind of method to solve RL with the hard constraints we studied.\n\n[R6] Bastani, O., Li, S., & Xu, A. (2021). Safe Reinforcement Learning via Statistical Model Predictive Shielding. In Robotics: Science and Systems (pp. 1-13).\n\n> **Q3**: The MDP formulated in this paper assumes a general stochastic environment; this could raise a problem - \"how could you let a random variable (s_t, a_t) be less than 0\" for Eq(3), I feel it should be within the probability format. Therefore, the authors may want to reconsider how to formulate their safe RL problem.\n\n**A3**: Thank you for raising the concern. As we explained in **A1, A2**, our RPO never aims to solve the Safe RL problem but attempts to solve MDP with hard constraints. In contrast to Safe RL methods, RPO focuses on decision problems with explicit instantaneous constraints like smart grid control. The difficulties of our studied problems are mainly from the tightness of the equality constraints and the requirement to maintain the satisfaction of both equality and inequality constraints rather than the randomness in constraints. For example, if the power balance constraints in the smart grid are modelled in probability form, it may lead to power unbalance in some states, and that will ruin the whole electrical grid.\n\n> **Q4**: What are the soft constraints mentioned in the introduction?\n\n**A4**: The soft constraints we mentioned are the constraints that are merely probably satisfied or satisfied with the expectation form. These kinds of constraints are more like multiple objectives, such as resource limitations. The soft constraint is different from the hard one. In some real-world applications like the smart grid, the hard constraints indicate the feasibility, which must be satisfied with high precision.\n\n> **Q5**: The reviewer cannot agree that \"... there is currently a lack of RL environments with complex hard constraints\". The reviewer acknowledges the efforts to develop test examples but cannot agree those three can be claimed as \"benchmarks.\"\n\n**A5**: Thank you for raising the concern. More precisely, most existing benchmarks only consider inequality constraints, lacking RL environments with both hard equality and inequality constraints at present. Besides, the general interface needs to be carefully designed that delivers explicit information on constraints to the RL agent. Hence, there is currently a lack of RL environments with complex hard constraints. That's why we believe that our three environments can be claimed as \"benchmarks\" for the further study of RL with hard constraints. \n\n> **Q6**: Even if the final action satisfies the constraints, how should I know it is because of the two-stage procedure or the training objective with penalty loss? \n\n**A6**: Thank you for raising the concern. As lines 258-260, SAC-L and DDPG-L are the versions of RPO without the two-stage procedure. You can find in Figure 3 and Table 2 that their performance of them is much poorer than RPO-SAC and RPO-DDPG.\n\n> **Q7**: How to enforce the policy network output basic actions?\n\n**A7**: The actions are divided into basic and nonbasic actions before the training, and the division is not changed after that. Therefore, we do not need to enforce the policy network output basic actions but define the outputs of the policy network are the basic actions. More details on division can be referred to in our response **A1** to reviewer ixcf and **Global Response A2**.\n"}]

Author rebuttal:
We thank the reviewers for their careful reading of considerate and meaningful suggestions to help us improve our paper. We sincerely appreciate that the reviewers find our work "innovative" (RHA4), "interesting" (u5yD,  ixcf) and "novel and well-motivated as the first attempt to introduce GRG to RL" (DYt8) to RL with hard constraints, which can be "very beneficial" (RHA4) in the RL field, and can handle both equality and inequality constraints in "a variety of" "complex" tasks (RHA4, u5yD). We are further glad that the reviewers agree unanimously that our manuscript is "well-written and "thorough" (RHA4, DYt8) and confirm our contributions on both theoretical analysis and "promising" empirical results (RHA4, u5yD, DYt8) to support our algorithm, and "new" RL benchmarks with hard constraint (ixcf).

In the following, we will try to address the concerns/questions of the reviewers and present a detailed item-by-item response to their comments.

Firstly, we would like to offer several clarifications about the common issues from the reviewers.

**Q1**: **(1)** The three test environments used seem simple and insufficient to support the RPO algorithm, and additional benchmark problems should be considered. **(2)** The inequality constraints of three test benchmarks are simple, and more challenging inequality constraints need to be considered. 

**A1**: **(1)** We believe our three test environments with both multiple equality and inequality constraints are complex enough to verify the efficiency of our algorithm empirically. For example, **the case of OPF with battery energy storage involves 57 states, 43 actions, and contains 28 equality constraints and 58 inequality constraints with high nonconvexity**. It's worth noting that there is a lack of such complex benchmarks to test our algorithm. We have to spend a lot of effort in developing these RL environments with both hard equality and inequality constraints. Current RL environments with hard constraints mentioned in Section 2 are either not open-source or require numerous modifications since they were designed just for one specific application. Implementing additional benchmark problems, such as UAVs, and robot dogs, is under consideration in our further work. 

**(2)** We would like to highlight that there is **no need to evaluate our RPO with more challenging inequality constraints**. As we mentioned in lines 131-135, any nonlinear inequality constraints can always be transformed into equality constraints plus inequality box constraints by adding slack variables. Hence, evaluating RPO with box inequality constraints and nonlinear equality constraints is sufficient to show RPO's generality. For example, if we have a complex inequality constraint $g(s, a) \le 0$. Then, we can always transform it into 
$$
    \begin{aligned}
        g(s,a) + \nu &= 0, \\\\
        \nu &\ge 0,
    \end{aligned}
$$
where $\nu$ is the slack variable, which can be viewed as the augmented actions in RL. 

**Q2**: How to divide the action into basic and nonbasic actions? What's the impact of different division choices? 

**A2**: The systematical way to divide the full action space into the basic and nonbasic subspaces below can ensure the solvability of the equality equation.

Firstly, assume there are $n-m$ equality constraints and the full action $a\in \mathbb{R}^n$. We can construct a $0-1$ relationship matrix $E$ with the shape of $(n-m)\times n$, and $E_{ij}$ is to describe whether the equality constraint $f_i$ is related to $a_j$.

For example, if we have 3 equality constraints on action $a\in \mathbb{R}^4$ like that 
$$
\begin{aligned}
f_1(a_1,a_2, a_4)=0, f_2(a_2,a_3)=0,f_3(a_1)=0,
\end{aligned}
$$

Then, the relationship matrix will be 
$$
\left(
\begin{array}{ccc}
1 & 1 & 0 & 1 \\\\
0 & 1 & 1 & 0 \\\\
1 & 0 & 0 & 0
\end{array}\right)
$$

Now, we need to choose the nonbasic actions that cover the equality constraints as many as possible. This is the maximal matching problem in the bipartite graph. Hence, (1,2,3), (1,2,4), (1,3,4) are valid choices of nonbasic actions here, and then the equations can be solved with such divisions. In contrast, if we choose (2,3,4) as the nonbasic actions, the equations cannot be solved. It is because if basic action $a_1$ is determined, the equations will be
$$
    f_1(a_2, a_4; a_1)=0, f_2(a_2,a_3)=0, f_3(\emptyset; a_1)=0,
$$
which is unsolvable.

Actually, part of our experiments is executed with random choices on valid divisions, and we do not observe large variances with different valid divisions. We will add the above analyses in the final version. 


Meta-review: The authors introduced a policy optimization methodology suitable for continuous control problems with hard constraints. The proposed methodology is innovative by introducing GRG to the RL setting with the goal to create a policy optimization with hard constraints in a problem-agnostic manner to solve a variety of complex tasks. Paper is well-written and easy to read, experiments present good results in comparisons with existing RL baselines.
Decision: Accept (poster)