SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shutong Ding $^{1}$  Jingya Wang $^{1}$  Yali Du $^{2}$  Ye Shi $^{1}$  *
$^{1}$ ShanghaiTech University  $^{2}$ King's College London
{dingsht, wangjingya, shiye}@shanghaiitech.edu.cn yali.du@kcl.ac.uk
Recent advances in constrained reinforcement learning (RL) have endowed reinforcement learning with certain safety guarantees. However, deploying existing constrained RL algorithms in continuous control tasks with general hard constraints remains challenging, particularly in those situations with non-convex hard constraints. Inspired by the generalized reduced gradient (GRG) algorithm, a classical constrained optimization technique, we propose a reduced policy optimization (RPO) algorithm that combines RL with GRG to address general hard constraints. RPO partitions actions into basic actions and nonbasic actions following the GRG method and output the basic actions via a policy network. Subsequently, RPO calculates the nonbasic actions by solving equations based on equality constraints using the obtained basic actions. The policy network is then updated by implicitly differentiating nonbasic actions with respect to basic actions. Additionally, we introduce an action projection procedure based on the reduced gradient and apply a modified Lagrangian relaxation technique to ensure inequality constraints are satisfied. To the best of our knowledge, RPO is the first attempt that introduces GRG to RL as a way of efficiently handling both equality and inequality hard constraints. It is worth noting that there is currently a lack of RL environments with complex hard constraints, which motivates us to develop three new benchmarks: two robotics manipulation tasks and a smart grid operation control task. With these benchmarks, RPO achieves better performance than previous constrained RL algorithms in terms of both cumulative reward and constraint violation. We believe RPO, along with the new benchmarks, will open up new opportunities for applying RL to real-world problems with complex constraints.
The past few years have witnessed the significant success of reinforcement learning (RL) [41] in various fields such as mastering GO [38], robotic manipulations [20, 24], autonomous driving [34], and smart grid controlling [52, 47, 33], etc. However, it is still challenging to deploy RL algorithms in real-world control tasks, such as operating robots on a specific surface, controlling power generation to fulfill the demand, etc. The principal reason here is that hard constraints must be taken into account in these control problems. Concretely, such constraints come in the form of both equality and inequality constraints and can be nonlinear or even nonconvex, which makes it difficult to handle them in RL. Moreover, unlike soft constraints, hard constraints take explicit form and require strict compliance, which poses additional challenges.
Existing work on constrained RL can be divided into two categories. The first category involves treating constraints as implicit or soft constraints and using Safe RL algorithms [4, 13, 14, 49, 25, 51, 26, 48]. These algorithms approximate the cumulative costs associated with the constraints and optimize the policy network to balance the trade-off between the cumulative rewards and costs. While Safe RL algorithms have provided certain guarantees on soft constraints, they cannot handle equality constraints since the constraints may not be satisfied due to approximation errors. Moreover, handling multiple constraints using Safe RL algorithms can be computationally expensive, and the existing benchmarks generally only involve simple soft constraints that do not reflect the complexity of real-world applications.
In the second category, the approaches treat the output of the policy network as a set of sub-optimal actions, correcting them to satisfy the constraints by adding an extra constrained optimization procedure. This technique has been explored in several works, such as [16, 29, 10, 33, 32, 24]. Compared to Safe RL algorithms, these algorithms can guarantee the satisfaction of hard constraints but are mostly designed for specific applications and hard constraints of a particular form. For instance, OptLayer [29] employs OptNet [8] into RL to ensure the satisfaction of linear constraints in robotic manipulation. As a result, these approaches to RL with hard constraints are limited in their ability to generalize and lack a generalized formulation.
To address the limitations of existing RL algorithms in handling hard constraints, we propose a constrained off-policy reinforcement learning algorithm called Reduced Policy Optimization (RPO). Our approach is inspired by Generalized Reduced Gradient (GRG), a classical optimization method. RPO partitions actions into basic actions and nonbasic actions following the GRG method and uses a policy network to output the basic actions. The nonbasic actions are then calculated by solving equations based on equality constraints using the obtained basic actions. Lastly, the policy network is updated by the reduced gradient with respect to basic actions, ensuring the satisfaction of the equality constraints. Moreover, we also incorporate a modified Lagrangian relaxation method with an exact penalty term into the loss function of the policy network to improve the initial actions. Our approach provides more confidence when deploying off-policy RL algorithms in real-world applications, as it ensures that the algorithms behave in a feasible and predictable manner. It is worth noting that there is currently a lack of RL environments with complex hard constraints. This motivates us to develop three new benchmarks to validate the performance of our proposed method. To summarize, our main contributions are as follows:
1) Reduced Policy Optimization. We present RPO, an innovative approach that introduces the GRG algorithm into off-policy RL algorithms. RPO treats the output of the policy network as a good initial solution and enforces the satisfaction of equality and inequality constraints via solving corresponding equations and applying reduced gradient projections respectively. To the best of our knowledge, this is the first attempt to fuse RL algorithms with the GRG method, providing a novel and effective solution to address the limitations of existing RL algorithms in handling hard constraints.
2) RL Benchmarks with Hard Constraints. We develop three benchmarks with hard constraints to validate the performance of our method, involving Safe CartPole, Spring Pendulum and Optimal Power Flow (OPF) with Battery Energy Storage. Comprehensive experiments on these benchmarks demonstrate the superiority of RPO in terms of both cumulative reward and constraint violation. We believe that these benchmarks will be valuable resources for the research community to evaluate and compare the performance of RL algorithm in environments with complex hard constraints.
In this section, we first review the existing methods in the field of constrained RL and divide them into different types according to different constraints. We summarize the differences between our method and previous works in Table 1. Besides, we also introduce the literature on the GRG method as the motivation and background of our method.
Soft-Constrained RL. RL with soft constraints is well-studied and also known as safe reinforcement learning (Safe RL). One of the principal branches in Safe RL methods is based on the Lagrangian relaxation such as [13, 39, 22], where the primal-dual update is used to enforce the satisfaction of constraints. Besides, different penalty terms [25, 35, 42] are designed to maintain the tradeoff between the optimality in reward and safety guarantees. Moreover, Notable classical methods of safe reinforcement learning include CPO [4] based on the local policy search; Lyapunov-based
approaches [14]; PCPO [49], FOCOPS [51], CUP [26] based on two-step optimization, EHCSB [46] based on bi-level optimization formulation, and SMPS [9] based on recovery and shielding. Besides, RL with soft instantaneous constraints was first studied separately in [16]. This approach adds a safety layer to the policy with a pre-trained constraint-violation classifier but can only handle linear constraints. Other approaches include [43] based on the Recovery Policy and [45, 44] based on the Gaussian Process. Very recently, Unrolling Safety Layer [50] was proposed to handle the soft instantaneous constraints in RL. However, these approaches in soft-constrained RL tackle constraints implicitly and cannot ensure strict compliance with the constraints, especially the equality ones. By contrast, our method can handle both hard equality and inequality constraints effectively under the RL framework.
Hard-Constrained RL. Compared to RL with soft constraints, RL with hard constraints is rarely studied, and most schemes are designed for some specific application. Pham et al. [29] proposed a plug-in architecture called OptLayer based on OptNet [8], which belongs to a special neural network called optimization layer [5, 40], to avoid infeasible actions in robotics manipulation. In studying resource allocation problems, Bhatia et al. [10] developed further Optlayer techniques to deal with hierarchical linear constraints. Liu et al. [24] investigated robotics manipulation tasks based on RL and used manifold optimization to handle the hard constraints with the inverse dynamic model of robots. Other researchers such as [47, 33] incorporated special optimization techniques into RL to handle power operation tasks in smart grids. However, these methods are designed solely for specific applications or constraints of special types. For example, OptLayer can only handle linear constraints in robotics manipulation. By contrast, RPO is not designed for one specific application and can handle general hard constraints in decision-making problems.
Generalized Reduced Gradient Method. GRG [3] is a classical constrained optimization technique and has the powerful capability to handle optimization with nonlinear hard constraints [27]. The basic idea of GRG is closely related to the simplex method in linear programming which divides variables into basic and nonbasic groups and then utilizes the reduced gradient to perform the update on the basic variables and nonbasic variables respectively. In the past decades, the GRG method has been applied to stock exchange [6], optimal control in very-large-scale robotic systems [31], optimal power flow models [17], and many other fields. Additionally, more recently, several works such as [37] also fuse the genetic algorithms with the GRG method. Besides, recent research like DC3 [18] in deep learning is also based on the idea of the GRG method. DC3 was proposed for learning to solve constrained optimization problems and has demonstrated its good capability to obtain near-optimal decisions with the satisfaction of nonlinear constraints.
Markov Decision Process. A classical Markov decision process (MDP) [41] can be represented as a tuple  $(S, A, R, P, \mu)$ , where  $S$  is the state space,  $A$  is the action space,  $R: S \times A \times S \to \mathbb{R}$  is the reward function,  $P: S \times A \times S \to [0, 1]$  is the transition probability function (where  $P(s' \mid s, a)$  is the transition probability from the previous state  $s$  to the state  $s'$  when the agent took action  $a$  in  $s$ ), and  $\mu: S \to [0, 1]$  is the distribution of the initial state. A stationary policy  $\pi: S \to \mathcal{P}(A)$  is a map from states to probability distributions over actions, and  $\pi(a|s)$  denotes the probability of taking action  $a$  in state  $s$ . The set of all stationary policies  $\pi$  is denoted by  $\Pi$ . The goal of RL is to find an optimal  $\pi^*$  that maximizes the expectation of the discounted cumulative reward, which is  $J_R(\pi) = \mathbb{E}_{\tau \sim \pi}[\sum_{t=0}^{\infty} \gamma^t R(s_t, a_t, s_{t+1})]$ . Here  $\tau = (s_0, a_0, s_1, a_1 \ldots)$  denotes a trajectory, and  $\tau \sim \pi$  is the distribution of trajectories when the policy  $\pi$  is employed. Then, the value function of
Constraint	Method	Multiple	Inequality	Equality	Generality	Model Agnostic
Soft, Cumulative	CPO [4], RCPO [42], PCPO [49]	X	✓	X	✓	✓
Soft, Cumulative	Lyapunov [14, 15]	X	✓	X	✓	✓
Soft, Cumulative	FOCOPS [51], CUP [48]	X	✓	X	✓	✓
Soft, Cumulative	IPO [25], P3O [35]	✓	✓	X	✓	✓
Soft, Cumulative/Instantaneous	Lagrangian [13, 11, 39], FAC [28]	X	✓	X	✓	✓
Soft, Instantaneous	Safety Layer [16]	✓	✓	X	X (Linear)	✓
Soft, Instantaneous	Recovery RL [43]	X	✓	X	✓	✓
Hard, Instantaneous	OptLayer [29], ReCO-RL [10]	✓	✓	✓	X (Specific Linear)	✓
Hard, Instantaneous	ATACOM [24]	✓	✓	✓	X (Specific Nonconvex)	X (Robotics)
Hard, Instantaneous	CC-SAC [33], Hybrid-DDDPG [47]	✓	✓	✓	X (Specific Nonconvex)	X (Power Grid)
Hard, Instantaneous	RPO(*)	✓	✓	✓	✓	✓
Table 1: Comparison among constrained RL algorithms of different categories
state  $s$  is  $V_{R}^{\pi}(s) = \mathbb{E}_{\tau \sim \pi}\left[\sum_{t = 0}^{\infty}\gamma^{t}R\left(s_{t},a_{t},s_{t + 1}\right)\mid s_{0} = s\right]$ , the action-value function of state  $s$  and action  $a$  is  $Q_R^\pi (s,a) = \mathbb{E}_{\tau \sim \pi}\left[\sum_{t = 0}^{\infty}\gamma^t R\left(s_t,a_t,s_{t + 1}\right)\mid s_0 = s,a_0 = a\right]$ .
Generalized Reduced Gradient Method. GRG considers the following nonlinear optimization problem:
$$
\min  _ {\mathbf {x} \in \mathbb {R} ^ {n}} f (\mathbf {x}), \text {s . t .} \mathbf {h} (\mathbf {x}) = \mathbf {0} \in \mathbb {R} ^ {m}, \mathbf {a} \leq \mathbf {x} \leq \mathbf {b} \tag {1}
$$
The optimization problem (1) is a general formulation of nonlinear optimization since any nonlinear inequality constraints can always be transformed into equality constraints with inequality box constraints by adding slack variables. GRG first partitions the variable  $\mathbf{x}$  into the basic variable  $\mathbf{x}^B$  and nonbasic variable  $\mathbf{x}^N$ . Then the reduced gradient with respect to  $\mathbf{x}^B$  is derived as follows [27]:
$$
\mathbf {r} ^ {T} = \nabla_ {\mathbf {x} ^ {B}} f (\mathbf {x} ^ {N}, \mathbf {x} ^ {B}) - \nabla_ {\mathbf {x} ^ {N}} f (\mathbf {x} ^ {N}, \mathbf {x} ^ {B}) [ \nabla_ {\mathbf {x} ^ {N}} \mathbf {h} (\mathbf {x} ^ {N}, \mathbf {x} ^ {B}) ] ^ {- 1} \nabla_ {\mathbf {x} ^ {B}} \mathbf {h} (\mathbf {x} ^ {N}, \mathbf {x} ^ {B}), \qquad (2)
$$
Finally, GRG defines the update step as  $\Delta \mathbf{x}^B = -\mathbf{r}$  and  $\Delta \mathbf{x}^N = -\left[\nabla_{\mathbf{x}^N}\mathbf{h}\right]^{-1}\nabla_{\mathbf{x}^B}\mathbf{h}\Delta \mathbf{x}^B$  to ensure the equality constraints still hold during the iterations. More details of the GRG method can be referred to in supplementary materials.
4 Reduced Policy Optimization
Problem Formulation. Although simple explicit constraints in neural networks can be easily handled by some specific activation functions (e.g., the Softmax operator for probability Simplex constraints and the ReLU operators for positive orthant constraints), it is hard to make the output of the policy network satisfy general hard constraints, especially for nonlinear and nonconvex constraints. In this section, we propose RPO to handle MDP with hard constraints formulated as follows:
$$
\max  _ {\theta} J _ {R} (\pi_ {\theta})
$$
$$
\text {s u b j e c t} f _ {i} \left(\pi_ {\theta} \left(s _ {t}\right); s _ {t}\right) = 0 \quad \forall i, t, \tag {3}
$$
$$
g _ {j} \left(\pi_ {\theta} (s _ {t}); s _ {t}\right) \leq 0 \quad \forall j, t,
$$
where  $f_{i}$  and  $g_{j}$  are the hard deterministic constraints that are only related to  $s_t$  and  $a_{t}$  in the current step, and they are required to be satisfied in all states for the policy. Notably, while this formulation is actually a special case of CMDP [7], it focuses more on the hard instantaneous constraints in constrained RL and is different from the cases [4] considered by previous works in Safe RL, where the constraints only involve implicit inequality ones. Besides, it is also possible to transform the action-independent constraints like  $g_{j}(s_{t + 1})\leq 0$  into  $g_{j}\left(\pi_{\theta}(s_{t});s_{t}\right)\leq 0$  in some real-world applications if the transition function is deterministic.
RPO consists of a policy network combined with an equation solver to predict the initial actions and a post-plugged GRG update procedure to generate feasible actions under a differentiable framework from end to end. Specifically, the decision process of RPO can be decomposed into a construction stage and a projection stage to deal with equality and inequality constraints respectively. In addition, we also developed practical implementation tricks combined with a modified Lagrangian relaxation method in order to further enforce the satisfaction of hard constraints and fuse the GRG method into RL algorithms appropriately. The pipeline of RPO is shown in Figure 1.
4.1 Construction Stage to Handle Equality Constraints
Recalling the nature of equality constraints, it can be viewed as the reduction of the freedom degree in actions. Hence, we follow the formulation in GRG method and divide actions  $a \in \mathbb{R}^n$  into basic actions  $a^B \in \mathbb{R}^m$  and nonbasic actions  $a^N \in \mathbb{R}^{n - m}$ , where  $n - m$  is defined as the number of equality constraints. Hence, the actual actions that we need to determine are the basic actions. Given that, we utilize the policy network to output this part of actions and then calculate the nonbasic actions, via solving a set of equations defined by equality constraints and the predicted basic actions, to guarantee the satisfaction of equality constraints. Additionally, we also present a correct gradient flow based on GRG method, which makes it possible to train the policy network in an end-to-end way. As shown in Proposition 1, we illustrate how to backpropagate from the nonbasic actions  $a^N$  to the basic actions  $a^B$ .
Proposition 1. (Gradient Flow in Construction Stage) Assume that we have  $(n - m)$  equality constraints denoted as  $F(a;s) = 0$  in each state  $s$ . Let  $a^B \in \mathbb{R}^m$  and  $a \in \mathbb{R}^n$  be the basic actions
Figure 1: The training and decision procedure of RPO.
and integrated actions, respectively. Let  $a^{N} = \phi_{N}(a^{B}) \in \mathbb{R}^{n - m}$  denote the nonbasic actions where  $\phi_{N}$  is the implicit function that determined by the  $n - m$  equality constraints. Then, we have
$$
\frac {\partial \phi_ {N} (a ^ {B})}{\partial a ^ {B}} = - \left(J _ {:, m + 1: n} ^ {F}\right) ^ {- 1} J _ {:, 1: m} ^ {F} \tag {4}
$$
where  $J^{F} = \frac{\partial F(a;s)}{\partial a} \in \mathbb{R}^{(n - m) \times n}$ ,  $J_{:,1:m}^{F}$  is the first  $m$  columns of  $J^{F}$  and  $J_{:,m + 1:n}^{F}$  is the last  $(n - m)$  columns of  $J^{F}$ .
The proof is provided in supplementary materials. Additionally, we denote the decision process of the construction stage, including the policy network  $\mu_{\theta}$  and the equation solving procedure  $\phi_N$  as  $\tilde{\pi}_{\theta}$ , and initial actions  $\tilde{a}$  as the concatenation of  $a^B$  and  $a^N$ , i.e.,  $\tilde{a} = (a^B, a^N)$ . Furthermore, with respect to the equation-solving procedure, we can always apply the modified Newton's method [27] in the construction stage. However, for some special equality constraints like linear constraints, we can obtain the analytical solutions directly instead of using the modified Newton's method.
4.2 Projection Stage to Handle Inequality Constraints
After the construction stage, the initial actions denoted as  $\tilde{a}$  may still fall out of a feasible region determined by inequality constraints. Hence, the principal difficulty here is to project the action into the feasible region without violating the equality constraints that have been done in the construction stage. To address this issue, the proposed projection stage is to correct the action in the null space defined by equality constraints. Specifically, this projection is an iterative procedure similar to GRG method, which regards the summation of violation in inequality constraints as the objective  $\mathcal{G}(a^B,a^N)\triangleq \sum_j\max \{0,g_j(a;s)\}$ . Since the update may destroy the satisfaction of equality constraints, we also need to consider equality constraints in the optimization problem. Then, we can employ GRG updates to the action until all the inequality constraints are satisfied, which means we find the optimal solution to this optimization problem illustrated above. Here the GRG update is defined as
$$
\nabla_ {a ^ {B}} \mathcal {G} \left(a ^ {B}, a ^ {N}\right) \triangleq \frac {\partial \mathcal {G} \left(a ^ {B} , a ^ {N}\right)}{\partial a ^ {B}} + \frac {\partial \phi_ {N} \left(a ^ {B}\right)}{\partial a ^ {B}} \frac {\partial \mathcal {G} \left(a ^ {B} , a ^ {N}\right)}{\partial a ^ {N}} \tag {5}
$$
$$
\Delta a ^ {B} = \nabla_ {a ^ {B}} \mathcal {G} (a ^ {B}, a ^ {N}), \Delta a ^ {N} = \frac {\partial \phi_ {N} (a ^ {B})}{\partial a ^ {B}} \Delta a ^ {B}.
$$
Then, the GRG updates are conducted as follows,
$$
a _ {k + 1} ^ {B} = a _ {k} ^ {B} - \eta_ {a} \Delta a ^ {B}, \quad a _ {k + 1} ^ {N} = a _ {k} ^ {N} - \eta_ {a} \Delta a ^ {N}. \tag {6}
$$
where  $\eta_{a}$  called projection step can be viewed as the learning rate of GRG update to control the step of update and  $a_{k}$  denotes the actions  $a$  at the  $k$ -th update where  $a_0 \triangleq \tilde{a}$ . After the projection stage, we obtain the feasible actions that can be deployed in the environment and denote the whole process of the construction stage and projection stage as  $\pi_{\theta}$ , which represents the complete policy performed.
Theorem 1. (GRG update in Tangent Space) If  $\Delta a^{N} = \frac{\partial\phi_{N}(a^{B})}{\partial a^{B}}\Delta a^{B}$ , the GRG update for inequality constraints is in the tangent space of the manifold determined by linear equality constraints.
The proof is referred to in supplementary materials. Theorem 1 indicates that the projection stage will not violate the linear equality constraints. With regard to nonlinear equality constraints, we can approximate this nonlinear equality manifold with the tangent space at each GRG update. In that case, we need to set the projection step  $\eta_{a}$  with a sufficiently small value, practically, smaller than  $10^{-3}$ , to avoid destroying the satisfaction of equality constraints. In this way, we can ensure that the GRG update is conducted in the manifold defined by the nonlinear equality constraints. Additionally, a similar projection method was proposed in [18] with  $\ell_2$  norm objective, which is likely to fail in the nonlinear situation as we analyze in the supplementary materials.
Here we present details about the implementation of our RPO model and some training designs. They mainly include two aspects: 1) how to achieve better initial actions from the policy network and 2) how to incorporate such a decision process into the training of off-policy RL algorithms.
Algorithm 1 Training Procedure of RPO
Input: Policy network  $\mu_{\theta}(s)$ , value network  $Q_{\omega}(s,a)$ , penalty factors  $\nu$ , replay buffer  $\mathcal{D}$ .  
1: for  $t$  in  $1,2,\dots,T$  do  
2: Sample the basic actions  $a_t^B$  with the output of  $\mu_{\theta}(s_t)$  and some random process.  
3: Calculate the action  $a_t$  according to the construction stage 4.1 and projection stage 4.2  
4: Take the action  $a_t$  in the environment and store the returned transition in  $\mathcal{D}$ .  
5: Sample a mini-batch of transitions in  $\mathcal{D}$ .  
6: Update the parameters of the policy network using 8 and the penalty factors using 9.  
7: Construct TD target as  $y_t = r_t + \gamma Q_{\omega}(s_{t+1},\pi_{\theta}(s_{t+1}))$ .  
8: Update the parameters of the value network using MSE loss.  
9: end for