SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. We convert the variables into binary representations. The number of bits used in the binary representation depends on the upper/ lower bounds of variables. For example, assuming  $x,y \in \{0,1,2,3\}$ , we can use  $A \cdot 2^{1} + B \cdot 2^{0}$  to represent  $x$  and  $C \cdot 2^{1} + D \cdot 2^{0}$  to represent  $y$  where  $A,B,C,D$  are Boolean variables.
2. We convert the given linear constraint into a Pseudo-Boolean (PB) constraint by using the binary representation of variables. The resulting PB constraint is  $A \cdot 2^{1} + B \cdot 2^{0} + C \cdot 2^{1} + D \cdot 2^{0} \leq 2$ . Note that the coefficients in a PB constraint can be real numbers.
3. We compile the PB constraint into an SDD using the method described in [6]. An SDD encodes all instantiations of Boolean variables that satisfy the constraint. Each valid instantiation is called a model of the SDD. Figure 1(a) shows a compiled SDD for the PB constraint. For the syntax and semantics of SDD, we refer to [8] for detailed elaboration. When there are multiple equalities/inequalities constraints, we create SDDs in this fashion for each constraint. We then conjoin all such SDDs (which is a poly-time operation) to encode all the constraints into a single SDD that represents the valid action space satisfying all the constraints.
4. We parameterize the final SDD using a standard package [5] to obtain a PSDD as shown in Figure 1(b), which represents a probability distribution over all models of the underlying SDD. A uniform distribution over all models is achievable by a special parameter initialization scheme as provided in [5]. Sampling models from the PSDD can be efficiently performed using a fast and top-down procedure, as described in [16].
We use this scheme to sample valid resource allocation actions in a bike sharing domain that has multiple equality and inequality constraints and has been used previously [4, 18].
3.3 Integrating DDPG with Normalizing Flows
In previous approaches to ACRL, addressing zero constraint violations often involves adding a projection layer to the original policy network. However, this method has drawbacks such as increased training time, slow convergence, and the issue of zero gradient when updating policy parameters, particularly during the early stages of training when pre-projection actions are likely to be invalid. In our work, we propose an integration of the DDPG algorithm with the learned mapping between the support of a uniform distribution  $p$  and the set of valid actions as in Section 3.1. This integration allows us to incorporate the learned mapping directly into the original policy network of DDPG, alleviating the aforementioned issues.
Figure 2(a) illustrates the architecture of our proposed policy network, which consists of two modules.
- The first module is the original parameterized policy network  $\mu_{\theta}$  from DDPG, which takes the state  $s$  as input and outputs  $\tilde{a}$ . To ensure that  $\tilde{a}$  belongs to a uniform distribution  $p$  used for learning the mapping  $f_{\psi}$ , we employ a Tanh activation function at the end of the original policy network.
- The second module is the learned mapping function  $f_{\psi}$  using normalizing flows. This mapping function takes inputs of  $\tilde{a}$  and  $s$ .
Since  $\tilde{a}$  is a member of the uniform distribution  $p$ , for a well trained  $f_{\psi}$ , chances of the mapped action outside of the valid action space will be low, which we also observe empirically. Our proposed policy network offers two significant advantages compared to previous ACRL methods. First, it eliminates the need to solve a QP to achieve zero constraint violation. Instead, the valid action is obtained by mapping a sample from the uniform distribution, resulting in a substantial increase in training speed. Note that in our work the mapping function  $f_{\psi}$  is pre-trained, and its parameters  $\psi$  remain fixed during the learning process of the original policy parameters  $\theta$ . However, if it becomes necessary for large state/action spaces, we can adapt our approach to refine the flow during the learning process. This can be achieved by performing a gradient update on  $f_{\psi}$ , using newly encountered states and valid actions, as described in Section 3.1. Through our experiments, we demonstrate the accuracy of the learned mapping, highlighting the effectiveness of our approach to satisfy action constraints. Second, the architecture of our policy network enables end-to-end updating of the original policy parameters  $\theta$  without solving a math program for action projection, thereby avoiding the issue of zero gradient. This advantage allows for smoother training and more stable convergence.
Policy update The objective is to learn a deterministic policy  $f_{\psi}(\mu_{\theta}(s), s)$  that gives the action  $a$  given a state  $s$  such that  $J(\mu_{\theta})$  is maximized. We assume that the  $Q$ -function is differentiable with respect to the action. Given that the learned mapping  $f_{\psi}$  is also differentiable and the parameters  $\psi$  are frozen, we can update the policy by performing gradient ascent only with respect to the original policy network parameters  $\theta$  to solve the following optimization problem:
$$
\max  _ {\theta} J (\mu_ {\theta}) = \mathbb {E} _ {s \sim \mathcal {B}} [ Q (s, f _ {\psi} (\mu_ {\theta} (s), s); \phi) ] \tag {10}
$$
where  $\mathcal{B}$  is the replay buffer and  $Q$ -function parameters  $\phi$  are treated as constants. Figure 2(b) shows the reversed gradient backpropagation path (in blue) for  $\theta$ . The new deterministic policy gradient for the update of  $\theta$  is then given as follows.
$$
\nabla_ {\theta} J (\mu_ {\theta}) = \mathbb {E} _ {s \sim \mathcal {B}} \left[ \nabla_ {a} Q ^ {\mu_ {\theta}} (s, a; \phi) \nabla_ {\tilde {a}} f _ {\psi} (\tilde {a}, s) \nabla_ {\theta} \mu_ {\theta} (s) | _ {\tilde {a} = \mu_ {\theta} (s), a = f _ {\psi} (\tilde {a}, s)} \right] \tag {11}
$$
Next, we derive the analytical form of the gradient term  $\nabla_{\tilde{a}}f_{\psi}(\tilde{a},s)$  in the new deterministic policy gradient, which is an additional component compared to the standard DDPG update (3). For notational simplicity, we focus on computing  $\nabla_{z^0}f_{\psi}(z^0,y)$ , which is same as  $\nabla_{\tilde{a}}f_{\psi}(\tilde{a},s)$  (as last layer of the policy maps to the domain of the latent uniform distribution, or  $\tilde{a} = z^0$ ). We also note  $\psi$  is the collection of parameters  $\psi_i\forall i = 1,\dots ,K$ . To compute this gradient, we apply the chain rule of differentiation. Considering the composition of functions  $f_{\psi}(z^{0},y) = f_{\psi_{K}}(f_{\psi_{K - 1}}(\dots f_{\psi_{1}}(z^{0},y)))$  we can express the derivative as a product of gradients:
$$
\begin{array}{l} \nabla_ {z ^ {0}} f _ {\psi} (z ^ {0}, y) = \frac {\partial f _ {\psi_ {K}} \circ f _ {\psi_ {K - 1}} \circ \ldots \circ f _ {\psi_ {1}} (z ^ {0} , y)}{\partial (z ^ {0}) ^ {T}} (z ^ {0}) \\ = \frac {\partial f _ {\psi_ {1}}}{\partial \left(z ^ {0}\right) ^ {T}} \left(z ^ {0}\right) \cdot \dots \cdot \frac {\partial f _ {\psi_ {K}}}{\partial \left(z ^ {K - 1}\right) ^ {T}} \left(z ^ {K - 1} = f _ {\psi_ {K - 1}} \left(z ^ {K - 2}, y\right)\right) \tag {12} \\ \end{array}
$$
The Jacobian of each bijective function  $f_{\psi_i}, i = 1, \ldots, K$  is
$$
\frac {\partial f _ {\psi_ {i}}}{\partial \left(z ^ {i - 1}\right) ^ {T}} = \frac {\partial x ^ {i}}{\partial \left(z ^ {i - 1}\right) ^ {T}} = \left[ \begin{array}{c c} \mathbb {I} _ {d} & \mathbf {0} \\ \frac {\partial x _ {d + 1 : D} ^ {i}}{\partial \left(z _ {1 : d} ^ {i - 1}\right) ^ {T}} & \operatorname {d i a g} \left(\exp \left[ - k \left(z _ {1: d} ^ {i - 1}, y\right) \right]\right) \end{array} \right] \tag {13}
$$
where  $\mathbb{I}_d$  is a  $d\times d$  identity matrix.  $\mathrm{diag}(\exp [-k(z_{1:d}^{i - 1},y)])$  is the diagonal matrix where the diagonal consists of elements corresponding to the vector  $\exp [-k(z_{1:d}^{i - 1},y)]$  and the other elements are all zeros.  $\partial x_{d + 1:D}^i /\partial (z_{1:d}^{i - 1})^T$  is computed as follows.
$$
\frac {\partial x _ {d + 1 : D} ^ {i}}{\partial \left(z _ {1 : d} ^ {i - 1}\right) ^ {T}} = - \exp \left[ - k \left(z _ {1: d} ^ {i - 1}, y\right) \right] \mathbf {1} ^ {T} \odot \left(\left[ z _ {d + 1: D} ^ {i - 1} - t \left(z _ {1: d} ^ {i - 1}, y\right) \right] \mathbf {1} ^ {T} \odot \frac {\partial k \left(z _ {1 : d} ^ {i - 1} , y\right)}{\partial \left(z _ {1 : d} ^ {i - 1}\right) ^ {T}} + \frac {\partial t \left(z _ {1 : d} ^ {i - 1} , y\right)}{\partial \left(z _ {1 : d} ^ {i - 1}\right) ^ {T}}\right) \tag {14}
$$
where  $\mathbf{1}^T$  denotes a row vector of length  $d$  whose elements are all one.  $\frac{\partial k(z_{1:d}^{i-1},y)}{\partial(z_{1:d}^{i-1})^T}$  and  $\frac{\partial t(z_{1:d}^{i-1},y)}{\partial(z_{1:d}^{i-1})^T}$  are the Jacobian of  $k$  and  $t$  respectively. Note that when  $k$  and  $t$  are complex functions modeled by neural networks, the Jacobian matrix can be computed efficiently using automatic differentiation tools such as Pytorch and Tensorflow.
Critic Update The update of  $Q$ -function parameters  $\phi$  in our approach follows the same update rule as in DDPG (4). However, in case the mapped action by the learned flow does not satisfy action constraints, we solve a QP to project it into the feasible action space as the environment only accepts valid actions. Therefore, the action stored in the replay buffer  $B$  is either the output from the flow model or the projected action, depending on whether flow mapped action is valid or not. We note that the probability of using the projected action during training is low as for all the tested instances as the normalizing flow had high accuracy. Even if the projection was required, the difference between the flow output and the projected action was small as we show empirically. We also highlight that for policy update, we use the flow mapped action without using any projection, which enables end-to-end training of the policy.
Other RL algorithms While our work focuses on DDPG, it can be extended to other RL algorithms such as SAC or PPO. This is possible because the normalizing flows enable the computation of log probabilities of actions, which is required during training in SAC or PPO. This showcases an additional advantage of using the normalizing flow model in our work compared to other generative models.
(b)
Figure 2: (a) Policy network; (b) The reversed gradient path of  $\theta$  (in blue). Nodes denote variables and edges denote operations. Paths in black are detached for  $\theta$ . The green block is a negative loss (assuming a minimization task).
**Environments** We evaluate our proposed approach on four continuous control RL tasks in the MuJoCo environment [31] and one resource allocation problem that has been used in previous works [4, 18].
- Reacher: The agent is required to navigate a 2-DoF robot arm to reach a target. The agent's actions  $(a_{1},a_{2})\in \mathbb{R}^{2}$  are 2D. The action space is subject to a nonlinear convex constraint  $a_1^2 +a_2^2\leq 0.05$
- Half Cheetah: The task is to make the cheetah run forward by applying torque on the joints. An action is a 6-dimensional vector  $(v_{1}, v_{2}, \ldots, v_{6})$ , bounded by  $[-1, 1]$ . The constraint is defined as  $\sum_{i=1}^{6} |v_i w_i| \leq 20$  where  $w_i$  is the velocity, which can be observed as part of the state.
- Hopper and Walker2d: The task involves controlling a robot to hop or walk forward by applying torques to its hinges. An action is a  $n$ -dimensional vector  $(v_{1},\dots,v_{n})$ , bounded by  $[-1,1]$  where  $n$  is the number of hinges on the robot (3 for Hopper and 6 for Walker2d). The constraint is defined as  $\sum_{i=1}^{n}\max\{w_{i}v_{i},0\} \leq 10$  where  $w_{i}$  is the angular velocity of the  $i^{th}$  hinge, observed in the state.
- Bike Sharing System (BSS): The environment consists of  $m$  bikes and  $n$  stations, each with a capacity  $c$ . At every RL step, the agent allocates  $m$  bikes among  $n$  stations based on previous allocation and demand while adhering to capacity constraints. The agent's action  $a = (a_{1}, a_{2}, \dots, a_{n})$  must satisfy  $\sum_{i=1}^{n} a_{i} = m$  and  $0 \leq a_{i} \leq c$ . We evaluate our approach on a specific scenario with  $n = 5$ ,  $m = 150$ , and  $c = 35$  as in [18]. It poses a significant challenge due to the large combinatorial action space of  $151^{5}$ .
Baselines We compare our approach FlowPG with the following two baselines.
-  $\mathbf{{DDPG}} + \mathbf{P} : \mathbf{{DDPG}} + \mathbf{{Projection}}$  is an extension of the vanilla DDPG [17], which introduces an additional step to ensure feasible actions. Invalid actions are projected onto the feasible action space by solving an optimization problem.
- NFWPO [18]: This algorithm efficiently explores feasible action space by using the Frank-Wolf method, and is also state-of-the-art approach for ACRL.
Learning the Normalizing Flows We employ different techniques to generate random samples from the valid action set in each environment. For the Reacher and Half Cheetah environments, we utilize HMC to generate random samples. Specifically, we generate 1 million random samples for training, which could take up to 5 minutes based on the constraint. HMC is more efficient than rejection sampling and a comparison is available in Appendix A. In the Reacher and Half Cheetah environments, all samples generated by HMC are valid actions, which shows the effectiveness of HMC compared to traditional rejection sampling which only produces  $3.93\%$ ,  $4.70\%$  valid actions respectively. For the BSS environment, we use a PSDD to sample valid actions and obtain all valid actions from the PSDD to train the flow. Additional information on compiling the linear constraints into a PSDD and statistics about the PSDD are provided in the Appendix A.
We apply batch gradient descent to train the conditional flow with Adam optimizer and a batch size of 5000. For Reacher, Half Cheetah, Hopper and Walker2d environments, we train the model for 5K epochs. For BSS environment, we train for 20K epochs. Further details about the training the flow such as learning rates, and neural network architecture of the model are provided in the Appendix B.
(a) WGAN
Figure 3: Mapping between a uniform distribution and action space of Reacher with constraint  $a_1^2 + a_2^2 \leq 0.05$
(b) VAE
(c) Normalizing Flow (Ours)
The source code of our implementation is publicly available<sup>1</sup>. To evaluate the accuracy of the flow model, we sample 100K random points from the uniform distribution  $[-1, 1]$ , and then apply the flow. We measure the accuracy based on whether the output lies within the feasible action region. Our flow model was able to produce  $99.98\%$ ,  $97.25\%$ ,  $87.89\%$ ,  $86.58\%$  and  $85.56\%$  accuracy respectively for Reacher, Half Cheetah, Hopper, Walker2d and BSS environments. Further, we compute the average recall over all uniformly sampled states to obtain the recall of our flow model using Equation 9. The achieved recall rates for our trained normalizing flow models were  $97.85\%$ ,  $78.01\%$ ,  $81.61\%$ ,  $83.58\%$ , and  $82.35\%$  respectively for Reacher, Half Cheetah, Hopper, Walker2d and BSS environments. More details such as time are available in Table 1 in Appendix B.
Comparing with other generative models We conduct an ablation study comparing two other generative models VAE and WGAN (which is more stable than GAN) in Reacher domain. We first evaluated the accuracy by calculating the percentage of valid actions among 100k generated actions. The accuracy rates were as follows: Normalizing flows:  $99.98\%$ ; WGAN  $98\%$ ; VAE:  $83\%$ . We then considered the recall rate. Our flow model can achieve a recall rate of  $97.85\%$ . In contrast, recall rate cannot be computed in a straightforward fashion in VAE and WGAN since determining the corresponding latent action for a given valid action is not possible [21]. Nonetheless, we still can visualize the coverage of VAE and WGAN. In Figure 3, we can see that the feasible region is not fully covered in both VAE and WGAN models, while it is well covered in the normalizing flow model.
Results We present the empirical results of our approach and the baselines on three different environments to show the effectiveness of our method in terms of low constraint violations and achieving fast training speeds. We report results using ten different random seeds. To make a fair comparison, we keep the architectures of our policy network (with the exception of the Tanh layer and the normalizing flow model) and the critic network identical to those used in the baseline approaches. For the detailed pseudo-code of our approach and the neural network architectures along with hyperparameters, please refer to the appendix.
Figure 4(a) shows the average return of all three approaches over the training steps. Our approach achieves a comparable average return to NFWPO in the challenging Half Cheetah, Hopper and Walker2d environments. Additionally, our approach outperforms the other two approaches in terms of average return in the Reacher and BSS environments. We note that DDPG+P suffers from the zero gradient issue, leading to convergence to a lower average return.
In Figure 4(b), we present the cumulative constraint violations before projection for all three approaches over the training steps, with the y-axis represented in log-scale. In all environments, except for the more challenging Walker2d, our approach demonstrates the fewest cumulative constraint violations before projection. In the Reacher and Half Cheetah environments, it notably reduces the violations by an order of magnitude. Moreover, although our approach sometimes generates infeasible actions, they tend to be located near the feasible region, which can be inferred from the average magnitude of constraint violations. To quantify the magnitude of constraint violations, we consider a constrained set  $\mathcal{C}$  defined with  $m$  inequality constraints and  $n$  equality constraints:  $\{x|x\in \mathbb{R}^d,f_i(x)\leq 0,h_j(x) = 0,i = 1,\ldots ,m,j = 1,\ldots ,n\}$ . We define the magnitude of constraint violations as  $CV(x) = \sum_{i = 1}^{m}\max (f_{i}(x),0) + \sum_{j = 1}^{n}\max (||h_{j}(x)| - \epsilon |,0)$ , where  $\epsilon = 0.1$  represents the error margin for the equality constraint. Figure 4(c) illustrates the average magnitude of constraint violations over training steps, with the y-axis presented in log-scale. Our approach exhibits the lowest average magnitude of constraint violations in all environments except Hopper and
----- Decision boundary Feasible actions Infeasible actions
$^{1}$ https://github.com/rlr-smu/flow-pg
Figure 4: Training curves for the Reacher, Half Cheetah, and BSS environments are displayed in columns from left to right, showcasing the Average Return(↑), Cumulative Constraint Violations(↓), Average Magnitude of Constraint Violations(↓), and Time Elapsed(↓).
Walker2d. In the Reacher, Half Cheetah, Hopper and Walker2d environments, the average magnitude of constraint violations is close to zero, indicating that the invalid actions are very close to the feasible region. In the BSS environment, our approach shows slightly higher average magnitude due to the integer property of the actions. The low number of cumulative constraint violations and lower average magnitude of constraint violations demonstrate the effectiveness of our learned flow model.
In Figure 4(d), we show the runtime of the training process. Our approach demonstrates a significantly faster training time compared to NFWPO, with a speed improvement of  $2 \sim 3$  times. This speed advantage is attributed to the fact that NFWPO requires computationally expensive QP solutions to determine the policy update direction within the feasible action space. In contrast, our approach utilizes the learned normalizing flow model to generate valid actions, with the flow model parameters frozen during the policy update. Although  $\mathrm{DDPG + P}$  shows a faster runtime, it comes at the expense of a lower average return and higher constraint violations.
In this work, we present a novel approach called FlowPG based on Normalizing Flows to address action constraints in RL. The architecture of FlowPG allows the policy network to generate actions within the feasible action region. Furthermore, our experimental results demonstrate that FlowPG effectively handles action constraints and outperforms the previous best method by significantly reducing the number of constraint violations while having faster training speeds.
— FlowPG — NFWPO — DDPG+P
This research/project is supported by the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme (AISG Award No: AISG2- RP-2020-017).
[1] Aaron D Ames. Human-inspired control of bipedal walking robots. IEEE Transactions on Automatic Control, pages 1115-1130, 2014.
[2] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, pages 136-145, 2017.
[3] Michael Betancourt. A Conceptual Introduction to Hamiltonian Monte Carlo. arXiv preprint arXiv:1701.02434, 2018.
[4] Abhinav Bhatia, Pradeep Varakantham, and Akshit Kumar. Resource constrained deep reinforcement learning. In International Conference on Automated Planning and Scheduling, pages 610-620, 2019.
[5] Arthur Choi. The PyPSDD Package. https://github.com/art-ai/pypsdd, 2018.
[6] Arthur Choi, Weijia Shi, Andy Shih, and Adnan Darwiche. Compiling neural networks into tractable boolean circuits. AAAI Spring Symposium on Verification of Neural Networks, 2017.
[7] Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration in continuous action spaces. arXiv preprint arXiv:1801.08757, 2018.
[8] Adnan Darwiche. Sdd: A new canonical representation of propositional knowledge bases. In International Joint Conference on Artificial Intelligence, pages 819-826, 2011.
[9] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference on Learning Representations, 2016.
[10] Marguerite Frank and Philip Wolfe. An algorithm for quadratic programming. Naval Research Logistics Quarterly, 3:95-110, 1956.
[11] Kurt Otto Friedrichs. The identity of weak and strong extensions of differential operators. Transactions of the American Mathematical Society, 55:132-151, 1944.
[12] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062, 2019.
[13] Leonard Jaillet and Josep M. Porta. Path Planning Under Kinematic Constraints by Rapidly Exploring Manifolds. IEEE Transactions on Robotics, 29(1):105-117, 2013.
[14] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980, 2017.
[15] Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible 1x1 Convolutions. In Advances in Neural Information Processing Systems, 2018.
[16] Doga Kisa, Guy Van Den Broeck, Arthur Choi, and Adnan Darwiche. Probabilistic sentential decision diagrams. In *Principles of Knowledge Representation and Reasoning*, pages 558-567, 2014.
[17] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
[18] Jyun-Li Lin, Wei Hung, Shang-Hsuan Yang, Ping-Chun Hsieh, and Xi Liu. Escaping from zero gradient: Revisiting action-constrained reinforcement learning via frank-wolfe policy optimization. In Uncertainty in Artificial Intelligence, pages 397-407, 2021.
Appendix A Sample Generation
HMC: We used Python implementation [23] of Hamiltonian Monte Carlo algorithm for sample generation of Reacher and Half-Cheetah environments. We set the starting point as the origin  $(0^{D})$  and used default hyperparameters such as  $\epsilon = 0.2$ , decay  $= 0.9$ .