FlowPG: Action-constrained Policy Gradient with Normalizing Flows
Authors: Janaka Chathuranga Brahmanage
Jiajing Ling
Akshat Kumar
Venue: NeurIPS 2023 poster (2023)
Keywords: action-constrained reinforcement learning
decision making

Abstract: Action-constrained reinforcement learning (ACRL) is a popular approach for solving safety-critical and resource-allocation related decision making problems. A major challenge in ACRL is to ensure agent taking a valid action satisfying constraints in each RL step. Commonly used approach of using a projection layer on top of the policy network requires solving an optimization program which can result in longer training time, slow convergence, and zero gradient problem. To address this, first we use a normalizing flow model to learn an invertible, differentiable mapping between the feasible action space and the support of a simple distribution on a latent variable, such as Gaussian. Second, learning the flow model requires sampling from the feasible action space, which is also challenging. We develop multiple methods, based on Hamiltonian Monte-Carlo and probabilistic sentential decision diagrams for such action sampling for convex and non-convex constraints. Third, we integrate the learned normalizing flow with the DDPG algorithm. By design, a well-trained normalizing flow will transform policy output into a valid action without requiring an optimization solver. Empirically, our approach results in significantly fewer constraint violations (upto an order-of-magnitude for several instances) and is multiple times faster on a variety of continuous control tasks.

FlowPG: Action-constrained Policy Gradient with Normalizing Flows
Janaka Chathuranga Brahmanage, Jiajing Ling, Akshat Kumar School of Computing and Information Systems
Singapore Management University
monakat.002. jjling 2018 2018)@phdcs smu. smu.edu. sg, akshatkumardsmu. edu.eeuuuuuuuuuuueee...... 5g
Abstract
retion-constrainee reinforcement learning (ACRL) is popular approach for solving safety-critical and resource-allocant related decision making problems. A major challenge in ACRL is ensure agent taking valid action satisfying constraints each RL step. Commonly used approach of using a projection layer on top of the policy network requires solving an optimization program which can result in longer training time, slow convergence, and zero gradient problem. To address this, first mapping between the feasible action space and the support of simple distribution latent variable, such as Gaussian. Second, learning the flow model requires sampling from the feasible action space, which is also challenging. We develop multiple methods, based on Hamiltonian Monte-Carlo and probabilistic sentential decision diagrams for such action sampling for convex and non-convex constraints. Third, we integrate the learned normalizing flow with the DDPG algorithm. By design, a well-trained normalizing flow will transform policy output into valid action without requiring an optimization solver. Empirically, our approach results in significantly fewer constraint violations (upto order-of- -f-emnnuu- for several instances) and is multiple times faster on a variety of continuous control tasks.
normalizing flow model to learn invertible, differentiable
Introduction
Action-constrained reinforcement learning (ACRL), where an agent's action taken each RL step should satisfy specified constraints, has been applied to solve diverse range of real-world applications [18]. One example is the resource allocation problem in supply-demand matching 4.
The allocation of resources must satisfy constraints such total assigned resources should be within a specific upper and lower limit. In robotics, kinematics constraints (e.g., limits on velocity, acceleration, torque among others) can be modeled effectively using action constraints 1 13 26 32)
One of the main challenges in solving ACRL to ensure that during training and evaluation, all the action constraints are satisfied at each time step, while simultancously improving the policy. Lagrangian Relaxation (L.) is a popular approach for solving constrained RL problems [30, however directly applying LR to ACRL is impractical due to the difficulty in defining cost functions (c(s, a)) that penalize infeasible actions for all (s, a), and the inability guarantee zero constraint violation during training and policy execution. Also, there are studies addressing ACRL with discrete action As they use propositional logic, is not clear how to extend them to continuous action spaces. Recently proposed approaches aim to satisfy continuous action constraints using an action projection step. One natural approach to add a differentiable projection layer at the end of the policy network to satisfy the constraints [2] 26). This projection layer projects the unconstrained policy action onto the feasible action space by solving quadratic program (QP). However, this approach has two primary limitations. Firstly, solving QP at each RL step can be
spaces [99 20
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

computationally expensive, particularly when dealing with non-convex action constraints or large action spaces. Secondly, the coupling between the policy network and projection layer can lead to a potential issue of zero gradient during end-to-end training, which can undermine the effectiveness of the projection layer approach, especially in the early training stages, as highlighted by recent studies [8]. This issue occurs when the policy output is far outside the feasible action space, thereby any small change in the policy parameters does not result in any change in the projected action.
To overcome the zero gradient issue, an ACRL algorithm has been proposed in 18. which decouples policy gradients from the projection layer. The policy parameters are updated by following the direction found by the Frank-Wol method [TO], which is within the feasible action space. However, this approach can also be computationally expensive. as solving a QP is still required during the update of policy parameters and during policy execution. To summarize, while there are several proposed approaches for solving ACRI, with continuous action space, all of them share the common drawback of requiring QP solving during either training or action execution, or both. This can cause a significant reduction in training speed, which we also validate empirically. Moreover, some of these approaches also suffer from the zero gradient issue, which can make training sample inefficient. Therefore, there is need for more efficient methods to tackle ACRL with continuous action spaces.
Often, action constraints are specified analytically using features from state and actions (e.g., x² + y² <= 1 where z, y can be features). Our key idea is to exploit given action constraints as domain knowledge and develop an effective, compact representation of all valid actions that can be integrated with RL algorithms. To find such a representation, generative models provide an attractive solution since they can generate valid actions by learning from a finite number of valid actions (which can be provided by sampling from the valid action space). This allows for compact feasible action space representation even for continuous action spaces. Furthermore, generative models can be easily integrated with RL algorithms for end-to-end training, as shown in previous work [22 33 even though they are not specifically solving ACRL problems. The idea of learning the representation of all valid actions also shares a connection with offline RL, where invalid actions are often treated as iiloof-distribution (OOD) actions [2] 35). However, notable difference between our work and offline RL lies in the availability of a dataset comprising valid actions. In offline RL, such a dataset is assumed to be available (c.g.. collected from random policy or expert). In ACRL, collecting data even from a random policy is not straightforward as random policy must also select actions uniformly from the feasible action space, which itself is the key problem we address in this work.
Our main contributions are:
We utilize the normalizing flows, a type of generative model, to learn a differentiable, invertible mapping between the data distribution of sampled valid actions and simple latent distribution (e.g.. Gaussian, uniform distribution) 9 (27]. When a member of the latent distribution is given, the flow model can transform into an element that conforms to the distribution of valid actions, assuming the model is well-trained. Compared to other generative models, such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), the normalizing flow model is known to be more efficient at completing the task of data generation [9]
Sampling uniformly from the feasible action space to train the flow model is challenging. For example, valid action space may be characterized using set of constraints over binary variables. We therefore develop multiple methods, based on Hamiltonian Monte-Cart decision diagrams (PSDD) [16, 28), for valid action sampling for (non)conwe) constraints. These methods are significantly more sample efficient than the standard rejection sampling.
B and probabilistic
We propose simple and easy to implement method to integrate the normalizing flow model with deep RL algorithms such as DDPG 17. Our approach general, requires change only in the last layer of the policy network, and can be integrated with other deep RL methods. In our method, the modified policy network outputs an element of the latent distribution used in the normalizing flow model, which is then transformed into valid action via the mapping provided by the normalizing flow. We also show gradients can be propagated through the normalizing flow to improve the policy. By integrating the normalizing flow with deep RL algorithms, we can avoid the zero gradient issue since there is no projection of invalid action onto the feasible action space. We show empirically that the flow model can be trained well on variety of action constraints used in the literature, therefore, probability of constraint violation occurring remains low. Empirically, approach results in significantly fewer constraint violations (upto 10x less), and is multiple times faster on a variety of continuous control tasks than the previous best method [18]
2

---

2 Preliminaries
Action-conss conosnconsttai Markov Decision Process We consider Markov decision process (MDP) model, which is defined by a tuple (S, A, r, 7, bo), where S is the set of possible states, A is the set of possible actions that an agent can take, p(8++1 8t, at) the probability distribution over the next state 8t-1 given the current state 8, and action Gr, r(sr, 0t) is the reward function that determines the reward received by the agent after taking action a state 8 at time step Y E [0. 1) is the discount factor, and by is the initial state distribution. We assume the action space A is continuous, and focus on deterministic policies. Specifically, let (-(()) denote a deterministic policy parameterized by 6; action taken in state 8 is given as 1o (s). Under policy /40, the state value function (V) and the state-action value function (Q) are defined as follows respectively.
V (s)
(r(ss, pol. Q (s,
E
rr((s) a0) s, 3, a: pه] (I)
An retion-constrained MDP extends the standard MDP by incorporating explicit action constraints that determine a valid action set C(s) C A for each state E S. In other words, the agent can only choose an action from the set of valid actions at each time step. The goal of an action-constrained MDP is to find a policy that maximizes the expected discounted total reward while ensuring that all chosen actions are valid with respect to the constraints. Formally, we have:
max ((0o = E, [V (s)] s.t. ag E C(ss) At 0
(2)
We conisder a RL setting where transition and reward functions are not known. The agent learns policy by interacting with the environment and using collected experiences (St, at Tt, 81+1).
Deep Deterministic Policy Gradient Deep Deterministic Policy Gradient (DDPG) is RL al- gorithm specifically designed to handle continuous action spaces
[17]. The algorithm combines deterministic policy gradient [29] with deep Q-learning (25). To solve unconstrained RL problem, DDPG applies stochastic gradient ascent to update policy parameters in the direction of the gradient (oJ(p)) The deterministic policy gradient theorem is used to compute the policy gradient as:
( E.-s(Ve V Q (s, a; 6) (elue(s) (s)pp=)))))
The state-action value function ((s, a; Φ) is approximated using a deep neural network, and the parameters of this neural network are updated by minimizing the following loss function:
Ers.a.e + Q ue (b));")> Q(s, az +))
where (s, 8',, r) is transition sample; B is the replay buffer that stores transition samples; O' and ø' are the parameters of the target policy and the target state-action value function respectively.
Normalizing Flows A normalizing flow model is a type of generative model that transforms simple distribution, such as a Gaussian or uniform distribution, into complex one by applying a sequence of invertible transformation functions 9 27 Let f = fK o fK -1 o fi R + R² denote RR i i= 1 K, an invertible transformation function. Starting from initial sample from a base distribution (or prior distribution), zo ~ p(="), the transformed sample from the model is = [K o f ....... o outputs 2², and J = = Given a training dataset DD the mapping function f learned by maximizing the log-likee ligeeiold of the data, which is defined as log p(D) = EveD log p(z). The log probability log p(x)) is computed by repeatedly applying the change of variables theorem, and is expressed as:
a normalizing flow model, where each ʃ :
(=²). Each fi takes input z and
logp(x) = llogp<>>) log det 0f((2--1)
(5)
0(2--17T
3 Normalizing Flow for ACRL
3.1 Learning Valid Action Mapping Using Normalizing Flows
We employ normalizing flows to establish an invertible mapping between the support of a simple distribution and the space of valid actions. While various transformation functions can be utilized
3

---

for implementing normalizing flows, we specifically focus on the conditional RealNVP model [4 due to its suitability for the general ACRL setting, where the set of valid actions is dependent on the state variable. The conditional RealNVP extends the original RealNVP [9] by incorporating the conditioning variable in both the prior distribution and the transformation functions. These transformation functions, which are implemented as affine coupling layers, possess the advantageous properties of enabling efficient forward propagation during model learning and efficient backward propagation for sample generation [9].
Given a dataset D consisting of valid actions (details in Section 3.2 we train the invertible mapping fo parameterized by v to capture the relationship between the support of a uniform distribution p and the elements in D. Learning this mapping provides an easy way to generate new valid actions. The policy network outputs an element from the domain of the uniform distribution, which is then mapped to valid action using the learned flow. This process of generating actions is much simpler than using math program for action projection, which can result in the zero gradient issue.
The flow learning process involves maximizing the log- lieelikei of the data, following the method- ology presented in [34). In contrast to using Gaussian distribution, we opt for uniform distribution as it demonstrated better empirical performance when combined with the DDPG algorithm. Once the normalizing flow model is learned, during backward propagation with conditioning variables, each bijective function fu., where 1..... ., K, takes inputs of 2²-1 R and a conditioning variable y E Rv (representing state features) and produces an output of x R as:
#i.d = zind
2.41DD = (#2+1:D - (( ,y) a
v))
where d D the index that splits dimensions of input x' into two parts; and are scale and translation functions that map from Rd+2, + D These two functions are modeled using neural networks. @ denotes the element-wise product (Hadamard product). Therefore, given a random member of the uniform distribution 20 ~ p(=0) and conditioning variable y. the mapped element is:
x Sox f o...o fox (2",,)
(8)
Note that the transformed output obtained through the normalizing flow model may not precisely match any specific element from the dataset D. However, when the mapping fo is effectively trained, the resulting x should exhibit similar characteristies to the elements in D, thereby satisfying the constraints with high probability, as we also observed empirically. In essence, the combination of the uniform distribution and the learned mapping serves as efficient representation of all valid actions, providing a means to generate actions that adhere to the constraints.
Unlike other generative models, normalizing flows provide the ability to measure the recall rate in learning the valid action mapping due to their invertible bijective transformations. The recall rate (we call it "coverage") indicates the fraction of valid actions that can be generated from the latent space. Given conditioning variable y, let C(y) denote the set of valid actions that are uniformly distributed in the feasible region. The recall can be computed as follows.
recallly) Exec(y) Idom sy for (x,9) 9)
C(y))
where fo is the inverse transformation function of the normalizing flows model fu. Recall is useful to characterize the learned generative model. If the recall is low, it would limit the feasible constrained space over which ACRL algorithm optimizes the policy. This can result in lower solution quality.
The Mollified Uniform Prior Most recent studies of normalizing flows employ a standard Gaussian distribution as the latent space distribution g 15 27,, which spans the entire real number space R. However, through empirical observation, we noticed that with Gaussian latent distribution, the feasible action region tends to be mapped closer to zero in the latent space, where the probability of the standard Gaussian is higher. On the other hand, points located far away from the center of the Gaussian (several standard deviations) often get mapped into the infeasible action space. Consequently, when coupling the DDPG policy network with the flow, it resulted in high constraint violations as the policy network output can be any number from the support of the Gaussian, and not just limited to the high probability region.
4

---

8
89
20
(a)
(h)
(c)
Figure 1: (a)An SDD representing the PB constraint +2++="40++++000 (b)A PSDD ; (c)
Samples generated with HMC
To address this issue, we decided to use a uniform distribution from [-1, 1]2 as the latent distribution instead of the standard Gaussian. In the uniform distribution, there no high or low probability region unlike the standard Gaussian. However, the uniform distribution is not differentiable, therefore we employ modified version called the Mollified uniform distribution [T]. In order train the normalizing flow, we require the probability density of the prior distribution as in Eq 5 Computing the probability density of manti-dimenniooall mollified distribution can be challenging since requires integrating over all dimensions []]. However, in the case of a single dimension, sample drawn from mollified uniform distribution can be seen as sample from a uniform distribution with added Gaussian noise. In other words, sample is obtained as 2n - 6, where zo follows a uniform distribution in the range [-1, 1]. and follows Gaussian distribution with mean 0 and standard deviation ø.
To calculate the overall probability density, we treat each dimension individually, compute their probability densities, and take their product as the overall probability. The probability density a single dimension can be easily calculated using the following formula: p(x)
202 (z-z0)² dzg = 4 --م + (=1=)) In this formula, & represents the cumulative density function of the standard Gaussian distribution.
E. 2
3.2 Generating Random Samples from the Valid Action Space
To effectively train the normalizing flow model, need to generate samples from the set of valid actions. Even with the known analytical form action constraints this a challenging problem. Standard rejection sampling is highly inefficient sample from the complex constraints that arise in ACRL benchmarks. Therefore, we use two methods described next.
Hamiltonian Monte-Carlo (HMC) HMC a Markov chain Monte Carlo method, which utilizes energy conservation effectively sample target distribution 3 It utilizes Hamiltonian-dynaiic to simulate the movement of a particle in a continues space based on a potential energy function. uses leap-frog algorithm with the first order integrator, estimate the next position of the particle. In our work, we need to generate samples in the feasible region. Therefore we define energy function to be a fixed value in the feasible region. Then we bound the feasible region with a hard wall of infinity potential by setting the potential to oo in the infeasible region. Figure [(c] depicts the distribution of feasible action samples generated by HMC in the action constraint of the Reacher Environment (aî + a² < 0.05). It presents a contour map of the sample density in Figure (cc) (ii), as well as their projection onto each dimension in Figure [(c) (i) and Figure [ie) respectively.
Probabilistic Sentential Decision Diagrams In many real-world applications such as resource allocation, the constraints are defined using set of linear equalities/inequalties over integer variables. However, sampling valid actions using HMC that satisfy such constraints is very challenging due to the presence of equality constraints. To address this challenge, we propose to use Probabilistic Sentential Decision Diagram (PSDD) [16] to encode uniform distribution over all the valid actions and then sample actions from the PSDD. For this setting, we assume that all the constraints are linear (in)equalities. We describe the process of constructing PSDD next. Consider an inequality constraint
2. We follow the following steps:
5

---

1. We convert the variables into binary representations. The number of bits used in the binary representation depends on the upper/lower bounds of variables. For example, assuming 2, y E {0, 2,3,3, we can use 4 to represent z and C222 D.22" to represent y where A, B, D are Boolean variables.
2. We convert the given linear constraint into Pseudo-Boolean (PB) constraint by using the binary representation of variables. The resulting PB constraint is 4C222 coefficients in a PB constraint can be real numbers.
Note that the
3. We compile the PB constraint into an SDD using the method described in [6. An SDD encodes all instantiations of Boolean variables that satisfy the constraint. Each valid instantiation is called a model of the SDD. Figure Ita) shows a compiled SDD for the PB constraint. For the syntax and semantics of SDD, refer to [8] for detailed elaboration. When there are multiple equalities/inequaittes constraints, we create SDDs in this fashion for each constraint. We then conjoin all such SDDs (which is a poly-time operation) to encode all the constraints into single SDD that represents the valid action space satisfying all the constraints.
4. We parameterize the final SDD using a standard package [5] to obtain PSDD as shown in Figure lb). which represents a probability distribution over all models of the underlying SDD. A uniform distribution over all models is achievable by special parameter initialization scheme as provided in (5) Sampling models from the PSDD can be efficiently performed using a fast and top-down procedure, described in [16].
We use this scheme to sample valid resource allocation actions in a bike sharing domain that has multiple equality and inequality constraints and has been used previously [1.88.
3.3 Integrating DDPG with Normalizing Flows
In previous approaches to ACRL, addressing zero constraint violations often involves adding a projection layer to the original policy network. However, this method has drawbacks such as increased training time, slow convergence, and the issue of zero gradient when updating policy parameters, particularly during the early stages of training when pre-projection actions are likely to be invalid. In our work, we propose an integration of the DDPG algorithm with the learned mapping between the support of a uniform distribution p and the of valid actions as in Section 3.1 This integration allows us to incorporate the learned mapping directly into the original policy network of DDPG, alleviating the aforementioned issues.
Figure 2(a) a) illustrates the architecture of our proposed policy network, which consists of two mod- ules.
The first module is the original parameterized policy network μe from DDPG, which takes the state s as input and outputs a. To ensure that à belongs to uniform distribution p used for learning the mapping fv. we employ a Tanh activation function at the end of the original policy network.
The second module is the learned mapping function fg using normalizing flows. This mapping function takes inputs of ä and 8.
Since ä is member of the uniform distribution p, for a well trained fax chances of the mapped action outside of the valid action space will be low, which we also observe empirically. Our proposed policy network offers two significant advantages compared previous ACRL methods. First, eliminates the need to solve a QP to achieve zero constraint violation. Instead, the valid action is obtained by mapping a sample from the uniform distribution, resulting in substantial increase in training speed. Note that in our work the mapping function fo is pre-trained, and its parameters & remain fixed during the learning process of the original policy parameters 0. However, it becomes necessary for large state/action spaces, we can adapt our approach to refine the flow during the learning process. This can be achieved by performing gradient update on fe. using newly encountered states and valid actions, as described in Section [.1 Through our experiments, we demonstrate the accuracy of the learned mapping, highlighting the effectiveness of our approach to satisfy action constraints. Second, the architecture of our policy network enables end-to-end updating of the original policy parameters 0 without solving a math program for action projection, thereby avoiding the issue of zero gradient. This advantage allows for smoother training and more stable convergence.

---

Policy update The objective is to learn a deterministic policy fo (H)(s,, s) that gives the action given a state such that ((1)) is maximized. We assume that the Q-function differentiable with respect to the action. Given that the learned mapping J is also differentiable and the parameters v
are frozen, we can update the policy by performing gradient ascent only with respect the original policy network parameters & to solve the following optimization problem:
max J(ue) = Fe.-sQ((e Jo (ne(()) 8); [)]
(10)
where B is the replay buffer and Q- Q-function parameters e are treated as constants. Figure 2 b) shows the reversed gradient backpropagation path (in blue) for The new deterministic policy gradient for the update of 0 is then given as follows.
( = B V a Q (s, ( Vafe (ā. s) V (=
(11)
Next, we derive the analytical form of the gradient term Vafu (ā, s) in the new deterministic policy gradient, which an additional component compared the standard DDPG update (3). For For notational simplicity, we focus on computing - fz y). which is same as Vafe s) (as last layer of the policy maps to the domain of the latent uniform distribution, or ā = 29). We also note &b is the collection of parameters t/, Vi 1,.... K. To compute this gradient, we apply the chain rule of differentiation. Considering the composition of functions [x(=" .V) = Sex (fox (... Jvr (2",,)))) ,))), we can express the derivative as a product of gradients:
V.ff(EE 0fax o Svx-_ o... S (2 .v) (x9)
0(20TT
a5 0(:0yZ
05nk (ath- fux (e (12)
9(=K-197*
The Jacobian of each bijective function fur I...., K
0fs.
Oxe
la
(13)
0(2--197
1(2
diag(expl - ,1I)
where IL is a d dl identity matrix. diag(expl- v)[) is the diagonal matrix where the diagonal
consist of elements corresponding the vector expl=k(2z = y)] and the other elements are all zeros. Or 0r/+1:1//(22 () is computed as follows.
0r.+1.D )
aklsi 2)
at 2)
= expl-k(etdd v @ (IITTL. - t())=
(14) and 01 are 8e
o( a(
where 1T denotes a row vector of length d whose elements are all one.
a 20) 8( )²
the Jacobian of & and respectively. Note that when and t are complex functions modeled by neural networks. The Jacobian matrix can be computed efficiently using automatic differentiation tools such as Pytorch and Tensorflow.
Critic Update The update of QQ-unccion parameters & in our approach follows the same update rule in DDPG 4.. However, in case the mapped action by the learned flow does not satisfy action constraints, we solve QP to project it into the feasible action space as the environment only accepts valid actions. Therefore, the action stored the replay buffer is either the output from the flow model or the projected action, depending whether flow mapped action is valid or not. We note that the probability of using the projected action during training is low as for all the tested instances as the normalizing flow had high accuracy. Even if the projection was required, the difference between the flow output and the projected action was small as we show empirically. We also highlight that for policy update, we use the flow mapped action without using any projection, which enables end-to-end training of the policy.
Other RL algorithms While our work focuses on DDPG, it can be extended to other RL algorithms such as SAC or PPO. This is possible because the normalizing flows enable the computation of log probabilities of actions, which is required during training in SAC or PPO. This showcases an additional advantage of using the normalizing flow model our work compared to other generative models.
7

---

Normalizing flow model Fo (4
Original policy (3(
O6x
dami
(b)
Figure 2: (a) Policy network; (b) The reversed gradient path of (in blue). Nodes denote variables and edges denote operations. Paths black are detached for B. The green block is a negative loss (assuming a minimization task).
4 Experiments
Environments We evaluate our proposed approach on four continuous control RL tasks in the MuloCo environment [1 and one resource allocation problem that has been used in previous works [8]
The agent required to navigate 2-DoF robot arm to reach a target. The agent's actions (a1, 02) I22 are 2D. The action space is subject nonlinear convex constraint
Reacher:
at + a3 2 0.05.
Half Cheetah: The task is make the cheetah forward by applying torque the joints. An action is a 6-dimensional vector (V1, o2....))). bounded by [-1, 1]. The constraint defined as Σ e < 20 where Wi is the velocity, which can be observed as part of the state.
Hopper and Walker2d: The task involves controlling robot to hop or walk forward by applying
torques to its hinges. An action is n- n-diimnnional vector (#1...., 2n). bounded by [-1, 1] where n is the number of hinges on the robot (3 for Hopper and for Walker2d). The constraint defined as Σ (w5e4 0} < 10 where w; is the angular velocity of the ith hinge, observed the state.
Bike Sharing System (BSS): The environment consists of m bikes and stations, each with capacity c. every RI step, the agent allocates m bikes among n stations based on previous al- location and demand while adhering to capacity constraints. The agent's 's action = (M1, 42 ..00g) must satisfy Σ a; = m and 0 S a; S c. We evaluate our approach on a specific scenario with 5, m = 150, and 35 as in [18] It poses significant challenge due to the large combinatorial action space of 1515.
Baselines We compare our approach FlowPG with the following two baselines.
DDPG+P: DDPG+Projection is extension of the vanilla DDPG [17), which introduces additional step to ensure feasible actions. Invalid actions are projected onto the feasible action space by solving an optimization problem.
NEWPO [8] This algorithm efficiently explores feasible action space by using the Frank Wolf method, and is also state-of-the-art approach for ACRL..
Learning the Normalizing Flows We employ different techniques to generate random samples from the valid action set in each environment. For the Reacher and Half Cheetah environments, we utilize HMC generate random samples. Specifically, we generate million random samples for training, which could take up to 5 minutes based on the constraint. HMC is more efficient than rejection sampling and a comparison is available in appendix[A] In the Reacher and Half Cheetah environments, all samples generated by HMC are valid actions, which shows the effecttiveness of HMC compared to traditional rejection sampling which only produces 3.93%, 4. 4.70% valid actions respectively. For the BSS environment, actions from the PSDD to train the flow. Additional information on compiling the linear constraints into PSDD and statistics about the PSDD are provided in the Appendix A
PSDD sample valid actions and obtain all valid
apply batch gradient descent to train the conditional flow with Adam optimizer and a batch size of 5000. For Reacher, Half Cheetah, Hopper and Walker2d environments, we train the model for 5K epochs. For BSS environment, we train for 20K epochs. Further details about the training the flow such learning rates, and neural network architecture of the model provided in the Appendix

---

Decision boundary
Feasible actions
Infeasible actions
Laconn: Space 1E
Action Space
Latert Space Ld1
Action Spacete)
Latam: Space 5X
Accin Sat
D1
D1
(a) WGAN
(b) VAE
(c) Normalizing Flow (Ours)
Figure 3: Mapping between a uniform distribution and action space of Reacher with constraint af + o3 S 0.05
The source code of our implementation is publicly available To evaluate the accuracy of the flow model, we sample 100K random points from the uniform distribution [-1. 1]. and then apply the flow. We measure the accuracy based on whether the output lies within the feasible action region. Our flow model was able to produce 99,98% 97.25% 87. 87.89%, 86.58% and 85.56% accuracy respectively for Reacher, Half Cheetah, Hopper, Walker2d and BSS environments. Further, compute the average recall over all uniformly sampled states to obtain the recall of our flow model using Equation 9 The achieved recall rates for our trained normalizing flow models were 97.85% 78.01 %, 81.61 %, 83.58% and 82.35% respectively for Reacher. Half Cheetah, Hopper, Walker2d and BSS environments. More details such as time available in Table in Appendix
Comparing with other generative models We conduct an ablation study comparing two other generative models VAE and WGAN (which is more stable than GAN) in Reacher domain. We first evaluated the accuracy by calculating the percentage of valid actions among 100k generated actions. The accuracy rates were as follows: Normalizing flows: 99.98%; WGAN 98%; VAE: 83%. We then considered the recall rate, Our flow model can achieve a recall rate of 97.85%. In contrast, recall rate cannot be computed in straightforward fashion in VAE and WGAN since determining the corresponding latent action for given valid action is not possible [2T] Nonetheless, we still can visualize the coverage of VAE and WGAN. In Figure 3 we can see that the feasible region is not fully covered in both VAE and WGAN models, while it is well covered in the normalizing flow model.
Results We present the empirical results of our approach and the baselines on three different environments show the effectiveness of our method in terms of low constraint violations and achieving fast training speeds. We report results using ten different random seeds. To make a fair comparison, we keep the architectures of our policy network (with the exception of the Tanh layer and the normalizing flow model) and the critic network identical to those used in the baseline approaches. For the detailed pseudo-code of our approach and the neural network architectures along with hyperparameters, please refer to the appendix.
Figure ()) shows the average return of all three approaches over the training steps. Our approach achieves comparable average return to NFWPO in the challenging Half Cheetah, Hopper and Walker2d environments. Additionally, our approach outperforms the other two approaches in terms of average return in the Reacher and BSS environments. We note that DDPG+P suffers from the zero gradient issue, leading to convergence
lower average return.
In Figure b)), we present the cumulative constraint violations before projection for all three ap- proaches over the training steps, with the y-axis represented in log-scale. In all environments, except for the more challenging Walker2d, our approach demonstrates the fewest cumulative constraint violations before projection. In the Reacher and Half Cheetah environments, it notably reduces the violations by an order of magnitude. Moreover, although our approach sometimes generates located near the feasible region, which can be inferred from
infeasible actions, they tend the average magnitude of constraint violations. To quantify the magnitude of constraint violations, we consider a constrained set C defined with m inequality constraints and n equality constraints:
{o|æ E Rd fz(z) < O. 0. (x) = 0. = 1,... m,j j = 1, I....,nl, .n]. We define the magnitude of con- straint violations as CV(x) = Σ i= max(fi(x). 0) + ビラー max(I[h,()) - cc. 0), where 0.1 represents the error margin for the equality constraint. Figure fe) illustrates the average magnitude of constraint violations over training steps, with the y-axis presented in lca-scale. Our approach exhibits the lowest average magnitude of constraint violations in all environments except Hopper and
9

---

FloWPG
NEWPO
DDPG+P
(a)
(0)
Constraint
Nagritude
(6)
100
100
10
10
100
100
6004
104
Half Chestath
10'
8000
2000
100
6004
3006
1500
10-1
10.4
2000
1300
8000
Wil Bo.oor2d
2:00
10'1
1700
200
-15033
110
1555
-12033
10
Dx30
Figure 4: Training curves for the Reacher, Half Cheetah, and BSS environments displayed in columns from left to right, showcasing the Average Return(f), Cumulative Constraint Violations({). Average Magnitude of Constraint Violations(L), and Time Elapsed(.)..
Walker2d. the Reacher, Half Cheetah, Hopper and Walker2d environments, the average magnitude of constraint violations is close zero, indicating that the invalid actions very close to the feasible region. In the BSS environment, our approach shows slightly higher average magnitude due to the integer property of the actions. The low number of cumulative constraint violations and lower average magnitude of constraint violations demonstrate the effectiveness of our learned flow model.
In Figure (gqure(d)) d)), we show the runtime of the training process. Our approach demonstrates significantly faster training time compared to NFWPO, with a speed improvement of 2~3 times. This speed advantage is attributed the fact that NFWPO requires computationally expensive QP solutions to determine the policy update direction within the feasible action space. In contrast, our approach utilizes the learned normalizing flow model generate valid actions, with the flow model parameters frozen during the policy update. Although DDPG+P shows a faster runtime, it comes at the expense of lower average return and higher constraint violations.
5 Conclusion
In this work, we present novel approach called FlowPG based on Normalizing Flows to address action constraints in RL. The architecture of FlowPG allows the policy network to generate actions within the feasible action region. Furthermore, our experimental results demonstrate that FlowPG effectively handles action constraints and outperforms the previous best method by significantly reducing the number of constraint violations while having faster training speeds.
10

---

Acknowledgement
This research/project is supported by the National Research Foundation Singapore and DSO National Laboratories under the Al Singapore Programme (AISG Award No: AISG2- RP-2020-017).
References
[1] Aaron D Ames. Human-inspired control of bipedal walking robots. JEEE Transactions on Automatic Control, pages 11 1115-1130 2014.
[2] Brandon Amos and J Zico Kolter. Optnet: Differentiable optimization layer in neural
networks. In International Conference on Machine Learning, pages 136-145, 2017.
[3] Michael Betancourt. A Conceptual Introduction to Hamiltonian Monte Carlo. arXiv preprint arXiv:/701.02434 2018.
[4] Abhinav Bhatia, Pradeep Varakantham, and Akshat Kumar. Resource constrained deep rein- forcement learning. In International Conference on Automated Planning and Scheduling, pages 610-620, 2019.
[5] Arthur Choi. The PyPSDD Package.
2018.
[6] Arthur Choi, Weijia Shi, Andy Shih, and Adnan Darwiche. Compiling neural networks into tractable boolean circuits. AAAI Spring Symposium on Verification Neural Networks, 2017.
[7] Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin Paduraru, and Yuval Tassa. Safe exploration continuous action spaces. arXiv preprint arXiv: 00757,, 2018.
[8] Adnan Darwiche. Sdd: A new canonical representation of propositional knowledge bases. In International Joint Conference Artificial Intelligence, pages 819-8266 2011.
[9] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference on Learning Representations, 2016.
[10] Marguerite Frank and Philip Wolfe. An algorithm for quadratic programming. Naval Research Logistics Quarterly, 199-110, 1956.
[11] Kurt Otto Friedrichs. The identity of weak and strong extensions of differential operators. Transactions of the American Mathematical Society, 55:132-15 151, 1944.
[12] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, pages 2052-2062, 2019.
[13] Leonard Jaillet and Josep M. Porta. Path Planning Under Kinematic Constraints by Rapidly Exploring Manifolds. IEEE Transactions on Robotics, 29(1):105-117, 2013.
[14] Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv: 1et.11422..,,... 2017.
[15] Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible 1x1 Convo- lutions. In Advances in Neural Information Processing Systems, 2018.
[16] Doga Kisa, Guy Van Den Broeck, Arthur Choi, and Adnan Darwiche. Probabilistic sentential decision diagrams. In Principles of Knowledge Representation and Reasoning, pages 558-567, 2014.
[17] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
[18] Jyun-Li Lin, Wei Hung, Shang- Hsuan Yang, Ping-Chun Hsieh, and Xi Liu. Escaping from zero gradient: Revisiting conttrronnt reinforcement learning via frank-wolfe policy optimization. In Uncertainty in Artificial Intelligence, pages 397-407, 2021.

---

[19] Jiajing Ling, Kushagra Chandak, and Akshat Kumar. Integrating knowledge compilation with reinforcement learning for routes. In International Conference on Automated Planning and Scheduling, pages 542-550, 2021.
[20] Jiajing Ling, Moritz Lukas Schuler, Akshat Kumar, and Pradeep Varakantham. Knowledge com- pilation for constrained combinatorial action spaces in reinforcement learning. International Conference Autonomous Agents and Multiagent Systems, pages 860-866. 2023.
[21] Tianci Liu and Jeffrey Regier. An Empirical Comparison of GANs and Normalizing Flows for Density Estimation. arXiv preprini arXiv:2006. 10/75, 2021.
[22] Bogdan Mazoure, Thang Doan, Audrey Durand, Joelle Pineau, and R Devon Hjelm. Leveraging exploration in off- off-policy algorithms via normalizing flows. In Conference on Robot Learning. pages 430 2020.
[23] Robert T. McGibbon. Pyhme: Hamiltonian Monte Carlo in Python. imagibbo/pyhhm/ 2023.
https://githu: com/
[24] Wannes Meert. PySDD httttpe//.trrr///hhhtooooe//a/aa 2017.
[25]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, pages 529-533, 2015.
[26] Tu-Hoa Pham, Giovanni De Magistris, and Ryuki Tachibana. Optlayer-practical constrained optimization for deep reinforcement learning in the real world. In International Conference on Robotics and Automation, pages 6236-6243, 2018.
[27] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538, 2015.
[28] Weijia Shi, Andy Shih, Adnan Darwiche, and Arthur Choi. On tractable representations of binary neural networks. In International Conference Principles of Knowledge Representation and Reasoning, pages 882-892, 2020.
[29] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference Machine Learning, pages 387-395, 2014.
[30] Chen Tessler, Daniel Mankowitz, and Shie Mannor. Reward constrained policy optimization. In International Conference on Learning Representations, 2019.
[31] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based control. In JEEE/RSJ International Conference Intelligent Robots and Systems, pages 0022-5033, 2012.
[32] Vassilios Tsounis, Mitja Alge, Joonho Lee, Farbod Farshidian, and Marco Hutter. DeepGait: Planning and Control of Quadrupedal Gaits Using Deep Reinforcement Learning. IEEE Robotics and Automation Letters, 5(2):3699-3706, 2020.
[33] Patrick Nadeem Ward, Ariella Smofsky, and Avishek Joey Bose. Improving exploration in soft-actor-critic with normalizing flows policies. arXiv preprint 1xXx::9906002771 2019.
[34] Christina Winkler, Daniel Worrall, Emiel Hoogeboom, and Max Welling. Learning likeli- hoods with conditional normalizing flows. arXiv preprint arXiv: 1912.0004, 2019.
[35] Jing Zhang, Chi Zhang, Wenjia Wang, and Bing- Yi Jing. Apac: Authorized probability- controlled actor-critic for offline reinforcement learning. arXiv preprint a1X1::2301.12130, 2023.
12

---

Appendix A
Sample Generation
HMC: We used Python implementation 23 of Hamiltonian Monte Carlo algorithm for sample generation of Reacher and Half-Cheetah environments. We set the starting point as the origin(OP) and used default hyperparameters such as = 0.2, decay- = 0.9.
PSDD: In the BSS environment, since there are capacity constraints, restricting each component of the action the range of [1, 35], we can represent each component of the action as 6-bit integers using Natural Binary Coded Decimal (NBCD). Specifically, we express ag. i = 1,..., 5 as a sum of its binary bits: a; ا j=1 26-3 x a². where a!...., af are Boolean variables representing each bit of a, By employing this encoding, we can convert the local constraint 1 S a; S 35 and the global constraint Σi= = 150 into Pseudo- Broude (PB) constraints by substituting each variable with its binary representation.
To compile these PB constraints, we utilize the method described in [6] to convert each constraint into a Sentential Decision Diagram (SDD). The SDDs are then conjoined using the package (24 to create a final SDD. The resulting SDD has node count of 733, indicating the number of decision nodes, and a model count of 23751, which is the total number of valid actions.
To obtain a Probabilistic Sentential Decision Diagram (PSDD), we parameterize the final SDD, The resulting PSDD has node count of 802 and size of 3138. Here, the node count refers to the number of decision nodes in the PSDD, while the size refers to the total number of decompositions.
Efficiency: To measure the efficiency of sample generation, we employ success rate metric, defined as the percentage of valid actions per 100 generated sample points. In both two domains, the HMC method achieves a success rate of 100%. For the rejection sampling, the success rates are 3.93% and 4.7% in the Reacher and Half Cheetah domains, respectively. Figure5Jsows the density of generated sample points within the feasible region. HMC method results in a significantly higher number of data points uniformly distributed across the feasible region. It indicates that HMC is more efficient in sample generation when compared to the rejection sampling method.
When action space constraints are expressed as (in)equalities (such as in the BSS environment), generating valid actions through either rejection sampling or HMC becomes challenging (e.g.. rejection/HMC sampling does not produce any action that satisfies all (in)equality constraints within practical time limit). The advantage of using PSDDs lies in their ability to represent a probability distribution over all valid actions, which implies any sampled action from PSDD is guaranteed to satisfy the constraint, Furthermore, PSDD enables fast sampling of actions with complexity linear in its size and can easily represent uniform distribution over the feasible action space (Section 3.2 in main paper).
Reacher (of of 2..05)
Halt Cheatah
HMC
Rejection Sampling
HMC
Rejection Sampling
0.7
41
14
Success Rate: 100, 00%
Sucowes Rate: 5.9%%
Sucowes Bate: 10. 00%
Sucowor Rate: 6.73%
Figure 5: Density map of generated valid actions using HMC and Rejection Sampling methods.
Appendix B Training RealNVP
RealNVP 9 was defined using coupling layers as shown in Figure We implemented A and t components as multi-layer perceptrons with 2 hidden layers of size 256 and ReLU activation functions. We used a uniform distribution with range 1), as the latent distribution. For the training purpose, we mollified it with Gaussian Noise of μ 0 and
0.01. We used Adam Optimizer [4] for learning the neural network parameters with the learning rate of 10-5, The trained Flow models were able to produce results mentioned in Table I Even when they produce infeasible
13

---

Pacsnagg if actors
35.0%
Pecomage af intosidle 8.210n
20.0%
25.0%
8.0%
20.0%
6.0%
10.0%
4.0%
10.0%
0.0%
0.0%
0.0%
0.0004
0.0007
1.0
nor
mom
1o narm
(a) Reacher
(b) Half Cheetah
(c) BSS
Figure 6: La distance from the infeasible actions to nearest feasible action.
Affine
Tart
Fial
Flid
suppi
ReiUU
Niddon 258
ReilU
Notden 255
Noden 256
Pisl
Saal
Figure 7: Affine Coupling Layers of RealNVP and k sub-component networks
Environment
Accuracy
Recall
Training Time (minutes)
Reacher 99.98%
97.85%
10
Half Cheetah
97.25%
78.01%
120
Hopper 87.89%
81.61% %
60
Walker2d
86.58%
83.58%
120
BSS
85.56%
82.35%
~
60
Table 1: Accuracy, recall of trained flow models and their training time
actions, they tend to be positioned closer to feasible regions. To measure that we filter infeasible actions generated by the trained Flow model and measure the distance (I22 norm) to the nearest feasible action. (ggure/]showe the histogram of the measured L2 norm. This shows that even when our model produces infeasible actions, they are not far away from the feasible region. For example in the Bike sharing environment(BSS) environment, most of the values are between 0 and 1, i.e. the error value no more than single bike.
Appendix C Learning the RL-Model
defined Actor and Critic networks as hidden layer neural networks with 400 and 300 units respectively as in Figure For the activation function, we used the ReL.U activation function. In the final layer of the actor-network, the tanh activation function was used map output the [-1, range before passing to the RealNVP. Adam [14] optimizer used to train both Actor and Critic
14

---

Adtr
CrRc
Pedkyp
Aftia Coupting Layer
oupporic
Reil
Nobtin 302
felilon
huttonn 402
PacLU
Nettnn 400
(a)
(b)
Figure 8: (a) Actor Network and (b) Critic Network
networks with learning rate of 10 and 10 respectively. For the soft target update, used T 0.001. We used Mani-batches of size 64 to train both Actor and Critic networks. We used Gaussian noise with
and 0.1 action noise for exploration. The replay buffer size was 10 The complete pseudo code of the FlowPG can be found in the Algorithm]
Algorithm FlowPG Algorithm
1: Input: Trained fgz
2: Randomly initialize critic network Q(s, a; σ) and actor (s)
3: Initialize target network Q' and μí with weights d' + o. 0' + 0
4: Initialize replay buffer B
5: for episode=1..... do
6: Initialize the random noise generator A for action exploration
7: Reset the environment and retrieve initial state
8: for 1=....,7 do
9:
Select action à = (s:) M₁ based on current policy and exploration noise Apply flow and get the environment action a, (āt, if 0, is invalid then
11:
12:
a: - @P_Soluer(a;: St, C(s:))
13:
end if
Execute action o; and observe reward r; and next state 81411
15:
Store transition (st, a, Ti, $t+1) in B
16:
Sample a random minibatch of transitions (5t, ai Ti, 8(41) from
17:
V1 = + (1 (Her((s+1)) (
18:
Update critic by minimizing the loss: L= N Σ, (21 Q(si, a;; o))²
19:
Update actor policy using the sampled policy gradient:
20:
VoJ(H N E, QaQ(s,, 0; Φ) Vafe Vo) a
21:
Update target networks:
22:
Ø' - TØ 0 70 + (1 0'
23:
24:
end for
25:
end for
15


Reviews:
[{"review_id": "pwf40pQAto", "reviewer": "Reviewer_NbGB", "summary": "The paper introduces a novel action-constrained reinforcement learning (ACRL) algorithm called FlowPG, which utilizes a normalizing flow model to generate actions within the feasible action region. Experimental results demonstrate that FlowPG effectively handles action constraints and outperforms two existing ACRL algorithms by reducing the number of constraint violations.", "questions": "Q1. The motivation behind the use of Hamiltonian Monte Carlo (HMC) and probabilistic sentential decision diagrams (PSDD) is not sufficiently clear. Additionally, an ablation study is necessary to validate the importance of these proposed techniques.\n\nQ2. The paper claims that the proposed method achieves faster training speeds compared to the baselines. However, Figure 4(a) shows no clear advantages in terms of convergence speed for FlowPG compared to the two baselines. This discrepancy needs to be addressed and clarified.\n\nQ3. The paper would benefit from including more comparisons between the proposed method and recent ACRL algorithms. By providing such comparisons, the authors can further highlight the strengths and weaknesses of their approach and provide a more comprehensive evaluation.\n", "limitations": "see Questions", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written and provides good motivation.", "weaknesses": "The paper lacks the ablation study to validate importance of the proposed HMC and PSDD. The advantages of training speeds are not obvious.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Motivation behind HMC and PSDD. Ablation study to justify the reason of using them \n\nPlease see the common response. (Sec. 2)\n\n2. Convergence speed of FlowPG \n\nFor half cheetah and reacher, we shall highlight that our approach has a similar training curve as NFWPO. However, the key difference is that each training step is much faster in our approach than NFWPO as shown in Figure 4 (last column). Therefore, in terms of runtime, our approach converges much faster than NFWPO. For BSS domain, our approach has better training curve than NFWPO (in terms of # of training steps), and is also significantly faster as shown in Figure 4 (first and last column).\n\n3. Comparison with recent ACRL algorithms \n\nPlease see the common response. (Sec. 4)\n\nCitations for all our responses:\n\n[1] Achiam, Joshua, et al. \"Constrained policy optimization.\" International conference on machine learning. PMLR, 2017.\n\n[2] Tessler, Chen, Daniel J. Mankowitz, and Shie Mannor. \"Reward constrained policy optimization.\" arXiv preprint arXiv:1805.11074 (2018).\n\n[3] Liu, Tianci, and Jeffrey Regier. \"An Empirical Comparison of GANs and Normalizing Flows for Density Estimation.\" arXiv preprint arXiv:2006.10175 (2020).\n\n[4] Lin, Jyun-Li, et al. \"Escaping from zero gradient: Revisiting action-constrained reinforcement learning via Frank-Wolfe policy optimization.\" Uncertainty in Artificial Intelligence. PMLR, 2021.\n"}, {"review_id": "I2lygDSmKV", "reviewer": "Reviewer_RnL4", "summary": "This paper provide a new method for ACRL, which incorporate normalized flow methods to alleviate the action violation problem. It achieves better results on MuJuCo compared with other methods.", "questions": "I am not an expert in ACRL so I wonder if there are other ACRL benchmarks? Only 3 environments seem not enough.", "limitations": "No obvious negative societal impacts.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Introduce normalized flow into action control -- which maps the original hard-to-control action space into another easy-to-control space.\n- Design a HMC-PSDD framework to efficiently train the flow model.\n- Find an appropriate prior distribution for the flow model.", "weaknesses": "Main issue:\n- An ablation study for PSDD may be necessary -- as we don't know the quality of generated valid actions. For example, a possible baseline can be: interact with the environments many times to get the possible valid actions.\n\n\nMinor issue:\n- DDPG is relatively outdated. More recent SoTA methods should be added into the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Ablation study for PSDD and HMC\n\nPlease see the common response.  (Sec. 2)\n\n2. Comparison with recent ACRL algorithms + more domains\n\nPlease see the common response.  (Sec. 3 and 4)\n"}, {"review_id": "t0KXRDNvIJ", "reviewer": "Reviewer_JVCX", "summary": "This paper solves the problem of action-constraint reinforcement learning. The author utilizes the Flow model to learn a projection from the action to the latent variable, and then integrate DDPG to construct the FlowPG framework. Empirically FlowPG outperforms its competitors in both fewer constraint violations and faster in elapsed time.", "questions": "1. There are some confusions in generating valid action space, even with HMC and PSDD, could it be assured that the $\\tilde{a}$ could be transformed to a valid action after the flow model projection in Figure 2(a). In addition, as we want to maximize the cumulative reward in RL problem, should the input of the flow model be ($\\tilde{a}, s, s'$) where $s'$ is the next state. It is also noted when $\\tilde{a}$ becomes a valid action $a$, the next state $s'$ may also changes\n\n2. The author claims that flow model is more efficient than VAE and GAN for data generation. Could the author provide some ablation study on this issue, especially for generating valid action samples.\n\n3. The experiments seem not convincing enough. The authors only considers Half-Cheetah and Bike Sharing System. Actually Half-Cheetah belongs to D4RL tasks, and there are many other tasks inside (some could be constraint RL tasks), it is suggested to consider more dataset for comparisons. \n\n4. In addition, the MCMC style methods are often time-consuming, how is the entire training time, compared with other competitors?", "limitations": "see weakness and questions", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The manuscript is clearly structured and well presented, and it is easy to read. \n2. It seems interesting to apply the generative model in constraints optimization problem. Introducing the flow model is novel in action-constraint scenarios, and it effectively avoids solving a QP problem after the policy network.\n3. The authors utilizes HMC and PSDD in sampling from the valid actions set. ", "weaknesses": "The action-constraint problems is actually somehow similar with constraint-RL problem, and there are some important works such as [1,2] and other related works. It is suggested that the authors should provide a discussion about whether other constraint RL methods could be applied in the action-constraint scenarios, and provide some experimental result if possible.\n\n[1] Constrained Policy Optimization https://arxiv.org/pdf/1705.10528.pdf\n\n[2] Safety-Constrained Reinforcement Learning for MDPs. https://arxiv.org/abs/1510.05880\n\nAnother issue is that the action-constraint RL problem could be seen as offline RL problem (invalid actions space could be OOD actions). There are various offline RL methods appearing [3,4] in recent years ([5] also applies the flow model), and the authors are suggested to consider the offline RL methods in this manuscript. \n\n[3] Off-Policy Deep Reinforcement Learning without Exploration. https://arxiv.org/abs/1812.02900\n\n[4] Stabilizing off- policy q-learning via bootstrapping error reduction. https://arxiv.org/abs/1906.00949\n\n[5] APAC: Authorized Probability-controlled Actor-Critic For Offline Reinforcement Learning. https://arxiv.org/abs/2301.12130", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Other constrained RL methods\n\nIn our paper, we consider the scenario where constraints are imposed on actions at each RL step, and these constraints have closed forms. Unlike the standard constrained MDP, we do not define cost functions for individual state-action pairs. Therefore, the direct application of well-known techniques like CPO[1] and Lagrangian Relaxation[2] becomes challenging for solving action-constrained RL. Additionally, these approaches do not provide a guarantee of constraint satisfaction. We shall clarify this point in the revision.\n\n2. Connection to Offline RL methods\n\nThanks for pointing to this line of work, we shall certainly discuss it in the revision. We do highlight some key differences from off-policy RL methods:\n\n– Off policy RL methods require a dataset for training from an expert or random policies. This is not required in our method as action constraints are known, and we use HMC/PSDD to sample feasible actions for different states.\n\n– Notice that collecting off-policy data from a random policy for offline RL is generally trivial. However, this is not trivial for our action constrained setting. Even a random policy must select actions uniformly from *only* the feasible action space, which itself is a key component we are trying to address.\n\n\n3. Input to flow? Is output of the flow guaranteed to be valid?\n\nWe shall clarify that the actions generated by the trained flow model are not guaranteed to be valid actions. However, a well-trained flow model exhibiting a high accuracy rate, such as 99.98% in the Reacher domain, is highly likely to generate valid actions. Even in instances where an invalid action is generated, our experiments show that these invalid actions are generally closer to the feasible region when compared to actions generated by the DDPG+P and NFWPO approaches. We also note that the projection of invalid actions into the feasible region is required in all approaches since the environment only accepts valid actions. \n\nThe input to the normalizing flows model are $s$ and $\\tilde{a}$, where $\\tilde{a}$ belongs to the latent distribution of the flow model. This $\\tilde{a}$ is transformed into the environment action $a$ by the flow model, which is then fed into the RL simulator (after projection, if required). The dynamics of the environment (i.e., the next state) only depends on $s$ and $a$. The input to the flow cannot include the next state s’ as the policy output $\\tilde{a}$ cannot be executed by the environment directly before mapping by the flow model. Thus s’ is not available as an input for the flow model.\n\nReward maximization is primarily handled by the policy gradient part connected with the flow model as shown in Figure 2a in the paper.\n\n4. Other generative models such as VAE and GAN \n\nPlease see the common response. (Sec. 1)\n\n5.  More domains in D4RL tasks \n\nPlease see the common response. (Sec. 3)\n\n6. Entire training time when considering HMC data generation\n\nIn our approach, generating valid actions and training a normalizing flows model are conducted prior to RL training. Specifically, the generation of data using HMC takes less than 5 minutes to produce 1 million samples for all the domains.This rapid data generation is attributed to the efficiency of HMC. An ablation study comparing the efficiency of HMC and traditional rejection sampling has been included in the general response. \n\nThe training time for the flow model for different domains is shown in Table 2 of rebuttal PDF. We do highlight that in our approach, once the flow is trained, retraining is not required if some problem parameters change such as the initial state, reward function etc. However, for approaches such as NFWPO, every time we run the approach, projection steps will be required, thus leading to higher runtime (as shown in figure 4, last column in main paper).\n"}, {"review_id": "l6mLQygfDk", "reviewer": "Reviewer_Vb4Z", "summary": "Handling constraints in reinforcement learning is a fundamental problem that has application in areas such as robotics and resource allocation. A common solution is to incorporate a projection step to compute feasible actions, which involves a computationally expensive optimization solver in the loop. This can be prohibitively slow, especially when constraints are non-convex. When used as part of the policy with a differentiable solver, it can led to the zero gradient problem if the policy output is far outside the feasible space. To circumvent the need for an optimization solver, the authors propose to learn an action mapping which respects the constraints with high probability by design. Specifically, they train a normalizing flow offline on samples from the distribution of feasible actions. These samples are generated via Hamiltonian Monte-Carlo for continuous action spaces and probabilistic sentential decision diagrams for discrete action spaces. During RL, the policy then outputs a latent action, which is mapped to the space of feasible actions via the normalizing flow. The weights of the flow are frozen, but the gradients are still propagated through the network to train the policy. They evaluate their approach, FlowPG, on two continuous control robotics tasks and a discrete resource allocation problem. FlowPG achieves the highest average return across ten random seeds for all problems and has the lowest number of constraint violations (prior to a projection step). When it does violate a constraint, the magnitude of the violation is lower, meaning it is closer to the feasible action space. And it achieves these benefits with reduced wall clock time compared to the best baseline.", "questions": "- How do you think this approach would scale to more complex, state-dependent constraints? Would it be too difficult to generate samples which cover the state space sufficiently? And if so, would an iterative approach which refines the flow on states encountered under the current policy be feasible?\n- How are the discrete actions in the BSS environment handled? If the output of the flow is rounded, does this lead to more constraint violations?\n- Is the form of the normalizing flow leveraged to more efficiently compute gradients while training? Or are we using standard auto-differentiation?\n- What is the rationale for using a normalizing flow over other generative models in the deterministic policy case? \n- How do you think this approach would work for an on-policy algorithm, such as PPO, which requires the log-likelihood? \n- How was the \"hard wall\" for handling constraints implemented with HMC, which needs a differentiable log-likelihood? Were they just barrier functions?", "limitations": "The authors discuss how their approach can still lead to constraint violations, requiring the use of an optimization solver in this case. However, they show that the probability of constraint violation is significantly reduced. A possible limitation not really discussed is scalability due to the need for sample generation to train the flow. It may be hard to scale this approach to higher-dimensional action spaces and constraints which heavily depend on the state. The Half Cheetah task definitely shows it can work, so at the very least, this is a good preliminary study to illustrate the potential of the approach.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Addressing action constraints in reinforcement learning is an important problem and critical for many real-world robotics and decision making tasks.\n- FlowPG is a novel solution to this problem and effective in improving performance while reducing constraint violations and wall clock time, all while being fairly straightforward to implement.\n- The finding that a uniform prior reduces constraint violations when used in conjunction with RL is a useful insight. The bounded support of the uniform distribution also works nicely with the fact that policy gradient algorithms often work well when the output is passed through a squashing function to respect box constraints on action limits.\n- Constraint violations which do occur, however infrequently, can still be remedied by solving an optimization problem. Importantly, this does not need to be done when performing the policy updates, just during rollouts, which makes training more efficient.\n- The paper is well organized and clearly written. It does a good job explaining the novelty and results and provides enough information to support its claims. In particular, I liked the visualizations in Figure 3 of the learned action space for the Reacher task.", "weaknesses": "- The tasks considered in the paper are fairly standard benchmarks and showcase the effectiveness of FlowPG. However, the constraints in each task are relatively simple, with only Half Cheetah having constraints which depend on a portion of state. This makes it difficult to gauge how this approach will scale to more challenging, state-dependent constraints. Tasks with more complex constraints would significantly strengthen the paper.\n- It seems difficult to scale the sample generation procedure to higher-dimensional action spaces. And for state-dependent constraints, it seems challenging to span the relevant portions of the state space which will be visited by the intermediate and final policies used during learning. Again, more challenging tasks would help prove the concept. It may be that an iterative approach which refines the flow on relevant portions of the state space will be necessary. \n- There does not appear to be any discussion of how the discrete actions in the BSS environment are handled. My guess is that the flow is trained on the integer values, but the output will still be continuous. We can then just round the output, but this could result in constraint violations. A better discussion of how this is handled in the main paper would help.\n- There is a lengthly discussion of gradient computation for the normalizing flow, but this is usually just handled by the deep learning framework. It seems unnecessary to get into these details unless the form of these gradients is leveraged to improve the speed of training. If auto-differentiation is still used as normal, then this feels a bit like filler and could be replaced with more relevant details about training, the tasks, or results.\n- The motivation for using a normalizing flow, rather than other generative models, seems missing in the paper. The authors do mention something, but it felt a bit hand-wavy. If we were optimizing a stochastic policy with an on-policy algorithm, such as PPO, having a tractable likelihood would be really important. This would be a great motivation for using a normalizing flow. But since we are using a deterministic policy and training with DDPG, it seems more arbitrary. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. How to generate samples to cover the state space sufficiently? Refine flow during training?\n\nIndeed, this is a good point. We note that our approach models the space of feasible actions for different states. Therefore, even if the feasible state space is large, but the relationship between different states and feasible action space is roughly similar, our approach can work well. \n\nIf the space of feasible actions varies significantly for different states, then training flow in a static manner would be challenging. Refining the flow during the training so that the flow can focus on reachable part of the state space, similar to training of the Q function using replay buffer, is indeed a good idea. Our overall approach can be easily modified to perform this online flow training. We shall explore it actively for larger problems with more complex state-action constraints. In our current settings, we were able to get good accuracy and recall without this online flow training as noted in table 1 in rebuttal pdf. \n\n\n2. How to handle discrete action? \n\nThe simulator for BSS is equipped with a rounding procedure that tries to apply a heuristic method to extract an integer solution from a given action. In our experience, this method was successful for most cases. When this rounding method did not give an integer solution satisfying all the constraints, we solved an integer program to perform the projection. This approach is applied consistently to all the baselines.\n\n3. Is the form of the normalizing flow leveraged to more efficiently compute gradients while training? Or are we using standard auto-differentiation?\n\nWe use Auto-differentiation. However, we wanted to explicitly highlight how the policy update in Eq 11 (main paper) utilizes the gradient from the flow model. In particular, in Eq 13, 14, highlight the policy gradient part coming from the flow. \n\n\n4. Rationale for using normalizing flows \n\nPlease see the common response. (Sec. 1)\n\n5. PPO+Flow \n\nAssuming the trained flow model achieves a high accuracy rate and recall rate, the outputs of the flow model are likely to be valid actions. In this case, the PPO approach can be effectively employed. This is due to the fact that the log probabilities of actions can be computed using the flow and that the optimization of policy is conducted within the feasible region. \n\nHowever, in cases where projection is required and the projected action is mapped using the inverse transformation of the flow to an area beyond the domain of the latent variable (e.g., outside $[-1, 1]$ for uniform prior), then we need to compute the closest point in the latent space and use this as a proxy to compute the log-probability of the action. \n\n\n6. How was the \"hard wall\" for handling constraints implemented with HMC, which needs a differentiable log-likelihood? Were they just barrier functions?\n\nWe initially tried with differentiable barrier functions. However, we found that better or comparable results could be achieved through a non-differentiable function with sharp barriers. Specifically, we employed a piecewise function that takes on a value of $-\\infty$ if there is constraint violation, and $0$ otherwise. We set the gradient to be zero everywhere. \nFor further details and precise implementation, you may refer to the `sample_generation/hmc.py` file in the supplementary materials."}]

Author rebuttal:
We thank all the reviewers for their thoughtful feedback and suggestions. We would like to address a few common questions as follows.

1. Rationale for using normalizing flows

Our reasons are:

High Accuracy: We conducted an ablation study comparing different generative models such as VAE and WGAN (which is more stable than GAN) in the Reacher domain with the constraint $a_1^2 + a_2^2 \leq 0.05$. We evaluated the accuracy by calculating the percentage of valid actions among 100k generated actions via such models respectively. The accuracy of the various generative models was as follows: Normalizing Flow: 99.98%; WGAN: 98%; VAE: 83%.

Recall Through Invertible Transformations: Normalizing flows provide the ability to measure recall due to their invertible bijective transformations. The recall rate (we also call it “coverage”) indicates the fraction of valid actions that can be generated from the latent space. Given a state $s$, it can be computed as follows $recall(s) = \frac{\sum_{a\in\tilde{\mathcal{C}}(s)} \mathbb{I}\_{\textbf{dom}f_{\psi}}\big(f^{-1}\_{\psi}(a,s) \big) }{|\tilde{\mathcal{C}}(s)|}$

Where $\tilde{\mathcal{C}}(s)$ is the set of valid actions that are uniformly distributed in feasible region. $f^{-1}\_{\psi}$ is the inverse transformation function of the normalizing flows model $f\_\psi$. We compute the average recall over all uniformly sampled states to obtain the recall of our flow model. The achieved recall rates for our trained normalizing flow model are as follows: 97.85% for the Reacher, 78.01% for the Half Cheetah, and 82.35% for the BSS environment. In contrast, recall rate cannot be computed in a straightforward fashion in VAE and WGAN since determining the corresponding latent action for a given valid action is not possible [3].  Nonetheless, we still can visualize the coverage of VAE and WGAN in Reacher. In Figure 3 in the rebuttal PDF, we can see that the feasible region is not fully covered in both VAE and WGAN models.
 
A higher accuracy rate indicates fewer projection operations. A higher recall rate indicates that the agent is able to explore a nearly complete feasible region. Table 1 in the rebuttal PDF summarizes the accuracy and recall


2. Ablation study on HMC and PSDD

To measure the efficiency of sample generation , we employ a success rate metric, defined as the percentage of valid actions per 100 generated sample points. In both two domains, the HMC method achieves a success rate of 100%. For the rejection sampling, the success rates are 3.93% and 4.7% in the Reacher and Half Cheetah domains, respectively. Figure 1 in the rebuttal PDF shows the density of generated sample points within the feasible region. HMC method results in a significantly higher number of data points uniformly distributed across the feasible region. It indicates that HMC is more efficient in sample generation when compared to the rejection sampling method. 

When action space constraints are expressed as (in)equalities  (such as in the BSS environment), generating valid actions through either rejection sampling or HMC becomes challenging (e.g., rejection/HMC sampling does not produce any action that satisfies all (in)equality constraints within a practical time limit). The advantage of using PSDDs lies in their ability to represent a probability distribution over all valid actions, which implies any sampled action from PSDD is guaranteed to satisfy the constraint. Furthermore, PSDD enables fast sampling of actions with complexity linear in its size and can easily represent uniform distribution over the feasible action space (section 3.2 in main paper). 

3. Other action-constrained RL domains 

We have evaluated our proposed framework in two more domains Hopper and Walker, which belong to the Gym-MuJoCo continuous control task. In Hopper domain, we consider the state-dependent constraint $\sum\_{i=1}^3 \text{max}(w\_i a\_i, 0) \leq 10$ , where $w\_i$ is the state feature. This is similar to the constraint in Half Cheetah.Similarly, we consider constraint $\sum\_{i=1}^6 \text{max}(w\_i a\_i, 0)\leq 10$ in Walker domain. All experimental settings remain the same as in the paper.

Figure 2 in the rebuttal PDF shows different curves during training in Hopper and Walker. In Hopper domain, when comparing to DDPG+P, our approach has less constraint violations before the projection operation, and achieves comparable results in terms of average return, magnitude of constraint violation, and running speed. In contrast to the NFWPO approach, our method performs better across all metrics, except for minor differences in the magnitude of constraint violation. In Walker domain, our approach outperforms DDPG+P across all metrics except running time. When compared with NFWPO, while our approach has slightly more cumulative constraint violations, the achieved magnitude of constraint violation and average return remain comparable. Significantly, our approach has much faster runtime compared to NFWPO.

To summarize, in the two new domains featuring state-dependent constraints, our approach consistently achieves a comparable or better average returns as other baselines. Moreover, our approach also demonstrates significantly reduced constraint violations, faster running time in comparison to other baselines.

4. Comparison with recent ACRL algorithms

In  constrained RL, most works focus on tackling cumulative constraints such as $\mathbb{E}\_{\pi}[\sum_{t=0}^\infty C(s\_t,a\_t)] \leq c$. There exist a limited number of works that specifically target ACRL where constraints are expressed as closed-form conditions on actions. Among the existing solutions for ACRL, representative approaches include DDPG+Projection, SAC+Projection, DDPG+OptLayer, and NFWPO. Among these, NFWPO has been demonstrated to have better performance in terms of return and constraint violations before projection [4]. This is why we have selected NFWPO as our primary baseline for comparison.  



Meta-review: The reviews and discussion offered enough positive support, with some concerns being good topics for future work. We encourage the authors to use the feedback to revise and strengthen the camera-ready.
Decision: Accept (poster)