SlowGuess/NeurIPS_2023_OCR huggingface dataset:
PSDD: In the BSS environment, since there are capacity constraints, restricting each component of the action to the range of [1, 35], we can represent each component of the action as 6-bit integers using Natural Binary Coded Decimal (NBCD). Specifically, we express  $a_i$ ,  $i = 1, \dots, 5$  as a sum of its binary bits:  $a_i = \sum_{j=1}^{6} 2^{6-j} \times a_i^j$ , where  $a_i^1, \ldots, a_i^6$  are Boolean variables representing each bit of  $a_i$ . By employing this encoding, we can convert the local constraint  $1 \leq a_i \leq 35$  and the global constraint  $\sum_{i=1}^{5} a_i = 150$  into Pseudo-Boolean (PB) constraints by substituting each variable with its binary representation.
To compile these PB constraints, we utilize the method described in [6] to convert each constraint into a Sentential Decision Diagram (SDD). The SDDs are then conjoined using the package [24] to create a final SDD. The resulting SDD has a node count of 733, indicating the number of decision nodes, and a model count of 23751, which is the total number of valid actions.
To obtain a Probabilistic Sentential Decision Diagram (PSDD), we parameterize the final SDD. The resulting PSDD has a node count of 802 and a size of 3138. Here, the node count refers to the number of decision nodes in the PSDD, while the size refers to the total number of decompositions.
Efficiency: To measure the efficiency of sample generation, we employ a success rate metric, defined as the percentage of valid actions per 100 generated sample points. In both two domains, the HMC method achieves a success rate of  $100\%$ . For the rejection sampling, the success rates are  $3.93\%$  and  $4.7\%$  in the Reacher and Half Cheetah domains, respectively. Figure 5 shows the density of generated sample points within the feasible region. HMC method results in a significantly higher number of data points uniformly distributed across the feasible region. It indicates that HMC is more efficient in sample generation when compared to the rejection sampling method.
When action space constraints are expressed as (in)equalities (such as in the BSS environment), generating valid actions through either rejection sampling or HMC becomes challenging (e.g., rejection/HMC sampling does not produce any action that satisfies all (in)equality constraints within a practical time limit). The advantage of using PSDDs lies in their ability to represent a probability distribution over all valid actions, which implies any sampled action from PSDD is guaranteed to satisfy the constraint. Furthermore, PSDD enables fast sampling of actions with complexity linear in its size and can easily represent uniform distribution over the feasible action space (Section 3.2 in main paper).
Figure 5: Density map of generated valid actions using HMC and Rejection Sampling methods.
Appendix B Training RealNVP
RealNVP [9] was defined using 6 coupling layers as shown in Figure 7. We implemented  $k$  and  $t$  components as multi-layer perceptrons with 2 hidden layers of size 256 and ReLU activation functions. We used a uniform distribution with range  $[-1,1]^D$  as the latent distribution. For the training purpose, we mollified it with Gaussian Noise of  $\mu = 0$  and  $\sigma = 0.01$ . We used Adam Optimizer [14] for learning the neural network parameters with the learning rate of  $10^{-5}$ . The trained Flow models were able to produce results mentioned in Table 1. Even when they produce infeasible
(a) Reacher
(b) Half Cheetah
(c) BSS
Figure 6:  $L_{2}$  distance from the infeasible actions to nearest feasible action.
Figure 7: Affine Coupling Layers of RealNVP and  $t, k$  sub-component networks
Environment	Accuracy	Recall	Training Time (minutes)
Reacher	99.98%	97.85%	~ 10
Half Cheetah	97.25%	78.01%	~ 120
Hopper	87.89%	81.61%	~ 60
Walker2d	86.58%	83.58%	~ 120
BSS	85.56%	82.35%	~ 60
Table 1: Accuracy, recall of trained flow models and their training time
actions, they tend to be positioned closer to feasible regions. To measure that we filter infeasible actions generated by the trained Flow model and measure the distance  $(L_{2}$  norm) to the nearest feasible action. Figure 6 shows the histogram of the measured  $L_{2}$  norm. This shows that even when our model produces infeasible actions, they are not far away from the feasible region. For example in the Bike sharing environment(BSS) environment, most of the values are between 0 and 1, i.e. the error value is no more than a single bike.
Appendix C Learning the RL-Model
We defined Actor and Critic networks as 2 hidden layer neural networks with 400 and 300 units respectively as in Figure 8. For the activation function, we used the ReLU activation function. In the final layer of the actor-network, the tanh activation function was used to map output to the  $[-1,1]$  range before passing it to the RealNVP. Adam [14] optimizer was used to train both Actor and Critic
(a)
Figure 8: (a) Actor Network and (b) Critic Network
networks with a learning rate of  $10^{-4}$  and  $10^{-3}$  respectively. For the soft target update, we used  $\tau = 0.001$ . We used mini-batches of size 64 to train both Actor and Critic networks. We used Gaussian noise with  $\mu = 0$  and  $\sigma = 0.1$  as action noise for exploration. The replay buffer size was  $10^{6}$ . The complete pseudo code of the FlowPG can be found in the Algorithm 1
Algorithm 1 FlowPG Algorithm
1: Input: Trained  $f_{\psi}$    
2: Randomly initialize critic network  $Q(s,a;\phi)$  and actor  $\mu_{\theta}(s)$    
3: Initialize target network  $Q^{\prime}$  and  $\mu^\prime$  with weights  $\phi^{\prime}\gets \phi ,\theta^{\prime}\gets \theta$    
4: Initialize replay buffer  $\mathcal{B}$    
5: for episode  $= 1,\dots ,M$  do   
6: Initialize the random noise generator  $\mathcal{N}$  for action exploration   
7: Reset the environment and retrieve initial state  $s_1$    
8: for t=1,...,T do   
9: Select action  $\tilde{a}_t = \mu_\theta (s_t) + \mathcal{N}_t$  based on current policy and exploration noise   
10: Apply flow and get the environment action  $a_{t} = f_{\psi}(\tilde{a}_{t},s_{t})$    
11: if  $a_{t}$  is invalid then   
12:  $a_{t}\gets QP\_ Solver(a_{t},s_{t},\mathcal{C}(s_{t}))$    
13: end if   
14: Execute action  $a_{t}$  and observe reward  $r_t$  and next state  $s_{t + 1}$    
15: Store transition  $(s_t,a_t,r_t,s_{t + 1})$  in  $\mathcal{B}$    
16: Sample a random minibatch of  $N$  transitions  $(s_i,a_i,r_i,s_{i + 1})$  from  $\mathcal{B}$    
17:  $y_{i} = r_{i} + \gamma Q^{\prime}(s_{i + 1},f_{\psi}(\mu_{\theta^{\prime}}(s_{i + 1}),s_{i + 1});\phi^{\prime})$    
18: Update critic by minimizing the loss:  $\mathcal{L} = \frac{1}{N}\sum_{i}(y_{i} - Q(s_{i},a_{i};\phi))^{2}$    
19: Update actor policy using the sampled policy gradient:   
20:  $\nabla_{\theta}J(\mu_{\theta}) = \frac{1}{N}\sum_{i}\nabla_{a}Q(s_{i},a;\phi)\nabla_{\tilde{a}}f_{\psi}(\tilde{a},s_{i})\nabla_{\theta}\mu_{\theta}(s_{i})|_{\tilde{a} = \mu_{\theta}(s_{i}),a = f_{\psi}(\tilde{a},s_{i})}$    
21: Update target networks:   
22:  $\phi^{\prime}\gets \tau \phi +(1 - \tau)\phi^{\prime}$    
23:  $\theta^{\prime}\gets \tau \theta +(1 - \tau)\theta^{\prime}$    
24: end for   
25: end for