SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We thank Günter Klambauer for useful discussions and comments. The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. We thank the projects AI-MOTION (LIT-2018-6-YOU-212), DeepFlood (LIT-2019-8-YOU-213), Medical Cognitive Computing Center (MC3), INCONTROL-RL (FFG-881064), PRIMAL (FFG-873979), S3AI (FFG-872172), DL for GranularFlow (FFG-871302), EPILEPSIA (FFG-892171), AIRI FG 9-N (FWF-36284, FWF-36235), AI4GreenHeatingGrids(FFG-899943), INTEGRATE (FFG-892418), ELISE (H2020-ICT-2019-3 ID: 951847), Stars4Waters (HORIZON-CL6-2021-CLIMATE-01-01). We thank Audi.JKU Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), FILL Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, GLS (Univ. Waterloo) Software Competence Center Hagenberg GmbH,TÜV Austria, Frauscher Sensonic, TRUMPF and the NVIDIA Corporation.
Nicholas C Wormald. Differential equations for random processes and random graphs. The annals of applied probability, pages 1217-1235, 1995.
Dian Wu, Lei Wang, and Pan Zhang. Solving statistical mechanics using variational autoregressive networks. Phys. Rev. Lett., 122:080602, Feb 2019. doi: 10.1103/PhysRevLett.122.080602. URL https://link.aps.org/doi/10.1103/PhysRevLett.122.080602.
Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=tW4QEInpni.
Ke Xu, Frédéric Boussemart, Fred Hemery, and Christophe Lecoutre. A simple model to generate hard satisfiable instances. In Leslie Pack Kaelbling and Alessandro Saffiotti, editors, *IJCAI-05*, Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence, Edinburgh, Scotland, UK, July 30 - August 5, 2005, pages 337-342. Professional Book Center, 2005. URL http://ijcai.org/Proceedings/05/Papers/0989.pdf.
Gal Yehuda, Moshe Gabel, and Assaf Schuster. It's not what machines can learn, it's what we cannot teach. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 10831-10841. PMLR, 2020. URL http://proceedings.mlr.press/v119/ yehuda20a.html.
Dinghuai Zhang, Hanjun Dai, Nikolay Malkin, Aaron C. Courville, Yoshua Bengio, and Ling Pan. Let the flows tell: Solving graph combinatorial optimization problems with gflownets. CoRR, abs/2305.17010, 2023. doi: 10.48550/arXiv.2305.17010. URL https://doi.org/10.48550/arXiv.2305.17010.
A.1 Evaluation Metrics
We use different metrics to evaluate the performance of models on CO problems. We assess the quality of an individual solution  $\sigma$  by the associated value of the energy function  $E_{m}(\sigma)$  which represents the size of the solution sets in MIS and MVC. Here  $m$  refers to the problem instance under consideration. Optimal solutions  $\sigma_{\mathrm{opt}}$  are obtained with the Gurobi solver [Gurobi Optimization, LLC, 2023]. For models that generate solutions indeterministically we sample  $n_S$  different solutions  $\sigma_{j}$  per problem instance and calculate the test dataset average of the best relative error  $\epsilon_{\mathrm{rel}}^*$ :
$$
\epsilon_ {\text {r e l}} ^ {*} = \frac {1}{M} \sum_ {m = 1} ^ {M} \min  _ {j} \frac {\left| E _ {m} \left(\boldsymbol {\sigma} _ {\text {o p t}}\right) - E _ {m} \left(\boldsymbol {\sigma} _ {j}\right) \right|}{\left| E _ {m} \left(\boldsymbol {\sigma} _ {\text {o p t}}\right) \right|}, \tag {9}
$$
where  $M$  denotes the number of problem instances in the test dataset. In case of deterministic algorithms only one sample is generated, i.e.  $n_S = 1$ . In several experiments it is insightful to investigate the average relative error  $\hat{\epsilon}_{\mathrm{rel}}$  for which we take the average instead of the minimum in Eq. 9. Analogously, we also define the best approximation ratio  $\mathrm{AR}^*$  by
$$
\mathrm {A R} ^ {*} = \frac {1}{M} \sum_ {m = 1} ^ {M} \min  _ {j} \frac {\left| E _ {m} (\boldsymbol {\sigma} _ {j}) \right|}{\left| E _ {m} (\boldsymbol {\sigma} _ {\text {o p t}}) \right|}, \tag {10}
$$
The average approximation ratio  $\widehat{\mathrm{AR}}$  is defined by taking the average instead of the minimum operation in Eq. 10. For VAG-CO we calculate the average over the ordered greedy (OG) sampling method (See. App A.7). We always report the evaluation metric together with the standard error over three different seeds except for the experiment on RRG-100 MIS, where only results on one seed are reported.
A.2 Linear Integer Program Formulations
For Gurobi we formulate the CO Problems studied in this paper in Linear Integer Program Formulation (LIP) (see Tab. 5).
Problem Type	ILP formulation q ∈ {0,1}N
MVC	minq ∑i=1N qi s.t. qi + qj ≥ 1 if (i,j) ∈ E
MIS	minq - ∑i=1N qi s.t. qi + qj ≤ 1 if (i,j) ∈ E
MaxCl	minq ∑i=1N qi s.t. qi + qj ≤ 1 if (i,j)∉ E
MaxCut	maxe∈{0,1}|E|,q ∑i,j∈E qi j
s.t. eij ≤ qi + qj if (i,j) ∈ E
eij ≤ 2 - (qi + qj) if (i,j) ∈ E
Table 5: Integer Linear Program (ILP) formulations of MIS, MVC, MaxCl and MaxCut.
A.3 Ensuring Feasible Solutions
Since there is no rigorous guarantee that the model samples only feasible solutions that satisfy the constraints, we use a fast post processing procedure to make sure that only feasible solutions are sampled. Here, we make use of our choice of the relative weighting of the energy terms  $A$  and  $B$  (see Tab. 1) in the Ising formulation, which ensures that only feasible solutions are minima in the energy landscape. Therefore, we can detect violations when  $E_B > 0$  and search for the spin that causes the largest amount of violations. Subsequently, we change the spin value of the node with the highest number of violations to satisfy the constraint and repeat the process until  $E_B = 0$ . We observe in our experiments that this post-processing step is typically unnecessary, since only in rare cases violating solutions are sampled.
A.4 Standardization of the Energy Scale
Since the energy scale of CO problems can vary significantly, a good choice of hyperparameters like the initial temperature  $T_{0}$ , learning rate and relative weighting between the policy and value loss can vary between different CO problem instances. Therefore, we standardize the energy scale that makes the choice of good hyperparameters easier. For this purpose we first express the binary energy function  $E(q)$  (see. Tab 1) in terms of spins, i.e. as  $E(\sigma)$  by substituting  $q_{i}$  with  $\frac{\sigma_i + 1}{2}$ . We then sample states for CO problem instances from the training set by using a Random Greedy Algorithm (RGA, see App. A.6.2). Since we use only a few RGA steps (see App. A.6.2), this algorithm performs only slightly better than a completely random algorithm. From these states the mean energy  $\hat{E}_{\mathrm{RGA}}$  and the standard deviation  $\mathrm{std}(E_{\mathrm{RGA}})$  over the training dataset is computed. Subsequently, we standardize the energy scale by:
$$
\hat {E} (\boldsymbol {\sigma}) = \frac {E (\boldsymbol {\sigma}) - \hat {E} _ {\mathrm {R G A}}}{\operatorname {s t d} \left(E _ {\mathrm {R G A}}\right)}. \tag {11}
$$
A.5 Baseline Methods
We compare VAG-CO with our own implementation of a Mean Field Approximation (MFA) and to Erdos Goes Neural (EGN). In both of these approaches we consider the case with and without annealing. In both methods the probability of a state  $\sigma$  factorizes into a product of independent Bernoulli probabilities. Therefore, the state probability is given by:
$$
p _ {\theta} (\boldsymbol {\sigma} | E) = \prod_ {i = 1} ^ {N} p _ {\theta} \left(\sigma_ {i} | E\right). \tag {12}
$$
While EGN and MFA both use the same probability distribution they differ in how the energy, or in case of annealing the free energy, is calculated. This is described in the following.
A.5.1 Mean Field Approximation
We estimate the energy by sampling states from the parameterized probability distribution and apply REINFORCE with variance reduction to update the network parameters. The network parameter gradients for one CO Problem Instance are then calculated with
$$
\Delta \theta (E) = \underset {\boldsymbol {\sigma} \sim p _ {\theta} (\boldsymbol {\sigma} | E)} {\mathbb {E}} \left[ (E (\boldsymbol {\sigma}) - b) \nabla_ {\theta} \log p _ {\theta} (\boldsymbol {\sigma} | E) \right],
$$
where  $b = \mathbb{E}_{\pmb{\sigma} \sim p_{\theta}(\pmb{\sigma}|E)}[E(\pmb{\sigma}]$ . For a batch of CO Problem Instances  $\Delta \theta(E)$  is averaged over multiple  $E$ .
A.5.2 MFA with Annealing
In MFA-Anneal we estimate the free energy instead and again update the network parameters with REINFORCE with variance reduction as explained above.
A.5.3 EGN
Since we consider cases, where the energy function is known in Ising formulation (see Sec. 1) the expectation  $\mathbb{E}_{\sigma \sim p_{\theta}(\sigma | E)}[E(\sigma)]$  can be written down in closed form by substituting  $\sigma_i \to 2 \cdot p_{\theta}(\sigma_i|E) - 1$  [Karalias and Loukas, 2020]. Therefore, the loss function for one CO Problem Instance can be computed with
$$
L (E) = E \left(p _ {\theta} (\sigma | E)\right)
$$
and the network parameters can be updated directly by computing the gradients of this loss function.
A.5.4 EGN with Annealing
In case of EGN-Anneal [Sun et al., 2022] the entropy of the parameterized probability distribution is calculated in closed form with the following formula:
$$
H \left(p _ {\theta} (\boldsymbol {\sigma} | E)\right) = - \sum_ {i} ^ {N} \left[ p _ {\theta} \left(\boldsymbol {\sigma} _ {i} | E\right) \cdot \log p _ {\theta} \left(\boldsymbol {\sigma} _ {i} | E\right) + \left(1 - p _ {\theta} \left(\boldsymbol {\sigma} _ {i} | E\right)\right) \cdot \log \left(1 - p _ {\theta} \left(\boldsymbol {\sigma} _ {i} | E\right) \right] \right.
$$
A.5.5 Conditional Expectation
To obtain good samples from the MFA approach in a deterministic way, we adopt the Conditional Expectation (CE) method [Raghavan, 1988] as described in Karalias and Loukas [2020].
To introduce randomness into the CE-based solution generation procedure we initialize the random node features (which are processed by the GNN) of every node with a random vector with six binary entries that are drawn from a uniform distribution. Unless stated otherwise, we follow the procedure in [Wang and Li, 2023] and report the best CE result of eight different random node feature initializations.
A.6 Greedy Algorithms
A.6.1 Degree Based Greedy Algorithm
The Degree Based Greedy (DB-Greedy) algorithm [Wormald, 1995] is a polynomial time algorithm for the MIS problem. The DB-Greedy algorithm works in the following way: At first all nodes are sorted according to their degrees. Then, starting from the smallest degree the node is chosen to be part of the independent set. In the next step this node, its neighbors and their corresponding edges are deleted from the graph. These steps are repeated until the graph is empty.
This algorithm can also be applied to the MVC problem by using the fact that the complement of an independent set is a vertex cover. In other words, nodes in the independent set are excluded from the vertex cover and nodes that are not part of the independent set are included into the vertex cover.
A.6.2 Random Greedy Algorithm
The Random Greedy Algorithm (RGA) is a general approach that can be utilized for solving a broad range of CO problems that can be mapped to the Ising model. Initially, the algorithm randomly samples spin values with uniform probability. Then, for a fixed number of iterations a spin is randomly selected and its value is changed if it decreases the energy value. We set the number of iterations to  $N \cdot n_{R}$ , where  $N$  is the number of nodes in the graph and  $n_{R}$  is the number of repetitions per node.
For the purpose of the standardization the energy scale (see. App. A.4), we employ this algorithm with  $n_R$  set to one.
A.7 Study on Sampling Methods
In order to find out, how the best samples can be obtained with VAG-CO we study in Fig. 3 three different sampling methods on the RRG-100 MIS and RB-200 MVC dataset. Here we evaluate  $\epsilon_{\mathrm{rel}}^*$  on the test dataset and plot it over the number of samples  $n_S$  that are used for each graph in the dataset. We also add MFA and DB-Greedy results, where we show for MFA how the method improves when more solutions are sampled. Since DB-Greedy is a deterministic algorithm, we draw a horizontal line that indicates the solution quality of the algorithm. To draw a relation to results that are presented in Fig. 1 (Left, Middle) we also add a vertical dotted line at  $n_S = 8$  samples. For VAG-CO we denote the first method as sampling (S), where for each graph  $n_S$  solutions are sampled according to the corresponding probability distribution. In the second VAG-CO sampling method called ordered sampling (OS), we use for each graph  $n_O$  different BFS node orderings and sample  $n_S / n_O$  states per graph ordering. Finally, we sample VAG-CO solutions with a method called ordered greedy (OG), where we generate solutions greedily for each BFS ordering with  $n_O = n_S$ . Results in Fig. 3 show that the sampling strategy OG always outperforms all other sampling strategies that we proposed for VAG-CO. Additionally, we see that as we increase  $n_O$  in the OS sampling strategy, the performance improves consistently. Remarkably, DB-Greedy exhibits the best solution quality when MFA and VAG-CO are allowed only one sample ( $n_S = 1$ ). However, DB-Greedy is outperformed by VAG-CO OG already with a modest amount of  $n_S > 1$  samples.
A.8 Experimental Details
In this section we provide additional details to experiments that are presented in Sec. 6.
Parameter Checkpointing. With VAG-CO we checkpoint over the course of training the parameters that obtain the best  $\epsilon_{\mathrm{rel}}^*$  and the best  $\hat{\epsilon}_{\mathrm{rel}}$  on the validation set. For testing we always use the checkpoint with the best  $\hat{\epsilon}_{\mathrm{rel}}$  except for the results in Fig. 3 when the sampling strategies (S) and (OS) are used. Here we use the checkpoints with the best  $\epsilon_{\mathrm{rel}}^*$ .
Hyperparameter Tuning. For MFA-Anneal, the learning rate and initial temperature are tuned on the validation dataset of Enzymes MIS via a grid search. We considered learning rates  $(\mathrm{lr} \in [5 \times 10^{-4}, 1 \times 10^{-4}, 5 \times 10^{-5}])$  and initial temperatures  $(T_0 \in [0.5, 0.25, 0.1, 0.05])$ . With that, the number of GNN layers  $(L \in [6, 8, 12])$  are tuned. Finally, the annealing duration  $(N_{\mathrm{anneal}})$  is increased until we observe that longer annealing does not lead
Figure 3: Ablation on different sampling strategies for VAG-CO on the RRG-100 MIS dataset (Left) and on the RB-200 MVC Dataset (Right). The averaged best relative error  $\epsilon_{\mathrm{rel}}^*$  on the test dataset is plotted over the number of solutions  $n_S$  per graph. Error bars indicate the standard error over the test dataset.
to improvements. For MFA, we tested on ENZYMES MIS the learning rates  $(\mathrm{lr} \in [5 \times 10^{-4}, 1 \times 10^{-4}, 5 \times 10^{-5}, 1 \times 10^{-5}])$  and with that tuned the number of GNN layers  $(L \in [6, 8, 10, 12])$  and stop training when no significant improvement on the validation set is observed. In MFA and MFA-Anneal, when training on the other datasets we started with the optimal ENZYMES MIS parameters and further tested different learning rates (lr), initial temperatures  $(T_0)$  and number of GNN layers  $(L)$  individually on each dataset. For experiments on the RRG-100 MIS and RB-200 MVC we make a more extensive hyperparameter search. In RRG-100 MIS for MFA-Anneal we first tuned the initial temperature in the range of  $T_0 \in [0.25, 0.1, 0.05, 0.035, 0.025, 0.015, 0.01]$  and in RB-200 MVC we search for the best initial temperature within the range of  $T_0 \in [0.5, 0.25, 0.1, 0.05]$ . For MFA and MFA-Anneal, we then iteratively search for the best learning rate in the range of  $\mathrm{lr} \in [5 \times 10^{-4}, 1 \times 10^{-4}, 5 \times 10^{-5}]$  and then for the best number of GNN layers  $L \in [6, 8, 12]$ .
In the experiments with EGN on RB-200 MVC we also performed a extensive hyperparameter search, where first the best temperature  $T_{0}$  was tuned within the range of  $T_{0} \in [0.25, 0.1, 0.05]$ . Then we tuned the best learning rate within the range of  $lr \in [5 \cdot 10^{-4}, 1 \cdot 10^{-1}, 5 \cdot 10^{-5}, 1 \cdot 10^{-5}]$ . Then we tuned for the best number of GNN layers within the range of  $L \in [8, 10, 12, 14]$ . In EGN without annealing the same hyperparameter search was performed but at  $T_{0} = 0$ . The best hyperparameters of all methods are listed in App. A.13.
On VAG-CO we iteratively tuned the learning rate  $(\mathrm{lr} \in [5 \cdot 10^{-4}, 1 \cdot 10^{-3}])$ , initial temperature  $(T_0 \in [0.05, 0.7, 0.1])$ , number of GNN layers  $(L \in [3, 4])$  and number of annealing steps  $(N_{\text{anneal}} \in [3000, 6000])$  on the ENZYMES MIS validation dataset. We then initially test these hyperparameters on other datasets and adapt the number of annealing steps and the initial temperature. The choice of all hyperparameters in VAG-CO is listed in Tab. 7.
Ablation on Subgraph Tokenization. In the ablation on subgraph tokenization in Fig. 1 we keep all hyperparameters the same, except for the time horizon  $\mathcal{T}$  (see Sec. A.9) and the hyperparameter  $\lambda$  in the PPO algorithm (see App. A.9). For the subgraph tokenization run we chose  $\mathcal{T} = 20$ ,  $\lambda = 0.95$  and  $k = 5$ . Therefore, the time horizon includes the generation of  $\mathcal{T} \cdot k = 100$  spins per graph during the data collection phase (see App. A.9). If we would keep  $\mathcal{T}$  the same, when we chose  $k = 1$  for the run without subgraph tokenization only  $\mathcal{T} \cdot k = 20$  spins would be generated for each graph. Therefore, we use  $\mathcal{T} = 100$  when no subgraph tokenization is used. As we always set  $\lambda = 1 - \frac{1}{\mathcal{T}}$  this hyperparameter is adapted accordingly.
Experiments on Random Regular Graphs. Since GNNs suffer from node ambiguity on Random Regular Graphs (RRGs) Wang and Li [2023] using random node features Abboud et al. [2021] can resolve this issue. Therefore, for the VAG-CO experiment in Fig. 1 (Left) we sample standard Gaussian random node features with the dimension of six and concatenate them to the graph representation of VAG-CO. For MFA and MFA-Anneal random node features are already used (see App. A.5.5) and had not to be added for the RRG experiments.
Averaged results on hard instances. In Fig. 1 (Left, Middle) only show results for specific generation parameters  $p$  on the RB-200 dataset and for specific values of  $d$  on the RRG-100 dataset. Here we report the corresponding averaged results on the RRG-100 MIS and RB-200 MVC dataset in Tab. 6. VAG-CO significantly outperforms all other methods for  $n_S = 8$ .
A.9 Proximal Policy Optimization
Proximal Policy Optimization (PPO) [Schulman et al., 2017] is a popular RL algorithm that has two main components. The policy is represented by the network  $p_{\theta}(\sigma_i|G_i)$  with parameters  $\theta$ . The expected future reward is estimated by the value network  $V_{\phi}(G_i)$  which is parameterized by  $\phi$ .
RB-200 MVC
RRG-100 MIS
Evaluation metric A.1	AR*	(s/graph)	AR*	(s/graph)
DB-Greedy	1.0088 ± 0.006	0.01	0.931	0.003
MFA: CE	1.0111 ± 0.0002	0.10	0.947	0.06
MFA-Anneal: CE	1.0098 ± 0.0001	0.10	0.955	0.06
EGN: CE	1.012 ± 0.0001	0.10	N/A	N/A
EGN-Anneal: CE	1.0087 ± 0.0002	0.10	N/A	N/A
VAG-CO (ours)	1.0059 ± 0.0001	0.20	0.975	0.08
Gurobi (tmax=0.08)	1.0092 ± 0.0001	0.06	0.945	0.07
Gurobi (tmax=0.1)	1.0073 ± 0.0001	0.07	0.954	0.09
Gurobi (tmax=0.15)	1.0052 ± 0.0001	0.09	0.974	0.12
Gurobi (tmax=0.2)	1.0037 ± 0.0001	0.11	0.987	0.16
Gurobi (tmax=0.3)	1.0017 ± 0.0001	0.13	0.995	0.22
Gurobi (tmax=0.5)	1.0006 ± 0.0001	0.176	0.998	0.31
Gurobi (tmax=1.)	1.0001 ± 0.0001	0.26	1.000	0.37
Table 6: Best approximation ratio  $AR^{*}$  on the test dataset of the RB-200 MVC and RRG-100 MIS dataset. Results where the  $AR^{*}$  is closer to one is better.
Data Collection. In PPO the policy is rolled out through the environment in order to collect and store the states, actions, probabilities and the output of the value function into the rollout buffer. The data collection procedure is completed after going through  $\mathcal{T}$  steps, where  $\mathcal{T}$  is the so-called time horizon. Then, the advantages  $A_{i} \coloneqq A(\sigma_{i}, G_{i})$  and value targets  $V_{i}^{T} \coloneqq V^{T}(G_{i})$  are calculated by making use of the Generalised Advantage Estimation (GAE, Schulman et al. [2016]), where  $A_{i}$  is given by
$$
A _ {i} = \delta_ {i} + (\gamma \lambda) \delta_ {i + 1} + \dots + (\gamma \lambda) ^ {\mathcal {T} - t + 1} \delta_ {\mathcal {T} - 1}, \tag {13}
$$
with  $\delta_i = R_i + \gamma V(G_{i+1}) - V(G_i)$  and  $V_i^T = A_i + V(G_i)$ . In our experiments the reward  $R_i$  is given by Eq. 6. In our experiments we chose  $\gamma = 1.0$  and  $\lambda = 1 - \frac{1}{\mathcal{T}}$ . We set  $\mathcal{T}$  to be approximately equal to the average graph size in the dataset. During the data collection phase we always collect data from  $H = 30$  different CO problem instances and for each instance we sample  $n_S = 30$  solutions.
Rollout Buffer. For the gradient updates in PPO, the loss is estimated with minibatches that are randomly sampled from the rollout buffer. As the rollout buffer contains data of size  $H \times n_S \times T$  we chose to sample data by first sampling  $H_{\mathrm{minib}}$  graphs and then for each graph, we chose to sample  $N_{\mathrm{minib}}$  solutions and for each solution  $S_{\mathrm{minib}}$  time steps are sampled. We report the minibatch settings of each experiment in Tab. 7.
Training. As described before, during training minibatches of data  $D_{\mathrm{minib}}$  is randomly sampled from the replay buffer. In PPO the overall loss that depends on the  $L(\theta, \phi; D_{\mathrm{minib}})$  is given by  $L(\theta, \phi; D_{\mathrm{minib}}) = L_1(\theta; D_{\mathrm{minib}}) + c_V L_2(\phi; D_{\mathrm{minib}})$ , where the first term depends on the policy and the second term on the value function. To specify a minibatch sample from  $D_{\mathrm{minib}}$ , we will use the index  $n$ .
The policy loss for one minibatch sample is then defined as
$$
L _ {1} (\theta) _ {n} = - \min  \left(I _ {n} (\theta) A _ {n}, \operatorname {c l i p} \left(I _ {n} (\theta), 1 - \epsilon , 1 + \epsilon\right) A _ {n}\right), \tag {14}
$$
where the importance sampling ratio  $I_{n}(\theta) = \frac{p_{\theta}(\sigma_{n}|G_{n})}{p_{\theta^{\prime}}(\sigma_{n}|G_{n})}$  is used to compute a weighted Monte Carlo estimate of  $A_{n}$ . Here,  $\theta^{\prime}$  represents the old policy parameters that were used to fill the rollout buffer during the data collection phase and  $\theta$  are the new parameters that are used in gradient descend. In Eq. 14 the clipping function is used to prevent that the probability of the current policy  $p_{\theta}(\sigma_n|G_n)$  differs by more than a factor  $1 \pm \epsilon$  from the old policy  $p_{\theta^{\prime}}(\sigma_n|G_n)$ .
Similarly, the value function loss for one sample is given by  $L_{2}(\phi)_{n} = (V_{\phi}(G_{n}) - V_{\phi^{\prime}}^{T}(G_{n}))^{2}$ .
The loss  $L(\theta, \phi; D_{\mathrm{minib}})$  is then updated for each minibatch in the rollout buffer so that each sample is used at least once. This procedure is overall repeated  $n_{\mathrm{repeat}}$  times, before we increment the epoch step  $N_{\mathrm{epoch}}$  and new data is collected with the updated set of parameters. In our experiments we always chose  $\epsilon = 0.1$ ,  $c_V = 0.5$  and  $n_{\mathrm{repeat}} = 2$ .
A.10 Model Details and GNN Architectures