SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Before processing the graph representation with GNNs, we encode its node features with an encoder MLP. To obtain node embeddings  $\mathcal{N}_l^l\in \mathbb{R}^{d(l)}$  with  $l\in [1,\dots ,L]$  that depend on the graph structure, the encoded node features are processed by  $L$  layers of GNNs. Afterwards, we apply a global sum aggregation to compute a global graph embedding. The global graph embedding is then concatenated to the node embedding  $\mathcal{N}_i^L$ , where  $i$  is the index of the spin whose value is to be generated (see Sec. 3). We then use this embedding to calculate the policy and value function outputs (see App. A.9) with two separate MLPs.
In our experiments, we employ two distinct message passing architectures. The first one is a Message-passing Neural Network (MPNN) [Battaglia et al., 2018] but with linear message functions. The second architecture also includes skip connections (MPNN-skip) [Battaglia et al., 2018] which we use when more than three GNN layers are used.
Message Passing Neural Network. Our MPNN layer is defined by the following update for node embeddings:
$$
\mathcal {N} _ {i} ^ {L + 1} = \ln \left[ \Psi \left(\mathcal {N} _ {i} ^ {L}, \sum_ {j \in N (i)} \Phi \left(\mathcal {N} _ {j} ^ {l}, \mathcal {N} _ {i} ^ {l}, \mathcal {E} _ {i j}\right)\right) + W _ {\mathrm {s k i p}} \mathcal {N} _ {i} ^ {L} \right], \tag {15}
$$
where  $\Phi (\mathcal{N}_j^l,\mathcal{N}_i^l,\mathcal{E}_{ij})$  is a message update MLP that computes messages with pairwise interactions between nodes. Skip connection are implemented by adding the term  $W_{\mathrm{skip}}\mathcal{N}_i^L$  after the node update, where  $W_{\mathrm{skip}}\in \mathbb{R}^{d(l + 1)\times d(l)}$  is a weight matrix. Finally, layer normalization  $\ln [\cdot ]$  [Ba et al., 2016] is applied as it is commonly done in Neural Network architectures when skip connections are used [Liu et al., 2021].
In VAG-CO we use a policy and value MLP with three layers. Both MLPs have 120 neurons in the first two layers. The value MLP has only one neuron in the output layer and the policy MLP has  $2^{k}$  output neurons. For the encoder MLP we use a two layer MLP with 40 neurons in each layer. In case of the node update MLP  $\Psi$  and the message update MLP  $\Phi$  we use 2 layers with 64 neurons in each layer. Layer normalization  $\ln[\cdot]$  is applied after every layer within a MLP except in the output layer of the policy and value MLP. We use Rectified Linear Unit (ReLU) activation functions Agarap [2018] except in the output of the policy MLP, where a softmax activation is used and except of the output of the value MLP where no activation is used.
In MFA with and without annealing the model uses an encoder MLP with one layer and 64 neurons. The node update MLP  $\Psi$  and the message update MLP  $\Phi$  have the same number of neurals as in VAG-CO. The output MLP that is applied on each node after  $n$  GNN message passing steps the has three layers with 64 neurons each and the final output layer has 2 neurons with a softmax activation.
A.10.1 Subgraph Tokenization
For subgraph tokenization we have to make changes to the one-hot encoding as described in Sec. 3 and also to the model architecture as described in App. A.10. The one-hot encoding in the graph representation is adapted so that spins  $\sigma_{i:i+k}$  that are going to be generated receive an enumeration that indicates their position in the sliced list of BFS-ordered (see Sec. 3]) indices  $i: i+k$ . Then, the graph  $G_i$  is processed by a GNN that provides a node embedding  $GNN_\theta(x_i)$  for each node. Along with a global sum aggregation, the node embeddings of the spins that are going to be generated are then concatenated according to their BFS order (Sec. 3) and further processed by the policy MLP that calculates the probability for each of the  $2^k$  spin configurations using a softmax output layer.
When the number of nodes  $N$  in the graph is not divisible by  $k$ , the number vertices in the CO problem instance description has to be increased without changing the inherent optimization objective. This can be realised by adding a sufficient amount of spins into the Ising model (see Sec. 2) with zero spin weight  $B$  and no connections to other spins.
A.11 Training Time and Computational Resources
All runs with VAG-CO were conducted either on A100 Nvidia GPU with 40 GB Memory or an A40 Nvidia GPUs with 48 GB Memory. In case of COLLAB MVC an A100 with 80 GB Memory is used.
The training time of our algorithm depends hyperparameters like the number of annealing steps  $N_{\mathrm{anneal}}$  (see App. A.14), the number of edges and nodes of graphs in the dataset, on the GPUs that are used during training, on the time horizon and on the minibatch size that is used for gradient updates in PPO (see App. A.9). For example the TWITTER MVC run with  $N_{\mathrm{anneal}} = 4000$ ,  $T = 30$ ,  $H_{\mathrm{minib}} = N_{\mathrm{minib}} = S_{\mathrm{minib}} = 10$  takes one day, when trained on an A100 40 GB Nvidia GPU. The run on ENZYMES MIS trained on a A40 GPU takes ten hours, when trained with  $N_{\mathrm{anneal}} = 6000$ ,  $T = 20$ ,  $H_{\mathrm{minib}} = N_{\mathrm{minib}} = 15$  and  $S_{\mathrm{minib}} = 10$ .
A.12 Time measurement
Gurobi. All Gurobi results are conducted on a Intel Xeon Platinum 8168 @ 2.70GHz CPU with 24 cores. All Problems expect for MaxCut are formualted as LIP and we set the number of threads for Gurobi to 26.
Learned Methods. For all learned methods (EGN, MFA, VAG-CO) we present the time per graph as if the algorithm are implemented in a parallelized manner, where each sampling process runs on a separate thread. We always report runtimes by using the JIT compiled performance of the Graph Neural Network.
A.13 Hyperparameters
VAG-CO Hyperparameters.
All hyperparameters that change across datasets are listed in Tab. 7, whereas hyperparameters that stay the same are specified in the corresponding section (e.g. App. A.10, A.14).
Dataset	CO Problem Instance	learning rate	\(N_{\text{anneal}}\)	\(T_0\)	\(L\)	\(N_{\text{minib}}\)	\(H_{\text{minib}}\)	\(S_{\text{minib}}\)	\(\mathcal{T}\)	\(k\)
TWITTER	MVC	\(1 \times 10^{-3}\)	4000	0.05	3	10	10	10	30	5
COLLAB	MVC	\(1 \times 10^{-3}\)	12000	0.05	3	10	10	6	35	5
MIS	\(1 \times 10^{-3}\)	4000	0.05	3	10	10	10	30	5
IMDB-BINARY	MVC	\(1 \times 10^{-3}\)	2000	0.05	3	10	10	10	5	5
MIS	\(5 \times 10^{-4}\)	2000	0.05	3	10	10	10	5	5
MaxCl	\(5 \times 10^{-4}\)	0.03	6000	3	6	10	7	20	6
ENZYMES	MIS	\(5 \times 10^{-4}\)	6000	0.07	4	15	15	15	25	5
MaxCl	\(5 \times 10^{-4}\)	6000	0.03	3	6	10	7	20	6
PROTEINS	MIS	\(5 \times 10^{-4}\)	10000	0.1	3	15	15	10	35	5
MUTAG	MIS	\(1 \times 10^{-3}\)	2000	0.05	3	10	10	10	7	5
RB-100	MVC	\(1 \times 10^{-3}\)	2000	\(5 \cdot 10^{-2}, 5 \cdot 10^{-3}, 0.0\)	3	10	10	5	20	5
RB-200	MVC	\(5 \times 10^{-4}\)	20000	0.05	3	15	10	10	30	5
RRG-100	MIS	\(5 \times 10^{-4}\)	20000	0.1	4	10	10	10	20	5
BA 200-300	MaxCut	\(1 \times 10^{-3}\)	7000	0.08	4	10	10	10	30	8
Table 7: Hyperparemeters that are used in VAG-CO on different datasets.
MFA Hyperparameters.
All runs with MFA used a batch size  $H$  of 32 and we sampled  $n_{S} = 30$  solutions. Furthermore, we used 8 GNN layers, and 6 random node features per node. In the RB-200 MVC experiment 10 GNN layers were used. For MFA-Anneal we used a learning rate of  $1 \times 10^{-4}$ , 2000 annealing steps  $N_{\text{anneal}}$  and a start temperature  $T_{0}$  of 0.1 except for RRG-100 MIS and for RB-200 MVC. In RRG-100 MIS a  $T_{0}$  of 0.015 is used and in RB-200 MVC  $T_{0} = 0.05$ . For MFA with and without annealing we trained for 2000 epochs and used a learning rate of  $5 \times 10^{-5}$  except for ENZYMS MIS and RB-200 MVC, where a learning rate of  $1 \times 10^{-4}$  is used.
EGN Hyperparameters.
All runs with EGN on RB-200 MVC have a batch size of  $H = 32$ . Here, in EGN-Anneal we use 12 GNN layers and in EGN without annealing 10 GNN layers performs best. In EGN-Anneal the initial temperature of  $T_0 = 0.1$  with the learning rate of  $5 \times 10^{-5}$  obtained the best results. With EGN without annealing the same learning rate is used. Both methods are trained for 2000 epochs.
A.14 Annealing Schedule
As described in Sec. 3, we change the temperature in the reward Eq. 6 according to a predefined annealing schedule. During the warm-up phase of  $N_{\mathrm{warmup}}$  epochs the temperature is held constant at the initial temperature  $T_0$ . Afterwards the following temperature schedule is applied:
$$
T \left(N _ {\mathrm {e p o c h}}\right) = T _ {\mathrm {a n n e a l}} \left(N _ {\mathrm {e p o c h}}\right) \cdot \left[ \cos \left(2 \pi \left(\lambda + \frac {1}{2}\right) \frac {N _ {\mathrm {e p o c h}} - N _ {\mathrm {w a r m u p}}}{N _ {\mathrm {a n n e a l}}}\right) + 1 \right]. \tag {16}
$$
Here,  $T_{\mathrm{anneal}}(N_{\mathrm{epoch}})$  is the gradually decreasing amplitude of the temperature oscillations:
$$
T _ {\text {a n n e a l}} \left(N _ {\text {e p o c h}}\right) = \frac {T _ {0}}{1 + c \cdot \frac {N _ {\text {e p o c h}} - N _ {\text {w a r m u p}}}{N _ {\text {a n n e a l}}}}. \tag {17}
$$
Figure 4: Illustration of the cosine modulated annealing schedule. The plot depicts the annealing schedule with  $N_{\mathrm{warmup}} = 200$  and  $N_{\mathrm{anneal}} = 1000$  steps. The vertical red line marks the end of the warmup phase.
Here  $c$  is a scaling factor that determines the slope of  $T_{\mathrm{anneal}}(N_{\mathrm{epoch}})$ . The parameter  $\lambda$  determines the number temperature oscillations in the schedule and  $N_{\mathrm{anneal}}$  is the total number of epochs that follow after the warmup phase. We use  $\lambda = 3$ ,  $N_{\mathrm{warmup}} = 400$  and  $c = 6$ . The course of the annealing schedule is illustrated in Fig. 4. One reason for the usage of cosine modulation function is rather practical, namely that  $T = 0$  is reached multiple times during training, which allows an assessment of the training success at an earlier stage for a given set of hyperparameters. In the absence of cosine modulation, we found it harder to assess the final performance before the entire annealing was finished. Similar periodic schedules for learning rates have been proposed as variants of Stochastic Gradient Descent [Loshchilov and Hutter, 2017].
A.15 Derivations
A.15.1 Free-Energy Decomposition into Rewards
In the following we show that using the reward defined in Eq. 6 is consistent with the goal of minimizing the free-energy defined in Eq. 2.
The right-hand side of Eq. 2 contains the expectation of the energy  $E(\sigma)$  and a term that is proportional to the entropy of  $p_{\theta}(\sigma)$ . For the energy we obtain the following decomposition into individual steps  $i$  of the solution generation process (Sec. 3):
$$
E (\boldsymbol {\sigma}) = \sum_ {i = 1} ^ {N} \left[ \sum_ {j <   i} J _ {i j} \sigma_ {j} \sigma_ {i} + B _ {i} \sigma_ {i} \right] = \sum_ {i = 1} ^ {N} \sigma_ {i} \left[ \sum_ {j <   i} J _ {i j} \sigma_ {j} + B _ {i} \right] = \sum_ {i = 1} ^ {N} \Delta E _ {i} \tag {18}
$$
By using an autoregressive factorization the entropy can also be decomposed in the following way:
$$
\begin{array}{l} S \big (p _ {\theta} (\boldsymbol {\sigma} | E) \big) = - \sum_ {\boldsymbol {\sigma}} p _ {\theta} (\boldsymbol {\sigma} | E) \log p _ {\theta} (\boldsymbol {\sigma} | E) = - \sum_ {\boldsymbol {\sigma}} p _ {\theta} (\boldsymbol {\sigma} | E) \left[ \sum_ {i = 1} ^ {N} \log p _ {\theta} \big (\sigma_ {i} | \sigma_ {<   i}, E \big) \right] \\ = - \underset {\boldsymbol {\sigma} \sim p _ {\theta}} {\mathbb {E}} \left[ \sum_ {i = 1} ^ {N} \log p _ {\theta} \left(\sigma_ {i} \mid \sigma_ {<   i}, E\right) \right] \\ \end{array}
$$
Therefore, we can use this decomposition by using the reward  $R_{i}(G_{i},\beta) = -\left[\Delta E_{i} + \frac{1}{\beta}\log p(\sigma_{i}|\sigma_{< i},E)\right]$ . By the relation  $F(\theta ;\beta ,E) = -\mathbb{E}_{\sigma \sim p_{\theta}}[\sum_{i}R_{i}(G_{i},\beta)]$ , maximizing this reward will then be equivalent to minimizing the free-energy.
A.15.2 Details on Remark 1
We first restate Corollary 5 of [Dudik et al., 2007] in our notation. In the context of this Corollary our energy function  $E$  can be regarded as a single feature and, therefore, we use their result for  $n = 1$ . In addition, we consider the case in which the samples  $\mathcal{S}$  are drawn from the target distribution, i.e.  $\pi = p(\beta^{*})$ . Since the terms Boltzmann distribution and Gibbs distribution can be used interchangeably we obtain:
Corollary 1 (Corollary 5 of [Dudík et al., 2007]). Assume that  $E$  is bounded in  $[0,1]$ . Let  $\hat{\beta}$  minimize  $\mathcal{L}_{\hat{p}}(\beta) + \lambda |\beta|$  with  $\lambda = \sqrt{\ln(2/\delta)/(2m)}$ . Then with probability at least  $1 - \delta$  for every Boltzmann distribution  $p(\beta)$ ,
$$
\mathcal {L} _ {p (\beta *)} (\hat {\beta}) \leqslant \mathcal {L} _ {p (\beta *)} (\beta) + \frac {| \beta |}{\sqrt {m}} \sqrt {2 \ln (2 / \delta)}.
$$
Now we consider the result for the case  $\beta = \beta^{*}$  and subtract  $\mathcal{L}_{p(\beta^{*})}(\beta^{*})$ . Remark 1 follows by applying the definition of the Kullback-Leibler divergence  $D_{KL}$  to the left-hand side.
We note that an equivalent statement, however, with a consideration the Rademacher complexity of  $E$  can be derived from Theorem 2 in Cortes et al. [2015].