Title: Variational Annealing on Graphs for Combinatorial Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Combinatorial Optimization

Entropy Regularization

Graph Neural Networks

Statistical Mechanics
Abstract: Several recent unsupervised learning methods use probabilistic approaches to solve combinatorial optimization (CO) problems based on the assumption of statistically independent solution variables. We demonstrate that this assumption imposes performance limitations in particular on difficult problem instances. Our results corroborate that an autoregressive approach which captures statistical dependencies among solution variables yields superior performance on many popular CO problems. We introduce Subgraph Tokenization in which the configuration of a set of solution variables is represented by a single token. This tokenization technique alleviates the drawback of the long sequential sampling procedure which is inherent to autoregressive methods without sacrificing expressivity. Importantly, we theoretically motivate an annealed entropy regularization and show empirically that it is essential for efficient and stable learning.

Variational Annealing on Graphs for Combinatorial Optimization

Abstract

Several recent unsupervised learning methods use probabilistic approaches to solve combinatorial optimization (CO) problems based on the assumption of statistically independent solution variables. We demonstrate that this assumption imposes performance limitations in particular on difficult problem instances. Our results corroborate that an autoregressive approach which captures statistical dependencies among solution variables yields superior performance on many popular CO problems. We introduce subgraph tokenization in which the configuration of a set of solution variables is represented by a single token. This tokenization technique alleviates the drawback of the long sequential sampling procedure which is inherent to autoregressive methods without sacrificing expressivity. Importantly, we theoretically motivate an annealed entropy regularization and show empirically that it is essential for efficient and stable learning. 1

1 Introduction

Combinatorial optimization (CO) problems are of central interest to a wide range of fields, including operations research, physics, and computational complexity [Papadimitriou and Steiglitz, 1998]. Since CO problems are typically NP-hard one would not expect that one can find the solutions to all arbitrarily large CO problem instances in polynomial time. However, these restrictions are concerned with worst case scenarios over entire problem families. For instance finding the Minimum Independent Set of any graph. Consequently, it is not surprising that a large body of work is focused on the design of solution strategies that yield a particularly good performance on a restricted subset of all possible instances. There is growing interest in exploring deep learning approaches to this restricted CO setting [Bello et al., 2017]. These methods aim to learn how to efficiently generate high quality solutions for CO problems and rely primarily on learned algorithmic components rather than on problem-specific, hand-crafted ones [Bengio et al., 2021]. Importantly, practicable methods should not rely on supervised training with solutions from other solvers since this limits the achievable performance of the learned model to that of the solver used to generate the training data [Yehuda et al., 2020]. As a result the development of unsupervised methods [Karalias and Loukas, 2020, Wang et al., 2022, Qiu et al., 2022, Karalias et al., 2022, Min et al., 2022, Wang and Li, 2023] and of Reinforcement Learning (RL) methods [Bello et al., 2017, Khalil et al., 2017, Kool et al., 2019, Ahn et al., 2020, Böther et al., 2022, Mazyavkina et al., 2021] for CO is an active field of research. In this work we are interested in training models that generate solutions at inference in contrast to approaches

Sepp Hochreiter 1 , 2

that require problem instance specific training. A popular approach is to take a representation of the CO problem instance as input to a deep learning model and to output the parameters of a distribution over solutions that has its probability mass concentrated on regions with high solution quality. This probabilistic optimization approach to CO is used in the recent works of e.g. [Karalias and Loukas, 2020, Min et al., 2022, Qiu et al., 2022, Sun et al., 2022]. In these works the distribution over solutions is built on the assumption of mutual statistical independence of the individual solution parameters and can, consequently, be represented by a product of Bernoulli distributions for the the individual solution variables. This simplification is typically refereed to as a mean-field approximation (MFA) and is frequently used in various fields including e.g. statistical mechanics [Parisi, 1988], Bayesian statistics [Wainwright and Jordan, 2008], and the analysis of neural networks [Mei et al., 2019]. However, the simplifying assumption of the MFA restricts the expressivity of the corresponding distributions which limits its applicability when the target distribution represents strong correlations [Jaakkola and Jordan, 1998]. However, replacing MFA with more expressive approaches is computationally expensive and requires careful regularization to ensure efficient and stable training. Based on these consideration our contributions can be summarized as follows.

We demonstrate that the frequently used MFA imposes limits on the attainable solution quality for CO problems. Importantly, we introduce Variational Annealing on Graphs for Combinatorial Optimization (VAG-CO) a method that achieves new state-of-the-art performances on popular CO problems by combining expressive autoregressive models with annealed entropy regularization. We provide a theoretical motivation for this entropy regularization via considerations of the sample complexity of related density estimation problems. By introducing sub-graph tokenization VAG-CO drastically reduces the number of necessary steps to generate solutions and therein significantly improves the training and inference efficiency.

2 Problem Description

Ising Formulation of CO. To introduce VAG-CO it is convenient to adopt a point of view on CO that is motivated by statistical mechanics. As shown in Lucas [2014] many frequently encountered NP-complete CO problems can be reformulated in terms of a particular class of models known as Ising models which therefore allow the representation of different CO problem types in a unified manner. In this context the CO problem is equivalent to finding the minimum of an energy function E : t´ 1 , 1 u N ÞÑ R . This function assigns an energy E p σ q to an N-tuple of discrete solution variables σ ' p σ 1 . . . σ N q P t´ 1 , 1 u N which are often referred to as spins. The family of aforementioned Ising models is characterized by the following form of the energy function:

where the first term represents the interaction between spins σ i P t´ 1 , 1 u through couplings J ij P R , while the second term determines the magnitude B i P R of the contribution of an individual spin to the energy of the system. For brevity we will denote the parameters of the energy function p J ij , B i q simply as E . Hence, E will represent a CO problem instance while E p σ q will denote the energy of a solution σ under an energy function with parameters E . The Ising formulation for the four CO problem types studied in this work is given in Tab. 1.

Variational Learning. Next, we specify how we approach the learning problem of approximating the optimal solution for a given problem instance. We follow the frequently taken variational learning approach of using a neural network with parameters θ to obtain for a given CO problem instance E an associated probability distribution p θ p σ | E q over the space of possible solutions t´ 1 , 1 u N . The problem instances E are assumed to be independently sampled from a probability distribution q p E q with Ω ' supp p q q . In this setting the learning problem can be formulated as finding argmin θ ř Ω q p E q ř σ p θ p σ | E q E p σ q . Here ř σ is a shorthand for the sum over all σ P t´ 1 , 1 u N . In practice both sums are approximated by empirical averages over samples from the corresponding distributions q and p θ . In CO E p σ q is typically characterized by many local minima. Therefore, directly approaching the learning problem described above via gradient descent methods is prone to getting stuck at sub-optimal parameters. As we demonstrate in Fig. 1 (Right) this problem can be alleviated by adding a regularization term to the optimization objective Eq. 1 which encourages p θ to avoid a collapse to local minima. The resulting loss function L : t´ 1 , 1 u N Ñ R associated to the

learning problem is given by:

where M is the number of CO problem instances E m in one batch and β plays the role of a regularization coefficient which is frequently referred to as the inverse temperature 1 { T . The loss for an individual problem instance E m is based on the so-called free-energy F p θ ; β, E m q :

In practice, we approximate the expectation value F p θ ; β, E m q with an empirical sample average ˆ F p θ ; β, E m q that is based on n S samples σ ' p θ p σ | E m q . The term E m p σ q on the right-hand side of Eq. 3 encourages the concentration of p θ p σ | E m q on low-energy solutions. Small values of the regularization parameter β P R ą 0 encourage p θ p σ | E m q to have a large entropy. For a given value of β the free-energy in Eq. 3 is known to be minimized by the Boltzmann distribution associated to E m (see e.g. [MacKay, 2003]):

Thus by minimizing L the model learns to approximate p B p σ | E m , β q for a given E m and β . For β Ñ inf the p B p σ | E m , β q has its mass concentrated on the global minima of E m and for β Ñ 0 it approaches the uniform distribution [Mézard and Montanari, 2009]. It is, therefore, to be expected that the minimization of L becomes harder at low temperatures, i.e. as β Ñ inf , which opens the opportunity to a principled curriculum learning approach (Sec. 4). Based on these considerations we reformulate CO problems as the variational problem of approximating p B p σ | E m , β q in the limit β Ñ inf with a variational ansatz p θ that has the variational parameters θ . This problem can be formalized as argmin θ lim β Ñ inf L p θ ; β q .

3 Variational Annealing on Graphs

Our method VAG-CO addresses CO as a variational learning problem on graphs. In particular, given a set of CO problem instances it learns to approximate the Boltzmann distribution of the corresponding Ising models (Sec. 2) with an autoregressive distribution. To obtain efficient training with this expressive model we apply an annealed entropy regularization. We formulate this learning problem in an RL setting and use PPO to train our model. To alleviate the lengthy sampling process of autoregressive approaches we introduce sub-graph tokenization which allows us to generate multiple solution variables at in a single step without loosing expressive power. We further improve the memory efficiency of our approach by dynamically pruning the graph which represents the CO problem instance.

Autoregressive Solution Generation. In the following we specify how we represent p θ p σ | E i q and how to sample it, i.e. how to generate solutions σ .

Draw a problem instance E ' p B i , J ij q ' q p E q from the data distribution q and construct a graph G ' p V, E q based on E . The nodes ν i P V correspond to the spins σ i and the set of edges E represents the non-zero components of J ij . The graph G is equipped with node features x i ' r B i , t i s that are associated to its nodes ν i . Here t i is a four dimensional one-hot encoding which indicates the four possible states (I-IV) of a node ν i namely whether the corresponding spin σ i is set to the value ` 1 (I) or ´ 1 (II) or whether it is to be generated in the current step (III) or afterwards (IV). The edges between nodes ν i and ν j are associated with the scalar edge features J ij .

Order the graph nodes according to the breadth-first search (BFS) algorithm. The i -th node in this ordering is denoted as σ i . Now i ' 1 and t i is set to the state (III) and all t ą i are set to (IV).

A GNN is used to parameterize a Bernoulli distribution p θ p σ i | G q ' GNN θ p G q from which the value ˘ 1 of σ i is sampled. The state encoding variables of G are updated by setting

t i associated to the graph is now updated accordingly and t i ` 1 is set to (III). Now i is incremented.

The previous step is repeated until the values of all σ i are set.

We note that at each step i the graph G depends on the problem instance E and the already generated spins σ ă i . Therefore this procedure represents an autoregressive parametrization of a distribution over the space of possible solutions t´ 1 , 1 u N . By denoting the graph G at step i as G i we get:

This approach is more expressive than MFA and can therefore be expected to be better suited to approximate the typically complex Boltzmann distributions (Eq. 4) encountered in CO. Next we specify how we realize a stable training procedure by employing RL methods. The model architecture and hyperparameters are detailed in App. A.10.

Reinforcement Learning Setting. To explain how we train our model it is convenient to adopt an RL perspective where an episode corresponds to the solution generation procedure described above. For this we consider a Markov Decision Process (MDP) that is given by p S, A, P, R q . Here, S is the set of possible states and the state at step i is denoted by s i P S . At each step s i is represented by the graph G i . Given a state s i an action a i that represents the assignments of the spin value σ i P t´ 1 , 1 u is sampled from a probability distribution which is parameterised by the policy p θ p σ i | G i q . After sampling an action the reward R i p G i , β q is computed according to an objective that is based on Eq. 3.

We use the relation F p θ ; β, E q ' ´ E σ ' p θ r ř N i ' 1 R i p G i , β qs and define the reward at step i as

where ∆ E i ' σ i ' ř j ă i J ij σ j ` B i ı . With this definition maximising the reward is equivalent to minimizing the free-energy in Eq. 3. We approximate F p θ ; β, E q with the empirical mean of ´ ř N i ' 1 R i p G i , β q based on n S samples σ ' p θ p σ i | σ ă i , E q . Finally, the state is changed deterministically, i.e. the spin σ i is set according to the sampled action a i . The state update corresponds to the updated of t i and t i ` 1 as described in step 3 of the solution generation procedure. Our model learns to solve this MDP via the popular PPO algorithm [Schulman et al., 2017] which is described in more detail in App. A.9.

Annealing. As discussed in Sec. 2 our training objective is based on Eq. 3 which contains an entropy regularization term. The strength of this regularization is determined by temperature T . At first, the temperature is kept constant at T ą 0 for N warmup steps. Then, the temperature is slowly adapted for N anneal steps by following a predefined annealing schedule (see App. A.14) that converges to T ' 0 as the end of training is reached. This reduction of the temperature throughout training is motivated from a theoretical point of view in Sec. 4.

Subgraph Tokenization. By introducing subgraph tokenization we decrease the number of forward passes per CO problem instance without sacrificing expressivity. Instead of modelling the probability for the two possible values ˘ 1 of a single spin, we let the policy represent a probability distribution p θ p σ i : i ` k | σ ă i , E q over all 2 k possible configurations of k consecutive spins in the BFS ordering (step 2 of the solution generation procedure). We represent p θ p σ i : i ` k | σ ă i , E q with a softmax function of a 2 k dimensional vector that is output by the GNN (App. A.10). Subgraph tokenization represents a modification of step 3 of the solution generation procedure and yields an improvement of the performance (Sec. 6).

Dynamic Graph Pruning. We note that once the spin value of σ i is sampled its interaction with adjacent spins σ j is fixed. Therefore, assuming that we have sampled the first spin σ 1 , its interaction with an adjacent spin σ j , that is yet to be generated, can be expressed as B p 1 ÞÑ j q j σ j , where we introduce B p 1 ÞÑ j q j ' J 1 j σ 1 . Therefore, we can immediately remove generated spins from the graph and update the node embeddings x j ' r B j , t j s of adjacent nodes with B j Ð B j ` B p i ÞÑ j q j . Reducing the graph size during the solution generation process has the benefit of reducing the memory requirements when processing graphs with a GNN as in step 3 of the solution generation procedure.

4 Theoretical Motivation of Annealing

In the following we show that the concept of an annealed entropy regularization in Eq. 3 can be motivated based the sample complexity of density estimation for Boltzmann distributions.

Consider the problem of approximating the probability density function of a Boltzmann distribution p B p s ; E,β q , where E : R N Ñr 0 , 1 s is a suitably normalized energy function. To solve this task we are given a set of samples S ' t s 1 , . . . , s m u which are independently drawn from the corresponding target Boltzmann distribution p B p s ; E,β ˚ q at a fixed inverse temperature β ˚ . For brevity we will denote the distribution associated to p B p s ; E,β q from now on as p p β q where E is suppressed since it is a fixed function. The empirical distribution corresponding to S will be denoted as ˆ p . Further assume that we can evaluate E p s i q for all s i P S .

In the present context a natural feasibility criterion for the estimated distribution is that the expectation value of the energy function for the estimated distribution should be compatible with the corresponding value E p p β ˚ q E p s q of the target distribution. We approach this problem with the maximum entropy principle [Jaynes, 1957]. Informally, this principle prescribes that among all feasible distributions one should prefer the one that yields the highest entropy. Since S contains only a finite number of samples we cannot determine E p p β ˚ q E p s q with arbitrary accuracy. As already stated in a similar form in [Dudík et al., 2007] one obtains with Hoeffding's inequality that with a probability of 1 ´ δ :

As shown in [Kazama and Tsujii, 2003] the resulting maximum entropy optimization problem with the inequality constraint Eq. (7) is equivalent to the following regularized maximum likelihood problem over the family of Boltzmann distributions: min β L ˆ p p β q ` λβ , where L ˆ p p β q ' ´ E ˆ p log p p β q is the cross-entropy between ˆ p and p p β q . Based on a closely related result in [Dudík et al., 2007] we obtain the following bound for the approximation of maximum entropy distributions:

Remark 1. Assume a bounded energy function E : R N Ñr 0 , 1 s and let ˆ β minimize L ˆ p p β q ` λ | β | where λ ' a ln p 2 { δ q{p 2 m q . Then with a probability at least 1 ´ δ :

See App. A.15.2 for further details. According to Eq. 8 the sample complexity of approximating a maximum entropy distribution at an inverse temperature β is in O p β 2 q .

Curriculum learning is a machine learning paradigm in which the difficulty of training tasks is gradually increased as training proceeds [Bengio et al., 2009]. In view of Theorem 1 the entropy annealing in VAG-CO (Sec. 3) can be regarded as a principled curriculum learning approach if an increased sample complexity is regarded as an indication of a more difficult learning problem. In supervised learning tasks Wu et al. [2021] find that the application of curriculum learning results in more resource efficient training but not necessarily in better performance. As we demonstrate in Sec. 6 our entropy annealing does actually yield better performing models.

5 Related Work

As pointed out in [Yehuda et al., 2020] supervised training of CO solvers faces the problem of expensive data generation and that data augmentation cannot circumvent this problem. Consequently, there is a growing interest in RL and unsupervised methods. In the following we focus on methods that attempt to learn how to generate solutions rather than how to improve existing ones.

Unsupervised Learning and Reinforcement Learning. The work of [Bello et al., 2017] proposes to use an actor-critic approach to learn how to solve CO problems. They were the first to show that deep RL is a promising approach for CO. Their method represents an autoregressive distribution over solutions. However, in their setting rewards are sparse since they are only available for complete solutions. In our approach rewards are dense since they are available after every state transition. Another influential work is [Khalil et al., 2017] who were the first to exploit the graph structure of CO problems by learning problem representations with GNNs. Applying GNNs to CO problem has become a common choice since then [Cappart et al., 2021]. In [Tönshoff et al., 2020] the method

RUN-CSP is introduced. The work of [Karalias and Loukas, 2020] proposes Erd˝ os Goes Neural (EGN) in which the goal is to minimize a loss that can be regarded as a certificate of the existence of a CO problem solution whose cost is upper bounded by the loss. By relying on an MFA they can calculate this loss efficiently. Also [Qiu et al., 2022] use an MFA to optimize a distribution over the solution space and optimize the corresponding parameters with gradient estimates that are obtained by REINFORCE. The work of [Min et al., 2022] uses the same MFA-based concept as EGN and focuses on alleviating the oversmoothing problem of GNNs by using modified Graph Convolutional Networks [Kipf and Welling, 2017]. An approach based on an entry-wise concave continuous relaxation of the discrete CO loss and a subsequent rounding procedure for integral solution generation is introduced in [Wang et al., 2022]. Here the concept of [Karalias and Loukas, 2020] is generalized to a wider class of problems and rather simple rounding procedure is introduced to generate solutions efficiently. This approach is further extended by [Wang and Li, 2023] to a meta-learning setting called Meta-EGN in which network parameters are updated for individual CO problem instances. They also argue that RL based CO methods suffer from unstable training. We call this claim into question by finding no stability issues with our RL-based method. Further V AG-CO outperforms EGN and even Meta-EGN despite not updating any parameters on test problem instances. The approach of extending functions on sets to continuous supports is taken in Neural Set Function Extensions (NSFE) [Karalias et al., 2022]. In NSFE the discrete CO loss function is replaced with a convex combination of the discrete loss function values at certain integral solutions. These extensions can be regarded as expectation values that can be calculated in closed form without sampling. Whether the lack of sampling noise in NSFE is beneficial for the optimization procedure is not obvious.

Variational Annealing. The concept of using autoregressive models in the variational problem of approximating Boltzmann distributions of Ising models was introduced by [Wu et al., 2019]. They show that variational annealing (V A), i.e. the combination of the variational approach and temperature annealing, is a highly performant method. The work of [Hibat-Allah et al., 2021] compares VA to other ones like Simulated Annealing (SA) [Kirkpatrick et al., 1983] and confirms its strong performance on problems related to spin glasses. In [Khandoker et al., 2023] the strong performance of VA compared to SA is confirmed on CO problems. A crucial aspect of these works on VA with respect to ours and the ones in the previous paragraph is that they optimize the parameters of their models only for individual problem instances. They do not attempt to learn how to generalize over a family of problems. The work of [Sun et al., 2022] aims at a generalizing application of VA in CO by using an MFA. Our experiments indicate that the simplicity of MFA-based methods leads to performance limitations in particular on hard CO problem instances.

6 Experiments

We evaluate VAG-CO on various CO problems that are studied in the recent literature. Additionally, we evaluate VAG-CO on synthetic datasets where solving the corresponding CO problem is known to be particularly hard. Finally, we discuss experiments on the impact of entropy regularization and subgraph tokenization. Our result tables also include inference runtimes. A quantitative runtime comparison is, however, difficult since the runtimes reported from other works and for Gurobi 10.0.0 [Gurobi Optimization, LLC, 2023] were obtained with differen setups. See App. A.12 for details on the runtime measurements. For Gurobi we report results for various different runtime limits.

Maximum Independent Set. In the following we will compare VAG-CO on the Maximum Independent Set (MIS) problem to recently published results from [Karalias et al., 2022], where the graph datasets COLLAB, ENZYMES and PROTEINS Morris et al. [2020] are used. In the MIS problem the task is to find the largest subset of independent, i.e. unconnected, nodes in a graph. As an optimization objective for V AG-CO we use Eq. 2 with the Ising energy function for MIS (see Tab. 1). Here, the energy function E p q q consists of two terms E A p q q and E B p q q , that depend on the binary representation of the solution q ' σ ` 1 2 with q P t 0 , 1 u N . When q i ' 1 the corresponding node is defined to be included in the set and it is excluded otherwise. The first term E A p q q is proportional to the number of nodes that are in the set, whereas E B p q q is proportional to the number of independence violations, i.e. the number of connected nodes within the set. By selecting A,B P R ` such that A ă B we ensure that all minima satisfy E B ' 0 since in this case excluding a violating node from the set always reduces the energy. In our experiments, we choose A ' 1 . 0 and B ' 1 . 1 . We follow the procedure of [Karalias et al., 2022] and use a 0 . 6 { 0 . 1 { 0 . 3 train/val/test split on the aforementioned datasets and use only the first 1000 graphs on the COLLAB dataset. The results

are shown in Tab. 2 where the test set average of the best approximation ratio AR ˚ out of n S ' 8 sampled solutions per graph is reported (see App. A.1). This metric was originally proposed in [Karalias and Loukas, 2020]. We also report results of our own implementation of an MFA-based method that is trained with REINFORCE. This method is used with (MFA-Anneal) and without (MFA) annealing (App. A.5). As in Wang and Li [2023] and Karalias and Loukas [2020] we use the conditional expectation procedure (CE, App. A.5.5) to sample solutions and report the corresponding results as (MFA: CE) and (MFA-Anneal: CE). We also add results obtained by the Degree Based Greedy (DB-Greedy) algorithm [Wormald, 1995] that is proposed by [Angelini and Ricci-Tersenghi, 2023] as a baseline for machine learning algorithms on MIS. For VAG-CO we greedily sample different states for a given problem instance by using different BSF orderings of the corresponding graph nodes. Our results show that VAG-CO significantly outperforms all competitors including the MFA on ENZYMES and PROTEINS. On IMDB-BINARY and MUTAG the MFA-based approaches and VAG-CO outperform all other machine learning methods and achieve an optimal AR ˚ . On COLLAB MFA-Anneal and VAG-CO exhibit the best results with insignificantly better results for MFA-Anneal. We also report results based on the test set average of the approximation ratio with n S ' 30 y AR (App. A.1). This metric shows the performance of the learned probability distribution for each model, when no post processing is applied. Our results show that V AG-CO always achieves a significantly better performance in terms of y AR than MFA-based approaches.

Minimum Vertex Cover. We compare VAG-CO to results from Wang and Li [2023] where the Minimum Vertex Cover (MVC) problem is solved on the TWITTER [Leskovec and Krevl, 2014], COLLAB and IMDB-BINARY Morris et al. [2020] graph datasets. The MVC problem is the task of finding the smallest subset of vertices such that each edge has at least one node in the subset. As for the MIS problem we formulate this CO problem in terms using the Ising energy function that is defined in Tab. 1 and set A ' 1 . 0 and B ' 1 . 1 . We follow the procedure of Wang and Li [2023] and use a 0 . 7 { 0 . 1 { 0 . 2 train/val/test split on these datasets. Their method Meta-EGN uses meta

learning in combination with the EGN method. They apply CE to generate solutions and report the best out of n S ' 8 sampled solutions. Additionally, they report fine tuning results denoted by (f-t), where they apply one step of fine tuning individually for each CO problem instance in the test dataset. Our results in Tab. 3 show that V AG-CO improves upon the results reported in Wang and Li [2023] across all datasets. We also show that the MFA with and without annealing performs remarkably well on the TWITTER and COLLAB dataset and even outperforms VAG-CO on the COLLAB dataset. The results in terms of y AR show that the strong performance of the MFA can be attributed to the post-processing method CE rather than to the learned MFA-based model.

Maximum Clique. The Maximum Clique (MaxCl) Problem is the problem of finding the largest set of nodes, in which each node is connected to all other nodes that are contained in the set. The MaxCl problem is equivalent to solving the MIS problem on the complementary graph. The complementary graph can be obtained by connecting nodes that are not connected in the original graph and by disconnecting nodes that are connected. We evaluate our method on the ENZYMES and IMDBBINARY datasets with the same train/val/test splits

that are used in the MIS problem (see Sec. 6). Results are shown in Tab. 4, where we compare to results that are reported in [Karalias et al., 2022] and to our own implementation of MFA and MFA-Anneal. Here, we see that VAG-CO is always among the best performing method in therms of AR ˚ and always the single best method in therms of y AR .

Maximum Cut. The Maximum Cut (MaxCut) problem is the problem of finding two sets of nodes so that the number of edges between these two sets is as high as possible. We evaluate our method on BA graphs [Barabási and Albert, 1999], where we follow [Zhang et al., 2023] and generate 4000/500/500 train/val/test graphs between the size of 200-300 nodes and set the generation parameter m to 4. Results on this dataset are shown in Tab. 4, where we compare our method to MaxCut Values ( MCut ' ř p i,j qP E 1 ´ σ i σ j 2 ) that are reported in [Zhang et al., 2023]. Our method achieves the best { MCut results among all learned methods.

1.000

1.035

100

VAG-CO (ours) t = 0.20

DB-Greedy t= 0.08

VAG-CO w/ ST: To = 5 ·10-2

0.975

1.030

MFA: CE t= 0.10

Gurobi tmax = 0.2

EGN: CE t= 0.10

Gurobi tmax = 0.3

VAG-CO w/ ST: To = 5 ·10-3

￥

0.925

0.950

1.025

1.020

EGN-Anneal: CE t = 0.10

MFA-Anneal: CE t = 0.10

Gurobi tmax = 1.0

VAG-CO w/ ST:To=0

VAG-CO w/o ST : To = 5 ·10-2

R

0.900

R

0.875

1.015

<E

10-1

VAG-CO (ours) t = 0.08

DB-Greedy t = 0.003

1.010

0.850

MFA: CE t= 0.06

Gurobi tmx = 0.08

0.825

EGN: CE t = 0.06

EGN-Anneal: CE t = 0.06

Gurobi tmax = 0.2

Gurobi fmax = 0.5

1.005

0.800

MFA-Anneal: CE t= 0.06

1.000

+--V--

2.5

5.0

7.5

10.012.515.017.5 20.0

d

0.30.40.50.6

p

0.7

0.8

0.91.0

0

number of gradient update steps 1e5

1

2

3

4

5

6

7

Evaluation on Synthetic Problems for MIS. On many of the benchmarks above nearly optimal results are obtained. Therefore, we conduct additional experiments on synthetic graph datasets that yield hard CO problems.

For graphs with a degree d larger than 16 the MIS problem on random regular graphs (RRGs) is known to be particularly hard [Barbier et al., 2013]. Therefore, we generate RRGs with an average size of 100 nodes (RRG-100), where we sample 4100 RRGs between the size of 80 to 120 with different degrees d P r 3 , 7 , 10 , 20 s . For training, validation and testing we use 3000 { 100 { 1000 graphs. Results for the MIS problem on the RRG-100 dataset are shown in Fig. 1 (Left), where we plot the test set average of the best relative error ϵ ˚ rel out of n S ' 8 sampled solutions per problem instance. Here the relative error is calculated with respect to exact solutions that are obtained with the commercial solver Gurobi. Error bars indicate the standard error over graphs with the corresponding node degree. As expected we find that for all methods ϵ ˚ rel increases for larger degrees. MFA and MFA-Anneal outperform the DB-Greedy method. Furthermore, VAG-CO is the best performing method for all graph degrees. We also show Gurobi performance for each d at different time limits t max . On this dataset VAG-CO outperforms Gurobi with the given compute budged (see App. A.12) by a high margin on hard instances.

AR

˚

in dependence

Evaluation on Synthetic Problems for MVC. We also conduct experiments on graphs that are generated by the so-called RB method [Xu et al., 2005]. This method allows the generation of graphs that are known to yield hard MVC problem instances [Wang and Li, 2023]. The RB model has three distinct generation parameters: n, k 1 , and p . Adjusting the values of n and k 1 allows us to control the expected size of the generated graphs, while p serves as a parameter to regulate the hardness of the graph instances. Specifically, when p is close to one, the generated graphs tend to be easier, whereas reducing p leads to increased difficulty. To ensure diversity in the RB dataset, we generate graphs with varying levels of hardness by randomly sampling p from the range of 0 . 25 to 1 . 0 . For our experiments, we utilize the RB-200 dataset, consisting of RB graphs with an average size of 200 nodes with a train/val/test split of 2000 { 500 { 1000 graphs. Following [Wang and Li, 2023] each graph in this dataset is generated by randomly selecting values for n from the range of 20 to 25 , and k 1 from the range of 9 to 10 . Figure 1 (Middle) shows the corresponding results in terms of

of p . We find that the DB-Greedy algorithm outperforms both MFA approaches. The MFA with annealing tends to perform worse than the MFA without annealing on this dataset. Here, we also add our own implementation of EGN and EGN-Anneal [Sun et al., 2022] and find that EGN performs similarly to MFA and EGN-Anneal achieves a similar performance as DB-Greedy. VAG-CO outperforms all other methods for all values of p on this dataset. We also show Gurobi performance for each p at different time limits t max . Here, we see that at high p values Gurobi achieves close to optimal results within a very small time budged and is the best performing method. Whereas for low p values Gurobi performance drops by a high margin and VAG-CO achieves similar results within a comparable time.

Ablation on Annealing. In the following we investigate whether the annealed entropy regularization in Eq. 3 is indeed beneficial. These experiment are conducted for the MVC problem on RB-100 graphs that are generated with n P r 9 , 15 s , k 1 P r 8 , 11 s and p P r 0 . 25 , 1 s . We compare VAG-CO test set learning curves with three different initial temperatures T 0 P t 5 ¨ 10 ´ 2 , 5 ¨ 10 ´ 3 , 0 u . The run with T 0 ' 0 is equivalent to V AG-CO without annealing (App. A.14). Figure 1 (right) shows the average relative error ˆ ϵ rel for n S ' 30 sampled solutions per problem instance on the test set over the number of gradient update steps. We find that the run without annealing starts to converge rather quickly but at a comparably bad relative energy. In contrast to that, the runs with annealing ( T 0 P t 5 ¨ 10 ´ 2 , 5 ¨ 10 ´ 3 uq keep improving over more steps and achieve a better final performance.

Ablation on Subgraph Tokenization. Next we study the impact of subgraph tokenization (see Sec. 3) on the performance of V AG-CO. The same problem setting as in the annealing ablation above is used. Figure 1 (right) compares V AG-CO with the default k ' 5 subgraph tokenization (w/ ST) to VAG-CO without subgraph tokenization (w/o ST). For a fair comparison the numbers of gradient update steps and the annealing schedules are set to be equal. We find that with subgraph tokenization we achieve a considerably better ˆ ϵ rel . These results underpin that subgraph tokenization does indeed yield an improved efficiency in terms of gradient update steps.

We additionally provide a more detailed investigation on ST, where we compare VAG-CO with four different values of k P t 1 , 2 , 5 , 10 u on the RB-100 MVC dataset. In our experiments we keep the number of gradient update steps constant and iteratively tune for the best learning rate and the best initial temperature in each setting of k . Results are shown in Fig. 2, where AR ˚ and the average time per graph are plotted over k . As k increases the performance in terms of AR ˚ does improve and the inference time is reduced considerably. This experiment underscores the performance and scalability improvement due to ST.

7 Limitations

While MFA-based approaches typically generate a solution to a CO problem in a single forward pass VAG-CO requires a number of forward passes that is proportional the problem size N . However, methods with solution generation in a single forward pass frequently rely on post-processing procedures like CE that have a time complexity which is also linear in N . In our approach the nodes in the graph are always processed in an order that is determined by a BFS. Therefore studying the impact of other node orderings or making the algorithm invariant to such orderings might be an interesting future research direction. For the annealing schedule we report comparisons for V AG-CO with three different initial temperatures but a comprehensive study on the optimization of the annealing schedule is left to future investigations.

8 Conclusion

Our results show that learning to solve hard CO problems requires sufficiently expressive approaches like autoregressive models. With VAG-CO we introduce a method that enables stable and efficient training of such a model through an annealed entropy regularization. We provide a theoretical motivation for this regularization method and demonstrate its practical benefit in experiments. In addition, we demonstrate that by systematically grouping solution variables VAG-CO achieves an improved performance with respect to a naive autoregressive model. Importantly VAG-CO outperforms recent approaches on numerous benchmarks and exhibits superior performance on synthetic CO problems that are designed to be hard.

References

Ralph Abboud, ˙ Ismail ˙ Ilkan Ceylan, Martin Grohe, and Thomas Lukasiewicz. The surprising power of graph neural networks with random node initialization. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021 , pages 2112-2118. ijcai.org, 2021. doi: 10.24963/ijcai.2021/291. URL https://doi. org/10.24963/ijcai.2021/291 .

Abien Fred Agarap. Deep learning using rectified linear units (relu). CoRR , abs/1803.08375, 2018. URL http://arxiv.org/abs/1803.08375 .

Sungsoo Ahn, Younggyo Seo, and Jinwoo Shin. Learning what to defer for maximum independent sets. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 134-144. PMLR, 2020. URL http://proceedings.mlr.press/v119/ahn20a.html .

Maria Chiara Angelini and Federico Ricci-Tersenghi. Modern graph neural networks do worse than classical greedy algorithms in solving combinatorial optimization problems like maximum independent set. Nat. Mac. Intell. , 5(1):29-31, 2023. doi: 10.1038/s42256-022-00589-y. URL https://doi.org/10.1038/ s42256-022-00589-y .

Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR , abs/1607.06450, 2016. URL http://arxiv.org/abs/1607.06450 .

Albert-László Barabási and Réka Albert. Emergence of scaling in random networks. Science , 286(5439): 509-512, 1999. doi: 10.1126/science.286.5439.509. URL https://www.science.org/doi/abs/10. 1126/science.286.5439.509 .

Jean Barbier, Florent Krzakala, Lenka Zdeborová, and Pan Zhang. The hard-core model on random graphs revisited. Journal of Physics: Conference Series , 473(1):012021, dec 2013. doi: 10.1088/1742-6596/473/1/ 012021. URL https://dx.doi.org/10.1088/1742-6596/473/1/012021 .

Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinícius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Çaglar Gülçehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew M. Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. CoRR , abs/1806.01261, 2018. URL http://arxiv.org/abs/1806.01261 .

Irwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi, and Samy Bengio. Neural combinatorial optimization with reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings . OpenReview.net, 2017. URL https: //openreview.net/forum?id=Bk9mxlSFx .

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning , ICML '09, page 41-48, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585161. doi: 10.1145/1553374.1553380. URL https://doi.org/10.1145/1553374.1553380 .

Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: A methodological tour d'horizon. Eur. J. Oper. Res. , 290(2):405-421, 2021. doi: 10.1016/j.ejor.2020.07.063. URL https://doi.org/10.1016/j.ejor.2020.07.063 .

Maximilian Böther, Otto Kißig, Martin Taraz, Sarel Cohen, Karen Seidel, and Tobias Friedrich. What's wrong with deep learning in tree search for combinatorial optimization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=mk0HzdqY7i1 .

Quentin Cappart, Didier Chételat, Elias B. Khalil, Andrea Lodi, Christopher Morris, and Petar Velickovic. Combinatorial optimization and reasoning with graph neural networks. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021 , pages 4348-4355. ijcai.org, 2021. doi: 10.24963/ijcai.2021/595. URL https: //doi.org/10.24963/ijcai.2021/595 .

Corinna Cortes, Vitaly Kuznetsov, Mehryar Mohri, and Umar Syed. Structural maxent models. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015 , volume 37 of JMLR Workshop and Conference Proceedings , pages 391-399. JMLR.org, 2015. URL http://proceedings.mlr.press/v37/cortes15.html .

Miroslav Dudík, Steven J. Phillips, and Robert E. Schapire. Maximum entropy density estimation with generalized regularization and an application to species distribution modeling. J. Mach. Learn. Res. , 8: 1217-1260, 2007. doi: 10.5555/1314498.1314540. URL https://dl.acm.org/doi/10.5555/1314498. 1314540 .

Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023. URL https://www.gurobi.com .

Mohamed Hibat-Allah, Estelle M. Inack, Roeland Wiersema, Roger G. Melko, and Juan Carrasquilla. Variational neural annealing. Nat. Mach. Intell. , 3(11):952-961, 2021. doi: 10.1038/s42256-021-00401-3. URL https://doi.org/10.1038/s42256-021-00401-3 .

Tommi S. Jaakkola and Michael I. Jordan. Improving the mean field approximation via the use of mixture distributions. In Michael I. Jordan, editor, Learning in Graphical Models , volume 89 of NATO ASI Series , pages 163-173. Springer Netherlands, 1998. doi: 10.1007/978-94-011-5014-9\_6. URL https://doi. org/10.1007/978-94-011-5014-9_6 .

T. Jaynes. Information theory and statistical mechanics. Phys. Rev. , 106:620-630, May 1957. doi: 10.1103/ PhysRev.106.620. URL https://link.aps.org/doi/10.1103/PhysRev.106.620 .

Nikolaos Karalias and Andreas Loukas. Erdos goes neural: an unsupervised learning framework for combinatorial optimization on graphs. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 49f85a9ed090b20c8bed85a5923c669f-Abstract.html .

Nikolaos Karalias, Joshua Robinson, Andreas Loukas, and Stefanie Jegelka. Neural set function extensions: Learning with discrete functions in high dimensions. In NeurIPS , 2022. URL http://papers.nips.cc/ paper_files/paper/2022/hash/6294a235c0b80f0a2b224375c546c750-Abstract-Conference. html .

Jun'ichi Kazama and Jun'ichi Tsujii. Evaluation and extension of maximum entropy models with inequality constraints. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP 2003, Sapporo, Japan, July 11-12, 2003 , 2003. URL https://aclanthology.org/W03-1018/ .

Elias B. Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial optimization algorithms over graphs. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 6348-6358, 2017. URL https://proceedings.neurips.cc/paper/2017/ hash/d9896106ca98d3d05b8cbdf4fd8b13a1-Abstract.html .

Shoummo Ahsan Khandoker, Jawaril Munshad Abedin, and Mohamed Hibat-Allah. Supplementing recurrent neural networks with annealing to solve combinatorial optimization problems. Mach. Learn. Sci. Technol. , 4(1): 15026, 2023. doi: 10.1088/2632-2153/acb895. URL https://doi.org/10.1088/2632-2153/acb895 .

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum? id=SJU4ayYgl .

Kirkpatrick, C. D. Gelatt, and M. P. Vecchi. Optimization by simulated annealing. Science , 220(4598): 671-680, 1983. doi: 10.1126/science.220.4598.671. URL https://www.science.org/doi/abs/10. 1126/science.220.4598.671 .

Wouter Kool, Herke van Hoof, and Max Welling. Attention, learn to solve routing problems! In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id=ByxBFsRqYm .

Jure Leskovec and Andrej Krevl. SNAP Datasets: Stanford large network dataset collection. http://snap. stanford.edu/data , June 2014.

Fenglin Liu, Xuancheng Ren, Zhiyuan Zhang, Xu Sun, and Yuexian Zou. Rethinking skip connection with layer normalization in transformers and resnets. CoRR , abs/2105.07205, 2021. URL https://arxiv.org/abs/ 2105.07205 .

Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=Skq89Scxx .

Andrew Lucas. Ising formulations of many np problems. Frontiers in Physics , 2, 2014. ISSN 2296-424X. doi: 10.3389/fphy.2014.00005. URL https://www.frontiersin.org/articles/10.3389/fphy.2014. 00005 .

David J. C. MacKay. Information theory, inference, and learning algorithms . Cambridge University Press, 2003. ISBN 978-0-521-64298-9.

Nina Mazyavkina, Sergey Sviridov, Sergei Ivanov, and Evgeny Burnaev. Reinforcement learning for combinatorial optimization: A survey. Comput. Oper. Res. , 134:105400, 2021. doi: 10.1016/j.cor.2021.105400. URL https://doi.org/10.1016/j.cor.2021.105400 .

Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In Alina Beygelzimer and Daniel Hsu, editors, Conference on Learning Theory, COLT 2019, 25-28 June 2019, Phoenix, AZ, USA , volume 99 of Proceedings of Machine Learning Research , pages 2388-2464. PMLR, 2019. URL http://proceedings.mlr.press/v99/mei19a.html .

Yimeng Min, Frederik Wenkel, Michael Perlmutter, and Guy Wolf. Can hybrid geometric scattering networks help solve the maximum clique problem? In NeurIPS , 2022. URL http://papers.nips.cc/paper_ files/paper/2022/hash/8ec88961d36d9a87ac24baf45402744f-Abstract-Conference.html .

Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. CoRR , abs/2007.08663, 2020. URL https://arxiv.org/abs/2007.08663 .

Marc Mézard and Andrea Montanari. Information, Physics, and Computation . Oxford University Press, 01 2009. ISBN 9780198570837. doi: 10.1093/acprof:oso/9780198570837.001.0001. URL https://doi.org/ 10.1093/acprof:oso/9780198570837.001.0001 .

Christos H Papadimitriou and Kenneth Steiglitz. Combinatorial optimization: algorithms and complexity . Courier Corporation, 1998.

G Parisi. Statistical field theory, vol. 66 in. Frontiers in Physics , page 343, 1988.

Ruizhong Qiu, Zhiqing Sun, and Yiming Yang. DIMES: A differentiable meta solver for combinatorial optimization problems. In NeurIPS , 2022. URL http://papers.nips.cc/paper_files/paper/2022/ hash/a3a7387e49f4de290c23beea2dfcdc75-Abstract-Conference.html .

Prabhakar Raghavan. Probabilistic construction of deterministic algorithms: approximating packing integer programs. Journal of Computer and System Sciences , 37(2):130-143, 1988.

John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings , 2016. URL http://arxiv.org/abs/1506.02438 .

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017. URL http://arxiv.org/abs/1707.06347 .

Haoran Sun, Etash Kumar Guha, and Hanjun Dai. Annealed training for combinatorial optimization on graphs. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop) , 2022. URL https: //openreview.net/forum?id=fo3b0XjTkU .

Jan Tönshoff, Martin Ritzert, Hinrikus Wolf, and Martin Grohe. Graph neural networks for maximum constraint satisfaction. Frontiers Artif. Intell. , 3:580607, 2020. doi: 10.3389/frai.2020.580607. URL https://doi. org/10.3389/frai.2020.580607 .

Martin J. Wainwright and Michael I. Jordan. Graphical models, exponential families, and variational inference. Found. Trends Mach. Learn. , 1(1-2):1-305, 2008. doi: 10.1561/2200000001. URL https://doi.org/10. 1561/2200000001 .

Haoyu Wang, Nan Wu, Hang Yang, Cong Hao, and Pan Li. Unsupervised learning for combinatorial optimization with principled objective relaxation. In NeurIPS , 2022. URL http://papers.nips.cc/paper_files/ paper/2022/hash/cbc1ad2066f0afebbcea930c5688fc1f-Abstract-Conference.html .

Haoyu Peter Wang and Pan Li. Unsupervised learning for combinatorial optimization needs meta learning. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview. net/forum?id=-ENYHCE8zBp .

Nicholas C Wormald. Differential equations for random processes and random graphs. The annals of applied probability , pages 1217-1235, 1995.

Dian Wu, Lei Wang, and Pan Zhang. Solving statistical mechanics using variational autoregressive networks. Phys. Rev. Lett. , 122:080602, Feb 2019. doi: 10.1103/PhysRevLett.122.080602. URL https://link.aps. org/doi/10.1103/PhysRevLett.122.080602 .

Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https://openreview.net/forum?id=tW4QEInpni .

Ke Xu, Frédéric Boussemart, Fred Hemery, and Christophe Lecoutre. A simple model to generate hard satisfiable instances. In Leslie Pack Kaelbling and Alessandro Saffiotti, editors, IJCAI-05, Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence, Edinburgh, Scotland, UK, July 30 - August 5, 2005 , pages 337-342. Professional Book Center, 2005. URL http://ijcai.org/Proceedings/05/Papers/ 0989.pdf .

Gal Yehuda, Moshe Gabel, and Assaf Schuster. It's not what machines can learn, it's what we cannot teach. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 10831-10841. PMLR, 2020. URL http://proceedings.mlr.press/v119/yehuda20a.html .

Dinghuai Zhang, Hanjun Dai, Nikolay Malkin, Aaron C. Courville, Yoshua Bengio, and Ling Pan. Let the flows tell: Solving graph combinatorial optimization problems with gflownets. CoRR , abs/2305.17010, 2023. doi: 10.48550/arXiv.2305.17010. URL https://doi.org/10.48550/arXiv.2305.17010 .

A Appendix

A.1 Evaluation Metrics

We use different metrics to evaluate the performance of models on CO problems. We assess the quality of an individual solution σ by the associated value of the energy function E m p σ q which represents the size of the solution sets in MIS and MVC. Here m refers to the problem instance under consideration. Optimal solutions σ opt are obtained with the Gurobi solver [Gurobi Optimization, LLC, 2023]. For models that generate solutions indeterministically we sample n S different solutions σ j per problem instance and calculate the test dataset average of the best relative error ϵ ˚ rel :

where M denotes the number of problem instances in the test dataset. In case of deterministic algorithms only one sample is generated, i.e. n S ' 1 . In several experiments it is insightful to investigate the average relative error ˆ ϵ rel for which we take the average instead of the minimum in Eq. 9. Analogously, we also define the best approximation ratio AR ˚ by

The average approximation ratio y AR is defined by taking the average instead of the minimum operation in Eq. 10. For VAG-CO we calculate the average over the ordered greedy (OG) sampling method (See. App A.7). We always report the evaluation metric together with the standard error over three different seeds except for the experiment on RRG-100 MIS, where only results on one seed are reported.

A.2 Linear Integer Program Formulations

For Gurobi we formulate the CO Problems studied in this paper in Linear Integer Program Formulation (LIP) (see Tab. 5).

A.3 Ensuring Feasible Solutions

Since there is no rigorous guarantee that the model samples only feasible solutions that satisfy the constraints, we use a fast post processing procedure to make sure that only feasible solutions are sampled. Here, we make use of our choice of the relative weighting of the energy terms A and B (see Tab. 1) in the Ising formulation, which ensures that only feasible solutions are minima in the energy landscape. Therefore, we can detect violations when E B ą 0 and search for the spin that causes the largest amount of violations. Subsequently, we change the spin value of the node with the highest number of violations to satisfy the constraint and repeat the process until E B ' 0 . We observe in our experiments that this post-processing step is typically unnecessary, since only in rare cases violating solutions are sampled.

A.4 Standardization of the Energy Scale

Since the energy scale of CO problems can vary significantly, a good choice of hyperparameters like the initial temperature T 0 , learning rate and relative weighting between the policy and value loss can vary between different CO problem instances. Therefore, we standardize the energy scale that makes the choice of good hyperparameters easier. For this purpose we first express the binary energy function E p q q (see. Tab 1) in terms of spins, i.e. as E p σ q by substituting q i with σ i ` 1 2 . We then sample states for CO problem instances from the training set by using a Random Greedy Algorithm (RGA, see App. A.6.2). Since we use only a few RGA steps (see App. A.6.2), this algorithm performs only slightly better than a completely random algorithm. From these states the mean energy ˆ E RGA and the standard deviation std p E RGA q over the training dataset is computed. Subsequently, we standardize the energy scale by:

A.5 Baseline Methods

We compare VAG-CO with our own implementation of a Mean Field Approximation (MFA) and to Erdos Goes Neural (EGN). In both of these approaches we consider the case with and without annealing. In both methods the probability of a state σ factorizes into a product of independent Bernoulli probabilities. Therefore, the state probability is given by:

While EGN and MFA both use the same probability distribution they differ in how the energy, or in case of annealing the free energy, is calculated. This is described in the following.

A.5.1 Mean Field Approximation

We estimate the energy by sampling states from the parameterized probability distribution and apply REINFORCE with variance reduction to update the network parameters. The network parameter gradients for one CO Problem Instance are then calculated with

where b ' E σ ' p θ p σ | E q r E p σ s . For a batch of CO Problem Instances ∆ θ p E q is averaged over multiple E .

A.5.2 MFA with Annealing

In MFA-Anneal we estimate the free energy instead and again update the network parameters with REINFORCE with variance reduction as explained above.

A.5.3 EGN

Since we consider cases, where the energy function is known in Ising formulation (see Sec. 1) the expectation E σ ' p θ p σ | E q r E p σ s can be written down in closed form by substituting σ i Ñ 2 ¨ p θ p σ i | E q ´ 1 [Karalias and Loukas, 2020]. Therefore, the loss function for one CO Problem Instance can be computed with

and the network parameters can be updated directly by computing the gradients of this loss function.

A.5.4 EGN with Annealing

In case of EGN-Anneal [Sun et al., 2022] the entropy of the parameterized probability distribution is calculated in closed form with the following formula:

A.5.5 Conditional Expectation

To obtain good samples from the MFA approach in a deterministic way, we adopt the Conditional Expectation (CE) method [Raghavan, 1988] as described in Karalias and Loukas [2020].

To introduce randomness into the CE-based solution generation procedure we initialize the random node features (which are processed by the GNN) of every node with a random vector with six binary entries that are drawn from a uniform distribution. Unless stated otherwise, we follow the procedure in [Wang and Li, 2023] and report the best CE result of eight different random node feature initializations.

A.6 Greedy Algorithms

A.6.1 Degree Based Greedy Algorithm

The Degree Based Greedy (DB-Greedy) algorithm [Wormald, 1995] is a polynomial time algorithm for the MIS problem. The DB-Greedy algorithm works in the following way: At first all nodes are sorted according to their degrees. Then, starting from the smallest degree the node is chosen to be part of the independent set. In the next step this node, its neighbors and their corresponding edges are deleted from the graph. These steps are repeated until the graph is empty.

This algorithm can also be applied to the MVC problem by using the fact that the complement of an independent set is a vertex cover. In other words, nodes in the independent set are excluded from the vertex cover and nodes that are not part of the independent set are included into the vertex cover.

A.6.2 Random Greedy Algorithm

The Random Greedy Algorithm (RGA) is a general approach that can be utilized for solving a broad range of CO problems that can be mapped to the Ising model. Initially, the algorithm randomly samples spin values with uniform probability. Then, for a fixed number of iterations a spin is randomly selected and its value is changed if it decreases the energy value. We set the number of iterations to N ¨ n R , where N is the number of nodes in the graph and n R is the number of repetitions per node.

For the purpose of the standardization the energy scale (see. App. A.4), we employ this algorithm with n R set to one.

A.7 Study on Sampling Methods

In order to find out, how the best samples can be obtained with VAG-CO we study in Fig. 3 three different sampling methods on the RRG-100 MIS and RB-200 MVC dataset. Here we evaluate ϵ ˚ rel on the test dataset and plot it over the number of samples n S that are used for each graph in the dataset. We also add MFA and DB-Greedy results, where we show for MFA how the method improves when more solutions are sampled. Since DB-Greedy is a deterministic algorithm, we draw a horizontal line that indicates the solution quality of the algorithm. To draw a relation to results that are presented in Fig. 1 (Left, Middle) we also add a vertical dotted line at n S ' 8 samples. For VAG-CO we denote the first method as sampling (S), where for each graph n S solutions are sampled according to the corresponding probability distribution. In the second V AG-CO sampling method called ordered sampling (OS), we use for each graph n O different BFS node orderings and sample n S { n O states per graph ordering. Finally, we sample V AG-CO solutions with a method called ordered greedy (OG), where we generate solutions greedily for each BFS ordering with n O ' n S . Results in Fig. 3 show that the sampling strategy OG always outperforms all other sampling strategies that we proposed for V AG-CO. Additionally, we see that as we increase n O in the OS sampling strategy, the performance improves consistently. Remarkably, DB-Greedy exhibits the best solution quality when MFA and VAG-CO are allowed only one sample ( n S ' 1 ). However, DB-Greedy is outperformed by VAG-CO OG already with a modest amount of n S ą 1 samples.

A.8 Experimental Details

In this section we provide additional details to experiments that are presented in Sec. 6.

Parameter Checkpointing. With VAG-CO we checkpoint over the course of training the parameters that obtain the best ϵ ˚ rel and the best ˆ ϵ rel on the validation set. For testing we always use the checkpoint with the best ˆ ϵ rel , except for the results in Fig. 3 when the sampling strategies (S) and (OS) are used. Here we use the checkpoints with the best ϵ ˚ rel .

Hyperparameter Tuning. For MFA-Anneal, the learning rate and initial temperature are tuned on the validation dataset of Enzymes MIS via a grid search. We considered learning rates ( lr P r 5 ˆ 10 ´ 4 , 1 ˆ 10 ´ 4 , 5 ˆ 10 ´ 5 s ) and initial temperatures ( T 0 P r 0 . 5 , 0 . 25 , 0 . 1 , 0 . 05 s ). With that, the number of GNN layers ( L P r 6 , 8 , 12 s ) are tuned. Finally, the annealing duration ( N anneal ) is increased until we observe that longer annealing does not lead

RRG-100 MIS Dataset

RB-200MVCDataset

10-1

DB-Greedy

VAG-CO:OG

2 × 102

DB-Greedy

VAG-CO:OG

VAG-CO:S

MFA-Anneal: CE

米

VAG-CO:S

MFA-Anneal: CE

VAG-CO: OS; no= 2

MFA: CE

VAG-CO: OS; no = 2

MFA: CE

VAG-CO:OS;no = 8

VAG-CO: OS;no = 8

102

3

6 ×10-3

102

4× 10-3

3 ×10-3

0

20

40

ns

60

80

100

0

20

40

ns

60

80

100

to improvements. For MFA, we tested on ENZYMES MIS the learning rates ( lr P r 5 ˆ 10 ´ 4 , 1 ˆ 10 ´ 4 , 5 ˆ 10 ´ 5 , 1 ˆ 10 ´ 5 s ) and with that tuned the number of GNN layers ( L P r 6 , 8 , 10 , 12 s ) and stop training when no significant improvement on the validation set is observed. In MFA and MFA-Anneal, when training on the other datasets we started with the optimal ENZYMES MIS parameters and further tested different learning rates ( lr ), initial temperatures ( T 0 ) and number of GNN layers ( L ) individually on each dataset. For experiments on the RRG-100 MIS and RB-200 MVC we make a more extensive hyperparameter search. In RRG-100 MIS for MFAAnneal we first tuned the initial temperature in the range of T 0 P r 0 . 25 , 0 . 1 , 0 . 05 , 0 . 035 , 0 . 025 , 0 . 015 , 0 . 01 s and in RB-200 MVC we search for the best initial temperature within the range of T 0 P r 0 . 5 , 0 . 25 , 0 . 1 , 0 . 05 s . For MFA and MFA-Anneal, we then iteratively search for the best learning rate in the range of lr P r 5 ˆ 10 ´ 4 , 1 ˆ 10 ´ 4 , 5 ˆ 10 ´ 5 s and then for the best number of GNN layers L P r 6 , 8 , 12 s .

In the experiments with EGN on RB-200 MVC we also performed a extensive hyperparameter seach, where first the best temperature T 0 was tuned within the range of T 0 P r 0 . 25 , 0 . 1 , 0 . 05 s . Then we tuned the best learning rate within the range of lr P r 5 ¨ 10 ´ 4 , 1 ¨ 10 ´ 1 , 5 ¨ 10 ´ 5 , 1 ¨ 10 ´ 5 s . Then we tuned for the best number of GNN layers withing the range of L P r 8 , 10 , 12 , 14 s . In EGN without annealing the same hayperparameter search was performed but at T 0 ' 0 . The best hyperparameters of all methods are listed in App. A.13.

On VAG-CO we iteratively tuned the learning rate ( lr P r 5 ¨ 10 ´ 4 , 1 ¨ 10 ´ 3 s ), initial temperature ( T 0 P r 0 . 05 , 0 . 7 , 0 . 1 s ), number of GNN layers ( L P r 3 , 4 s ) and number of annealing steps ( N anneal P r 3000 , 6000 s ) on the ENZYMES MIS validation dataset. We then initially test these hyperparameters on other datasets and adapt the number of annealing steps and the initial temperature. The choice of all hyperparameters in V AG-CO is listed in Tab. 7.

Ablation on Subgraph Tokenization. In the ablation on subgraph tokenization in Fig. 1 we keep all hyperparameters the same ,except for the time horizon T (see Sec. A.9) and the hyperparameter λ in the PPO algorithm (see App. A.9). For the subgraph tokenization run we chose T ' 20 , λ ' 0 . 95 and k ' 5 . Therefore, the time horizon includes the generation of T ¨ k ' 100 spins per graph during the data collection phase (see App. A.9). If we would keep T the same, when we chose k ' 1 for the run without subgraph tokenization only T ¨ k ' 20 spins would be generated for each graph. Therefore, we use T ' 100 when no subgraph tokenization is used. As we always set λ ' 1 ´ 1 T this hyperparameter is adapted accordingly.

Experiments on Random Regular Graphs. Since GNNs suffer from node ambiguity on Random Regular Graphs (RRGs) Wang and Li [2023] using random node features Abboud et al. [2021] can resolve this issue. Therefore, for the V AG-CO experiment in Fig. 1 (Left) we sample standard Gaussian random node features with the dimension of six and concatenate them to the graph representation of V AG-CO. For MFA and MFA-Anneal random node features are already used (see App. A.5.5) and had not to be added for the RRG experiments.

Averaged results on hard instances. In Fig. 1 (Left, Middle) only show results for specific generation parameters p on the RB-200 dataset and for specific values of d on the RRG-100 dataset. Here we report the corresponding averaged results on the RRG-100 MIS and RB-200 MVC dataset in Tab. 6. VAG-CO significantly outperforms all other methods for n S ' 8 .

A.9 Proximal Policy Optimization

Proximal Policy Optimization (PPO) [Schulman et al., 2017] is a popular RL algorithm that has two main components. The policy is represented by the network p θ p σ i | G i q with parameters θ . The expected future reward is estimated by the value network V ϕ p G i q which is parameterized by ϕ .

Data Collection. In PPO the policy is rolled out through the environment in order to collect and store the states, actions, probabilities and the output of the value function into the rollout buffer. The data collection procedure is completed after going through T steps, where T is the so-called time horizon. Then, the advantages A i : ' A p σ i , G i q and value targets V T i : ' V T p G i q are calculated by making use of the Generalised Advantage Estimation (GAE, Schulman et al. [2016]), where A i is given by

with δ i ' R i ` γV p G i ` 1 q ´ V p G i q and V T i ' A i ` V p G i q . In our experiments the reward R i is given by Eq. 6. In our experiments we chose γ ' 1 . 0 and λ ' 1 ´ 1 T . We set T to be approximately equal to the average graph size in the dataset. During the data collection phase we always collect data from H ' 30 different CO problem instances and for each instance we sample n S ' 30 solutions.

Rollout Buffer. For the gradient updates in PPO, the loss is estimated with minibatches that are randomly sampled from the rollout buffer. As the rollout buffer contains data of size H ˆ n S ˆ T we chose to sample data by first sampling H minib graphs and then for each graph, we chose to sample N minib solutions and for each solution S minib time steps are sampled. We report the minibatch settings of each experiment in Tab. 7.

Training. As described before, during training minibatches of data D minib is randomly sampled from the replay buffer. In PPO the overall loss that depends on the L p θ, ϕ ; D minib q is given by L p θ, ϕ ; D minib q ' L 1 p θ ; D minib q ` c V L 2 p ϕ ; D minib q , where the first term depends on the policy and the second term on the value function. To specify a minibatch sample from D minib , we will use the index n .

The policy loss for one minibatch sample is then defined as

where the importance sampling ratio I n p θ q ' p θ p σ n | G n q p θ 1 p σ n | G n q is used to compute a weighted Monte Carlo estimate of A n . Here, θ 1 represents the old policy parameters that were used to fill the rollout buffer during the data collection phase and θ are the new parameters that are used in gradient descend. In Eq. 14 the clipping function is used to prevent that the probability of the current policy p θ p σ n | G n q differs by more than a factor 1 ˘ ϵ from the old policy p θ 1 p σ n | G n q .

Similarly, the value function loss for one sample is given by L 2 p ϕ q n ' p V ϕ p G n q ´ V T ϕ 1 p G n qq 2 .

The loss L p θ, ϕ ; D minib q is then updated for each minibatch in the rollout buffer so that each sample is used at least once. This procedure is overall repeated n repeat times, before we increment the epoch step N epoch and new data is collected with the updated set of parameters. In our experiments we always chose ϵ ' 0 . 1 , c V ' 0 . 5 and n repeat ' 2 .

A.10 Model Details and GNN Architectures

Before processing the graph representation with GNNs, we encode its node features with an encoder MLP. To obtain node embeddings N ⟩ l P R d p l q with l P r 1 , . . . , L s that depend on the graph structure, the encoded node features are processed by L layers of GNNs. Afterwards, we apply a global sum aggregation to compute a global graph embedding. The global graph embedding is then concatenated to the node embedding N L i , where i is the index of the spin whose value is to be generated (see Sec. 3). We then use this embedding to calculate the policy and value function outputs (see App. A.9) with two separate MLPs.

In our experiments, we employ two distinct message passing architectures. The first one is a Message-passing Neural Network (MPNN) [Battaglia et al., 2018] but with linear message functions. The second architecture also includes skip connections (MPNN-skip) [Battaglia et al., 2018] which we use when more than three GNN layers are used.

Message Passing Neural Network. Our MPNN layer is defined by the following update for node embeddings:

where Φ p N l j , N l i , E ij q is a message update MLP that computes messages with pairwise interactions between nodes. Skip connection are implemented by adding the term W skip N L i after the node update, where W skip P R d p l ` 1 qˆ d p l q is a weight matrix. Finally, layer normalization ln r¨s [Ba et al., 2016] is applied as it is commonly done in Neural Network architectures when skip connections are used [Liu et al., 2021].

In VAG-CO we use a policy and value MLP with three layers. Both MLPs have 120 neurons in the first two layers. The value MLP has only one neuron in the output layer and the policy MLP has 2 k output neurons. For the encoder MLP we use a two layer MLP with 40 neurons in each layer. In case of the node update MLP Ψ and the message update MLP Φ we use 2 layers with 64 neurons in each layer. Layer normalization ln r¨s is applied after every layer within a MLP except in the output layer of the policy and value MLP. We use Rectified Linear Unit (ReLU) activation functions Agarap [2018] except in the output of the policy MLP, where a softmax activation is used and except of the output of the value MLP where no activation is used.

In MFA with and without annealing the model uses an encoder MLP with one layer and 64 neurons. The node update MLP Ψ and the message update MLP Φ have the same number of neurals as in VAG-CO. The output MLP that is applied on each node after n GNN message passing steps the has three layers with 64 neurons each and the final output layer has 2 neurons with a softmax activation.

A.10.1 Subgraph Tokenization

For subgraph tokenization we have to make changes to the one-hot encoding as described in Sec. 3 and also to the model architecture as described in App. A.10. The one-hot encoding in the graph representation is adapted so that spins σ i : i ` k that are going to be generated receive an enumeration that indicates their position in the sliced list of BFS-ordered (see Sec. 3]) indices i : i ` k . Then, the graph G i is processed by a GNN that provides a node embedding GNN θ p x i q for each node. Along with a global sum aggregation, the node embeddings of the spins that are going to be generated are then concatenated according to their BFS order (Sec. 3) and further processed by the policy MLP that calculates the probability for each of the 2 k spin configurations using a softmax output layer.

When the number of nodes N in the graph is not dividable by k , the number vertices in the CO problem instance description has to be increased without changing the inherent optimization objective. This can be realised by adding a sufficient amount of spins into the Ising model (see Sec. 2) with zero spin weight B and no connections to other spins.

A.11 Training Time and Computational Resources

All runs with VAG-CO were conducted either on A100 Nvidia GPU with 40 GB Memory or an A40 Nvidia GPUs with 48 GB Memory. In case of COLLAB MVC an A100 with 80 GB Memory is used.

The training time of our algorithm depends hyperparameters like the number of annealing steps N anneal (see App. A.14), the number of edges and nodes of graphs in the dataset, on the GPUs that are used during training, on the time horizon and on the minibatch size that is used for gradient updates in PPO (see App. A.9). For example the TWITTER MVC run with N anneal ' 4000 , T ' 30 , H minib ' N minib ' S minib ' 10 takes one day, when trained on an A100 40 GB Nvidia GPU. The run on ENZYMES MIS trained on a A40 GPU takes ten hours, when trained with N anneal ' 6000 , T ' 20 , H minib ' N minib ' 15 and S minib ' 10 .

A.12 Time measurement

Gurobi. All Gurobi results are conductedon a Intel Xeon Platinum 8168 @ 2.70GHz CPU with 24 cores. All Problems expect for MaxCut are formualted as LIP and we set the number of threads for Gurobi to 26.

Learned Methods. For all learned methods (EGN, MFA, VAG-CO) we present the time per graph as if the algorithm are implemented in a parallelized manner, where each sampling process runs on a separate thread. We always report runtimes by using the jit compiled performance of the Graph Neural Network.

A.13 Hyperparameters

VAG-CO Hyperparameters.

All hyperparameters that change across datasets are listed in Tab. 7, whereas hyperparameters that stay the same are specified in the corresponding section (e.g. App. A.10, A.14).

MFA Hyperparameters.

All runs with MFA used a batch size H of 32 and we sampled n S ' 30 solutions. Furthermore, we used 8 GNN layers, and 6 random node features per node. In the RB-200 MVC experiment 10 GNN layers were used. For MFA-Anneal we used a learning rate of 1 ˆ 10 ´ 4 , 2000 annealing steps N anneal and a start temperature T 0 of 0 . 1 except for RRG-100 MIS and for RB-200 MVC. In RRG-100 MIS a T 0 of 0 . 015 is used and in RB-200 MVC T 0 ' 0 . 05 . For MFA with and without annealing we trained for 2000 epochs and used a learning rate of 5 ˆ 10 ´ 5 except for ENZYMS MIS and RB-200 MVC, where a learning rate of 1 ˆ 10 ´ 4 is used.

EGN Hyperparameters.

All runs with EGN on RB-200 MVC have a batch size of H ' 32 . Here, in EGN-Anneal we use 12 GNN layers and in EGN without annealing 10 GNN layers performes best. In EGN-Anneal the initial temperature of T 0 ' 0 . 1 with the learning rate of 5 ˆ 10 ´ 5 obtained the best results. With EGN without annealing the same learning rate is used. Both methods are trained for 2000 epochs.

A.14 Annealing Schedule

As described in Sec. 3, we change the temperature in the reward Eq. 6 according to a predefined annealing schedule. During the warm-up phase of N warmup epochs the temperature is held constant at the initial temperature T 0 . Afterwards the following temperature schedule is applied:

Here, T anneal p N epoch q is the gradually decreasing amplitude of the temperature oscillations:

1.0

0.8

Temperature

0.6

0.4

0.2

0.0

0

200

400

600

800

1000

1200

Nepoch

Here c is a scaling factor that determines the slope of T anneal p N epoch q . The parameter λ determines the number temperature oscillations in the schedule and N anneal is the total number of epochs that follow after the warmup phase. We use λ ' 3 , N warmup ' 400 and c ' 6 . The course of the annealing schedule is illustrated in Fig. 4. One reason for the usage of cosine modulation function is rather practical, namely that T ' 0 is reached multiple times during training, which allows an assessment of the training success at an earlier stage for a given set of hyperparmeters. In the absence of cosine modulation, we found it harder to assess the final performance before the entire annealing was finished. Similar periodic schedules for learning rates have been proposed as variants of Stochastic Gradient Descent [Loshchilov and Hutter, 2017].

A.15 Derivations

A.15.1 Free-Energy Decomposition into Rewards

In the following we show that using the reward defined in Eq. 6 is consistent with the goal of minimizing the free-energy defined in Eq. 2.

The right-hand side of Eq. 2 contains the expectation of the energy E p σ q and a term that is proportional to the entropy of p θ p σ q . For the energy we obtain the following decomposition into individual steps i of the solution generation process (Sec. 3):

By using an autoregressive factorization the entropy can also be decomposed in the following way:

Therefore, we can use this decomposition by using the reward R i p G i , β q ' ´ ' ∆ E i ` 1 β log p p σ i | σ ă i , E q ı .

By the relation F p θ ; β, E q ' ´ E σ ' p θ r ř i R i p G i , β qs , maximizing this reward will then be equivalent to minimizing the free-energy.

A.15.2 Details on Remark 1

We first restate Corollary 5 of [Dudík et al., 2007] in our notation. In the context of this Corollary our energy function E can be regarded as a single feature and, therefore, we use their result for n ' 1 . In addition, we consider the case in which the samples S are drawn from the target distribution, i.e. π ' p p β ˚ q . Since the terms Boltzmann distribution and Gibbs distribution can be used interchangeably we obtain:

Corollary 1 (Corollary 5 of [Dudík et al., 2007]) . Assume that E is bounded in r 0 , 1 s . Let ˆ β minimize L ˆ p p β q ` λ | β | with λ ' a ln p 2 { δ q{p 2 m q . Then with probability at least 1 ´ δ for every Boltzmann distribution p p β q ,

Now we consider the result for the case β ' β ˚ and subtract L p p β ˚ q p β ˚ q . Remark 1 follows by applying the definition of the Kullback-Leibler divergence D KL to the left-hand side.

We note that an equivalent statement, however, with a consideration the Rademacher complexity of E can be derived from Theorem 2 in Cortes et al. [2015].

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses RL and Annealing to train a learned CO solver. They demonstrate that their algorithm empirically performs well over many problem types and dataset types.\n\nSTRENGTHS:\n1. This paper uses several important baselines.\n2. The breadth of the datasets chosen is good.\n\nWEAKNESSES:\n1. You are missing a comma in the first sentence.\n2. Use \\citep to put parentheses around your citations\n3. You misspell Reinforce in the table.\n4. The experiments seem relatively limited in that comparison is only for two problem types. Moreover, the authors do not compare with Gurobi to get a range on how solvers that are not learned perform on these problems. Also, the time taken by each algorithm is not used to compare these algorithms. Indeed, these metrics are important to see to compare.\n5. I do not understand the contribution of this paper. Indeed, it seems that this paper combines annealing from Sun et al. and RL for CO, which is also known. The subgraph solving seems to reflect many classical CO algorithms and is not new. The subgraph tokenization may be novel. In general, I'm a little confused about what the overall contribution of this paper is in the context of the literature.\n6. Also, the improvement in the solving ratio seems very small, enough such that hyperparameter tuning your algorithm and not tuning the baselines can make this difference.  \n\nI might also change the title of the paper. Currently, it closely resembles \"Annealed Training for Combinatorial Optimization on Graphs\" by Sun et. al.\n\nQUESTIONS:\n1. During the RL process, how do you compute the reward of an intermediate graph where spins for certain nodes have not been assigned? Do you only compute the reward for nodes that have assigned spins?\n\nOverall, due to the limitations in the experiments, issues with experiments, and lack of contribution, I am voting to reject this paper. However, if the authors can run some further experiments comparing this paper with Gurobi and including the time taken by each algorithm, I believe this paper will be improved. I am flexible, so if the authors adequately address my concerns, I will increase my score accordingly.\n\nLIMITATIONS:\nThey have addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present Variational Annealing on Graphs for Combinatorial Optimization (VAG-CO), a novel method for tackling combinatorial optimization problems. This method presumably combines elements of variational inference, annealing, and graph theory to form an effective optimization approach.\n\n=====\nAfter reading the rebuttal, the authors have convinced me to increase my score.\n\nSTRENGTHS:\nNovel Methodology: The paper presents a new approach to solve combinatorial optimization problems, Variational Annealing on Graphs for Combinatorial Optimization (VAG-CO), which is innovative and adds value to the existing literature.\n\nOvercoming Limitations of Existing Methods: The authors identify the limitations of the widely used Mean-Field Approximation (MFA) and propose a way to overcome these, which demonstrates a deep understanding of the problem space.\n\nImprovement in Efficiency: The use of sub-graph tokenization and entropy regularization to improve the efficiency of the training and inference process could potentially revolutionize the way these types of problems are solved.\n\nWEAKNESSES:\n1. The experiments are only completed on synthetic data. It will be more convincing if the proposed method can be applied and compared on some real data.\n2. Maximum Independent Set (MIS) and Minimum Vertex Cover (MVC) are well-known problems in graph theory and are widely used as benchmark problems in combinatorial optimization. However, they only represent a subset of the vast array of combinatorial optimization problems. A more extensive evaluation of the proposed method would involve its application to a larger and more diverse set of problems. This would enable a more comprehensive understanding of its capabilities and limitations. If the authors want to claim the wide application of the proposed method, they may want to conduct experiments for Traveling Salesman Problem, Knapsack Problem, Vehicle Routing Problem etc.\n\nQUESTIONS:\nHow do the authors define hard problems in the paper? What are the criteria?\n\nLIMITATIONS:\nThe experiments are not extensive.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the use of autoregressive deep generative models for solving combinatorial problems.\n\nStarting from an optimization problem, the authors reformulate it using a Boltzmann distribution on the solution set.\nThis distribution can then be approximated with a parametric family, typically a neural network, following a variational approach (minimization of the free energy).\n\nWhile many previous works use the independent (or \"mean field\") approximation, it is suggested that building the solution iteratively is more expressive, and thus performs better.\nIn this case, the autoregressive model used is a graph neural network, which gives a conditional probability for the current spin given past choices.\nTo increase inference speed, two tricks are introduced: subgraph tokenization (making several decisions at once) and dynamic graph pruning (removing fixed vertices from the graph). \n\nTraining is achieved using a temperature annealing scheme which favors gradual concentration around global optima.\nThis annealing process enjoys an interpretation in terms of sample complexity and curriculum learning.\n\nNumerical experiments on the maximum independent set and the minimum vertex cover problem show the promise of this new method.\n\nSTRENGTHS:\n### Originality\n\nThis paper describes a new combination of known ideas, coherently motivated by the specific goal of solving hard optimization problems:\n\n- formulation of optimization in the language of statistical physics\n- variational approximation of the Boltzmann distribution\n- iterative construction of a solution\n- RL framework with partial rewards\n- temperature annealing \n\nAs far as I can tell, the training tricks related to tokenization and pruning seem to be novel.\n\n### Quality\n\nThe proposed model makes theoretical sense, and rests on solid foundations of previous work.\nI particularly enjoyed the interpretation of annealing as curriculum learning: as the temperature gets lower, the learning gets harder.\n\nAs for the numerical experiments, they appear to be thorough and include a wide variety of algorithms.\nThe authors also took care to test on harder instances whenever nearly optimal solutions were too easy to reach.\nThey do however exhibit a few inconsistencies, which I point out below.\n\n### Clarity\n\nn.a.\n\n### Significance\n\nThis paper contributes to a wider discussion on the best variational ansatz for combinatorial optimization.\nComparing mean field with autoregression is an important first step, and hopefully the field can evolve towards more generic structured representations.\n\nSubgraph tokenization is also an interesting idea, which deserves to be pushed further.\n\nWEAKNESSES:\n### Originality\n\n### Quality\n\nA questionable algorithmic choice is the arbitrary order in which nodes are processed.\nIt isn't obvious why BFS ordering of the vertices makes more sense than other options.\nThe same goes for subgraph tokenization: why take the $k$ next vertices in the BFS order, even though they might be very far apart from each other?\n\nAs for the experiments, the realistic datasets of the first batch are seemingly too easy, since every method implemented by the authors (including the baseline DB-Greedy) shows near-perfect accuracy.\nThe results of Table 2 are very surprising to me, as both EGN and MFA are mean field methods with conditional expectation decoding, yet they exhibit wildly different performances on the minimum AR.\nOn the other hand, the average AR is only reported for new methods, not for the state of the art.\nI would welcome more detail from the authors on this point.\nIn any case, the benefits of autoregression as opposed to independence are not definitively proven by this series of benchmarks.\n\nThe synthetic datasets of the second batch are designed to be harder to solve, but the associated plots do not include other methods from the state of the art, which were analyzed on realistic datasets only.\nAgain, clarification from the authors would be very welcome.\n\n### Clarity\n\nThe notations are sometimes a bit hard to follow.\nSeveral algorithms included in the benchmark suite are only mentioned earlier in the text.\n\n### Significance\n\nDepending on the validity and fairness of the benchmarks, the results may be less significant than announced by the authors. \nI look forward to the rebuttal period to enlighten this aspect.\n\nQUESTIONS:\nL121: Why BFS in particular? Is there a way to make the order itself parametric?\n\nL160: Is there a way to perform subgraph tokenization that does not scale exponentially with $k$?\n\nL198: Are there other insights from curriculum learning that we could draw inspiration from? I'm not at all familiar with this literature\n\nL215: What if our problem provides no natural way to define partial rewards?\n\nL312: Why change the evaluation metric from approximation ratio to relative error? Not sure I grasp the difference\n\nLIMITATIONS:\nThe authors do not discuss many limitations of their work, although they mention the additional complexity.\n\nSocietal impact is irrelevant here.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This submission proposes a novel unsupervised framework for solving graph combinatorial optimization problems. The proposed method is coined as VAG-CO, which autoregressively generates solutions to CO problems via annealing / reinforcement learning. Corresponding theoretical analysis is provided and numerical demonstration on various datasets are conduct.

STRENGTHS:
- This paper is well written, and the technique is solid.
- The proposed autoregressive generation is appropriate for solving graph CO problems whose solutions are a set over the nodes, which could be represented with a binary vector. 
- The authors use RL to optimize and carefully design an MDP to achieve that.
- An open sourced implementation is also provided.
- Solid technical analysis is provided.
- Various experiments on both simulated and realistic benchmark and on both MIS and MVC tasks.

WEAKNESSES:
This submission is great. A few issues are listed as follows.
- It seems the method is also applicable for other CO problems such as Minimum Dominate Set. Would it possible for the authors to elaborate about at least the possibility?
- It would be also good to evaluate under some subset of the MIS benchmark (https://github.com/maxiboether/mis-benchmark-framework).

QUESTIONS:
See above.

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-14