Title: Are Graph Neural Networks Optimal Approximation Algorithms?
Venue: NeurIPS 2024
Accepted: yes
Keywords: Combinatorial Optimization

Graph Neural Networks

Unsupervised Learning
Abstract: In this work we design graph neural network architectures that capture optimal
approximation algorithms for a large class of combinatorial optimization problems,
using powerful algorithmic tools from semidefinite programming (SDP). Concretely, we prove that polynomial-sized message-passing algorithms can represent
the most powerful polynomial time algorithms for Max Constraint Satisfaction
Problems assuming the Unique Games Conjecture. We leverage this result to
construct efficient graph neural network architectures, OptGNN, that obtain high quality approximate solutions on landmark combinatorial optimization problems
such as Max-Cut, Min-Vertex-Cover, and Max-3-SAT. Our approach achieves
strong empirical results across a wide range of real-world and synthetic datasets
against solvers and neural baselines. Finally, we take advantage of OptGNN’s
ability to capture convex relaxations to design an algorithm for producing bounds
on the optimal solution from the learned embeddings of OptGNN.

Are Graph Neural Networks Optimal Approximation Algorithms?

Abstract

In this work we design graph neural network architectures that capture optimal approximation algorithms for a large class of combinatorial optimization problems, using powerful algorithmic tools from semidefinite programming (SDP). Concretely, we prove that polynomial-sized message-passing GNN's can learn the most powerful polynomial time algorithms for Max Constraint Satisfaction Problems assuming the Unique Games Conjecture. We leverage this result to construct efficient graph neural network architectures, OptGNN, that obtain high-quality approximate solutions on landmark combinatorial optimization problems such as Max-Cut, Min-Vertex-Cover, and Max-3-SAT. Our approach achieves strong empirical results across a wide range of real-world and synthetic datasets against solvers and neural baselines. Finally, we take advantage of OptGNN's ability to capture convex relaxations to design an algorithm for producing bounds on the optimal solution from the learned embeddings of OptGNN.

1 Introduction

Combinatorial Optimization (CO) is the class of problems that optimize functions subject to constraints over discrete search spaces. They are often NP-hard to solve and to approximate, owing to their typically exponential search spaces over nonconvex domains. Nevertheless, their important applications in science and engineering (Gardiner et al., 2000; Zaki et al., 1997; Smith et al., 2004; Du et al., 2017) has engendered a long history of study rooted in the following simple insight. In practice, CO instances are endowed with domain-specific structure that can be exploited by specialized algorithms (Hespe et al., 2020; Walteros & Buchanan, 2019; Ganesh & Vardi, 2020). In this context, neural networks are natural candidates for learning and then exploiting patterns in the data distribution over CO instances.

The emerging field at the intersection of machine learning (ML) and combinatorial optimization (CO) has led to novel algorithms with promising empirical results for several CO problems. However, similar to classical approaches to CO, ML pipelines have to manage a tradeoff between efficiency and optimality. Indeed, prominent works in this line of research forego optimality and focus on parametrizing heuristics (Li et al., 2018; Khalil et al., 2017; Yolcu & Póczos, 2019; Chen & Tian, 2019) or by employing specialized models (Zhang et al., 2023; Nazari et al., 2018; Toenshoff et al., 2019; Xu et al., 2021; Min et al., 2022) and task-specific loss functions (Amizadeh et al., 2018;

Stefanie Jegelka

Karalias & Loukas, 2020; Wang et al., 2022; Karalias et al., 2022; Sun et al., 2022). Exact ML solvers that can guarantee optimality often leverage general techniques like branch and bound (Gasse et al., 2019; Paulus et al., 2022) and constraint programming (Parjadis et al., 2021; Cappart et al., 2019), which offer the additional benefit of providing approximate solutions together with a bound on the distance to the optimal solution. The downside of those methods is their exponential worst-case time complexity. Striking the right balance between efficiency and optimality is quite challenging, which leads us to the central question of this paper:

Are there neural architectures for efficient combinatorial optimization that can learn to adapt to a data distribution over instances yet capture algorithms with optimal worst-case approximation guarantees?

OptGNN

Minimize loss over

training data

Training:

Inference:

OptGNN

To answer this question, we build on the extensive literature on approximation algorithms and semidefinite programming (SDP) which has led to breakthrough results for NP-hard combinatorial problems, such as the Goemans-Williamson approximation algorithm for Max-Cut (Goemans & Williamson, 1995) and the use of the Lovász theta function to find the maximum independent set on perfect graphs (Lovász, 1979; Grötschel et al., 1981). For several problems, it is known that if the Unique Games Conjecture (UGC) is true, then the approximation guarantees obtained through SDP relaxations are indeed the best that can be achieved (Raghavendra, 2008; Barak & Steurer, 2014). The key insight of our work is that a polynomial time message-passing algorithm (see Definition G) approximates the solution of an SDP with the opti- mal integrality gap for the class of Maximum Constraint Satisfaction Problems (Max-CSP), assuming the UGC. This algorithm can be naturally parameterized to build a graph neural network, which we call OptGNN.

Our contributions can be summarized as follows:

We construct a polynomial-time message passing algorithm (see Definition G) for solving the SDP of Raghavendra (2008) for the broad class of maximum constraint satisfaction problems (including Max-Cut, Max-SAT, etc.), that is optimal barring the possibility of significant breakthroughs in the field of approximation algorithms.

We construct a graph neural network architecture which we call OptGNN, to capture this message-passing algorithm. We show that OptGNN is PAC-learnable with polynomial sample complexity.

We describe how to generate optimality certificates from the learned representations of OptGNN i.e., provable bounds on the optimal solution.

Empirically, OptGNN is simple to implement (see pseudocode in Appendix 5) and we show that it achieves competitive results on 3 landmark CO problems and several datasets against classical heuristics, solvers and state-of-the-art neural baselines 3 . Furthermore, we provide out-of-distribution (OOD) evaluations and ablation studies for OptGNN that further validate our parametrized message-passing approach.

2 Background and Related Work

Optimal Approximation Algorithms. Proving that an algorithm achieves the best approximation guarantee for NP-hard problems is an enormous scientific challenge as it requires ruling out that better algorithms exist (i.e., a hardness result). The Unique Games Conjecture (UGC) (Khot, 2002) is a striking development in the theory of approximation algorithms because it addresses this obstacle. If true, it implies approximation hardness results for a large number of hard combinatorial problems that match the approximation guarantees of the best-known algorithms (Raghavendra & Steurer, 2009b; Raghavendra et al., 2012). For that reason, those algorithms are also sometimes referred to as UGC-optimal. More importantly, the UGC implies that for all Max-CSPs there is a general UGC-optimal algorithm based on semidefinite programming (Raghavendra, 2008). For a complete exposition on the topic of UGC and approximation algorithms, we refer the reader to Barak & Steurer (2014).

Neural approximation algorithms and their limitations. There has been important progress in characterizing the combinatorial capabilities of modern deep learning architectures, including bounds on the approximation guarantees achievable by GNNs on bounded degree graphs (Sato et al., 2019, 2021) and conditional impossibility results for solving classic combinatorial problems such as Max-Independent-Set and Min-Spanning-Tree (Loukas, 2019). It has been shown that a GNN can implement a distributed local algorithm that straightforwardly obtains a 1/2-approximation for MaxSAT (Liu et al., 2021), which is also achievable through a simple randomized algorithm (Johnson, 1973). Recent work proves there are barriers to the approximation power of GNNs for combinatorial problems including Max-Cut and Min-Vertex-Cover (Gamarnik, 2023) for constant depth GNNs. Other approaches to obtaining approximation guarantees propose avoiding the dependence of the model on the size of the instance with a divide-and-conquer strategy; the problem is subdivided into smaller problems which are then solved with a neural network (McCarty et al., 2021; Kahng et al., 2023).

Convex Relaxations and Machine Learning. Convex relaxations are crucial in the design of approximation algorithms. In this work, we show how SDP-based approximation algorithms can be incorporated into the architecture of neural networks. We draw inspiration from the algorithms that are used for solving low-rank SDPs (Burer & Monteiro, 2003; Wang et al., 2017; Wang & Kolter, 2019; Boumal et al., 2020). Beyond approximation algorithms, there is work on designing differentiable Max-SAT solvers via SDPs to facilitate symbolic reasoning in neural architectures (Wang et al., 2019). This approach uses a fixed algorithm for solving a novel SDP relaxation for Max-SAT, and aims to learn the structure of the SAT instance. In our case, the instance is given, but our algorithm is learnable, and we seek to predict the solution. SDPs have found numerous applications in machine learning including neural network verification (Brown et al., 2022), differentiable learning with discrete functions (Karalias et al., 2022), kernel methods (Rudi et al., 2020; Jethava et al., 2013) and quantum information tasks (Kriváchy et al., 2021). Convex relaxation are also instrumental in integer programming where branch-and-bound tree search is guaranteed to terminate with optimal integral solutions to Mixed Integer Linear Programs (MILP). Proposals for incorporating neural networks into the MILP pipeline include providing a 'warm start' (Benidis et al., 2023) to the solver, learning branching heuristics (Gasse et al., 2019; Nair et al., 2020; Gupta et al., 2020; Paulus et al., 2022), and learning cutting plane protocols (Paulus et al., 2022). A recent line of work studies the capabilities of neural networks to solve linear programs (Chen et al., 2022; Qian et al., 2023). It is shown that GNNs can represent LP solvers, which may in turn explain the success of learning branching heuristics (Qian et al., 2023). In a similar line of work, neural nets are used to learn branching heuristics for CDCL SAT solvers (Selsam & Bjørner, 2019; Kurin et al., 2020; Wang et al., 2021). Finally, convex optimization has also found applications (Numeroso et al., 2023) in the neural algorithmic reasoning paradigm (Veliˇ ckovi´ c et al., 2022) where neural networks are trained to solve problems by learning to emulate discrete algorithms in higher dimensional spaces.

Learning frameworks for CO. A common approach to neural CO is to use supervision either in the form of execution traces of expert algorithms or labeled solutions (Li et al., 2018; Selsam et al., 2018; Prates et al., 2019; Vinyals et al., 2015; Joshi et al., 2019, 2020; Gasse et al., 2019; Ibarz et al., 2022; Georgiev et al., 2023). Obtaining labels for combinatorial problems can be computationally costly which has led to the development of neural network pipelines that can be trained without access to labels or partial solutions. This includes approaches based on Reinforcement Learning (Ahn et al., 2020; Böther et al., 2022; Barrett et al., 2020, 2022; Bello et al., 2016; Khalil et al., 2017; Yolcu &

Póczos, 2019; Chen & Tian, 2019), and other self-supervised methods (Brusca et al., 2023; Karalias et al., 2022; Karalias & Loukas, 2020; Tönshoff et al., 2022; Schuetz et al., 2022a,b; Amizadeh et al., 2019; Dai et al., 2020; Sun et al., 2022; Wang et al., 2022; Amizadeh et al., 2018; Gaile et al., 2022). Our work falls into the latter category since only the problem instance is sufficient for training and supervision signals are not required. For a complete overview of the field, we refer the reader to the relevant survey papers (Cappart et al., 2023; Bengio et al., 2021).

3 Optimal Approximation Algorithms with Neural Networks

We begin by showing that solving the Max-Cut problem using a vector (low-rank SDP) relaxation and a simple projected gradient descent scheme amounts to executing a message-passing algorithm on the input graph. We then generalize this insight to the entire class of Max-CSPs. We reformulate the UGC-optimal SDP for Max-CSP in SDP 1. Our main Theorem 3.1 exhibits a message passing algorithm (Algorithm 1) for solving SDP 1. We then capture Algorithm 1 via a message passing GNN with learnable weights (see Definition G for definition of Message Passing GNN). Thus, by construction OptGNN captures algorithms with UGC-optimal approximation guarantees for MaxCSP. Furthermore, we prove that OptGNN is efficiently PAC-learnable (see Lemma 3.1) as a step towards explaining its empirical performance.

3.1 Solving Combinatorial Optimization Problems with Message Passing

In the Max-Cut problem, we are given a graph G = ( V, E ) with N vertices V and edges E . The goal is to divide the vertices into two sets that maximize the number of edges going between them. This corresponds to the quadratic integer program

The global optimum of the integer program is the Max-Cut. Noting that discrete variables are not amenable to the tools of continuous optimization, a standard technique is to 'lift' the quadratic integer problem: replace the integer variables x i with vectors v i ∈ R r and constrain v i to lie on the unit sphere

This nonconvex relaxation of the problem admits an efficient algorithm Burer & Monteiro (2003). Furthermore, all local minima are approximately global minima (Ge et al., 2016) and variations of stochastic gradient descent converge to its optimum (Bhojanapalli et al., 2018; Jin et al., 2017) under a variety of smoothness and compactness assumptions. Specifically, for large enough r (Boumal et al., 2020; O'Carroll et al., 2022), simple algorithms such as block coordinate descent (Erdogdu et al., 2019) can find an approximate global optimum of the objective. Projected gradient descent is a natural approach for solving the minimization problem in equation 1. In iteration t (for T iterations), update vector v i as

where NORMALIZE enforces unit Euclidean norm, η ∈ R + is an adjustable step size, and N ( i ) the neighborhood of node i . The gradient updates to the vectors are local, each vector is updated by aggregating information from its neighboring vectors (i.e., it is a message-passing algorithm).

OptGNN for Max-Cut. Our main contribution in this paper builds on the following observation. We may generalize the dynamics described above by considering an overparametrized version of the gradient descent updates in equations 2. Let M 1 , M 2 , ..., M T ∈ R r × 2 r be a set of T learnable matrices corresponding to T layers of a neural network. Then for layer t and embedding v i we define a GNN update

More generally, we can write the dynamics as v t +1 i := NORMALIZE ( M t ( AGG ( v t i , { v t j } j ∈ N ( i ) )) , where AGG is a function of the node embedding and its neighboring embeddings. We will present a message passing algorithm 1 that generalizes the dynamics of 2 to the entire class of Max-CSPs (see example derivations in Appendix A and Appendix B), which provably converges in polynomial iterations for a reformulation of the canonical SDP relaxation of Raghavendra (2008) (see SDP 1). Parametrizing this general message-passing algorithm will lead to the definition of OptGNN (see Definition 3.3).

3.2 Message Passing for Max-CSPs

Given a set of constraints over variables, Max-CSP asks to find a variable assignment that maximizes the number of satisfied constraints. Formally, a Constraint Satisfaction Problem Λ = ( V , P , q ) consists of a set of N variables V := { x i } i ∈ [ N ] each taking values in an alphabet [ q ] and a set of predicates P := { P z } z ⊂V where each predicate is a payoff function over k variables X z = { x i 1 , x i 2 , ..., x i k } . Here we refer to k as the arity of the Max-k-CSP. We adopt the normalization that each predicate P z returns outputs in [0 , 1] . We index each predicate P z by its domain z . The goal of Max-k-CSP is to maximize the payoff of the predicates.

where we normalize by the number of constraints so that the total payoff is in [0 , 1] . Therefore we can unambiguously define an ϵ -approximate assignment as an assignment achieving a payoff of OPT -ϵ . Since our result depends on a message-passing algorithm, we will need to define an appropriate graph structure over which messages will be propagated. To that end, we will leverage the constraint graph of the CSP instance: Given a Max-k-CSP instance Λ = ( V , P , q ) a constraint graph G Λ = ( V, E ) is comprised of vertices V = { v ϕ,ζ } for every subset of variables ϕ ⊆ z for every predicate P z ∈ P and every assignment ζ ∈ [ q ] | z | to the variables in z . The edges E are between any pair of vectors v ϕ,ζ and v ϕ ′ ,ζ ′ such that the variables in ϕ and ϕ ′ appear in a predicate together. For instance, for a SAT clause ( x 1 ∨ x 2 ) ∧ x 1 ∧ x 3 there are four nodes v 1 , v 12 , v 3 and v ∅ with a complete graph between { v 1 , v 12 , v ∅ } and v 3 an isolated node.

Let SDP (Λ) be the optimal value of the SDP 1 on instance Λ . The approximation ratio for the Max-k-CSP problem achieved by the SDP 1 is min Λ ∈ Max-k-CSP OPT (Λ) SDP (Λ) , where the minimization is taken over all instances Λ with arity k . There is no polynomial time algorithm that can achieve a larger approximation ratio assuming the truth of the UGC Raghavendra (2008). We construct our message passing algorithm as follows. First we introduce the definition of the vector form SDP and its associated quadratically penalized Lagrangian.

Definition (Quadratically Penalized Lagrangian) . Any standard form SDP Λ can be expressed as the following vector form SDP for some matrix V = [ v 1 , v 2 , . . . , v N ] ∈ R N × N .

For any SDP in vector form we define the ρ quadratically penalized Lagrangian to be

We show that gradient descent on this Lagrangian L ρ ( V ) for the Max-CSP SDP 1 takes the form of a message-passing algorithm on the constraint graph that can provably converge to an approximate global optimum for the SDP (see algorithm 1). We see that gradient descent on L ρ takes the form of a simultaneous message passing update on the constraint graph. See equation 60 and algorithm 3 for analytic form of the Max-CSP message passing update. See appendix A and B for analytic form of Min-Vertex-Cover and Max-3-SAT message passing updates. Our main theorem is then following.

Theorem 3.1. [Informal] Given a Max-k-CSP instance Λ represented in space Φ = O ( |P| q k ) , there is a message passing Algorithm 3 on constraint graph G Λ with a per iteration update time of O (Φ) that computes in O ( ϵ -4 Φ 4 ) iterations an ϵ -approximate solution (solution satisfies constraints to error ϵ achieving objective value within ϵ of optimum) to SDP 1. For the formal theorem and proof see Theorem C.1.

3.3 OptGNN for Max-CSP

Next we define OptGNN for Max-CSP. See A and B for OptGNN for Vertex Cover and 3-SAT.

Definition (OptGNN for Max-CSP) . Let Λ be a Max-CSP instance on a constraint graph G Λ with N nodes. Let U be an input matrix of dimension r × N for N nodes with embedding dimension r . Let L ρ be the penalized lagrangian loss defined as in equation 6 associated with the Max-CSP instance Λ . Let M be the OptGNN weights which are a set of matrices M := { M 1 , M 2 , ..., M T } ∈ R r × 2 r . Let LAYER M i : R r × N → R r × N be the function LAYER M i ( U ) = M i ( AGG ( U )) , where AGG : R r × N → R 2 r × N is the aggregation function AGG ( U ) := [ U, ∇L ρ ( U )] . We define OptGNN ( M, Λ) : R r × N → R to be the function

The per iteration update time of OptGNN is O (Φ r ω ) where ω is the matrix multiplication exponent. We update the parameters of OptGNN by inputting the output of the final layer LAYER M T into the Lagrangian L ρ and backpropagate its derivatives. We emphasize the data is the instance Λ and not the collection of vectors U which can be chosen entirely at random. The form of the gradient ∇L ρ is a message passing algorithm over the nodes of the constraint graph G Λ . Therefore, OptGNN is a message passing GNN over G Λ (see Definition G). This point is of practical importance as it is what informs out implementation of the OptGNN architecture. We then arrive at the following corollary.

Corollary 1 (Informal) . Given a Max-k-CSP instance Λ represented in space Φ = O ( |P| q k ) , there is an OptGNN ( M, Λ) with T = O ( ϵ -4 Φ 4 ) layers, and embeddings of dimension Φ such that there is an instantiation of learnable parameters M = { M t } t ∈ [ T ] that outputs a set of vectors V satisfying the constraints of SDP 1 and approximating its optimum to error ϵ . See formal statement 2

Moving on from our result on representing approximation algorithms, we ask whether OptGNN is learnable. That is to say, does OptGNN approximate the value of SDP 1 when given a polynomial amount of data? We provide a perturbation analysis to establish the polynomial sample complexity of PAC-learning OptGNN. The key idea is to bound the smoothness of the polynomial circuit AGG used in the OptGNN layer which is a cubic polynomial analogous to linear attention. We state the informal version below. For the formal version see Lemma E.5.

Lemma 3.1 (PAC learning) . Let Q be a dataset of Max-k-CSP instances over an alphabet of size [ q ] with each instance represented in space Φ . Here the dataset D := Λ 1 , Λ 2 , ..., Λ Γ ∼ D is drawn i.i.d from a distribution over instances D . Let M be a set of parameters M = { M 1 , M 2 , ..., M T } in a parameter space Θ . Then for T = O ( ϵ -4 Φ 4 ) , for Γ = O ( ϵ -4 Φ 6 log 4 ( δ -1 )) , let ̂ M be the empirical loss minimizing weights for arbitrary choice of initial embeddings U in a bounded norm ball. Then we have that OptGNN is (Γ , ϵ, δ ) -PAC learnable. That is to say the empirical loss minimizer EMP ( Q ) is within ϵ from the distributional loss with probability greater than 1 -δ :

We note that this style of perturbation analysis is akin to the VC theory on neural networks adapted to our unsupervised setting. Although it's insufficient to explain the empirical success of backprop, we believe our analysis sheds light on how architectures that capture gradient iterations can successfully generalize.

OptGNN in practice . Figure 1 summarizes the OptGNN pipeline for solving CO problems. OptGNN computes node embeddings V ∈ R N × r as per equation 7 which feeds into the loss L ρ . For pseudocode, please refer to the appendix (Max-Cut: algorithm 4 and general SDPs: algorithm 5). We use Adam (Kingma & Ba, 2014) to update the parameter matrices M . Given a training distribution D , the network is trained in a completely unsupervised fashion by minimizing E G ∼D [ L ( v ; G )] with a standard automatic differentiation package like PyTorch (Paszke et al., 2019). A practical benefit of our approach is that users do not need to reimplement the network to handle each new problem. Users need only implement the appropriate loss, and our implementation uses automatic differentiation to compute the messages in the forward pass. At inference time, the output embeddings must be rounded to a discrete solution. To do this, we select a random hyperplane vector y ∈ R r , and for each node with embedding vector v i , we calculate its discrete assignment x i ∈ {-1 , 1 } as x i = sign ( v ⊤ i y ) . We use multiple hyperplanes and pick the best resulting solution.

SDP 1 SDP for Max-k-CSP (Equivalent to UGC-optimal)

SDP Vector Formulation Λ = ( V , P , { 0 , 1 } ) . Multilinear formulation of objective.

Notation: For a Max-CSP instance Λ , there is a canonical boolean polynomial representing the objective denoted P ( x 1 , ..., x N ) = ∑ P z ∈P P z ( X ) where P z is the canonical polynomial associated to the predicate over the variables z . Conversely, for each predicate P ∈ P we let S ( P ) denote the variables in the domain of P . We write out the polynomial P z with coefficients { w s } s ⊆ z as follows P z = ∑ s ⊆ z w s ∏ i ∈ s x i . We introduce the set notation ζ ( A,B ) := A ∪ B/A ∩ B and use C ( s ) to denote the size of the set { g, g ′ ⊆ s : ζ ( g, g ′ ) = s } .

Algorithm 1 Message Passing for Max-CSP (informal).

Input: Max-CSP Instance Λ with constraint graph G Λ v 0 ← Uniform ( S n -1 ) {Initialization on the unit sphere} for t = 1 , 2 , . . . , T do for v t i ∈ v t {message passing on the constraint graph} do

end for end for

Output: vector solution v t for SDP 1

Notation: L ρ ( V ) can be written as the sum of functions across pairs of edges in the constraint graph G Λ i.e L ρ ( V ) = ∑ i ∈ V ∑ e,e ′ ∈ E ( i ) Y ρ ( e, e ′ ) where E ( i ) is the set of edges involving vertex i , and e denotes the vector embeddings of its two corresponding vertices. For analytic form of the messages see algorithm 3

Obtaining neural certificates. We construct dual certificates of optimality from the output embeddings of OptGNN. The certificates provide a bound on the optimal solution of the SDP. The key idea is to estimate the dual variables of SDP 1. Since we use a quadratic penalty for constraints, the natural dual estimate is one step of the augmented method of Lagrange multipliers on the SDP solution which can be obtained straightforwardly from the primal problem variables V . We then analytically bound the error in satisfying the KKT conditions of SDP 1. See C.1 for derivations and extended discussion, and Figure 3 for an experimental demonstration.

Adaptation to Data vs. Theoretical Result . The theory result concerns worst case optimality where the embedding dimension is as large as N . In practice we deploy OptGNN with varying choices of the embedding dimension to obtain the best performance. For low dimensional embeddings, the corresponding low rank SDP objective is NP-hard to optimize. As such, training OptGNN is like training any other GNN for CO in that the weights adapt to the data. This is highlighted in our experiments where OptGNN generally outperforms SDP solvers.

4 Experiments

First, we provide a comprehensive experimental evaluation of the OptGNN pipeline on several problems and datasets and compare it against heuristics, solvers, and neural baselines. We then describe a model ablation study, an out-ofdistribution performance (OOD) study, and an experiment with our neural certification scheme. We empirically test the performance of OptGNN on NP-Hard combinatorial optimization problems: Maximum Cut , Minimum Vertex Cover , and Maximum 3-SAT . We obtain results for several datasets and compare against greedy algorithms, local search, a state-of-the-art MIP solver (Gurobi), and various neural baselines. For Max-3-SAT and Min-Vertex-Cover we adopt the quadratically penalized Lagrangian loss of their respective SDP relaxations. For details of the setup see Appendix D.

Min-Vertex-Cover experiments. We evaluated OptGNN on forced RB instances, which are hard vertex cover instances from the RB random CSP model that contain hidden optimal solutions (Xu et al., 2007). We use two distributions specified in prior work (Wang & Li, 2023), RB200 and RB500. The results are in Figure 2b, which also includes the performance of several neural and classical baselines as reported in (Wang & Li, 2023; Brusca et al., 2023). OptGNN consistently outperforms state-of-the-art unsupervised baselines on this task and is able to match the performance of Gurobi with a 0.5s time limit.

Max-3-SAT experiment and ablation. The purpose of this experiment is twofold: to demonstrate the

viability of OptGNN on the Max-3-SAT problem and to examine the role of overparameterization in OptGNN. We generate 3-SAT formulae on the fly with 100 variables and a random number of clauses in the [400 , 430] interval, and train OptGNN for 100,000 iterations. We then test on instances with 100 variables and { 400 , 415 , 430 } clauses. The results are Table 2. We compare with WalkSAT, a classic local search algorithm for SAT, a low-rank SDP solver (Wang et al., 2019), Survey Propagation (Braunstein et al., 2005), and ErdosGNN Karalias & Loukas (2020), a neural baseline trained in the same way. We also compare with a simple baseline reported as 'Autograd' that directly employs gradient descent on the penalized Lagrangian using the autograd functionality of Pytorch. For details see D.2. OptGNN is able to outperform Erd˝ osGNN consistently and improves significantly over Autograd, which supports the overparameterized message passing of OptGNN. OptGNN performs better than the low-rank SDP solver, though does not beat WalkSAT/Survey Prop. It is worth noting that the performance of OptGNN could likely be further improved without significant computational cost by applying a local search post-processing step to its solutions but we did not pursue this further in order to emphasize the simplicity of our approach.

Max-Cut experiments. Table 1 presents a comparison between OptGNN, a greedy algorithm, and Gurobi for Max-Cut. OptGNN clearly outperforms greedy on all datasets and is competitive with Gurobi when Gurobi is restricted to a similar runtime. For results on more datasets see subsection D.3. Following the experimental setup of ANYCSP Tönshoff et al. (2022), we also tested OptGNN on the GSET benchmark instances (Benlic & Hao, 2013). We trained an OptGNN for 20k iterations on generated Erd˝ os-Renyi graphs G n,p for n ∈ [400 , 500] and p = 0 . 15 . Figure 2a shows the results. We have included an additional low-rank SDP baseline to the results, while the rest of the baselines are reproduced as they appear in the original ANYCSP paper. These include state-of-the-art neural baselines, the Goemans-Williamson algorithm, and a greedy heuristic. We can see that OptGNN outperforms the SDP algorithms and the greedy algorithm, while also being competitive with the

0

100

200

300

400

500

600

700

800

Mean deviation from known optimal (number of edges; lower is better)

Greedy

SDP

Low-Rank SDP

RUNCSP

ECO-DQN

ECORD

ANYCSP

OptGNN

N

=800

N

=1000

N

=2000 (NA shown as 0)

N

3000 (NA shown as 0)

(a) Comparison on GSET Max-Cut instances against state-of-the-art neural baselines. Numbers reported are the mean (over the graphs in the test set) deviations from the best-known Max-Cut values, reported in Benlic & Hao (2013).

1.00

1.02

1.04

1.06

1.08

1.10

1.12

1.14

Approximation ratio (values closer to 1 are better)

CMP

Erd sGNN

Meta-EGN

RUN-CSP

Greedy MVC

Gurobi (0.5s)

Gurobi (1s)

Gurobi (5s)

OptGNN

RB200

RB500

(b) Average approximation ratio and standard deviation over the test set for vertex covers on forced RB instances. A ratio of 1.000 represents finding the minimum vertex cover.

neural baselines. However, OptGNN does not manage to outperform ANYCSP, which to the best of our knowledge achieves the current state-of-the-art results for neural networks.

Out of distribution generalization. We test OptGNN's ability to perform on data distributions (for the same optimization problem) that it's not trained on. The results can be seen in table 7 and subsection D.6. The results show that the model is capable of performing well even on datasets it has not been trained on.

Model ablation. We train modern GNN architectures from the literature with the same loss function and compare them against OptGNN. Please see Appendix D.4 for more details and results on multiple datasets for two different problems. Overall, OptGNN is the best performing model on both problems across all datasets.

Experimental demonstration of neural certificates. Next, we provide a simple experimental example of our neural certificate scheme on small synthetic instances. Deploying this scheme on Max-Cut on random graphs, we find this dual certificate to be remarkably tight. figure 3 shows an example. For 100 node graphs with 1000 edges our certificates deviate from the SDP certificate by about 20 nodes but are dramatically faster to produce. The runtime is dominated by the feedforward of OptGNN which is 0 . 02 seconds vs. the SDP solve time which is 0 . 5 seconds on cvxpy. See C.1 for extensive discussion and additional results.

5 Conclusion

We presented OptGNN, a GNN that can capture provably optimal message passing algorithms for a large class of combinatorial optimization problems. OptGNN achieves the appealing combination of obtaining approximation guarantees while also being able to adapt to the data to achieve improved results. Empirically, we observed that the OptGNN architecture achieves strong performance on a wide range of datasets and on multiple problems. OptGNN opens up several directions for future exploration, such as the design of powerful rounding procedures that can secure approximation guarantees, the construction of neural certificates that improve upon the ones we described in Appendix C.1, and the design of neural SDP-based branch and bound solvers.

6 Acknowledgment

The authors would like to thank Ankur Moitra, Sirui Li, and Zhongxia Yan for insightful discussions in the preparation of this work. Nikolaos Karalias is funded by the SNSF, in the context of the project "General neural solvers for combinatorial optimization and algorithmic reasoning" (SNSF grant number: P500PT_217999). Stefanie Jegelka and Nikolaos Karalias acknowledge support from NSF AI Institute TILOS (NSF CCF-2112665). Stefanie Jegelka acknowledges support from NSF award 2134108.

Certificate Quality:SDPvs.Opt-GNN-certN=100

Opt-GNN-cert

680

SDP-cert

660

640

Value

620

600

580

560

580

600

620

640

Opt-GNN

References

Sungsoo Ahn, Younggyo Seo, and Jinwoo Shin. Learning what to defer for maximum independent sets. In International Conference on Machine Learning , pp. 134-144. PMLR, 2020.

Saeed Amizadeh, Sergiy Matusevych, and Markus Weimer. Learning to solve circuit-sat: An unsupervised differentiable approach. 2018.

Saeed Amizadeh, Sergiy Matusevych, and Markus Weimer. Pdp: A general neural framework for learning constraint satisfaction solvers, 2019.

Boaz Barak and David Steurer. Sum-of-squares proofs and the quest toward optimal algorithms. arXiv preprint arXiv:1404.5236 , 2014.

Thomas Barrett, William Clements, Jakob Foerster, and Alex Lvovsky. Exploratory combinatorial optimization with reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pp. 3243-3250, 2020.

Thomas D Barrett, Christopher WF Parsonson, and Alexandre Laterre. Learning to solve combinatorial graph partitioning problems via efficient exploration. arXiv preprint arXiv:2205.14105 , 2022.

Irwan Bello, Hieu Pham, Quoc V Le, Mohammad Norouzi, and Samy Bengio. Neural combinatorial optimization with reinforcement learning. arXiv preprint arXiv:1611.09940 , 2016.

Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: a methodological tour d'horizon. European Journal of Operational Research , 290(2):405-421, 2021.

Konstantinos Benidis, Ugo Rosolia, Syama Rangapuram, George Iosifidis, and Georgios Paschos. Solving recurrent mips with semi-supervised graph neural networks. arXiv preprint arXiv:2302.11992 , 2023.

Una Benlic and Jin-Kao Hao. Breakout local search for the max-cutproblem. Engineering Applications of Artificial Intelligence , 26(3):1162-1173, 2013.

Srinadh Bhojanapalli, Nicolas Boumal, Prateek Jain, and Praneeth Netrapalli. Smoothed analysis for low-rank solutions to semidefinite programs in quadratic penalty form, 2018.

Maximilian Böther, Otto Kißig, Martin Taraz, Sarel Cohen, Karen Seidel, and Tobias Friedrich. What's wrong with deep learning in tree search for combinatorial optimization. arXiv preprint arXiv:2201.10494 , 2022.

Nicolas Boumal, Vladislav Voroninski, and Afonso S Bandeira. Deterministic guarantees for burermonteiro factorizations of smooth semidefinite programs. Communications on Pure and Applied Mathematics , 73(3):581-608, 2020.

Alfredo Braunstein, Marc Mézard, and Riccardo Zecchina. Survey propagation: An algorithm for satisfiability. Random Structures & Algorithms , 27(2):201-226, 2005.

Robin A Brown, Edward Schmerling, Navid Azizan, and Marco Pavone. A unified view of sdp-based neural network verification through completely positive programming. In International Conference on Artificial Intelligence and Statistics , pp. 9334-9355. PMLR, 2022.

Lorenzo Brusca, Lars CPM Quaedvlieg, Stratis Skoulakis, Grigorios G Chrysos, and Volkan Cevher. Maximum independent set: Self-training through dynamic programming. In Advances in neural information processing systems (NeurIPS) , 2023.

Samuel Burer and Renato Monteiro. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming, Series B , 95:329-357, 02 2003. doi: 10.1007/s10107-002-0352-8.

Quentin Cappart, Emmanuel Goutierre, David Bergman, and Louis-Martin Rousseau. Improving optimization bounds using machine learning: decision diagrams meet deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pp. 1443-1451, 2019.

Quentin Cappart, Didier Chételat, Elias B Khalil, Andrea Lodi, Christopher Morris, and Petar Velickovic. Combinatorial optimization and reasoning with graph neural networks. J. Mach. Learn. Res. , 24:130-1, 2023.

Xinyun Chen and Yuandong Tian. Learning to perform local rewriting for combinatorial optimization. In Advances in Neural Information Processing Systems , pp. 6278-6289, 2019.

Ziang Chen, Jialin Liu, Xinshang Wang, Jianfeng Lu, and Wotao Yin. On representing linear programs by graph neural networks. arXiv preprint arXiv:2209.12288 , 2022.

Hanjun Dai, Xinshi Chen, Yu Li, Xin Gao, and Le Song. A framework for differentiable discovery of graph algorithms. 2020.

Jiaoman Du, Xiang Li, Lean Yu, Ralescu Dan, and Jiandong Zhou. Multi-depot vehicle routing problem for hazardous materials transportation: A fuzzy bilevel programming. Information sciences , 399:201-218, 2017.

Vijay Prakash Dwivedi, Chaitanya K Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. arXiv preprint arXiv:2003.00982 , 2020.

Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Graph neural networks with learnable structural and positional representations. arXiv preprint arXiv:2110.07875 , 2021.

Murat A. Erdogdu, Asuman Ozdaglar, Pablo A. Parrilo, and Nuri Denizcan Vanli. Convergence rate of block-coordinate maximization burer-monteiro method for solving large sdps, 2019.

El¯ ıza Gaile, Andis Draguns, Em¯ ıls Ozolin , š, and K¯ arlis Freivalds. Unsupervised training for neural tsp solver. In International Conference on Learning and Intelligent Optimization , pp. 334-346. Springer, 2022.

David Gamarnik. Barriers for the performance of graph neural networks (gnn) in discrete random structures. Proceedings of the National Academy of Sciences , 120(46):e2314092120, 2023.

Vijay Ganesh and Moshe Y Vardi. On the unreasonable effectiveness of sat solvers. Beyond the Worst-Case Analysis of Algorithms , pp. 547-566, 2020.

Eleanor J Gardiner, Peter Willett, and Peter J Artymiuk. Graph-theoretic techniques for macromolecular docking. Journal of Chemical Information and Computer Sciences , 40(2):273-279, 2000.

Maxime Gasse, Didier Chételat, Nicola Ferroni, Laurent Charlin, and Andrea Lodi. Exact combinatorial optimization with graph convolutional neural networks. arXiv preprint arXiv:1906.01629 , 2019.

Rong Ge, Jason D. Lee, and Tengyu Ma. Matrix completion has no spurious local minimum. CoRR , abs/1605.07272, 2016. URL http://arxiv.org/abs/1605.07272 .

Dobrik Georgiev, Danilo Numeroso, Davide Bacciu, and Pietro Liò. Neural algorithmic reasoning for combinatorial optimisation. arXiv preprint arXiv:2306.06064 , 2023.

Michel X Goemans and David P Williamson. Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM (JACM) , 42(6): 1115-1145, 1995.

Martin Grötschel, László Lovász, and Alexander Schrijver. The ellipsoid method and its consequences in combinatorial optimization. Combinatorica , 1:169-197, 1981.

Prateek Gupta, Maxime Gasse, Elias Khalil, Pawan Mudigonda, Andrea Lodi, and Yoshua Bengio. Hybrid models for learning to branch. Advances in neural information processing systems , 33: 18087-18097, 2020.

Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart. Exploring network structure, dynamics, and function using networkx. In Gaël Varoquaux, Travis Vaught, and Jarrod Millman (eds.), Proceedings of the 7th Python in Science Conference , pp. 11 - 15, Pasadena, CA USA, 2008.

Demian Hespe, Sebastian Lamm, Christian Schulz, and Darren Strash. Wegotyoucovered: The winning solver from the pace 2019 challenge, vertex cover track. In 2020 Proceedings of the SIAM Workshop on Combinatorial Scientific Computing , pp. 1-11. SIAM, 2020.

Borja Ibarz, Vitaly Kurin, George Papamakarios, Kyriacos Nikiforou, Mehdi Bennani, Róbert Csordás, Andrew Joseph Dudzik, Matko Bošnjak, Alex Vitvitskyi, Yulia Rubanova, et al. A generalist neural algorithmic learner. In Learning on Graphs Conference , pp. 2-1. PMLR, 2022.

Vinay Jethava, Anders Martinsson, Chiranjib Bhattacharyya, and Devdatt Dubhashi. Lovász ϑ function, svms and finding dense subgraphs. The Journal of Machine Learning Research , 14(1): 3495-3536, 2013.

Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M. Kakade, and Michael I. Jordan. How to escape saddle points efficiently, 2017.

David S Johnson. Approximation algorithms for combinatorial problems. In Proceedings of the fifth annual ACM symposium on Theory of computing , pp. 38-49, 1973.

Chaitanya K Joshi, Thomas Laurent, and Xavier Bresson. An efficient graph convolutional network technique for the travelling salesman problem. arXiv preprint arXiv:1906.01227 , 2019.

Chaitanya K Joshi, Quentin Cappart, Louis-Martin Rousseau, and Thomas Laurent. Learning tsp requires rethinking generalization. arXiv preprint arXiv:2006.07054 , 2020.

Andrew B Kahng, Robert R Nerem, Yusu Wang, and Chien-Yi Yang. Nn-steiner: A mixed neural-algorithmic approach for the rectilinear steiner minimum tree problem. arXiv preprint arXiv:2312.10589 , 2023.

Nikolaos Karalias. Probabilistic methods for neural combinatorial optimization. Technical report, EPFL, 2023.

Nikolaos Karalias and Andreas Loukas. Erdos goes neural: an unsupervised learning framework for combinatorial optimization on graphs. arXiv preprint arXiv:2006.10643 , 2020.

Nikolaos Karalias, Joshua Robinson, Andreas Loukas, and Stefanie Jegelka. Neural set function extensions: Learning with discrete functions in high dimensions. Advances in Neural Information Processing Systems , 35:15338-15352, 2022.

Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial optimization algorithms over graphs. In Advances in Neural Information Processing Systems , pp. 6348-6358, 2017.

Subhash Khot. On the power of unique 2-prover 1-round games. In Proceedings of the thiry-fourth annual ACM symposium on Theory of computing , pp. 767-775, 2002.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Tamás Kriváchy, Yu Cai, Joseph Bowles, Daniel Cavalcanti, and Nicolas Brunner. High-speed batch processing of semidefinite programs with feedforward neural networks. New Journal of Physics , 23(10):103034, 2021.

Vitaly Kurin, Saad Godil, Shimon Whiteson, and Bryan Catanzaro. Can q-learning with graph networks learn a generalizable branching heuristic for a sat solver? Advances in Neural Information Processing Systems , 33:9608-9621, 2020.

Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493 , 2015.

Zhuwen Li, Qifeng Chen, and Vladlen Koltun. Combinatorial optimization with graph convolutional networks and guided tree search. In Advances in Neural Information Processing Systems , pp. 539-548, 2018.

Minghao Liu, Fuqi Jia, Pei Huang, Fan Zhang, Yuchen Sun, Shaowei Cai, Feifei Ma, and Jian Zhang. Can graph neural networks learn to solve maxsat problem? arXiv preprint arXiv:2111.07568 , 2021.

Andreas Loukas. What graph neural networks cannot learn: depth vs width, 2019.

László Lovász. On the shannon capacity of a graph. IEEE Transactions on Information theory , 25(1): 1-7, 1979.

Evan McCarty, Qi Zhao, Anastasios Sidiropoulos, and Yusu Wang. Nn-baker: A neural-network infused algorithmic framework for optimization problems on geometric intersection graphs. Advances in Neural Information Processing Systems , 34:23023-23035, 2021.

Yimeng Min, Frederik Wenkel, Michael Perlmutter, and Guy Wolf. Can hybrid geometric scattering networks help solve the maximum clique problem? Advances in Neural Information Processing Systems , 35:22713-22724, 2022.

Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pp. 4602-4609, 2019.

Vinod Nair, Sergey Bartunov, Felix Gimeno, Ingrid Von Glehn, Pawel Lichocki, Ivan Lobov, Brendan O'Donoghue, Nicolas Sonnerat, Christian Tjandraatmadja, Pengming Wang, et al. Solving mixed integer programs using neural networks. arXiv preprint arXiv:2012.13349 , 2020.

Mohammadreza Nazari, Afshin Oroojlooy, Lawrence Snyder, and Martin Takác. Reinforcement learning for solving the vehicle routing problem. In Advances in Neural Information Processing Systems , pp. 9839-9849, 2018.

Danilo Numeroso, Davide Bacciu, and Petar Veliˇ ckovi´ c. Dual algorithmic reasoning. arXiv preprint arXiv:2302.04496 , 2023.

Liam O'Carroll, Vaidehi Srinivas, and Aravindan Vijayaraghavan. The burer-monteiro sdp method can fail even above the barvinok-pataki bound. Advances in Neural Information Processing Systems , 35:31254-31264, 2022.

Augustin Parjadis, Quentin Cappart, Louis-Martin Rousseau, and David Bergman. Improving branchand-bound using decision diagrams and reinforcement learning. In Integration of Constraint Programming, Artificial Intelligence, and Operations Research: 18th International Conference, CPAIOR 2021, Vienna, Austria, July 5-8, 2021, Proceedings 18 , pp. 446-455. Springer, 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Max B Paulus, Giulia Zarpellon, Andreas Krause, Laurent Charlin, and Chris Maddison. Learning to cut by looking ahead: Cutting plane selection via imitation learning. In International conference on machine learning , pp. 17584-17600. PMLR, 2022.

Marcelo Prates, Pedro HC Avelar, Henrique Lemos, Luis C Lamb, and Moshe Y Vardi. Learning to solve np-complete problems: A graph neural network for decision tsp. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pp. 4731-4738, 2019.

Chendi Qian, Didier Chételat, and Christopher Morris. Exploring the power of graph neural networks in solving linear optimization problems. arXiv preprint arXiv:2310.10603 , 2023.

Prasad Raghavendra. Optimal algorithms and inapproximability results for every csp? In Proceedings of the fortieth annual ACM symposium on Theory of computing , pp. 245-254, 2008.

Prasad Raghavendra and David Steurer. How to round any csp. In 2009 50th Annual IEEE Symposium on Foundations of Computer Science , pp. 586-594, 2009a. doi: 10.1109/FOCS.2009.74.

Prasad Raghavendra and David Steurer. Towards computing the grothendieck constant. In Proceedings of the Twentieth Annual ACM-SIAM Symposium on Discrete Algorithms , pp. 525-534. SIAM, 2009b.

Prasad Raghavendra, David Steurer, and Madhur Tulsiani. Reductions between expansion problems. In 2012 IEEE 27th Conference on Computational Complexity , pp. 64-73. IEEE, 2012.

Alessandro Rudi, Ulysse Marteau-Ferey, and Francis Bach. Finding global minima via kernel approximations. arXiv preprint arXiv:2012.11978 , 2020.

Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Approximation ratios of graph neural networks for combinatorial problems, 2019.

Ryoma Sato, Makoto Yamada, and Hisashi Kashima. Random features strengthen graph neural networks. In Proceedings of the 2021 SIAM international conference on data mining (SDM) , pp. 333-341. SIAM, 2021.

Martin JA Schuetz, J Kyle Brubaker, and Helmut G Katzgraber. Combinatorial optimization with physics-inspired graph neural networks. Nature Machine Intelligence , 4(4):367-377, 2022a.

Martin JA Schuetz, J Kyle Brubaker, Zhihuai Zhu, and Helmut G Katzgraber. Graph coloring with physics-inspired graph neural networks. Physical Review Research , 4(4):043131, 2022b.

Daniel Selsam and Nikolaj Bjørner. Guiding high-performance sat solvers with unsat-core predictions. In Theory and Applications of Satisfiability Testing-SAT 2019: 22nd International Conference, SAT 2019, Lisbon, Portugal, July 9-12, 2019, Proceedings 22 , pp. 336-353. Springer, 2019.

Daniel Selsam, Matthew Lamm, Benedikt Bünz, Percy Liang, Leonardo de Moura, and David L. Dill. Learning a sat solver from single-bit supervision, 2018.

Alexander Smith, Andreas Veneris, and Anastasios Viglas. Design diagnosis using boolean satisfiability. In ASP-DAC 2004: Asia and South Pacific Design Automation Conference 2004 (IEEE Cat. No. 04EX753) , pp. 218-223. IEEE, 2004.

Haoran Sun, Etash K Guha, and Hanjun Dai. Annealed training for combinatorial optimization on graphs. arXiv preprint arXiv:2207.11542 , 2022.

Jan Toenshoff, Martin Ritzert, Hinrikus Wolf, and Martin Grohe. Run-csp: Unsupervised learning of message passing networks for binary constraint satisfaction problems. arXiv preprint arXiv:1909.08387 , 2019.

Jan Tönshoff, Berke Kisin, Jakob Lindner, and Martin Grohe. One model, any csp: Graph neural networks as fast global search heuristics for constraint satisfaction. arXiv preprint arXiv:2208.10227 , 2022.

Petar Veliˇ ckovi´ c, Rex Ying, Matilde Padovano, Raia Hadsell, and Charles Blundell. Neural execution of graph algorithms. In International Conference on Learning Representations .

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations , 2018.

Petar Veliˇ ckovi´ c, Adrià Puigdomènech Badia, David Budden, Razvan Pascanu, Andrea Banino, Misha Dashevskiy, Raia Hadsell, and Charles Blundell. The clrs algorithmic reasoning benchmark. In International Conference on Machine Learning , pp. 22084-22102. PMLR, 2022.

Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Advances in Neural Information Processing Systems , pp. 2692-2700, 2015.

Jose L Walteros and Austin Buchanan. Why is maximum clique often easy in practice. Oper. Res , 2019.

Haoyu Wang and Pan Li. Unsupervised learning for combinatorial optimization needs meta-learning. International Conference on Learning Representations , 2023.

Haoyu Peter Wang, Nan Wu, Hang Yang, Cong Hao, and Pan Li. Unsupervised learning for combinatorial optimization with principled objective relaxation. In Advances in Neural Information Processing Systems , 2022.

Po-Wei Wang and J Zico Kolter. Low-rank semidefinite programming for the max2sat problem. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pp. 1641-1649, 2019.

Po-Wei Wang, Priya Donti, Bryan Wilder, and Zico Kolter. Satnet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. In International Conference on Machine Learning , pp. 6545-6554. PMLR, 2019.

Wenxi Wang, Yang Hu, Mohit Tiwari, Sarfraz Khurshid, Kenneth McMillan, and Risto Miikkulainen. Neurocomb: Improving sat solving with graph neural networks. arXiv preprint arXiv:2110.14053 , 2021.

Yiming Wang, Austin Buchanan, and Sergiy Butenko. On imposing connectivity constraints in integer programs. Mathematical Programming , 166(1-2):241-271, 2017.

Ke Xu, Frédéric Boussemart, Fred Hemery, and Christophe Lecoutre. Random constraint satisfaction: Easy generation of hard (satisfiable) instances. Artificial intelligence , 171(8-9):514-534, 2007.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations , 2019. URL https:// openreview.net/forum?id=ryGs6iA5Km .

Yunqiu Xu, Meng Fang, Ling Chen, Gangyan Xu, Yali Du, and Chengqi Zhang. Reinforcement learning with multiple relational attention for solving vehicle routing problems. IEEE Transactions on Cybernetics , 52(10):11107-11120, 2021.

Emre Yolcu and Barnabás Póczos. Learning local search heuristics for boolean satisfiability. Advances in Neural Information Processing Systems , 32, 2019.

Mohammed Javeed Zaki, Srinivasan Parthasarathy, Mitsunori Ogihara, Wei Li, et al. New algorithms for fast discovery of association rules. In KDD , volume 97, pp. 283-286, 1997.

Dinghuai Zhang, Hanjun Dai, Nikolay Malkin, Aaron Courville, Yoshua Bengio, and Ling Pan. Let the flows tell: Solving graph combinatorial optimization problems with gflownets. arXiv preprint arXiv:2305.17010 , 2023.

A Min-Vertex-Cover OptGNN

Min-Vertex-Cover can be written as the following integer program

To deal with the constraint on the edges (1 -x i )(1 -x j ) = 0 , we add a quadratic penalty to the objective with a penalty parameter ρ > 0 yielding

Analogously to Max-Cut, we adopt a natural low rank vector formulation for vectors v = { v i } i ∈ [ N ] in r dimensions.

Now we can design a simple projected gradient descent scheme as follows. For iteration t in max iterations T , and for vector v i in v we perform the following update.

We can then define a OptGNN ( M,G ) analogously with learnable matrices M = { M t } t ∈ [ T ] ∈ R r × 2 r which are each sets of T learnable matrices corresponding to T layers of neural network. Let the message from node v j to node v i be

Let the aggregation function AGG be defined as

Then for layer t in max iterations T , for v i in v , we have

This approach can be straightforwardly adopted to compute the maximum clique and the maximum independent set.

B Max-3-SAT OptGNN

Our formulation for Max 3-SAT directly translates the OptGNN architecture from Definition 3.3. Let Λ be a 3-SAT instance with a set of clauses P over a set of binary literals V = { x 1 , x 2 , ..., x N } ∈ {-1 , 1 } . Here -1 corresponds to assigning the literal to False whilst 1 corresponds to True. Each clause P z ∈ P can be specified by three literals ( x i , x j , x k ) and a set of three 'tokens' ( τ i , τ j , τ k ) ∈ {-1 , 1 } 3 which correspond to whether the variable x i , x j , x k are negated in the clause. For instance the clause ( x 1 ∨ ¬ x 2 ∨ x 3 ) is translated into three literals ( x 1 , x 2 , x 3 ) and tokens (1 , -1 , 1) .

The 3-SAT problem on instance Λ is equivalent to maximizing the following polynomial

Subject to the constraint x 1 , x 2 , ..., x N ∈ { 1 , -1 } . Now we're reading to define the OptGNN for 3-SAT

Objective: First we define the set of vector embeddings. For every literal x i we associate an embedding v i ∈ R r . For every pair of literals that appear in a clause ( x i , x j ) we associate a variable v ij ∈ R r . Finally we associate a vector v ∅ to represent 1 . Then the unconstrained objective SAT is defined as

Constraints: The constraints are then as follows. For every clause involving variables ( x i , x j , x k ) we impose the following constraints on v i , v j , v k , v ij , v ik , v jk and v ∅ . Note that these constraints are exactly the ones listed in algorithm 2 which we organize here for convenience. The naming convention is the degree of the polynomial on the left to the degree of the polynomial on the right.

1. pair-to-pair

triplets-to-triplet

triplet-to-single

4. quad-to-pair

(34)

⟨

v

ij

, v ik

⟩

=

⟨

v

j

, v

k

⟩

(36)

(38)

5. unit norm

Then for completeness we write out the full penalized objective. We perform this exercise in such great detail to inform the reader of how this could be set up for other Max-CSP's.

Definition (OptGNN for 3-SAT) . Let Λ be a 3-SAT instance. Let L ρ be defined as

Then taking the gradient of L gives us the precise form of the message passing. We list the forms of the messages from adjacent nodes in the constraint graph G Λ .

Message: Pair to Singles for Single in Pair. This is the message MESSAGE [ v ij → v i ] for each pair node v ij to a single v i node.

Message: Pair to Singles for Single not in Pair. This is the message MESSAGE [ v ij → v k ] for each pair node v ij to a single v k node.

Message: Single to Pair for Single in Pair. This is the message MESSAGE [ v i → v ij ] for each single node v i to a pair v k

Message: Single to Pair for Single not in Pair. This is the message MESSAGE [ v ij → v k ] for each pair node v ij to a single v k node.

Message: Single to Single This is the message MESSAGE [ v i → v j ] for each single node v i to a single v j node.

Message: Pair to Pair This is the message MESSAGE [ v ij → v jk ] for each pair node v ij to a pair v jk node.

MESSAGE [ v ij → v jk ] = |{ c ∈ P : x i , x j , x k ∈ c }| 2 ρ [( ⟨ v ij , v jk ⟩ - ⟨ v i , v k ⟩ ) v ij ] (48) Then the OptGNN ( M,G Λ ) is defined with the following functions UPDATE, AGGREGATE, and NONLINEAR. For any node v , the OptGNN update is

Finally, we note that many of the signs in the forms of messages could have been chosen differently i.e ⟨ v i , v j ⟩ - ⟨ v ij , v ∅ ⟩ produces different gradients from ⟨ v ij , v ∅ ⟩ - ⟨ v i , v j ⟩ . We leave small choices like this to the reader.

C Optimality of Message Passing for Max-CSP

Our primary theoretical result is that a polynomial time message passing algorithm on an appropriately defined constraint graph computes the approximate optimum of SDP 2 which is notable for being an SDP that achieves the Unique Games optimal integrality gap.

Our proof roadmap is simple. First, we design an SDP relaxation SDP 2 for Max-k-CSP that is provably equivalent to the SDP of Raghavendra (2008) and therefore inherits its complexity theoretic optimality. Finally, we design a message passing algorithm to approximately solve SDP 2 in polynomial time to polynomial precision. Our message passing algorithm has the advantage of being formulated on an appropriately defined constraint graph. For a Max-k-CSP instance Λ with N variables, |P| predicates, over an alphabet of size q , it takes |P| q k space to represent the Max-CSP. To declutter notation, we let Φ be the size of the Max-CSP which is equal to |P| q k . Our message passing algorithm achieves an additive ϵ approximation in time poly ( ϵ -1 , Φ , log( δ -1 )) which is then polynomial in the size of the CSP and inverse polynomial in the precision.

Here we briefly reiterate the definition of Max-k-CSP. A Max-k-CSP instance Λ = ( V , P , q ) consists of a set of N variables V := { x i } i ∈ [ N ] each taking values in an alphabet [ q ] and a set of predicates P := { P z } z ⊂V where each predicate is a payoff function over k variables denoted z = { x i 1 , x i 2 , ..., x i k } . Here we refer to k as the arity of the Max-k-CSP, and we adopt the normalization that each predicate P z returns outputs in [0 , 1] . We index each predicate P z by its domain z and we will use the notation S ( P ) to denote the domain of a predicate P . The goal of Max-k-CSP is to maximize the payoff of the predicates.

Where X z denotes the assignment of variables { x i } i ∈ z .

There is an SDP relaxation of equation 52 that is the "qualitatively most powerful assuming the Unique Games conjecture" Raghavendra (2008). More specifically, the integrality gap of the SDP achieves the Unique Games optimal approximation ratio. Furthermore, there exists a rounding that achieves its integrality gap.

SDP Reformulation: Next we will introduce the SDP formulation we adopt in this paper. For the sake of exposition and notational simplicity, we will work with binary Max-k-CSP's where q = { 0 , 1 } . The extension to general q is straightforward and detailed in the appendix.

We will adopt the standard pseudoexpectation and pseudodistribution formalism in describing our SDP. Let ˜ E µ [ x ] be a matrix in dimension R ( N +1) d/ 2 × ( N +1) d/ 2 of optimization variables defined as follows

Where we use ⊗ to denote tensor product. It is convenient to think of ˜ E µ [ x ] as a matrix of variables denoting the up to d multilinear moments of a distribution µ over the variables V . A multilinear polynomial is a polynomial of the form X ϕ := ∏ i ∈ ϕ x i for some subset of the variables ϕ ⊆ V . We index the variables of the matrix ˜ E µ [ x ] by the multilinear moment that it represents. Notice that this creates repeat copies as their are multiple entries representing the same monomial. This is dealt with by constraining the repeated copies to be equal with linear equality constraints.

Specifically, let z be a subset of the CSP variables z ⊂ { x i } i ∈ [ N ] of size k . Let X z denote the multilinear moment X z := ∏ i ∈ z x i . Then ˜ E µ [ X z ] denotes the SDP variable corresponding to the multilinear moment E µ [ X z ] . Of course optimizing over the space of distributions µ over V is intractable, and so we opt for optimizing over the space of low degree pseudodistributions and their associated pseudoexpecation functionals. See Barak & Steurer (2014) for references therein.

In particular, for any subset of variables X z := { x i 1 , ..., x i k } ∈ V we let ˜ E µ [ x ] ∣ ∣ z,d denote the matrix of the up to degree up to d multilinear moments of the variables in z .

We refer to the above matrix as a degree d pseudoexpectation funcitonal over X z . Subsequently, we describe a pseudoexpectation formulation of our SDP followed by a vector formulation.

Multilinear Formulation: A predicate for a boolean Max-k-CSP P z ( X z ) can be written as a multilinear polynomial

For some real valued weights w z,τ and y s which are simply the fourier coefficients of the function P z . Then the pseudoexpectation formulation of our SDP is as follows

subject to the following constraints

Unit: ˜ E µ [1] = 1 , ˜ E µ [ x 2 i ] = 1 for all x i ∈ V , and ˜ E µ [ ∏ i ∈ s x 2 i ∏ j ∈ s ′ x j ] = ˜ E µ [ ∏ j ∈ s ′ x j ] for all s, s ′ ⊆ S ( P ) for every predicate P ∈ P such that 2 s + s ′ ≤ k . In expectation, the squares of all multilinear polynomials are equal to 1 .

Positive Semidefinite: ˜ E µ [ x ] | V , 2 ⪰ 0 i.e the degree two pseudoexpectation is positive semidefinite. ˜ E µ [ x ] ∣ ∣ z, 2 k ⪰ 0 for all z = S ( P ) for all P ∈ P . The moment matrix for the multilinear polynomials corresponding to every predicate is positive semidefinite.

Equivalently we can view the SDP in terms of the vectors in the cholesky decomposition of ˜ E µ [ x ] . We rewrite the above SDP accordingly. For this purpose it is useful to introduce the notation ζ ( A,B ) := A ∪ B/A ∩ B . It is also useful to introduce the notation C ( s ) for the size of the set { g, g ′ ⊆ s : ζ ( g, g ′ ) = s } .

SDP 2 SDP for Max-k-CSP (Equivalent to UGC-optimal)

SDP Vector Formulation Λ = ( V , P , { 0 , 1 } ) . Multilinear formulation of objective.

First constraint is the square of multilinear polynomials are unit.

Second constraint are degree 2 k SoS constraints for products of multilinear polynomials.

Lemma C.1. For Max-k-CSP instance Λ , The SDP of SDP 2 is at least as tight as the SDP of Raghavendra (2008).

Proof. The SDP of Raghavendra (2008) is a degree 2 SoS SDP augmented with k -local distributions for every predicate P ∈ P . By using the vectors of the cholesky decomposition and constraining them to be unit vectors we automatically capture degree 2 SoS. To capture k local distributions we simply enforce degree 2 k SoS on the boolean hypercube for the domain of every predicate. This can be done with the standard vector formulation written in SDP 2. See Barak & Steurer (2014) for background and references.

Moving forward, the goal of Algorithm 3 is to minimize the loss L ρ ( v ) which is a function of the Max-CSP instance Λ .

The loss L ρ has gradient of the form

Noticing that the form of the gradient depends only on the vectors in the neighborhood of the constraint graph G Λ we arrive at our message passing algorithm. The key to our proof is bounding the number of iterations required to optimize equation 60 to sufficient accuracy to be an approximate global optimum of SDP 2.

Theorem C.1. Algorithm 3 computes in O ( ϵ -4 Φ 4 log( δ -1 )) iterations a set of vectors v := { ˆ v s } for all s ⊆ S ( P ) for all P ∈ P that satisfy the constraints of SDP 2 to error ϵ and approximates the

Algorithm 3 Message Passing for Max-CSP

1: Inputs: Max-CSP instance Λ

2: n ← Φlog( δ - 1 )

3: η, ψ, σ ← n variance}

2 {Initialize step size, noise threshold, and noise

4: v 0 = { v s } s ⊆ z : P z ∈P ← Uniform ( S n - 1 ) {Initialize vectors to uniform on the unit sphere} 5: for t ∈ [ O ( ϵ - 4 Φ 4 log( δ - 1 ))] do

6: for v t w ∈ v t {Iterate over vectors and update each vector with neighboring vectors in constraint graph} do

7:

8: if ∥ v t +1 w -v t w ∥ ≤ ψ then 9: ζ ← N (0 , σI )

10: else

11: ζ ← 0

12: end if

13: v t +1 w ← v t +1 w + ζ

14: end for

15: end for

16: return v t

17: Output: vectors corresponding to solution to SDP 2

optimum of SDP 2 to error ϵ with probability 1 -δ

where SDP (Λ) is the optimum of SDP 2.

Proof. We begin by writing down the objective penalized by a quadratic on the constraints.

For any monomial X s = ∏ i ∈ s x i in P z ( X z ) we write

Where C ( s ) is the size of the set { g, g ′ ⊆ s : ζ ( g, g ′ ) = s } . In a small abuse of notation, we regard this as the definition of ˜ E µ [ X s ] but realize that we're referring to the iterates of the algorithm before they've converged to a pseudoexpectation. Now recall equation 55, we can expand the polynomial P z ( X z ) along its standard monomial basis

where we have defined coefficients y s for every monomial in P z ( X z ) . Plugging equation 64 and equation 65 into equation 63 we obtain

Taking the derivative with respect to any v w ∈ v we obtain

The gradient update is then what is detailed in Algorithm 3

Thus far we have established the form of the gradient. To prove the gradient iteration converges we reference the literature on convergence of perturbed gradient descent (Jin et al., 2017) which we rewrite in Theorem F.1. First we note that the SDP 2 has ℓ smooth gradient for ℓ ≤ poly ( ρ, B ) and has γ lipschitz Hessian for γ = poly ( ρ, B ) which we arrive at in Lemma E.2 and Lemma F.1. The proofs of Lemma E.2 and Lemma F.1 are technically involved and form the core hurdle in arriving at our proof.

Then by Theorem F.1 the iteration converges to an ( ϵ ′ , γ 2 ) -SOSP Definition F.1 in no more than ˜ O ( 1 ϵ ′ 2 ) iterations with probability 1 -δ . Now that we've established that the basic gradient iteration converges to approximate SOSP, we need to then prove that the approximate SOSP are approximate global optimum. We achieve this by using the result of Bhojanapalli et al. (2018) lemma 3 which we restate in Lemma C.2 for convenience.

The subsequent presentation adapts the proof of Lemma C.2 to our setting. To show that ( ϵ ′ , γ 2 ) -SOSP are approximately global optimum, we have to work with the original SDP loss as opposed to the nonconvex vector loss. For subsequent analysis we need to define the penalized loss which we denote H ρ ( ˜ E [ x ]) in terms of the SDP moment matrix ˜ E [ x ] .

Here we use the notation ˜ E ˆ µ [ X ζ ( g,g ′ ) ] and ˜ E ˆ µ [ X ζ ( h,h ′ ) ] to denote ⟨ v g , v ′ g ⟩ and ⟨ v h , v ′ h ⟩ respectively. Note that although by definition H ρ ( ˜ E [ x ]) = L ρ ( v ) , their gradients and hessians are distinct because L ρ ( v ) is overparameterized. This is the key point. It is straightforward to argue that SOSP of a convex optimization are approximately global, but we are trying to make this argument for SOSP of the overparameterized loss L ρ ( v ) .

Let the global optimum of SDP 2 be denoted ˜ E ˜ µ [˜ x ] with a cholesky decomposition ˜ v . Let ˆ v be the set of vectors outputted by Algorithm 3 with associated pseudoexpectation ˜ E ˆ µ [ˆ x ] . Then, we can bound

Here the first equality is by definition, and the inequality is by the convexity of H ρ . Moving on, we use the fact that the min eigenvalue of the hessian of overparameterized loss ∇ 2 L ρ (ˆ v ) ⪰ -γ √ ϵ ′ implies the min eigenvalue of the gradient of the convex loss λ min ( ∇H ρ ( ˜ E [ˆ x ])) ≥ -γ √ ϵ ′ . This fact is invoked in Bhojanapalli et al. (2018) lemma 3 which we adapt to our setting in Lemma C.2. Subsequently, we adapt the lines of their argument in Lemma C.2 most relevant to our analysis which we detail here for the sake of completeness.

Here the first inequality follows by a standard inequality of frobenius inner product, the second inequality follows by Cauchy-Schwarz, the third inequality follows by the ( ϵ ′ , γ 2 ) -SOSP conditions on both the min eigenvalue of the hessian and the norm of the gradient, the final two inequalities follow from knowing the main diagonal of ˜ E [ˆ x ] is the identity and that every vector in ˜ v is a unit vector up to inverse polynomial error poly ( ρ -1 , 2 k ) . For this last point see the proof in Lemma E.1. Therefore if we set ϵ ′ = poly ( ϵ, 2 -k ) we arrive at any ϵ error. Therefore we have established our estimate ˆ v is approximates the global optimum of the quadratically penalized objective i.e H ρ ( ˜ E [ˆ x ]) -H ρ ( ˜ E [˜ x ]) ≤ ϵ . To finish our proof, we have to bound the distance between the global optimum of the quadratically penalized objective H ρ ( ˜ E [˜ x ]) and SDP (Λ) the optimum of SDP 2. This is established for ρ a sufficiently large poly ( ϵ -1 , 2 k ) in Lemma E.1. This concludes our proof that the iterates of Algorithm 3 converge to the solution of SDP 2.

In our proof above, the bound on approximate SOSP being approximate global optimum is built on the result of Bhojanapalli et al. (2018) which we rephrase for our setting below.

Lemma C.2. [Bhojanapalli et al. (2018) lemma 3 rephrased] Let L ρ ( · ) be defined as in equation 63 and let H ρ ( · ) be defined as in equation 69. Let U ∈ R Φ × Φ be an ( ϵ, γ ) -SOSP of L ρ ( · ) , then

Furthermore, the global optimum ˜ X obeys the optimality gap

The following Lemma E.1 establishes that for a sufficiently large penalty parameter ρ = poly ( ϵ -1 , 2 k ) the optimum of the penalized problem and the exact solution to SDP 2 are close.

Lemma C.3. Let Λ be a Max-k-CSP instance, and let SDP (Λ) be the optimum of SDP 2. Let L ρ ( v ) be the quadratically penalized objective

Let ˜ v be the argmin of the unconstrained minimization

Then we have

Proof. We begin the analysis with the generic equality constrained semidefinite program of the form

For an objective matrix C and constraint matrices { A i } i ∈F in some constraint set F . We will invoke specific properties of SDP 2 to enable our analysis. First we define the penalized objective in this generic form

Let ˜ X be the minimizer of the penalized problem.

Let X ∗ be the minimizer of the constrained problem equation 87. Let τ i be the error ˜ X has in satisfying constraint ⟨ A i , ˜ X ⟩ = b i .

We will show that τ i scales inversely with ρ . That is, τ i ≤ poly ( k, ρ -1 ) . Notice that the quadratic penalty on the violated constraints must be smaller than the decrease in the objective for having violated the constraints. So long as the objective is not too sensitive i.e 'robust' to perturbations in the constraint violations the quadratic penalty should overwhelm the decrease in the objective. To carry out this intuition, we begin with the fact that the constrained minimum is larger than the penalized minimum.

Applying the definitions of H ρ ( X ∗ ) and H ρ ( ˜ X ) we obtain

Rearranging LHS and RHS and dividing by |F| we obtain

We upper bound the RHS using the robustness theorem of Raghavendra & Steurer (2009a) restated in the appendix Theorem F.2 which states that an SDP solution that violates the constraints by a small perturbation changes the objective by a small amount. Thus we obtain,

Therefore combining equation 79 with equation 80 we obtain

Taking Cauchy-Schwarz on the RHS we obtain

Noticing that 1 |F| ∑ i τ 2 i appears in both LHS and RHS we rearrange to obtain

Where for ρ = poly ( k ) ϵ 3 / 4 we have the average squared error of the constraints is upper bounded by ϵ . Then via Markov's no more than √ ϵ fraction of the constraints can be violated by more than squared error √ ϵ . We label these 'grossly' violated constraints. The clauses involved in these grossly violated constraints contributes no more than poly ( k ) √ ϵ to the objective. On the other hand, the constraints that are violated by no more than squared error √ ϵ contributed no more than poly ( k ) ϵ 1 / 8 to the objective which follows from the robustness theorem Theorem F.2. Taken together we conclude that

Finally we show it's not hard to generalize our algorithm to alphabets of size [ q ] .

Notation for General Alphabet. For any predicate P ∈ P , let D ( P ) be the set of all variable assignment tuples indexed by a set of variables s ⊆ S ( P ) and an assignment τ ∈ [ q ] | s | . Let x ( i,a ) denote an assignment of value a ∈ [ q ] to variable x i .

SDP 3 SDP Vector Formulation for Max-k-CSP General Alphabet (Equivalent to UGC optimal)

SDP Vector Formulation General Alphabet Λ = ( V , P , q ) . Pseudoexpectation formulation of the objective.

̸

(84)

First constraint corresponds to booleanity of each value in the alphabet.

Second constraint corresponds to a variable taking on only one value in the alphabet.

Third constraint corresponds to a variable taking on only one value in the alphabet.

Fourth constraint corresponds to local distribution on the variables in each predicate.

Fifth constraint corresponds to the positivity of every degree two sum of squares of polynomials.

Lemma C.4. There exists a message passing algorithm that computes in poly ( ϵ -1 , Φ , log( δ -1 )) iterations a set of vectors v := { ˆ v ( i,a ) } for all ( i, a ) ∈ ϕ , for all ϕ ⊆ D ( P ) , for all P ∈ P that satisfy

the constraints of Algorithm 3 to error ϵ and approximates the optimum of Algorithm 3 to error ϵ with probability 1 -δ

where SDP (Λ) is the optimum of Algorithm 3.

Proof. The proof is entirely parallel to the proof of Theorem C.1. We can write Algorithm 3 entirely in terms of the vector of its cholesky decomposition where once again we take advantage of the fact that SoS degree 2 kq distributions are actual distributions over subsets of kq variables over each predicate. Given the overparameterized vector formulation, we observe that once again we are faced with equality constraints that can be added to the objective with a quadratic penalty. Perturbed gradient descent induces a message passing algorithm over the constraint graph G Λ , and in no more than poly ( ϵ -1 Φ) iterations reaches an ( ϵ, γ ) -SOSP. The analysis of optimality goes along the same lines as Lemma E.1. For sufficiently large penalty ρ = poly ( ϵ -1 , q k ) the error in satisfying the constraints is ϵ and the objective is robust to small perturbations in satisfying the constraint. That concludes our discussion of generalizing to general alphabets.

C.1 Neural Certification Scheme

An intriguing aspect of OptGNN is that the embeddings can be interpreted as the solution to a low rank SDP which leaves open the possibility that the embeddings can be used to generate a dual certificate i.e a lower bound on a convex relaxation. First, we define the primal problem

Lemma C.5. Let OPT be the minimizer of the SDP equation 87. Then for any ˜ X ∈ R N × N ⪰ 0 and any λ ∗ ∈ R |F| , we define F λ ∗ ( X ) to be

We require SDP to satisfy a bound on its trace Tr ( X ) ≤ Y for some Y ∈ R + . Then the following is a lower bound on OPT.

Proof. Next we introduce lagrange multipliers λ ∈ R k and Q ⪰ 0 to form the lagrangian

We lower bound the optimum of OPT defined to be the minimizer of equation 87

Where in the first inequality we replaced X ⪰ 0 with V V T which is a lower bound as every psd matrix admits a cholesky decomposition. In the second inequality we flipped the order of min and max, and in the final inequality we chose a specific set of dual variables λ ∗ ∈ R |F| which lower bounds the maximization over dual variables. The key is to find a good setting for λ ∗ .

Next we establish that for any choice of λ ∗ we can compute a lower bound on inequality 92 as follows. Let F λ ∗ ( V V T ) be defined as the funciton in the RHS of 92.

Then equation 92 can be rewritten as

Now let V ∗ be the minimizer of equation 92 and let X ∗ = V ∗ ( V ∗ ) T . We have by convexity that

In the first inequality we apply the convexity of F λ ∗ . In the second inequality we apply a standard inequality of frobenius inner product. In the last inequality we use the fact that Tr( X ∗ ) = N . Rearranging we obtain for any X

Therefore it suffices to upper bound the two terms above ⟨∇ F λ ∗ ( X ) , X ⟩ and λ min ( ∇ F λ ∗ ( X )) which is an expression that holds for any X . Given the output embeddings ˜ V of OptGNN (or indeed any set of vectors ˜ V ) let ˜ X = ˜ V ˜ V T . Then we have concluded

Up to this point, every manipulation is formal proof. Subsequently we detail how to estimate the dual variables λ ∗ . Although any estimate will produce a bound, it won't produce a tight bound. To be clear, solving for the optimal λ ∗ would be the same as building an SDP solver which would bring us back into the expensive primal dual procedures that are involved in solving SDP's. We are designing quick and cheap ways to output a dual certificate that may be somewhat looser. Our scheme is simply to set λ ∗ such that ∥∇ F λ ∗ ( ˜ X ) ∥ is minimized, ideally equal to zero. The intuition is that if ( ˜ X,λ ∗ ) were a primal dual pair, then the lagrangian would have a derivative with respect to X evaluated at ˜ X would be equal to zero. Let H λ ( V ) be defined as follows

We know the gradient of H λ ( ˜ V )

Therefore it suffices to find a setting of λ ∗ such that ∥∇ F λ ( ˜ X ) ˜ V ∥ is small, ideally zero. This would be a simple task, indeed a regression, if not for the unfortunate fact that OptGNN explicitly projects the vectors in ˜ V to be unit vectors. This creates numerical problems such that minimizing the norm of ∥∇ F λ ( ˜ X ) ˜ V ∥ does not produce a ∇ F λ ( ˜ X ) with a large minimum eigenvalue.

To fix this issue, let R η,ρ ( V ) denote the penalized lagrangian with quadratic penalties for constraints of the form ⟨ A i , X ⟩ = b i and linear penalty η i for constraints along the main diagonal of X of the form ⟨ e i e T i , X ⟩ = 1 .

Taking the gradient of R η,ρ ( V ) we obtain

Our rule for setting dual variables δ i for i ∈ J is

our rule for setting dual variables η j for j ∈ [ N ] is

Then our full set of dual variables λ ∗ is simply the concatenation ( δ, η ) . Writing out everything explicitly we obtain the following matrix for ∇ F λ ∗ ( ˜ V ˜ V T )

Plugging this expression into Lemma C.5 the final bound we evaluate in our code is

Which is entirely computed in terms of ˜ V the output embeddings of OptGNN. The resulting plot is as follows.

CertificateQuality:SDPvs.Opt-GNN-certN=50

Opt-GNN-cert

200

SDP-cert

190

180

Value

170

160

150

140

130

140

150

160

170

180

190

Opt-GNN

Note: The reason for splitting the set of dual variables is because the projection operator onto the unit ball is hard coded into the architecture of the lift network. Satisfying the constraint set via projection is different from the soft quadratic penalties on the remaining constraints and require separate handling.

̸

Max-Cut Certificate For Max-Cut our dual variables are particularly simple as there are no constraints ⟨ A i , X ⟩ = b i for b i = 0 . The dual variables for Max-Cut take on the form for all i ∈ [ N ]

It's certainly possible to come up with tighter certification schemes which we leave to future work.

Intuition: Near global optimality one step of the augmented method of lagrange multipliers ought to closely approximate the dual variables. After obtaining a estimate for the penalized lagrange multipliers we estimate the lagrange multipliers for the norm constraint by approximating ∇ R λ ( V ) = 0 . The alternative would have been to solve the linear system for all the lagrange multipliers at once but this runs into numerical issues and degeneracies.

Certificate Experiment: We run our certification procedure which we name OptGNN-cert and compare it to the SDP certificate. Note, that mathematically we will always produce a larger (i.e inferior) dual certificate in comparison to the SDP because we are bounding the distance to the SDP optimum with error in the gradients and hessians of the output embeddings of OptGNN. Our advantage is in the speed of the procedure. Without having to go through a primal dual solver, the entire time of producing OptGNN-cert is in the time required to feedforward through OptGNN. In this case we train an OptGNN-Max-Cut with 10 layers, on 1000 Erdos-Renyi graphs, with N = 100 nodes and edge density p = 0 . 1 . We plot the OptGNN Max-Cut value (an actual integer cut) on the x-axis and in the y-axis we plot the dual certificate value on the same graph where we compare the SDP certificate with the OptGNN-cert. See 4 for the N = 50 graphs and 3 for the N = 100 graphs.

Note of course the dual certificate for any technique must be larger than the cut value outputted by OptGNN so the scatter plot must be above the x = y axis of the plot. We see as is mathematically necessary, the OptGNN-cert is not as tight as the SDP certificate but certainly competitive and more importantly it is arrived at dramatically faster. Without any runtime optimizations, the OptGNN feedforward and certification takes no more than 0 . 02 seconds whereas the SDP takes 0 . 5 seconds on N = 100 node graphs.

D Experiment details

In this section we give further details on our experimental setup.

Datasets. Our experiments span a variety of randomly generated and real-world datasets. Our randomly generated datasets contain graphs from several random graph models, in particular Erd˝ osRényi (with p = 0 . 15 ), Barabási-Albert (with m = 4 ), Holme-Kim (with m = 4 and p = 0 . 25 ), Watts-Strogatz (with k = 4 and p = 0 . 25 ), and forced RB (with two sets of parameters, RB200 and RB500). Our real-world datasets are ENZYMES, PROTEINS, MUTAG, IMDB-BINARY, COLLAB (which we will together call TU-small ), and REDDIT-BINARY, REDDIT-MULTI-5K, and REDDIT-MULTI-12K (which we will call TU-REDDIT ).

We abbreviate the generated datasets using their initials and the range of vertex counts. For example, by ER (50,100) we denote Erd˝ os-Rényi random graphs with a vertex count drawn uniformly at random from [50, 100]. In tables, we mark generated datasets with superscript a , TU-small with b , and TU-REDDIT with c .

For Figure 2b, we display results for forced RB instances drawn from two different distributions. For RB200, we select N uniformly in [6 , 15] and K uniformly in [12 , 21] . For RB500, we select N uniformly in [20 , 34] and K uniformly in [10 , 29] .

In Table 2, random 3-SAT instances are generated by drawing three random variables for each clause and negating each variable with p = 0 . 5 . We trained on instances with 100 variables and clause count drawn uniformly from the interval [400 , 430] , and tested on instances with 100 variables and 400, 415, and 430 clauses respectively.

Baselines. We compare the performance of our approach against known classical algorithms. In terms of classical baselines, we run Gurobi with varying timeouts and include SDP results on smaller datasets.

Figure 5: Hyperparameter range explored for each group of datasets. For each NN architecture, when training on a dataset, we explored every listed hyperparameter combination in the corresponding column.

We also include a greedy baseline, which is the function one_exchange (for Max-Cut) or min_weighted_vertex_cover (for Min-Vertex-Cover) from networkx (Hagberg et al., 2008).

Validation and test splits. For each dataset we hold out a validation and test slice for evaluation. In our generated graph experiments we set aside 1000 graphs each for validation and testing. Each step of training ran on randomly generated graphs. For TU-small , we used a train/validation/test split of 0.8/0.1/0.1. For TU-REDDIT , we set aside 100 graphs each for validation and testing.

Scoring. To measure a model's score on a graph, we first run the model on the graph with a random initial vector assignment to generate an SDP output, and then round this output to an integral solution using 1,000 random hyperplanes. For the graph, we retain the best score achieved by any hyperplane.

We ran validation periodically during each training run and retained the model that achieved the highest validation score. Then for each model and dataset, we selected the hyperparameter setting that achieved the highest validation score, and we report the average score measured on the test slice. Please see D for further details on the hyperparameter ranges used.

In Figure 2b and Figure 2a, at test time, we use 100 random initial vector assignments instead of just 1, and retain the best score achieved by any hyperplane on any random initial vector assignment. We use 1 random initial vector assignment for validation as in other cases.

Hardware. Our training runs used 20 cores of an Intel Xeon Gold 6248 (for data loading and random graph generation) and a NVIDIA Tesla V100 GPU. Our Gurobi runs use 8 threads on a Intel Xeon Platinum 8260. Our KaMIS runs use an Intel Core i9-13900H. Our LwD and DGL-TREESEARCH runs use an Intel Core i9-13900H and an RTX 4060.

Hyperparameters. We ran each experiment on a range of hyperparameters. See Figure 5 for the hyperparameter listing. For all training runs, we used the Adam optimizer Kingma & Ba (2014) with a learning rate of 0.001. We used Laplacian eigenvector Dwivedi et al. (2020) (LE) or random walk Dwivedi et al. (2021) (RW) positional encoding with dimensionality of half the rank, except for rank 32 where we used 8 dimensions. For Max-3-SAT, we set the penalty term ρ = 0 . 003 .

D.1 Max-Cut

Low-rank SDP .We have included an additional baseline that solves a low-rank SDP using coordinate descent (Wang & Kolter, 2019) to the Max-Cut benchmark that originally appears in Any-CSP (Tönshoff et al., 2022). We have used the publicly available implementation provided by the authors. For each instance, the rank is automatically selected by the package to be ⌈ √ 2 N ⌉ .

D.2 Max-3-SAT

Erd˝ osGNN . The Erd˝ osGNN baseline accepts as input a clause-variable bipartite graph of the SAT formula and the node degrees as input attributes. Each variable x i is set to TRUE with probability p i . A graph neural network (GatedGCNN) is trained until convergence ( ∼ 50k iterations) with randomly generated formulae with clause to variable ratiot in the range [4 , 4 . 3] . The neural network is trained to produce output probabilities p i by minimizing the expected number of unsatisfied clauses. The exact closed form of the expectation can be found in (Karalias, 2023). At inference time, the probabilities are rounded to binary assignments using the method of conditional expectation.

Survey propagation . We report the results of survey propagation Braunstein et al. (2005). Typically, algorithms like survey propagation are accompanied by a local search algorithm like WalkSAT. Here we report the results obtained by the algorithm without running any local search on its output.

Low-rank SDP . This is identical to the baseline that we used for Max-3-SAT (see Max-Cut section above).

WalkSAT. We use a publicly available python implementation of WalkSAT and we set the max number of variable flips to 4 × number of variables, for a total of 400 flips on the instances we tested.

Autograd. The autograd comparison measures the performance of autograd on the same loss as OptGNN. Starting with some initial node embeddings for the instance, the Lagrangian is computed for the problem and the node embeddings are updated by minimizing the Lagrangian using Adam. This process is run for several iterations. After it is concluded, the vectors are rounded with hyperplane rounding, yielding a binary assignment for the variables of the formula.

Autograd parameters. We use the Adam optimizer with learning rate 0.1 for 1000 epochs with penalty 0.01 and round with 10,000 hyperplanes, which is 10 times as many as that used by OptGNN.

D.3 Additional Results

D.3.1 Comparisons with Gurobi and Greedy

Table 4 and Table 3 contain comparisons on additional datasets with Gurobi and Greedy.

D.3.2 Ratio tables

In Figure 6 and Figure 7 we supply the performance of OptGNN as a ratio against the integral value achieved by Gurobi running with a time limit of 8 seconds. These tables include the standard deviation in the ratio. We note that for Maximum Cut, OptGNN comes within 1.1% of the Gurobi 8s value, and for Min-Vertex-Cover, OptGNN comes within 3.1%.

D.4 Model ablation study

Here we provide the evaluations of several models that were trained on the same loss as OptGNN. We see that OptGNN consistently achieves the best performance among different neural architectures. Note that while OptGNN was consistently the best model, other models were able to perform relatively well; for instance, GatedGCNN achieves average cut values within a few percent of OptGNN on nearly all the datasets (excluding COLLAB). This points to the overall viability of training using an SDP relaxation for the loss function.

Table 6 presents the performance of alternative neural network architectures on Min-Vertex-Cover.

(a) On TU-small datasets.

TU-Small dataset hyperparameter performance for vertex_cover

Dataset: MUTAG

Dataset: ENZYMES

Dataset: PROTEINS

Dataset: IMDB-BINARY

Dataset: COLLAB

1.7.5

16

66

-20.0

68

10

25.0

18

72

70

12

27.5

SE

19

-19,07

74

14

B0.0

76

40

20

78

16

16

16

B

16

8

16

8

16

Safeunu

num_layers

num_layers

num_layers

num_layers

10

74

14

10.0

76

40

20

16

16

16

rank

rank

17.5

.84

784

20.0 -

21.38

10

22.5

70

72

12

-27.5

74

14

-B0.0

61

76

40

20

84

16

positional_encoding

RW

LE

positional_encoding

LE

RW

positional_encoding

LE

RW

positional_encoding

LE

RW

positional_encoding

LE

RW

Generated dataset hyperparameter performance for vertex_cover

Dataset: BarabasiAlbert

Dataset: ErdosRenyi

Dataset: PowerlawCluster

Dataset: WattsStrogatz

100

100

0

100

100

200

100

200

100

-105.57

200

200

300

300

400

400

300

500

400

500

400

8

16

009

16

500

16

600

16

num_layers

num_layers

num_layers

num_layers

100

100

100

100

200

100

200

300

400

300

300

500

400

400

600

500

600

32

32

16

ZE

rank

rank

rank

rank

100

200

100

200

00E-

200

100

200

100

300

400

300

300

-500

400

RW

RW

RW

600

RW

5upoouajeuog/sod

buipoouajeuogisod

positional_encoding

positional_encoding

(b) On generated datasets.

D.5 Effects of hyperparameters on performance

Figure 8 and Figure 9 present overall trends in model performance across hyperparameters.

TU-Small dataset hyperparameter performance for max_cut

(a) On

TU-small

datasets.

Dataset: MUTAG

Dataset: ENZYMES

Dataset: PROTEINS

Dataset: IMDB-BINARY

Dataset: COLLAB

30

110

110 -

1000

100

90

80

70

60

60

50

40

1250

000

Safeunu

16

8

Safeunu

16

suaAejunu

16

8

num_layers

16

num_layers

16

110

28

80

70

60

20

1500

18

40

16

000

rank

16

ZE

16

rank

16

rank

16

OE

110

110

100

100

70

20

60

08

1500

750

18

50

40

1250

16

1000

buposuajeuogisod

LE

RW

positional_encoding

RW

LE

positiona_encoding

LE

RW

positional_encoding

LE

RW

fuposuajeuogisod

LE

RW

Generated dataset hyperparameter performance for max_cut

Dataset: BarabasiAlbert

Dataset: ErdosRenyi

Dataset: PowerlawCluster

Dataset: WattsStrogatz

1750

0000

0000

0000

1500

2000

2500

5000

2000

1250

1000

1500

750

1000

500

500

500

250

500

500

250

16

16

8

16

8

16

num_layers

num_layers

num_layers

num_layers

3000

2000

1000

1000

32

16

32

rank

rank

rank

rank

3000

2500

1250

2000

1000

1500

1000

000

500

500

500

250

0

-500

RW

RW

RW

RW

positional_encoding

positional_encoding

positional_encoding

positional_encoding

(b) On generated datasets.

D.6 Out of distribution testing

OptGNN trained on one dataset performed quite well on other datasets without any finetuning, suggesting that the model can generalize to examples outside its training distribution. For each dataset in our collection, we train a model and then test the trained model on a subset of datasets in the collection. The results are shown in Table 7. It is apparent from the results that the model performance generalizes well to different datasets. Interestingly, we frequently observe that the model reaches its peak performance on a given test dataset even when trained on a different one. This suggests that the model indeed is capturing elements of a more general process instead of just overfitting the training data.

D.7 Pseudocode for OptGNN training and inference

In algorithm 4, we present pseudocode for OptGNN in the Max-Cut case and in algorithm 5 pseudocode for the forward pass of a general SDP.

Algorithm 4

Algorithm 5 OptGNN pseudocode for implementing a general SDP forward pass

Require: graph G

1: v = { v , v , . . . , v } (random initial feature vectors and/or positional encodings)

E Generalization Analysis

In this section we produce a generalization bound for OptGNN. First we restate our result that a penalized loss approximates the optimum of SDP 2. Our analysis follows from a standard perturbation argument where the key is to bound the lipschitzness of the OptGNN aggregation function. Here we will have to be more precise with the exact polynomial dependence of the lipschitzness of the gradient ∇L ρ and the smoothness of the hessian ∇ 2 L ρ .

Notation: For the convenience of the proofs in this section, with a slight abuse of notation, we will define loss functions L ρ ( V ) that take matrix arguments V instead of collections of vectors v where V is simply the vectors in v concatenated row by row. We will also refer to rows of V by indexing them with set notation v i ∈ V where v i denotes the i 'th row of V . Furthermore, let every vector be bounded in norm by some absolute constant B .

We begin by recomputing precisely the number of iterations required for algorithm 3 to approximate the global optimum of SDP 2. Note that this was done in the proof of Theorem C.1 but we do it here with explicit polynomial dependence.

Lemma E.1 (gradient descent lemma restated) . Algorithm 3 computes in O (Φ 4 ϵ -4 log 4 ( δ -1 )) iterations a set of vectors v := { ˆ v s } for all s ⊆ S ( P ) for all P ∈ P that satisfy the constraints of SDP 2 to average error ϵ and approximates the optimum of SDP 2 to error ϵ with probability 1 -δ

where SDP (Λ) is the optimum of SDP 2.

Proof. To apply the gradient descent lemma of Jin et al. (2017) Theorem F.1 we need a bound on the lipschitzness of the gradient and the smoothness of the hessian of the loss equation 72. By the lipschitz gradient Lemma E.2 we have that the loss is ℓ := poly ( B ) ρ lipschitz, and by the smooth hessian Lemma F.1 we have the loss is γ := poly(B) ρ smooth. Then we have by Theorem F.1 that perturbed gradient descent in

iterations can achieve a ( γ 2 , ϵ ′ ) -SOSP. In our setting | f ( X 0 ) -f ∗ | ≤ 1 because the loss is normalized between [0 , 1] . Our desired accuracy ϵ ′ is poly ( B -1 , ρ -1 )Φ -2 = poly ( B -1 , ϵ -1 )Φ -2 where we take ρ = poly (2 k , ϵ -1 ) as in Lemma E.1. For these settings we achieve an ϵ approximation in ˜ O (Φ 4 ϵ -4 log 4 ( δ -1 )) iterations.

Next we move on to prove the lipschitzness of the Max-CSP gradient. This is important for two reasons. First we need it to bound the number of iterations required in the proof of Lemma E.1. Secondly, the lipschitzness of the hessian will be the key quantity

Lemma E.2 (Lipschitz Gradient Lemma Max-CSP) . For a Max-CSP instance Λ , Let L ρ ( v ) be the normalized loss defined in equation 72. Then the gradient satisfies

Proof. We begin with the form of the Max-CSP gradient.

Where recall

We break the gradient ∇L ρ ( V ) up into three terms T 1 ( V ) , T 2 ( V ) , and T 3 ( V ) such that ∂ L ρ ( V ) /∂v w = T 1 ( V ) | w + T 2 ( V ) | w + T 3 ( V ) | w Where T 1 ( V ) | w , T 2 ( V ) | w , T 3 ( V ) | w are defined as follows

Such that by triangle inequality we have

Where the three terms in equation 104 are as follows.

We bound the terms one by one. First we bound term 1 .

Term 1:

We will need to define some matrices so that we can write the above expression as the frobenius inner product of matrices. First we define G Λ to be the adjacency matrix of the constraint graph. In particular we denote the ( w,w ′ ) entry of G Λ as G Λ | w,w ′ defined as follows.

Furthermore, we define M y s /C ( s ) to be a matrix comprised of a set of coefficients y s / |C ( s ) | corresponding to every edge in the consraint graph G Λ . The ( w,w ′ ) entry of M y s /C ( s ) is denoted M y s /C ( s ) | w,w ′ and defined as follows.

Then rewriting equation 109

By Cauchy-Schwarz we obtain

Next we move on to bound term 2 .

Term 2:

Let the vector δ w ′ = v w -v w ′ and let the scalar δ w,w ′ = ⟨ v w , v w ′ ⟩ - ⟨ ˆ v w , ˆ v w ′ ⟩ . Then we have by plugging definitions that

Substituting equation 115 into equation 114 in the square root we obtain

Applying triangle inequality we obtain

Let M ⟨ v w ,v w ′ ⟩ ∈ R |P| 2 k ×|P| 2 k be the matrix whose w,w ′ entry is ⟨ v w , v ′ w ⟩ . Let M δ w,w ′ be the matrix whose w,w ′ entry is δ w,w ′ . Let ⊙ denote the entrywise product of two matrices.

Applying Cauchy-Schwarz we obtain

We apply entrywise upper bound ⟨ v w , v w ′ ⟩ ≤ B 2 which can be done because we're taking a frobenius norm so the sign of each entry does not matter. Likewise we apply a crude entrywise upper bound of δ w,w ′ ≤ B 2 again because the sign of each entry does not matter in the frobenius norm (for each row this is euclidean norm).

Using the fact that ∥ G λ ∥ ≤ 2 k

Therefore we have established

Finally we move on to bound term 3 .

Term 3:

Applying triangle inequality we obtain

Let δ v w := v ′ w -v w then substituting the definition we obtain = 2 ρ |P| √ ∑ w ∈F [ ∥ ∥ ∥ ∥ v w ∥ 2 v w -∥ v w + δ v w ∥ 2 ( v w + δ v w ) ∥ ∥ ∥ 2 ] + 2 ρ |P| ∥ V -ˆ V ∥ F (127)

expanding the expression in the square root we obtain

By triangle inequality we upper bound by

For the first term consider by Cauchy-Schwarz

Consider the second term which we upper bound via the norm bound on ∥ v w ∥ ≤ B

Consider the third term which we upper bound via the norm bound on ∥ v w ∥ ≤ B

Consider the fourth term which we upper bound via Cauchy-Schwarz

Consider the fifth term. We apply a crude upper bound of ∥ δ v w ∥ 2 ≤ B 2

Therefore we conclude

So we conclude our bound on term 3

as desired. Putting our bounds for terms 1 , 2 , and 3 into equation 104 we obtain

as desired.

Lemma E.3 (Max-CSP gradient perturbation analysis) . For any set of matrices M := { M 1 , M 2 , ..., M T } ∈ R 2 r × r a perturbation U := { U 1 , U 2 , ..., U T } ∈ R 2 r × r that satisfies ∥ U i ∥ 1 , 1 ≤ ϵ for all i ∈ [ T ] . Let M + U denote the elementwise addition of M and U as such M + U := { M 1 + U 1 , M 2 + U 2 , ..., M T + U T } . Then for a matrix V ∈ R r × N satisfying ∥ V ∥ F ≤ √ d we define the aggregation function AGG : R r × N → R 2 r × N as such

Furthermore, let LAYER M i ( V ) denote

Finally, let OptGNN M : R r × N → R be defined as

Here we feed the output of the final layer LAYER T to the loss function L : R r × N → R . Then

Proof. We begin by analyzing how much the gradient perturbs the input to a single layer. First we apply the definition of LAYER and AGG to obtain

Here the first inequality follows by triangle inequality. The second inequality follows by CauchySchwarz. Finally the frobenius norm of U is ϵr the frobenius norm of V is Then applying the lipschitzness of the gradient Lemma E.2 we obtain.

To conclude, we've established that

Next we upper bound the lipschitzness of the LAYER function by the frobenius norm of ∥ M ∥ F ≤ O ( r ) multiplied by the lipschitzness of ∇L ρ which is O ( B 2 ρ ) . Taken together we find the the lipschitzness of LAYER is upper bounded by O ( B 2 ρr ) .

Note that OptGNN is comprised of T layers of LAYER functions followed by the evaluation of the loss L . The T layers contribute a geometric series of errors with the dominant term being O ( ϵr 3 / 2 B 2 ρ ∗ ( B 2 ρr ) T ) = O ( ϵ ( B 2 ρr ) 2 T ) . The smaller order terms contribute no more than an additional multiplicative factor of T leading to an error of O ( Tϵ ( B 2 ρr ) 2 T ) . At any rate, the dominant

term is the exponential dependence on the number of layers. Finally, OptGNN's final layer is the evaluation of the loss L ρ . The lipschitzness of the loss L ρ can be computed as follows. For any pair of matrices V and ˆ V both in R r × r we have

where we used the fact that the loss is dominated by its quadratic penalty term ρ times the square of the violations where each row in V can be counted in up to 2 k constraints normalized by the number of predicates |P| . Putting this together applying the LAYER function over T layers we obtain

At this point we restate some elementary theorems in the study of PAC learning adapted for the unsupervised learning setting.

Lemma E.4 (Agnostic PAC learnability (folklore)) . Let x 1 , x 2 , ..., x N ∼ D be data drawn from a distribution D . Let H be any finite hypothesis class comprised of functions h ∈ H where the range of h in bounded in [0 , 1] . Let ˆ h ∈ H be the empirical loss minimizer

Let δ be the failure probability and let ϵ be the approximation error. Then we say H is ( N,ϵ, δ ) -PAC learnable if

Furthermore, H is ( N,ϵ, δ ) -PAC learnable so long as

The proof is a standard epsilon net union bound argument which the familiar reader should feel free to skip.

Proof. For any fixed hypothesis h the difference between the distributional loss between [0 , 1] and the empirical loss can be bounded by Hoeffding.

Let ˆ h ∈ H be the empirical risk minimizer within the hypothesis class H on data { x 1 , x 2 , .., x N } . What is the probability that the empirical risk deviates from the distributional risk by greater than ϵ ? i.e we wish to upper bound the quantity

Of course the biggest caveat is that ˆ h depends on x 1 , ..., x N and thus Hoeffding does not apply. Therefore we upper bound by the probability over draws x 1 , ..., x N ∼ D that there exists ANY hypothesis in H that deviates from its distributional risk by more than ϵ .

This follows because the event that ˆ h deviates substantially from its distributional loss is one of the elements of the union on the right hand side.

Then we have via union bound that

where the last line follows by Hoeffding. In particular if we want the failure probability to be δ then

Suffices for the ˆ h ∈ H to deviate from its empirical risk by less than ϵ .

Finally putting our perturbation analysis Lemma E.3 together with the agnostic PAC learnability Lemma E.4

Lemma E.5. Let Λ 1 , Λ 2 , ..., Λ Γ ∼ D be Max-CSP instances drawn from a distribution over instances D with no more than |P| predicates. Let M be a set of parameters M = { M 1 , M 2 , ..., M T } in a parameter space Θ . Then for T = O (Φ 4 ) , for Γ = O ( 1 ϵ 4 Φ 6 log 4 ( δ -1 )) , let ˆ M be the empirical loss minimizer

Then we have that OptGNN is (Γ , ϵ, δ ) -PAC learnable

Proof. The result follows directly from the agnostic PAC learning Lemma E.4 and the perturbation analysis Lemma E.3. We have that for a net of interval size ϵ r 2 T suffices for an ϵ approximation. The cardinality of the net is then r 2 T /ϵ per parameter raised the power of the number of parameters required for OptGNN to represent an ϵ approximate solution to SDP (Λ) . Each LAYER is comprised of a matrix M i of dimension r × 2 r for r = Φ . By Lemma E.1 we need a total of T = O (Φ 4 ϵ -4 log 4 ( δ -1 )) layers to represent an ϵ optimal solution with probability 1 -δ . Then the total number of parameters in the network is O (Φ 6 ϵ -4 log 4 ( δ -1 )) as desired.

which implies

F Hessian Lemmas

This section is to prove the hessian ∇ 2 L ρ is smooth. This is relevant for bounding the number of iterations required for algorithm 3.

Lemma F.1 (Smooth Hessian Lemma) . For a Max-CSP instance Λ , Let L ρ ( v ) be the normalized loss defined in equation 72. Then the hessian satisfies

In particular for ρ = poly ( k, ϵ -1 ) we have that the hessian is poly ( B,k,ϵ -1 ) smooth.

Proof. Next we verify this for the Max-CSP hessian. The form of its hessian is far more complex. To simplify matters we first consider the hessian of polynomials of the form T ( v i , v j , v k , v ℓ ) = ( ⟨ v i , v j ⟩ - ⟨ v k , v ℓ ⟩ ) 2

We can decompose the Hessian as a sum of 10 matrices corresponding to the cases enumerated above.

Then we can compute

Noticing that the first four terms have the same upper bound, terms 5 and 6 have the same upper bound, and terms 7 through 10 share the same upper bound we obtain

Now consider the first term

Let δ j = ˆ v j -v j and δ ℓ = ˆ v ℓ -v ℓ . We expand equation 154 to obtain

where we apply Cauchy-Schwarz as needed. The second term is bounded similarly. The last term can be bounded

Upper bounding v ja + ˆ v ja ≤ 2 B which applies for all a ∈ [ r ] we obtain

Putting the terms together we obtain the following bound for

Now we can perform the analysis for the hessian of the entire Max-CSP loss

We break the loss into three terms

such that L ρ ( v ) := W 1 ( v ) + W 2 ( v ) + W 3 ( v ) . We break the hessian apart into three terms

The first term is zero as its the hessian of a quadratic which is a constant. The difference is then zero. We bound the second term as follows.

Noticing that each δ g can be involved in no more than |P| sums by being variables in every single predicate.

Now we move on to bound W 3

Putting all three terms together we obtain the smoothness of the hessian is dominated by W 2 .

F.1 Miscellaneous Lemmas

Theorem F.1 (perturbed-gd Jin et al. (2017)) . Let f be ℓ -smooth (that is, it's gradient is ℓ -Lipschitz) and have a γ -Lipschitz Hessian. There exists an absolute constant c max such that for any δ ∈ (0 , 1) , ϵ ≤ ℓ 2 γ , ∆ f ≥ f ( X 0 ) -f ∗ , and constant c ≤ c max , PGD ( X 0 , ℓ, γ, ϵ, c, δ, ∆ f ) applied to the cost function f outputs a ( γ 2 , ϵ ) SOSP with probability at least 1 -δ in

iterations.

Definition. [ ( γ, ϵ ) -second order stationary point] A ( γ, ϵ ) second order stationary point of a function f is a point x satisfying

∥∇

(

λ

min

∇

f

2

f

(

x

(

)

x

))

ϵ

≥ - √

γϵ.

Theorem F.2. (Robustness Theorem 4.6 (Raghavendra & Steurer, 2009a) rephrased) Let v be a set of vectors satisfying the constraints of SDP 2 to additive error ϵ with objective OBJ ( v ) , then

Corollary 2. Given a Max-k-CSP instance Λ , there is an OptGNN M, Λ ( v ) with T = O ( ϵ -4 Φ 4 log( δ -1 )) layers, r = O (Φ) dimensional embeddings, with learnable parameters M = { M 1 , ..., M T } that outputs a set of vectors V satisfying the constraints of SDP 2 and approximating its objective, SDP (Λ) , to error ϵ with probability 1 -δ over random noise injections.

Proof. The proof is by inspecting the definition of OptGNN in the context of Theorem 3.1.

Corollary 3. The OptGNN of Corollary 2 , which by construction is equivalent to Algorithm 3, outputs a set of embeddings v such that the rounding of Raghavendra & Steurer (2009a) outputs an integral assignment V with a Max-k-CSP objective OBJ( V ) satisfying OBJ ( V ) ≥ S Λ ( SDP (Λ) -ϵ ) -ϵ in time exp ( exp ( poly ( kq ϵ ))) which approximately dominates the Unique Games optimal approximation ratio.

Proof. The proof follows from the robustness theorem of Raghavendra & Steurer (2009a) which states that any solution to the SDP that satisfies the constraints approximately does not change the objective substantially Theorem F.2.

G Definitions

In this section we introduce precise definitions for message passing, message passing GNN, and SDP's. First we begin with the SDP.

Definition (Standard Form SDP) . An SDP instance Λ is comprised of objective matrix C ∈ R N × N and constraint matrices { A i } i ∈F and constants { b i } i ∈F over a constraint set F

For any standard form SDP, there is an equivalent vector form SDP with an identical optimum.

Definition (Vector Form SDP) . Any standard form SDP Λ is equivalent to the following vector form SDP.

For the SDP's corresponding to CSP's, there is a natural graph upon which the CSP instance is defined. Next we define what it means for a message passing algorithm on a graph to solve a SDP.

Definition (Message Passing) . A T iteration message passing algorithm denoted MP is a uniform circuit family that takes as input a graph G = ( V, E ) and initial embeddings U = { u 0 i } i ∈ [ N ] ∈ R r × N and outputs MP ( G,U 0 ) ∈ R r × N . The evaluation of MP involves the use of a uniform circuit family defined for each iteration UPDATE = { UPDATE j } j ∈ [ T ] . At iteration ℓ ∈ [ T ] , for each node i ∈ V , the function UPDATE ℓ : R r × ( | N ( i ) | +1) → R r takes the embeddings of nodes i and its neighbors at iteration ℓ -1 , denoted { u ℓ -1 j } j ∈ N ( i ) ∪ i ∈ R r , and outputs embedding u ℓ i ∈ R r .

We additionally require some mild restrictions on the UPDATE function to be polynomially smooth in its inputs and computable in polynomial time. That is for any ∥ U ∥ ≤ B in the B norm ball we have

and the UPDATE ℓ circuit is polynomial time computable poly( | N ( i ) | , r ) .

We impose some mild restrictions on the form of the message passing algorithm to capture algorithms that could reasonably be executed by a Message Passing GNN. In practice the UPDATE circuit for OptGNN is a smooth cubic polynomial similar to linear attention and therefore of practical value. Next we define Message Passing GNN.

Definition (Message Passing GNN) . A T layer Message Passing GNN is a uniform circuit family that takes as input a graph G = ( V, E ) , a set of parameters M := { M 1 , M 2 , ..., M T } , and initial embeddings U 0 = { u 0 i } i ∈ V ∈ R r × N and outputs GNN ( G,M,U 0 ) ∈ R r × N . The GNN circuit is evaluated as follows. For each node i ∈ V , each layer ℓ ∈ [ T ] , there is a uniform circuit family AGG ℓ : R | M ℓ | × R r ×| N ( i ) | → R r that takes as input a set of parameters M ℓ ∈ M and a set of

embeddings of node i and its neighbors at layer ℓ -1 denoted { u ℓ -1 j } j ∈ N ( i ) ∪ i , and outputs an embedding u ℓ i . This update equation is represented as follows

For some functions UPD and MSG parameterized by M ℓ . To capture meaningful models of GNNs, we require AGG ( M,U ) to be polynomially lipschitz and computable in polynomial time. That is, for weights ˆ M and M ′ and for inputs ˆ U and U ′ all in the B norm ball, we require

and we require AGG ( ˆ M, ˆ U ) to be computable in poly ( | N ( i ) | , r ) time.

In practice the AGG circuit for OptGNN is a smooth cubic polynomial similar to linear attention and therefore of practical value. In almost the same way, we define a message passing algorithm see Definition G.

Definition (Output of GNN solves SDP) . We say a GNN ϵ -approximately solves an SDP with optimum OPT if its output embeddings U ℓ = { u i } i ∈ [ N ] approximately optimize the vector form SDP objective SDP ( { u i } i ∈ [ N ] ) ≥ OPT -ϵ and approximately satisfy the vector form constraints ∣ ∣ ⟨ A i , V T V ⟩ -b i ∣ ∣ ≤ ϵ .

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Main claims are message passing GNN for Max-CSP Theorem C.1 Corollary 1, OptGNN experiments for Max Cut, Vertex-Cover, 3-SAT (Figure 2b, Table 1, Table 2) neural certification experiment Figure 3.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We compare our results against a broad variety of neural and classical baselines (WalkSAT, Survey Propagation, Gurobi, CVXPY, SDP's, etc.), perform extensive ablations on a host of datasets Tables 5,6 and Figures 5-9 and situate our theory within the existing literature on approximation algorithms with SDP's (see references in intro). Computational efficiency of OptGNN and message passing algorithm are stated adjacent to their definitions.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Main theorem informally stated Theorem 3.1 is restated formally in body of paper as Theorem C.1 with accompanying proof. Main PAC learning result is stated in paper 3.1 and restated in appendix and proven as E.5. Certification proof is provided in C.1.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper provides a thorough description of the experimental details (see Appendix D), as well as pseudocode for the proposed method (see Algorithm 5). Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Link to the code along with instructions is provided in the main text.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes, the experimental setup is thoroughly described in the Appendix D and in the main experiments section of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports standard deviations for several experiments (e.g., Figure 3b, Table 2, Figure 6, Figure 7). In cases such as Figure 3a where the setup is identical to previous published work, the deviations are not available because they were not provided in the original paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The resource that were used are detailed in Appendix D. Execution time is also discussed there but also explicitly provided in tables 1,2,3,4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: After reviewing the code of Ethics, the authors find that the paper does not violate it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper provides a general theoretical result about graph neural network architectures and experimental backup for its claims on general benchmark instances. There are no immediate societal implications to this line of work. On the other hand, improving aspects of neural networks such as their combinatorial problem-solving (which is the case here) can open up avenues for their misuse. However, we believe that those consequences are rather indirect and beyond the scope of our paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The released models and the data used do not have a high risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper provides appropriate citations for packages and data that have been used and their licenses are properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Yes we provide a detailed description of the code that is being provided with the submission.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not include any such experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper has no such research or experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper draws analogy between the optimization of, e.g., Max-Cut and Max-SAT problems, and the message-passing algorithm, and accordingly constructs OptGNN to implement the message-passing algorithms towards the problems. The generated optimal solutions are shown with provable bounds. Finally, empirical studies show the effectiveness and efficiency of the proposed OptGNN.

STRENGTHS:
1. The logic of the paper in writing/presentation are absolutely easy to read and understand, and the project in design are reasonable.
2. The proposal is presented with theoretical guarantees, i.e. the bounds of the optimal solutions are shown.
3. The proposed framework OptGNN is easy to implement and the empirical studies are consistent and demonstrate the effectiveness of the proposal.
4. The discussion of related work is extensive and satisfactory.

WEAKNESSES:
1. Take the example of Max-Cut problem as an example, how to show the optimization problem are the same in equation (2) and the message-passing diagram shown in equation (3)? Same for other optimization problems discussed in the paper.

QUESTIONS:
1. In Table 2, what's the usual setting for the clause? Since the ratio is relatively high. What's the relationship between the total computational complexity and this ratio?

LIMITATIONS:
Based on my review, this part is not well addressed by the authors. But since myself is not an expert in optimization approximation algorithms, please the authors/ACs kindly refer to the weaknesses and questions raised by pother reviewers.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-04

comments: SUMMARY:
The papers propose graph neural architectures that can be used to capture optimal approximation algorithms for a large class of combinatorial optimization problems.

STRENGTHS:
- The paper for the most part is well written with clear motivation.
- The contributions made in the paper are manifold across various optimization problems like Max-cut, Min-vertex cover etc. and have shown commendable performance.
- The paper also showcases a good theoretical basis.

WEAKNESSES:
No significant weakness as such. 
Fig1. and Fig. 2 the visibility could be improved.

QUESTIONS:
NA.

LIMITATIONS:
Yes, the authors has provided limitations section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-13

comments: SUMMARY:
The paper establishes that polynomial-sized message-passing GNNs, can learn and replicate the optimal approximation capabilities of traditional algorithms based on SDP relaxations for Max-CSP under the assumption of the UGC.  The authors propose OptGNN, which effectively integrates the theoretical framework of SDP to produce high-quality solutions for combinatorial problems such as Max-Cut, Min-Vertex-Cover, and Max-3-SAT.

STRENGTHS:
The paper explores integrating semidefinite programming into GNNs under the Unique Games Conjecture to optimally approximate combinatorial optimization problems. It introduces the OptGNN, a model designed to utilize polynomial-time message-passing algorithms and low-rank SDP relaxations to effectively solve problems such as Max-Cut and Max-3-SAT. The paper establishes OptGNN's potential through PAC learning arguments and empirical validation against traditional solvers and neural baselines. The paper is well-structured and mathematically sound.

WEAKNESSES:
- The paper lacks a comprehensive analysis of how well OptGNN scales with increasing graph sizes and complexity.
- Although the out-of-distribution generalization of OptGNN has been tested, an analysis of its generalization over problem size is missing.

QUESTIONS:
1. Could you clarify how the proposed approach scales with increasingly large problem instances, particularly regarding computational complexity and performance stability?
2. What are the generalization capabilities over problem scale of the proposed approach?
3. What are the limitations of OptGNN? please explain in the text.

LIMITATIONS:
Authors need to discuss the limitation of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a significant advancement in the field of combinatorial optimization by developing a graph neural network (GNN) architecture named OptGNN. The authors demonstrate that OptGNN can capture optimal approximation algorithms for a broad class of combinatorial optimization problems, leveraging the power of semidefinite programming (SDP) and the Unique Games Conjecture (UGC). They prove that polynomial-sized message-passing GNNs can learn the most powerful polynomial-time algorithms for Max Constraint Satisfaction Problems, resulting in high-quality approximate solutions for challenges such as Max-Cut, Min-Vertex-Cover, and Max-3-SAT. Additionally, OptGNN provides a method for generating provable bounds on the optimal solution from the learned embeddings. The empirical results show that OptGNN outperforms classical heuristics, solvers, and state-of-the-art neural baselines across various datasets, making it a robust tool for efficient and optimal combinatorial optimization.\n\nSTRENGTHS:\n1. The paper is theoretically solid. It builds on well-established concepts in approximation algorithms and semidefinite programming, providing a strong theoretical basis for its claims. \n2. The development of OptGNN, a graph neural network that can capture optimal message-passing algorithms for combinatorial optimization problems, seems to be a significant innovation. This architecture leverages message-passing updates use SDP relaxations to prove its high-quality approximate solutions.\n3. The paper includes a variety of evaluations, such as out-of-distribution tests and ablation studies, which help validate the practical utility and robustness of OptGNN.\n\nWEAKNESSES:\n1. Although OptGNN shows strong performance on several benchmarks, it does not outperform all state-of-the-art methods. This indicates room for improvement in the model’s optimization and training processes.\n2. The theoretical guarantees provided by OptGNN rely on the truth of the UGC, which, while widely believed, remains unproven. This dependence could limit the certainty of the results. That being said, I consider this more like a limitation than weakness.\n3. The writing needs to be improved, there are multiple typos and inconsist notations.\n\nQUESTIONS:\n1. Can you elaborate on the scalability of OptGNN for very large graphs or more complex real-world problems? What are the practical limitations in terms of computational resources and time?\n2. How adaptable is OptGNN to new combinatorial optimization problems that were not directly addressed in the paper? What modifications would be necessary to extend its applicability?\n3. Can you also point out some future directions for this research? Both application and theoretical wise are good.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-26"}