Title: On the Power of Small-size Graph Neural Networks for Linear Programming
Venue: NeurIPS 2024
Accepted: yes
Keywords: Learning to optimize

graph neural network

linear programming

gradient descent

efficiency
Abstract: Graph neural networks (GNNs) have recently emerged as powerful tools for addressing complex optimization problems. It has been theoretically demonstrated that GNNs can universally approximate the solution mapping functions of linear programming (LP) problems. However, these theoretical results typically require GNNs to have large parameter sizes. Conversely, empirical experiments have shown that relatively small GNNs can solve LPs effectively, revealing a significant discrepancy between theoretical predictions and practical observations. In this work, we aim to bridge this gap by providing a theoretical foundation for the effectiveness of small-size GNNs. We prove that polylogarithmic-depth, constant-width GNNs are sufficient to solve packing and covering LPs, two widely used classes of LPs. Our proof leverages the capability of GNNs to simulate a variant of the gradient descent algorithm on a carefully selected potential function. Additionally, we introduce a new GNN architecture, termed GD-Net. Experimental results demonstrate that GD-Net significantly outperforms conventional GNN structures while using fewer parameters.

On the Power of Small-size Graph Neural Networks for Linear Programming

Abstract

Graph neural networks (GNNs) have recently emerged as powerful tools for addressing complex optimization problems. It has been theoretically demonstrated that GNNs can universally approximate the solution mapping functions of linear programming (LP) problems. However, these theoretical results typically require GNNs to have large parameter sizes. Conversely, empirical experiments have shown that relatively small GNNs can solve LPs effectively, revealing a significant discrepancy between theoretical predictions and practical observations. In this work, we aim to bridge this gap by providing a theoretical foundation for the effectiveness of smaller GNNs. We prove that polylogarithmic-depth, constant-width GNNs are sufficient to solve packing and covering LPs, two widely used classes of LPs. Our proof leverages the capability of GNNs to simulate a variant of the gradient descent algorithm on a carefully selected potential function. Additionally, we introduce a new GNN architecture, termed GD-Net. Experimental results demonstrate that GD-Net significantly outperforms conventional GNN structures while using fewer parameters.

1 Introduction

Learning to Optimize (L2O) has emerged as a compelling research area, leveraging machine learning techniques to enhance the efficiency of optimization processes. Unlike traditional theory-driven optimization methods, L2O approaches are primarily data-driven, learning optimization strategies from existing problem instances. This new paradigm has yielded notable advancements in both continuous optimization [22] and combinatorial optimization [4, 21].

Recently, graph neural networks (GNN) have become increasingly popular in the field of L2O [6, 27]. GNNs are a class of neural networks specifically designed to process and analyze data structured as graphs, leveraging the relationships between nodes to perform tasks such as node classification, link prediction, and graph classification [35]. Due to their properties of permutation equivariance and natural adaptation to varying input dimensions, GNNs are well-suited for graph-related optimization problems such as minimum vertex covering [30] and traveling salesman [13]. Recent research has demonstrated that GNNs can effectively accelerate the solving process for both linear programming (LP) and mixed integer linear programming (MILP) problems, which are among the most important and widely applied types of optimization problems. For instance, Li et al. [17] proposed a GNN-based reformulation method for LP to enhance the solver performance. Furthermore, Chen et al. [8] and

Qian et al. [29] explored the potential of GNNs to approximate the solution mapping of LP and reported encouraging results. Additionally, several studies have proposed various approaches to guide MILP solvers with GNNs [10, 11, 14, 18, 26, 32].

Despite the numerical success, the theoretical foundation of using GNN to solve optimization problems remains less clear. Initial steps towards a theoretical understanding have been made by Chen et al. [8] and Qian et al. [29]. They demonstrated that with a sufficient number of parameters, GNNs can approximate the solution mapping of LPs with arbitrary precision. However, a significant gap persists between the theoretical progress and empirical evidence. The proof in [8] relies heavily on the universal approximation theorem for multi-layer perceptions (MLP), which necessitates a large number of parameters in principle. Similarly, the result of [29] requires the depth of GNN to be polynomial in the problem dimension. In practice, however, GNN with a modest width and fewer than ten layers often suffice to achieve good performance in approximating the optimal solution of LP with hundreds of nodes and constraints.

This gap between theory and practice raises an intriguing open question:

When and why can small-size GNNs effectively solve LPs?

As larger networks usually require more training examples and higher computational resources, addressing this question is not only of theoretical interest but also provides potential guidance to L2O practitioners. While current GNNs may not yet rival theoretically grounded LP solvers in precision, they could still accelerate LP solving by providing high-quality initializations to warm-start traditional solvers. Understanding the principles behind the success of small-sized GNNs can potentially lead to more parameter-efficient models, thus reducing the computational resources required for LP solving. Moreover, these insights could potentially enhance MILP solvers. Specifically, the leading approach for MILP is the branch-and-bound framework, which involves selecting variables and dividing the search space. A prevalent variable selection approach is strong branching, where variables are scored by solving associated LPs to guide the branching decisions. Researchers have exploited GNNs to score these variables, with GNNs essentially solving the associated LPs [9-11, 23, 31]. By uncovering the mechanism behind the success of small-sized GNNs in solving LPs, these models could be further improved, leading to more efficient MILP solving.

1.1 Our contribution

In this paper, we show that polylogarithmic-depth constant-width GNNs can approximate the solution mapping for a broad class of LPs, namely packing LPs and covering LPs, with arbitrary precision (Theorem 3 and Theorem 5). This result advances previous theoretical results on general LPs [8, 29], narrowing the gap between existing theoretical progress and empirical evidence.

Packing and covering LPs and their norm form. A packing LP and its dual covering LP are nonnegative LPs of the canonical form: max { c T · x | Ax ≤ b , x ≥ 0 } and min { b T · y | A T y ≥ c , y ≥ 0 } respectively. Here, A ∈ R n × m ≥ 0 , b ∈ R n ≥ 0 , and c ∈ R m ≥ 0 . Packing LPs and covering LPs are broad classes of linear programming problems that can be used to approximate or relax a wide range of fundamental problems in combinatorial optimization. They include fractional versions of the vertex cover problem, the set cover problem, the hypergraph matching problem, the dominating set problem, and the maximum independent set problem. Without loss of generality, we assume the packing and covering LPs are presented in their normal form [2], where the vectors b and c are all-ones vectors and A ij is either zero or greater than 1. The details of the reduction to the normal form can be found in the appendix.

Our results. We present a new theoretical explanation of the phenomenon that small-size GNNs can solve general packing LPs and covering LPs, by unrolling gradient descent algorithms. Specifically, we first propose a variant of the gradient descent (GD) algorithm proposed by Awerbuch and Khandekar [2] for solving packing and covering LPs (Algorithms 1 and 2), so that our variant can be more naturally simulated by GNNs, where one iteration of the algorithm can be simulated by one GNN layer of constant width. Importantly, our variant of GD is guaranteed to output a (1 + ϵ ) -approximate solution in polylog( mnA max /ϵ ) iterations. Therefore, we affirm the feasibility of solving general packing and covering LPs with polylogarithmic-depth constant-width GNNs. Here, we remark that the polylogarithmic dependency on depth is also necessary. Specifically, Kuhn et

al. [16] showed that: for the fractional maximum matching problem, a special kind of packing LP, every constant-factor approximation distributed algorithm requires at least Ω( √ log n/ log log n ) rounds. Moreover, since one layer of GNNs can be naturally simulated by one round of distributed LP algorithms (see, e.g., the second paragraph on page 5 in [16]), we conclude that GNNs need at least Ω( √ log n/ log log n ) layers.

Building on our theoretical proof, we propose a new GNN architecture, termed GD-Net, for solving packing and covering LPs. We empirically demonstrate that, when appropriately trained, GD-Net can effectively adapt to given problem instance distributions. Experiments across various datasets demonstrate that GD-Net outperforms classical graph convolutional network (GCN), a predominant GNN architecture in L2O for LP and MILP solving. Specifically, GD-Net generates better solutions with an order of magnitude fewer parameters than GCN. These performance enhancements become increasingly significant as the problem dimensions expand. The numerical success of GD-Net not only corroborates our theoretical framework but also suggests a potential direction for designing more parameter-efficient GNNs for L2O. This direction involves developing architectures that can simulate well-established, theoretically grounded algorithms, potentially leading to improvements in both computational efficiency and solution quality.

1.2 Related works

The design of GD-Net is based on the concept of unrolling iterative algorithms as GNNs. Indeed, there is a body of research that has explored this approach. For example, Velickovic et al. [33] investigated solving basic graph problems (e.g., the shortest path, the minimum spanning tree) by GNNs. By unrolling classical graph algorithms (e.g., breadth-first search, Prim's algorithm) as GNNs, they suggest that message-passing neural networks with a maximization aggregator may be best suited for such graph problems. Aiming at mitigating oversmoothing, long-range dependencies, and spurious edges issues of GNNs, Yang et al. [36] proposed a new family of GNN layers by unrolling and integrating the update rules of two classical iterative algorithms, namely the proximal gradient descent and iterative reweighted least squares. Papers [20, 38] showed that many existing GNN models (such as GCN, GAT, APPNP) can be viewed as unrolling gradient descent serving specific graph signal denoising problems. Chen et al. [7] proposed new GNNs to improve graph signal denoising by unrolling sparse coding and trend filtering algorithms. Papers [24, 39, 37] bridge the gap between graph convolution and iterative algorithms by providing a unified optimization framework for GNNs.

2 Gradient descent algorithm for packing and covering LPs

Throughout the paper, we use boldface to represent vectors or matrices, e.g., x and A . Let 1 p × q denote the p × q -dimensional all-ones matrix, and 1 p denote the p -dimensional all-ones column vector. Similarly, we can define 0 p × q and 0 p .

Algorithm for packing LPs. Awerbuch and Khandekar [2] proposed a simple (1+ ϵ ) -approximation algorithm for packing LPs of the normal form:

where A ij , b i , c j are all non-negative and A ij is either 0 or ≥ 1 . The algorithm is described as follows: given parameters µ = 1 ϵ · ln mA max ϵ , α = ϵ 4 , β = α 10 µ , and δ = α 10 µnA max ;

Start with x := 0 m ; Then, repeat the following procedure:

y i := exp[ µ · ( A i x -1)] for each i ∈ [ n ] ;

For any j ∈ [ m ] do

(a) If A T j y ≤ 1 -α , then x j := max { x j · (1 + β ) , δ } ;

(b) If A T j y ≥ 1 + α , then x j := x j · (1 -β ) .

Algorithm 1: Our variant of Awerbuch-Khandekar algorithm for packing LPs

1 Input : A n × m matrix A where A ij is either zero or no less than 1, and ϵ > 0 .

2 Parameter: µ := 1 ϵ · ln mA max ϵ , α = ϵ 4 , β = α 10 µ , and δ = α 10 µnA max ; 3 Initialize x 0 := 0 ; 4 for k = 0 to K -1 do 5 y k i := exp[ µ ( A i x k -1)] for any i ∈ [ n ] ; 6 for each j ∈ [ m ] do 7 if A T j y k ≤ 1 -α then x k +1 j := x k j · (1 + β ) + δ ; 8 if A T j y k ≥ 1 + α then x k +1 j := x k j · (1 -β ) ; 9 Output : x K .

The intuition behind this algorithm is that it can be viewed as applying gradient descent on a carefully chosen potential function defined as

which enjoys the following nice properties [2]: (i) Φ p ( x ) is differentiable and convex; (ii) ∇ Φ p ( x ) = A T y ; and (iii) any stationary point of Φ p is a nearly optimal solution to the packing LP (1): For any x where ∇ Φ p ( x ) = 0 , x is feasible and 1 T · x ≥ (1 -4 ϵ ) · OPT . It turns out that this algorithm has polylogarithmic convergence [2]. Precisely, given any ϵ > 0 , this algorithm always maintains a feasible solution (i.e. x ≥ 0 and Ax ≤ 1 always hold), and returns a (1 + ϵ ) -approximation solution to the packing LP (1) in ˜ O (ln 2 ( mA max ) · ln 2 ( nA max ) /ϵ 5 ) iterations. Here ˜ O hides lower order terms like ln ln( mnA max ) and ln(1 /ϵ ) .

Algorithm for covering LPs. In the same paper [2], Awerbuch and Khandekar also considered solving the dual covering LP of the normal form

and designed the following algorithm: given µ := 1 ϵ · ln nA max ϵ , α = ϵ 4 , β = α 20 µ , and δ = α 20 µmA max ,

Start with y := 1 n ; Then repeat the following procedure:

x j := exp[ µ · (1 -A T j y )] for each j ∈ [ m ] ;

For any i ∈ [ n ] do

(a) If A i x ≥ 1 + α , then y i := max { y i · (1 + β ) , δ } ;

(b) If A i x ≤ 1 -α , then y i := y i · (1 -β ) .

Similarly, this algorithm can also be thought of as applying a variant of gradient descent on a carefully selected potential function [2]:

which satisfies that: (i) Φ c ( y ) is differentiable and convex; (ii) ∇ Φ c ( y ) = 1 -Ax ; and (iii) for any y where ∇ Φ d ( y ) = 0 , y is feasible and 1 T · y ≤ (1 + 4 ϵ ) · OPT . This algorithm was shown to enjoy polylogarithmic convergence as well [2]. Formally, this algorithm always maintains a feasible solution (i.e. y ≥ 0 and A T y ≥ 1 always hold), and returns a (1 + ϵ ) -approximation solution to the covering LP (2) in ˜ O (ln 2 ( nA max ) · ln 2 ( mnA max ) /ϵ 5 ) iterations.

Avariant of Awerbuch-Khandekar algorithms. Wepropose a variant of the Awerbuch-Khandekar algorithms, specifically Algorithm 1 for solving packing LPs (1) and Algorithm 2 for covering LPs (2). The motivation and advantage of our variant is that it can be simulated more naturally by GNNs.

Basically, we replace x j ← max { x j (1 + β ) , δ } with x j ← x j (1 + β ) + δ in their algorithm for packing LPs, and y j ← max { y j (1 + β ) , δ } with y j ← y j (1 + β ) + δ in their algorithm for covering LPs. Through almost the same proof, one can verify that the polylogarithmic convergence still holds.

Theorem 1. Algorithm 1 always maintains a feasible solution and returns a (1 + ϵ ) -approximation solution to the packing LP (1) in ˜ O (ln 2 ( mA max ) · ln 2 ( nA max ) /ϵ 5 ) iterations.

Algorithm 2 always maintains a feasible solution and returns a (1 + ϵ ) -approximation solution to the covering LP (2) in ˜ O (ln 2 ( nA max ) · ln 2 ( mnA max ) /ϵ 5 ) iterations.

Connection to GNN. For LPs, an instance can be naturally encoded as a labeled bipartite graph [1, 3, 15, 25], where (a) a left node represents a variable, (b) a right node represents a constraint (and equivalently, the associated dual variable), and (c) a left node and a right node are connected if the corresponding variable participates the corresponding constraint. In our GD algorithms, the matrix-vector multiplication Ax (or A T y ) can be interpreted as a message passing step from left nodes to right nodes (or from right nodes to left nodes). In the next sections, we will demonstrate how one iteration of our GD algorithms can be transformed into a single layer of GNN.

3 Design of packing GD-Net

By combining the idea of Algorithm 1 and techniques from graph neural networks, we design a graph neural network architecture, named packing GD-Net, for solving packing LPs.

One iteration of Algorithm 1 consists of two steps: first, the algorithm updates y from x , and calculates the gradient ∇ Φ p ( x ) = A T y -1 ; second, using ∇ Φ p ( x ) , it applies a variant of gradient descent to update x . Our packing GD-Net modifies the second step by replacing it with a learnable neural network block, while leaving the first step unchanged. Thus, the packing GD-Net can also be viewed as utilizing a neural network block to accelerate the convergence of the gradient descent.

ELU activation for y -update. We apply the Exponential Linear Unit (ELU) activation function to replicate the y -update: y k i = exp[ µ ( A i x k -1)] . We will fix the parameter α in ELU to 1 , then ELU( t ) + 1 = exp( t ) for t ≤ 0 . In addition, as mentioned in Theorem 1, A i x k -1 ≤ 0 always holds in the execution of Algorithm 1. Thus the ELU function can exactly replicate the y -update. Specifically, our packing GD-Net updates y as follows:

Since α is fixed to 1 and µ is fixed to 1 ϵ · ln mA max ϵ , no learnable parameters are involved here.

Learnable gradient descent procedure. The gradient descent in Algorithm 1 can be rewritten as

-1nxd

1nxd

A

μ

xk

ELU

yk

-1mxd

AT

fok

十

ggk

ReLU

Wk

-1nxd

1nxd

A

μ

Xk+1

ELU

yk+1

1mxd

1mxd

AT

μ

yk

ELU

十

Xk

1nxd

fok

+

ggk

ReLU

Wk

1mxd

1mxd

-AT

μ

yk+1

ELU

十

Xk+1

where f, g : R → R are sums of Heaviside step functions defined as:

As the Heaviside step function can be naturally approximated by the sigmoid function σ ( t ) = 1 1+exp( -t ) , the packing GD-Net adopts the following learnable functions as substitutes for f and g :

and

Note that it is guaranteed that f θ k ( t ) = -f θ k ( -t ) , in particular f θ k (0) = 0 , and g θ k (0) = 0 . So if the packing GD-Net reaches a stationary point of Φ p , it will remain there. Additionally, the packing GD-Net applies the ReLU activation to keep x k j ≥ 0 .

Channel expansion and architecture of packing GD-Net We integrate the channel expansion technique to further strengthen the expressive power. Specifically, we expand the m -dimensional column vector x k into a ( m × d ) -dimensional matrix X k , and n -dimensional column vector y k into a ( n × d ) -dimensional matrix Y k . Now, we are ready to present the packing GD-Net architecture:

Initialize X 0 := 0 m × d and Y 0 = 0 n × d .

For k = 0 , 1 , 2 , · · · , K -1

Y k := ELU [ µ ( A i X k -1 n × d ) ] + 1 n × d .

Output x final := X K · w K ∈ R m .

Here, ◦ denotes entry-wise multiplication (a.k.a. Hadamard product), and f θ k , g θ k : R → R are applied entrywise to the matrix A T Y k -1 m × d . The learnable parameter is Θ := { θ k , W k } K -1 k =0 ∪ { w K } , where θ k = { θ k ℓ, 1 , θ k ℓ, 2 , θ k ℓ, 3 } L 1 ℓ =1 ∪ { θ k ℓ, 4 , θ k ℓ, 5 , θ k ℓ, 6 } L 2 ℓ =1 , W k ∈ R d × d , and w K ∈ R d × 1 . So the total number of parameters is K · ( d 2 +3 L 1 +3 L 2 ) + d . The following theorem reveals the ability of the packing GD-Net to reproduce Algorithm 1 with arbitrarily small precision.

Theorem 2. Given any network depth K , any network width d ≥ 1 , and any precison η > 0 , there exists a K -layer packing GD-Net with a specific choice of parameters such that, for any packing LP instance (1) , ∥ x final -x alg ∥ ≤ η . Here, x alg is the output of K -iteration Algorithm 1.

By combining Theorems 1 and 2, we conclude that the packing GD-Nets with polylogarithmic depth and constant width are sufficient to solve packing LPs.

Theorem 3. Given any ϵ > 0 and η > 0 , there exists a packing GD-Net of ˜ O (ln 2 ( mA max ) · ln 2 ( nA max ) /ϵ 5 ) depth and constant width, using the parameter assignment Θ GD , that satisfies the following condition: For any packing LP instance (1) , x final is η -close to being a (1+ ϵ ) -approximate solution.

Remark 1. We can establish a similar theorem with general-purpose GNNs (such as GCNs) where we can still derive polylogarithmic depth in the bound but constant width is no longer guaranteed. Specifically, by the universal approximation theorem, the ELU function can be simulated with arbitrary precision by a 2-layer and sufficiently wide perceptron. So if we replace each occurrence of ELU with this 2-layer perceptron, we then obtain a GCNs. Note that this GCN still has polylogarithmic depth but the required width is no longer a constant.

Network training. The training data set is a set I = { ( A , x ∗ ) } of packing LP instances in the normal form. More specifically, the input of an instance is identified by the constraint matrix A , with b and c being both all-ones vectors; the label x ∗ represents the corresponding optimal solution. Let x final ( Θ , A ) denote the output of the packing GD-Net parameterized by Θ running on the input A . The goal of the training process is to find a parameter Θ ∗ minimizing loss function defined as:

Feasibility resortation. Note that the x final returned by packing GD-Net may be infeasible. To restore feasibility, we implement the following post-processing procedure:

First, for each j ∈ [ m ] , update x j := max(0 , min(1 , x j )) ;

Then, for i = 1 to n do

If A i x ≥ 1 , then update x j := x j A i x for each j with A ij = 0 .

4 Design of covering GD-Net

In one iteration of Algorithm 2, the process begins by updating x from y , followed by the calculation of the gradient ∇ Φ d ( y ) = 1 -Ax . Finally, it applies a variant of gradient descent to update y . Our covering GD-Net retains the original x -update and gradient-calculation modules, but replaces the gradient descent with a learnable neural network block to accelerate convergence.

Architecture of covering GD-Net. Similarly, we substitute the ELU activation with α fixed to 1 for the exp( · ) function, and replace the x -update x k j = exp[ µ (1 -A T j y k )] with

As we will show in Theorem 4, (7) can exactly simulate the x -update in Algorithm 2.

The gradient descent procedure in Algorithm 2 can be rewritten as

where f, g : R → R are the same functions as those defined in (4) for packing GD-Net. So, in the covering GD-Net, we also substitute f and g with the learnable functions f θ k and g θ k defined in (5) and (6) respectively. In addition, we also apply ReLU to keep y k i ≥ 0 .

Besides, the channel expansion technique is also incorporated: similarly, we expand x k ∈ R m into X k ∈ R m × d , and y k ∈ R n into Y k ∈ R n × d . Then, we propose our covering GD-Net architecture:

Initialize Y 0 := 1 n × d and X 0 = 1 m × d .

For k = 0 , 1 , 2 , · · · , K -1

̸

X k := ELU [ µ ( 1 m × d -A T Y k ) ] + 1 m × d ;

Output y final := Y K · w K ∈ R n .

The learnable parameter is Θ := { θ k , W k } K -1 k =0 ∪ { w K } , where θ k = { θ k ℓ, 1 , θ k ℓ, 2 , θ k ℓ, 3 } L 1 ℓ =1 ∪ { θ k ℓ, 4 , θ k ℓ, 5 , θ k ℓ, 6 } L 2 ℓ =1 , W k ∈ R d × d , and w K ∈ R d × 1 . So the total number of parameters is K · ( d 2 + 3 L 1 +3 L 2 ) + d , the same as in packing GD-Net.

Theorem 4. Given any network depth K , any network width d ≥ 1 , and any precison η > 0 , there exists a K -layer covering GD-Net with a specific choice of parameters such that, for any covering LP instance (2) , ∥ y final -y alg ∥ ≤ η . Here, y alg is the output of K -iteration Algorithm 2.

Theorems 1 and 4 together imply the capacity of polylogarithmic-depth constant-width covering GD-Nets for solving covering LPs.

Theorem 5. Given any ϵ > 0 and η > 0 , there exists a covering GD-Net of ˜ O (ln 2 ( nA max ) · ln 2 ( mnA max ) /ϵ 5 ) depth and constant width, using the parameter assignment Θ GD , that satisfies the following condition: For any covering LP instance (2) , y final is η -close to being a (1 + ϵ ) -approximate solution.

Remark 2. Similar to Remark 1, we can establish a similar theorem with general-purpose GNNs where the polylogarithmic bound on the depth still holds but constant width is no longer guaranteed.

Network training. The training dataset is a set I = ( A , y ∗ ) consisting of instances where y ∗ is the optimal solution to the covering LP (2) with constraint matrix A . The goal of the training process is to find a parameter Θ ∗ minimizing the following loss function:

Feasibility resortation. Since the y final may be infeasible, we implement the following postprocessing procedure to restore feasibility:

First, for each i ∈ [ n ] , update y i := max(0 , min(1 , y i )) ;

Then, for j = 1 to m do

If A T j y ≤ 1 , then update y i := y i A T j y for each i with A ij = 0 .

5 Experimental study

5.1 Experemental Setup

Datasets. We utilized four LP relaxations of publicly available mixed-integer optimization instances as benchmarks. Specifically, we included the Maximal Independent Set (IS) , Packing Problem (Packing) , Edge Covering Problem (ECP) , and Set Covering (SC) . The problem definitions are adopted from [9, 29]. Each benchmark comprises four sets of problem instances with varying sizes, including one set designated for the generalization experiment. Detailed information regarding the problem sizes and data splitting ratios can be found in the appendix.

To construct datasets for training, each instance M i ≡ { A i , b i , c i } undergoes normalization to ˜ M i ≡ { ˆ A i , 1 n , 1 m } . Subsequently, via the optimization solver SCIP [5], we obtain the optimal solution and optimal objective value pair { x i , obj ∗ i } for each instance ˜ M i . Finally, we utilize the input-target pair to compose the dataset D ≡ { ˆ A i , x i } | M | i =1 .

Models and Training settings. For comparison, our experiments also include the graph convolutional network (GCN), a predominant GNN architecture in L2O for LP and MILP. Specifically,

̸

we adopt the GCN implementation from [29], which is tailored for predicting the optimal solutions for LPs. Both the GCN and the proposed GD-Net utilize a four-layer architecture with 64 hidden units in each layer. Consequently, the number of parameters is 1,656 for GD-Net and 34,306 for GCN. Note that our GD-Net has an order of magnitude fewer parameters compared to GCN. For GD-Net, we set ϵ = 0 . 2 . All models were trained using a learning rate of 10 -3 . We trained each model for 10,000 epochs, and the checkpoint with the lowest validation loss was saved for evaluation. For reproducibility, our code to conduct the experiments can be found at https://anonymous.4open.science/r/GD-Net-6FC7/ .

Metrics. To effectively evaluate the GNN's ability to solve LP problems, we employed two distinct metrics: the relative gap R. Gap = | ˜ obj i -obj ∗ i | /obj ∗ i and the absolute gap (A. Gap = | ˜ obj i -obj ∗ i | ), where ˜ obj i denotes the predicted objective value of the respective approach after feasibility restoration.

5.2 Comparing against GCNs

In this section, we assess the effectiveness of the proposed GD-Net and the GCNs adopted from [29] in predicting the optimal solution for LP problems. As shown in Table 1, we report relative and absolute gaps, along with validation and test errors, to evaluate the quality of solutions generated by each model. The results indicate that GD-Net typically achieves narrower gaps compared

to GCNs. Even in instances where GD-Net does not surpass GCNs, the performance discrepancy remains minimal, which is noteworthy given GD-Net's significantly fewer parameters. Notably, in scenarios like Packing-L , although GD-Net records a higher test error, it still outperforms GCNs. This suggests that GD-Net may better capture the structural nuances of the problem. Overall, GD-Net generally demonstrates superior performance over conventional GCNs.

5.3 Generalization to larger instances

In this section, we evaluate the generalization capability of our proposed GD-Net, trained on smaller instances, to larger problem domains. We train GD-Nets on the largest dataset available for each problem and subsequently test these models on problem instances with 10% more constraints and variables. Table 2 presents the results, showcasing relative and absolute gaps, along with validation and test errors. The results reveal that GD-Nets possess a notable ability to generalize to larger problem instances with only minimal performance degradation. This suggests robustness in handling increased problem complexity, underscoring the adaptability and scalability of the proposed GD-Nets.

5.4 Comparing against more Baselines

To further demonstrate the effectiveness of the proposed framework, we include two additional baselines: the traditional first-order solver PDLP [19] and the commercial solver Gurobi [12]. Specifically, we used Gurobi's primal simplex method, which efficiently produces feasible primal solutions. The table below shows the time taken by each method to achieve solutions with the same precision level as GD-Net's solutions. The experiment was conducted on both the SC and Packing datasets across all three size variants.

As shown in Table 3, GD-Net consistently outperforms both general-purpose solvers on all datasets. Notably, PDLP, a first-order method known for its fast early-stage convergence, is unable to produce solutions of comparable quality to those of GD-Net in shorter time. This further highlights GD-Net's ability to efficiently generate high-quality solutions. Additionally, as previously mentioned, the simplex method requires matrix factorization, which is computationally expensive. In this case, it required up to 300 × more time to converge to a solution of the same quality as GD-Net. These findings strongly support the effectiveness of GD-Net, demonstrating its capability to consistently generate high-quality solutions.

Moreover, we also include experiments under a more practical setting and compare the inference time, which can be found in Appendix G and H.

6 Conclusion

Inspired by Awerbuch and Khandekar's gradient descent algorithms for packing and covering LPs, we introduce packing and covering GD-Net, and prove that they can approximate the solution mapping of packing and covering LPs respectively. Importantly, they only need polylogarithmic depth and constant width, significantly narrowing the gap between existing theoretical prediction and empirical evidence. Experiments are also conducted to demonstrate their effectiveness. We list some directions for future work: (1) How to further reduce the size of GNNs theoretically, since there is still a gap between our theoretical progress and empirical evidence; (2) How low the size of GNNs can go for solving general LPs, noting that our nets only work for packing and covering LPs; (3) To explore our nets in L2O for MILP. Recall that GD-Nets can be viewed as unrolling the gradient descent on a carefully selected potential function with some good properties. For (2) and (3), a natural direction is to design other potential functions that still enjoy those good properties.

Acknowledgments

This paper is supported by the National Key Research and Development Project under grant 2022YFA1003900; Hetao Shenzhen-Hong Kong Science and Technology Innovation Cooperation Zone Project (No.HZQSWS-KCCYB-2024016); University Development Fund UDF01001491, the Chinese University of Hong Kong, Shenzhen; Guangdong Provincial Key Laboratory of Mathematical Foundations for Artificial Intelligence (2023B1212010001); the Guangdong Major Project of Basic and Applied Basic Research (2023B0303000001). Qian Li and Minghui Ouyang's work was additionally supported by the National Natural Science Foundation of China Grants No.62002229. Tian Ding's work was additionally supported by the Internal Project of Shenzhen Research Institute of Big Data under Grant J00220240005.

References

M. Ahmadi, F. Kuhn, and R. Oshman. Distributed approximate maximum matching in the CONGEST model. In U. Schmid and J. Widder, editors, 32nd International Symposium on Distributed Computing, DISC 2018, New Orleans, LA, USA, October 15-19, 2018 , volume 121 of LIPIcs , pages 6:1-6:17. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2018.

B. Awerbuch and R. Khandekar. Stateless distributed gradient descent for positive linear programs. SIAM J. Comput. , 38(6):2468-2486, 2009.

Y. Bartal, J. W. Byers, and D. Raz. Global optimization using local information with applications to flow control. In 38th Annual Symposium on Foundations of Computer Science, FOCS '97, Miami Beach, Florida, USA, October 19-22, 1997 , pages 303-312. IEEE Computer Society, 1997.

Y. Bengio, A. Lodi, and A. Prouvost. Machine learning for combinatorial optimization: a methodological tour d'horizon. European Journal of Operational Research , 290(2):405-421, 2021.

S. Bolusani, M. Besançon, K. Bestuzheva, A. Chmiela, J. Dionísio, T. Donkiewicz, J. van Doornmalen, L. Eifler, M. Ghannam, A. Gleixner, C. Graczyk, K. Halbig, I. Hedtke, A. Hoen, C. Hojny, R. van der Hulst, D. Kamp, T. Koch, K. Kofler, J. Lentz, J. Manns, G. Mexi, E. Mühmer, M. E. Pfetsch, F. Schlösser, F. Serrano, Y. Shinano, M. Turner, S. Vigerske, D. Weninger, and L. Xu. The SCIP Optimization Suite 9.0. Technical report, Optimization Online, February 2024. URL https://optimization-online.org/2024/02/ the-scip-optimization-suite-9-0/ .

Q. Cappart, D. Chételat, E. B. Khalil, A. Lodi, C. Morris, and P. Velickovic. Combinatorial optimization and reasoning with graph neural networks. J. Mach. Learn. Res. , 24:130:1-130:61, 2023.

S. Chen, Y. C. Eldar, and L. Zhao. Graph unrolling networks: Interpretable neural networks for graph signal denoising. IEEE Trans. Signal Process. , 69:3699-3713, 2021. doi: 10.1109/TSP. 2021.3087905. URL https://doi.org/10.1109/TSP.2021.3087905 .

Z. Chen, J. Liu, X. Wang, and W. Yin. On representing linear programs by graph neural networks. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023.

M. Gasse, D. Chételat, N. Ferroni, L. Charlin, and A. Lodi. Exact combinatorial optimization with graph convolutional neural networks. Advances in neural information processing systems , 32, 2019.

P. Gupta, M. Gasse, E. B. Khalil, P. K. Mudigonda, A. Lodi, and Y. Bengio. Hybrid models for learning to branch. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

P. Gupta, E. B. Khalil, D. Chételat, M. Gasse, A. Lodi, Y. Bengio, and M. P. Kumar. Lookback for learning to branch. Trans. Mach. Learn. Res. (TMLR) , 2022, 2022.

Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2024. URL https://www. gurobi.com .

B. Hudson, Q. Li, M. Malencia, and A. Prorok. Graph neural network guided local search for the traveling salesperson problem. In International Conference on Learning Representations , 2021.

E. B. Khalil, C. Morris, and A. Lodi. Mip-gnn: A data-driven framework for guiding combinatorial solvers. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 10219-10227, 2022.

F. Kuhn, T. Moscibroda, and R. Wattenhofer. The price of being near-sighted. In Proceedings of the Seventeenth Annual ACM-SIAM Symposium on Discrete Algorithms, SODA 2006, Miami, Florida, USA, January 22-26, 2006 , pages 980-989. ACM Press, 2006.

F. Kuhn, T. Moscibroda, and R. Wattenhofer. Local computation: Lower and upper bounds. J. ACM , 63(2):17:1-17:44, 2016.

X. Li, Q. Qu, F. Zhu, J. Zeng, M. Yuan, K. Mao, and J. Wang. Learning to reformulate for linear programming. arXiv preprint arXiv:2201.06216 , 2022.

D. Liu, M. Fischetti, and A. Lodi. Learning to search in local branching. In Proceedings of the aaai conference on artificial intelligence , volume 36, pages 3796-3803, 2022.

H. Lu and J. Yang. A practical and optimal first-order method for large-scale convex quadratic programming, 2024. URL https://arxiv.org/abs/2311.07710 .

Y. Ma, X. Liu, T. Zhao, Y. Liu, J. Tang, and N. Shah. A unified view on graph neural networks as graph signal denoising. In G. Demartini, G. Zuccon, J. S. Culpepper, Z. Huang, and H. Tong, editors, CIKM '21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021 , pages 1202-1211. ACM, 2021. doi: 10.1145/3459637.3482225. URL https://doi.org/10.1145/3459637. 3482225 .

N. Mazyavkina, S. Sviridov, S. Ivanov, and E. Burnaev. Reinforcement learning for combinatorial optimization: A survey. Computers & Operations Research , 134:105400, 2021.

V. Monga, Y. Li, and Y. C. Eldar. Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. IEEE Signal Processing Magazine , 38(2):18-44, 2021.

V. Nair, S. Bartunov, F. Gimeno, I. von Glehn, P. Lichocki, I. Lobov, B. O'Donoghue, N. Sonnerat, C. Tjandraatmadja, P. Wang, R. Addanki, T. Hapuarachchi, T. Keck, J. Keeling, P. Kohli, I. Ktena, Y . Li, O. Vinyals, and Y . Zwols. Solving mixed integer programs using neural networks. CoRR , abs/2012.13349, 2020.

X. Pan, X. Han, C. Wang, Z. Li, S. Song, G. Huang, and C. Wu. A unified framework for convolution-based graph neural networks. Pattern Recognit. , 155:110597, 2024. doi: 10.1016/J. PATCOG.2024.110597. URL https://doi.org/10.1016/j.patcog.2024.110597 .

C. H. Papadimitriou and M. Yannakakis. Linear programming without the matrix. In S. R. Kosaraju, D. S. Johnson, and A. Aggarwal, editors, Proceedings of the Twenty-Fifth Annual ACM Symposium on Theory of Computing, May 16-18, 1993, San Diego, CA, USA , pages 121-129. ACM, 1993.

M. B. Paulus, G. Zarpellon, A. Krause, L. Charlin, and C. Maddison. Learning to cut by looking ahead: Cutting plane selection via imitation learning. In International conference on machine learning , pages 17584-17600. PMLR, 2022.

Y. Peng, B. Choi, and J. Xu. Graph learning for combinatorial optimization: a survey of state-of-the-art. Data Science and Engineering , 6(2):119-141, 2021.

A. Prouvost, J. Dumouchelle, L. Scavuzzo, M. Gasse, D. Chételat, and A. Lodi. Ecole: A gym-like library for machine learning in combinatorial optimization solvers. In Learning Meets Combinatorial Algorithms at NeurIPS2020 , 2020. URL https://openreview.net/forum? id=IVc9hqgibyB .

C. Qian, D. Chételat, and C. Morris. Exploring the power of graph neural networks in solving linear optimization problems. In International Conference on Artificial Intelligence and Statistics , pages 1432-1440. PMLR, 2024.

R. Sato, M. Yamada, and H. Kashima. Approximation ratios of graph neural networks for combinatorial problems. Advances in Neural Information Processing Systems , 32, 2019.

M. Seyfi, A. Banitalebi-Dehkordi, Z. Zhou, and Y. Zhang. Exact combinatorial optimization with temporo-attentional graph neural networks. In D. Koutra, C. Plant, M. G. Rodriguez, E. Baralis, and F. Bonchi, editors, Machine Learning and Knowledge Discovery in Databases: Research Track - European Conference, ECML PKDD 2023, Turin, Italy, September 18-22, 2023, Proceedings, Part IV , volume 14172 of Lecture Notes in Computer Science , pages 268-283. Springer, 2023.

Y. Shen, Y. Sun, A. Eberhard, and X. Li. Learning primal heuristics for mixed integer programs. In 2021 international joint conference on neural networks (ijcnn) , pages 1-8. IEEE, 2021.

P. Velickovic, R. Ying, M. Padovano, R. Hadsell, and C. Blundell. Neural execution of graph algorithms. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview. net/forum?id=SkgKO0EtvS .

L. Wang, H. Wu, W. Wang, and K.-C. Chen. Socially enabled wireless networks: Resource allocation via bipartite graph matching. IEEE Communications Magazine , 53(10):128-135, 2015.

Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang, and S. Y. Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems , 32(1):4-24, 2020.

Y. Yang, T. Liu, Y. Wang, J. Zhou, Q. Gan, Z. Wei, Z. Zhang, Z. Huang, and D. Wipf. Graph neural networks inspired by classical iterative algorithms. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event , volume 139 of Proceedings of Machine Learning Research , pages 1177311783. PMLR, 2021. URL http://proceedings.mlr.press/v139/yang21g.html .

H. Zhang, T. Yan, Z. Xie, Y. Xia, and Y. Zhang. Revisiting graph convolutional network on semi-supervised node classification from an optimization perspective. CoRR , abs/2009.11469, 2020. URL https://arxiv.org/abs/2009.11469 .

Z. Zhang and Z. Zhao. Towards understanding graph neural networks: An algorithm unrolling perspective. CoRR , abs/2206.04471, 2022. doi: 10.48550/ARXIV.2206.04471. URL https: //doi.org/10.48550/arXiv.2206.04471 .

M. Zhu, X. Wang, C. Shi, H. Ji, and P. Cui. Interpreting and unifying graph neural networks with an optimization framework. In J. Leskovec, M. Grobelnik, M. Najork, J. Tang, and L. Zia, editors, WWW '21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021 , pages 1215-1226. ACM / IW3C2, 2021. doi: 10.1145/3442381.3449953. URL https://doi.org/10.1145/3442381.3449953 .

A Packing and covering LP: reduction to the normal form

Recall that a packing LP and its dual covering LP are nonnegative LPs of the canonical form:

where all A ij , b i , and c j are non-negative. Their norm forms are those where b and c are both all-ones vectors and A ij is either zero or greater than 1.

The reduction to the normal form proceeds as follows. First, we can assume that each b i > 0 since otherwise all variables x j with A ij > 0 have to be zero and y i can be set to zero; similarly, we can assume each c j > 0 since otherwise x j can be set to zero and all variables y i with A ij > 0 have to be zero. Then, we can replace A ij by ˆ A ij = A ij b i c j , replace b and c by all-ones vector, and work with variables ˆ x j = c j x j and ˆ y i = b i y i . Finally, we replace ˆ A ij by ˜ A ij = ˆ A ij min { ˆ A i ′ j ′ | ˆ A i ′ j ′ > 0 } and work with ˜ x j = ˆ x j · min { ˆ A i ′ j ′ | ˆ A i ′ j ′ > 0 } and ˜ y i = ˆ y i · min { ˆ A i ′ j ′ | ˆ A i ′ j ′ > 0 } .

B Proof of Theorem 2

Proof. As increasing the network width d does not decrease the expressive power, it suffices to show the theorem for d = 1 . When d = 1 and w K = 1 , then packing GD-Net reduces to

Initialize x 0 := 0 m and y 0 = 0 n .

For k = 0 , 1 , 2 , · · · , K -1

y k i := ELU [ µ ( A i x k -1) ] +1 for each i ∈ [ n ] ;

Output x final := x K .

For the y -update part, recalling that in the execution of Algorithm 1, x is always feasible, thus A i x -1 ≤ 0 (Theorem 1). Additionally, since α is fixed to 1 , ELU( t ) + 1 = exp[ t ] for t ≤ 0 . Therefore, in Algorithm 1, we can replace y k i := exp[ µ ( A i x k -1)] with y k i := ELU [ µ ( A i x k -1) ] +1 without altering the algorithm's behavior.

For the x -update part, we rewrite this update in Algorithm 1 as

where f and g are defined in (4). Both f and g can be expressed as the sum of at most two Heaviside step functions, which can be naturally simulated by a sigmoid function with arbitrarily small error. Specifically, by setting L 1 = 1 , θ 1 , 1 = -β , θ 1 , 3 = α · θ 1 , 2 , and making θ k 1 , 2 sufficiently large, f θ k can approximate f with arbitrary precision. Similarly, by setting L 2 = 1 , θ k 1 , 4 = -δ , θ k 1 , 6 = α · θ k 1 , 5 , and making θ k 1 , 5 large enough, g can be approximated arbitrarily well by g θ k . Therefore, in Algorithm 1, if we substitute

we will get the above reduced packing GD-Net, and the change of the output x K can be made arbitrarily small by appropriately choosing the parameter Θ .

C Proof of Theorem 4

Proof. The proof is very similar to that of Theorem 2. First, we only need to show the theorem for d = 1 , because it does not decrease the expressive power to increase the network width d . When d = 1 , and by letting w K = 1 , the covering GD-Net reduces to

Initialize y 0 := 1 n and x 0 = 1 m .

For k = 0 , 1 , 2 , · · · , K -1

x k j := ELU [ µ (1 -A T j y k ) ] +1 for each j ∈ [ m ] ;

y k +1 i := ReLU [ y k i + f θ k (1 -A i x ) · y k i + g θ k ( 1 -A i x )] for each i ∈ [ n ] ;

Output y final := y K .

For the x -update part, recalling that in the execution of Algorithm 2, y is always feasible, thus 1 -A T j y ≤ 0 (Theorem 1). Additionally, since α is fixed to 1 , ELU( t )+1 = exp[ t ] for t ≤ 0 . Therefore, in Algorithm 2, we can replace x k j := exp[ µ (1 -A T j y k )] with x k j := ELU [ µ (1 -A T j y k ) ] + 1 without altering the algorithm's behavior.

For the y -update part, we rewrite this update in Algorithm 1 as

where f and g are defined in (4). As shown in the proof of Theorem 2, f θ k and g θ k can approximate f and g respectively with arbitrary precision. Therefore, in Algorithm 1, if we substitute

then we will reach the above reduced covering GD-Net, and the change of the output y K can be made arbitrarily small by appropriately choosing the parameter Θ .

D Dataset specification

For all experiments, involved models are trained with datasets split into 5,000 training instances, 100 validation instances, and 100 test instances. Each problem contains 4 different sizes, namely small (S), medium (M), large (L), and generalization (Gen). The detailed sizes of each size can be found in Table 4-7. For the IS benchmarks, we generated instances of specified sizes using the Ecole

library [28]. For ECP , we initially created IS instances, then converted these into their dual forms to obtain ECP instances. Regarding the SC and Packing datasets, we produced matrices of size m × n with a density of only density % non-zero entries. We then formulated the corresponding problems by transposing the matrix A or leaving it as is, depending on the dataset requirements.

E Hardware/Software specification

All experiments were performed on a server machine equipped with an Intel(R) Xeon(R) Platinum 8280 CPU @ 2.70GHz and 2.95 TB RAM. Data collection for training utilized Ecole 0.7.3 and Pyscipopt 4.2.0 for generating and solving instances. Model implementations were developed using PyTorch 2.1.

F Problem Definitions

In this section, we give the problem definitions and the specific formulations, as well as their LP relaxations of the original mixed-integer optimization problems used to generate data.

Maximum Independent Set (IS). The goal is to select as many vertices as possible from an undirected graph G = { V, E } such that, no two vertices form a edge.

Packing Problem. The goal is to maximize the profit from selecting from m items, while the selected items must fit in n resources constraints.

Edge Covering Problem (SC). This is the dual problem of the Maximum Independent Set problem, which minimizes the number of chosen edges such that every vertice touches at least one edge.

Set Covering (SC). Given a family of subsets S = { s 1 , · · · , s m } where s i ⊆ [ n ] , the goal is to select as few subsets as possible to cover all elements in [ n ] .

G Practical Problem

For the sake of more practical settings, we also included the Bipartite Maxflow problem (BMP) [34], which is a common model formulation applied to areas such as wireless communication. In our dataset, each bipartite graph is obtained by deleting all edges between V ′ and U ′ from a fully connected bipartite graph, and then randomly sample from the remaining edges with a probability of 60%, V ′ (and U ′ resp.) is a random subset consisting of half of the left nodes (and the right nodes). Based on the definition, we generated two sets of BMP problems with 1200 and 2000 nodes, respectively. In Table 8, we report the performance of GD-Net and compare it against GCNs.

Based on the results, GD-Net consistently achieves better predictions compared to GCN, with an average of only a 1% optimality gap from the optimal solutions.

Additionally, we evaluated GD-Net against the conventional Ford-Fulkerson method, which is specifically designed for solving Maxflow problems. In Table 9, we report the time for Ford-Fulkerson to find solutions with the same quality of solutions predicted by GD-Net.

The results highlight that GD-Net is significantly faster than the Ford Fulkerson heuristic in achieving high-quality solutions, demonstrating its efficiency and effectiveness.

Table 10: Inference profiling of the proposed GD-Net and GCNs from [29]. Both the Inference time and the number of parameters are reported. All times are reported in seconds. Results are averaged across 100 instances.

H Inference time profiling

To demonstrate the efficiency and scalability of GD-Nets, we present the average inference times of two models in Table 10. We see that GD-Nets consistently achieve faster inference times than

GCNs do. Furthermore, GD-Nets display strong scalability; their inference times remain comparably acceptable even as problem sizes increase. In contrast, GCNs require substantially more time to process and predict for larger problem instances.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In Sections 2, 3, and 4, we prove theoretical theorems and propose NN architecture. In Section 5, we provide the experimental results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Section 6 discusses the limitations.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We put the proof of theoretical results in the appendix, and the algorithm we cited was accurately described in Section 2 and followed by a brief explanation of why it is correct.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all information of our GNN architectures in Sections 3 and 4, and the experiment setting and code in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our code can be found at https://anonymous.4open.science/r/GD-Net-6FC7/. The dataset is provided in Section 5.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify all the training and test details in Section 5 and the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We did not provide the error bar since the experiments are relatively expensive to repeat for multiple runs.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Section 5 and the appendix provide sufficient information on the computer resources needed to reproduce the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper conforms with the NeurIPS Code of Ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper is about GNN's ability to solve LPs, which doesn't have a direct societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All assets are properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing experiments nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing experiments nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the effectiveness of small-sized GNNs in solving LP problems, addressing the discrepancy between theoretical predictions requiring large GNNs and empirical observations showing small GNNs' capability. The authors provide a theoretical foundation for the success of compact GNNs by proving that GNNs with polylogarithmic depth and constant width can solve specific LP classes, such as packing and covering LPs, by simulating a variant of the gradient descent algorithm. They introduce a novel GNN architecture, GD-Net, which outperforms traditional GNN models with fewer parameters. This understanding could lead to more efficient MILP solvers and reduce computational resources needed for solving LPs.\n\nSTRENGTHS:\n1. The paper successfully establishes a novel and intriguing link between the AK algorithm and GNNs. This connection is not only innovative but also has the potential to spark interest and pave the way for future research in the field. \n\n2. The introduction of the unrolled GNN architecture represents a contribution to GNN-based LP and MILP solvers.\n\nWEAKNESSES:\nThere are several areas where improvements could significantly enhance its contributions. Addressing these points satisfactorily would make a strong case for elevating the paper's status to \"Accept\".\n\n1. The presentation of Theorems 2-4, given that GD-Net is an unrolled version of the AK algorithm, appears somewhat straightforward (Question 1,2).\n\n2. To convincingly demonstrate the performance superiority of the proposed methods, it is crucial to include more realistic benchmarks and baselines for comparison (Question 3, 4).\n\n3. Given that unrolled neural networks typically enjoy several advantages over more general neural architectures, it would be highly beneficial for the paper to explicitly verify these benefits in the context of the proposed GD-Net (Question 5).\n\nQUESTIONS:\n1. The paper claims that it closes the gap between theoretical and empirical perspectives in GNN-based LP solvers by illustrating the relationship between GD-Net and the AK algorithm. However, it has been observed that previous empirical studies predominantly utilized general-purpose GNNs, such as GCNs. Could similar theorems be achieved with general GNN backbones?\n\n2. The demonstration that polylogarithmic-depth and constant-width GNNs can approximately solve LP problems is compelling. To better understand the significance of this result, could the authors discuss how close these findings are to the theoretical lower bound of what is required to solve LP problems?\n\n3. This paper relaxes MILP problems to LPs and uses them as benchmarks, which may not fully reflect the practical significance of the proposed method. I suggest adopting more standard LP benchmarks and comparing GD-Net with heuristic algorithms.\n\n4. In the specific setting described by [23], it is noted that GCNs underperform compared to GEN. To provide a comprehensive evaluation landscape, would it be possible for the authors to include GEN as a baseline in Table 1? This addition would help readers better assess the relative performance enhancements introduced by GD-Net.\n\n5. The paper highlights the unique advantage of unrolling methods in achieving exceptional size generalization. However, the current presentation in Table 2 might not fully capture this advantage. Would the authors consider augmenting their experimental setup to include:\n- Training on medium-scale datasets and testing on both small and large-scale datasets;\n- Introducing the performance of GCNs as baselines for a direct comparison;\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines the expressive power of GNNs in representing linear programs (LPs). The authors first introduce a first-order iterative algorithm for packing and covering LPs, conceptualizing this algorithm as a GNN called GD-Net, applied to these LP types. They then provide a convergence rate of the proposed first-order algorithm, deriving a complexity upper bound for GNNs representing LPs. Finally, numerical experiments compare the performance of GD-Net with a general GCN.\n\nSTRENGTHS:\nThe writing is clear and easy to follow. The mathematical aspects of the paper are technically correct.\n\nWEAKNESSES:\nThe paper lacks significant contributions.\n\n- The idea of conceptualizing iterative algorithms as GNNs is not novel:\n  - Yang et al. \"Graph Neural Networks Inspired by Classical Iterative Algorithms.\" ICML 2021.\n  - Zhang and Zhao. \"Towards Understanding Graph Neural Networks: An Algorithm Unrolling Perspective.\" 2022.\n\n- The convergence rate of first-order algorithms for LPs is not new, even though GD-Net is not exactly the same with existing algorithms. For example:\n  - Wang and Shroff. \"A New Alternating Direction Method for Linear Programming.\" NeurIPS 2017.\n  - Applegate et al. \"Faster First-Order Primal-Dual Methods for Linear Programming using Restarts and Sharpness.\" Mathematical Programming 2023.\n\nNote that these papers provide exact linear convergence rates, which are better than GD-Net's sublinear rate that only converges to an approximate solution. Additionally, the sublinear rate itself is not new, as it is provided by Awerbuch and Khandekar.\n\n- The empirical performance of GD-Net is not convincing. The paper only compares GD-Net with GCN, without showing advantages over traditional algorithms like ADMM or PDHG. Furthermore, comparing with commercial LP solvers like CPLEX or Gurobi would be a more ambitious but necessary goal to demonstrate real-world applicability.\n\nQUESTIONS:\nRefer to \"Weaknesses\".\n\nLIMITATIONS:\nThis theoretical paper appears to have no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the capability of small-sized Graph Neural Networks (GNNs) to solve linear programming (LP) problems, specifically focusing on polylogarithmic-depth, constant-width GNNs. It provides both theoretical proofs and empirical evidence demonstrating that these GNN architectures can effectively solve packing and covering LPs, which are common in various optimization contexts. The introduction of a novel GNN architecture, termed GD-Net, is highlighted, showing superior performance in terms of parameter efficiency and problem-solving capability compared to traditional GNNs.\n\nSTRENGTHS:\n1. **Theoretical Foundation**: The paper successfully bridges the gap between theoretical predictions and empirical results by proving that small-sized GNNs can efficiently solve packing and covering LPs.\n2. **Innovative Architecture**: The introduction of GD-Net, which significantly outperforms existing GNN models using fewer parameters, provides practical insights into the design of efficient neural network architectures for optimization problems.\n3. **Comprehensive Experiments**: Extensive empirical evaluations demonstrate the effectiveness of the proposed approach across different datasets and problem sizes, which solidifies the paper’s claims.\n4. **Impactful**: As you noted, the results are inspiring and provide valuable guidance for the L2O community on GNN structure and size design.\n\nWEAKNESSES:\n**Scope of Applicability**: The paper primarily focuses on packing and covering LPs. The generalizability of the proposed methods to other types of LPs or optimization problems is not addressed.\n\nQUESTIONS:\n1. Could the authors elaborate on potential modifications or extensions of the GD-Net architecture that might enable it to handle a broader range of LPs or even mixed integer linear programming problems?\n2. What are the limitations in terms of scalability when dealing with extremely large datasets or more complex network architectures?\n\nLIMITATIONS:\n**Generalizability**: The techniques are validated primarily on packing and covering LPs, and there is limited discussion on their effectiveness for other classes of LPs or more complex optimization tasks.\n\nAlthough I am not an expert in this specific area, the encouraging results give the L2O community some guidance on designing GNN architectures and sizes. I defer to other reviewers for a more detailed critique.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}