Title: A Deep Instance Generative Framework for MILP Solvers Under Limited Data Availability
Venue: NeurIPS 2023
Accepted: yes
Keywords: Learning to Optimize

Machine Learning for Combinatorial Optimization

Mixed-Integer Linear Programming

Graph Generation
Abstract: In the past few years, there has been an explosive surge in the use of machine learning (ML) techniques to address combinatorial optimization (CO) problems, especially mixed-integer linear programs (MILPs). Despite the achievements, the limited availability of real-world instances often leads to sub-optimal decisions and biased solver assessments, which motivates a suite of synthetic MILP instance generation techniques. However, existing methods either rely heavily on expert-designed formulations or struggle to capture the rich features of real-world instances. To tackle this problem, we propose G2MILP, *the first* deep generative framework for MILP instances. Specifically, G2MILP represents MILP instances as bipartite graphs, and applies a masked variational autoencoder to iteratively corrupt and replace parts of the original graphs to generate new ones. The appealing feature of G2MILP is that it can learn to generate novel and realistic MILP instances without prior expert-designed formulations, while preserving the structures and computational hardness of real-world datasets, simultaneously. Thus the generated instances can facilitate downstream tasks for enhancing MILP solvers under limited data availability. We design a suite of benchmarks to evaluate the quality of the generated MILP instances. Experiments demonstrate that our method can produce instances that closely resemble real-world datasets in terms of both structures and computational hardness. The deliverables are released at [https://miralab-ustc.github.io/L2O-G2MILP](https://miralab-ustc.github.io/L2O-G2MILP).

A Deep Instance Generative Framework for MILP Solvers Under Limited Data Availability

Abstract

In the past few years, there has been an explosive surge in the use of machine learning (ML) techniques to address combinatorial optimization (CO) problems, especially mixed-integer linear programs (MILPs). Despite the achievements, the limited availability of real-world instances often leads to sub-optimal decisions and biased solver assessments, which motivates a suite of synthetic MILP instance generation techniques. However, existing methods either rely heavily on expert-designed formulations or struggle to capture the rich features of real-world instances. To tackle this problem, we propose G2MILP, the first deep generative framework for MILP instances. Specifically, G2MILP represents MILP instances as bipartite graphs, and applies a masked variational autoencoder to iteratively corrupt and replace parts of the original graphs to generate new ones. The appealing feature of G2MILP is that it can learn to generate novel and realistic MILP instances without prior expert-designed formulations, while preserving the structures and computational hardness of real-world datasets, simultaneously. Thus the generated instances can facilitate downstream tasks for enhancing MILP solvers under limited data availability. We design a suite of benchmarks to evaluate the quality of the generated MILP instances. Experiments demonstrate that our method can produce instances that closely resemble real-world datasets in terms of both structures and computational hardness. The deliverables are released at https://miralab-ustc.github.io/L2O-G2MILP .

1 Introduction

Mixed-integer linear programming (MILP)-a powerful and versatile modeling technique for many real-world problems-lies at the core of combinatorial optimization (CO) research and is widely adopted in various industrial optimization scenarios, such as scheduling [1], planning [2], and portfolio [3]. While MILPs are NP -hard problems [4], machine learning (ML) techniques have recently emerged as a powerful approach for either solving them directly or assisting the solving process [5, 6]. Notable successes include [7] for node selection, [8] for branching decision, and [9] for cut selection, etc.

Despite the achievements, the limited availability of real-world instances, due to labor-intensive data collection and proprietary issues, remains a critical challenge to the research community [5, 10, 11]. Developing practical MILP solvers usually requires as many instances as possible to identify issues through white-box testing [12]. Moreover, machine learning methods for improving MILP solvers often suffer from sub-optimal decisions and biased assessments under limited data availability, thus

compromising their generalization to unseen problems [13]. These challenges motivate a suite of synthetic MILP instance generation techniques, which fall into two categories. Some methods rely heavily on expert-designed formulations for specific problems, such as Traveling Salesman Problems (TSPs) [14] or Set Covering problems [15]. However, these methods cannot cover realworld applications where domain-specific expertise or access to the combinatorial structures, due to proprietary issues, is limited. Other methods construct general MILP instances by sampling from an encoding space that controls a few specific statistics [16]. However, these methods often struggle to capture the rich features and the underlying combinatorial structures, resulting in an unsatisfactory alignment with real-world instances.

Developing a deep learning (DL)-based MILP instance generator is a promising approach to address this challenge. Such a generator can actively learn from real-world instances and generate new ones without expert-designed formulations. The generated instances can simulate realistic scenarios, cover more cases, significantly enrich the datasets, and thereby enhance the development of MILP solvers at a relatively low cost. Moreover, this approach has promising technical prospects for understanding the problem space, searching for challenging cases, and learning representations, which we will discuss further in Section 5. While similar techniques have been widely studied for Boolean satisfiability (SAT) problems [17], the development of DL-based MILP instance generators remains a complete blank due to higher technical difficulties, i.e., it involves not only the intrinsic combinatorial structure preservation but also high-precision numerical prediction. This paper aims to lay the foundation for the development of such generators and further empower MILP solver development under limited data availability.

In this paper, we propose G2MILP, which is the first deep generative framework for MILP instances. Werepresent MILP instances as weighted bipartite graphs, where variables and constraints are vertices, and non-zero coefficients are edges. With this representation, we can use graph neural networks (GNNs) to effectively capture essential features of MILP instances [8, 18]. Using this representation, we recast the original task as a graph generation problem. However, generating such complex graphs from scratch can be computationally expensive and may destroy the intrinsic combinatorial structures of the problems [19]. To address this issue, we propose a masked variational autoencoder (VAE) paradigm inspired by masked language models (MLM) [20, 21] and VAE theories [22-24]. The proposed paradigm iteratively corrupts and replaces parts of the original graphs using sampled latent vectors. This approach allows for controlling the degree to which we change the original instances, thus balancing the novelty and the preservation of structures and hardness of the generated instances. To implement the complicated generation steps, we design a decoder consisting of four modules that work cooperatively to determine multiple components of new instances, involving both structure and numerical prediction tasks simultaneously.

We then design a suite of benchmarks to evaluate the quality of generated MILP instances. First, we measure the structural distributional similarity between the generated samples and the input training instances using multiple structural statistics. Second, we solve the instances using the advanced solver Gurobi [12], and we report the solving time and the numbers of branching nodes of the instances, which directly indicate their computational hardness [19, 25]. Our experiments demonstrate that G2MILP is the very first method capable of generating instances that closely resemble the training sets in terms of both structures and computational hardness. Furthermore, we show that G2MILP is able to adjust the trade-off between the novelty and the preservation of structures and hardness of the generated instances. Third, we conduct a downstream task, the optimal value prediction task, to demonstrate the potential of generated instances in enhancing MILP solvers. The results show that using the generated instances to enrich the training sets reduces the prediction error by over 20% on several datasets. The deliverables are released at https://miralab-ustc.github.io/L2O-G2MILP .

2 Related Work

Machine Learning for MILP Machine learning (ML) techniques, due to its capability of capturing rich features from data, has shown impressive potential in addressing combinatorial optimization (CO) problems [26-28], especially MILP problems [5]. Some works apply ML models to directly predict the solutions for MILPs [29-31]. Others attempt to incorporate ML models into heuristic components in modern solvers [7, 9, 32, 33]. Gasse et al. [8] proposed to represent MILP instances as bipartite graphs, and use graph neural networks (GNNs) to capture features for branching decisions. Our

proposed generative framework can produce novel instances to enrich the datasets, which promises to enhance the existing ML methods that require large amounts of i.i.d. training data.

MILP Instance Generation Many previous works have made efforts to generate synthetic MILP instances for developing and testing solvers. Existing methods fall into two categories. The first category focuses on using mathematical formulations to generate instances for specific combinatorial optimization problems such as TSP [14], set covering [15], and mixed-integer knapsack [34]. The second category aims to generate general MILP instances. Bowly [16] proposed a framework to generate feasible and bounded MILP instances by sampling from an encoding space that controls a few specific statistics, e.g., density, node degrees, and coefficient mean. However, the aforementioned methods either rely heavily on expert-designed formulations or struggle to capture the rich features of real-world instances. G2MILP tackles these two issues simultaneously by employing deep learning techniques to actively generate instances that resemble real-world problems, and it provides a versatile solution to the data limitation challenge. In [35], we further extend G2MILP to learn to generate challenging MILP instance.

Deep Graph Generation A plethora of literature has investigated deep learning models for graph generation [36], including auto-regressive methods [37], varational autoencoders (V AEs) [23], and generative diffusion models [38]. These models have been widely used in various fields [39] such as molecule design [21, 40, 41] and social network generation [42, 43]. G2SAT [17], the first deep learning method for SAT instance generation, has received much research attention [19, 44]. Nevertheless, it is non-trivial to adopt G2SAT to MILP instance generation (see Appendix C.1), as G2SAT does not consider the high-precision numerical prediction, which is one of the fundamental challenges in MILP instance generation. In this paper, we propose G2MILP-the first deep generative framework designed for general MILP instances-and we hope to open up a new research direction for the research community.

3 Methodology

In this section, we present our G2MILP framework. First, in Section 3.1, we describe the approach to representing MILP instances as bipartite graphs. Then, in Section 3.2, we derive the masked variational autoencoder (VAE) generative paradigm. In Section 3.3, we provide details on the implementation of the model framework. Finally, in Section 3.4, we explain the training and inference processes. The model overview is in Figure 1. More implementation details can be found in Appendix A. The code is released at https://github.com/MIRALab-USTC/L2O-G2MILP .

3.1 Data Representation

A mixed-linear programming (MILP) problem takes the form of:

where c ∈ R n , A ∈ R m × n , b ∈ R m , l ∈ ( R ∪ {-∞} ) n , u ∈ ( R ∪ { + ∞} ) n , and the index set I ⊂ { 1 , 2 , · · · , n } includes those indices j where x j is constrained to be an integer.

To represent each MILP instance, we construct a weighted bipartite graph G = ( V ∪ W , E ) as follows [18, 29].

The constraint vertex set V = { v 1 , · · · , v m } , where each v i corresponds to the i th constraint in Equation 1. The vertex feature v i of v i is described by the bias term, i.e., v i = ( b i ) .

The variable vertex set W = { w 1 , · · · , w n } , where each w j corresponds to the j th variable in Equation 1. The vertex feature w j of w j is a 9 -dimensional vector that contains information of the objective coefficient c j , the variable type, and the bounds l j , u j .

The edge set E = { e ij } , where an edge e ij connects a constraint vertex v i ∈ V and a variable vertex w j ∈ W . The edge feature e ij is described by the coefficient, i.e., e ij = ( a ij ) , and there is no edge between v i and w j if a ij = 0 .

As described above, each MILP instance is represented as a weighted bipartite graph, equipped with a tuple of feature matrices ( V , W , E ) , where V , W , E denote stacks of vertex features v i , w j and

Ƹ

Masking

Process

𝒢

variables

constraints

𝑤1

𝑤2

𝑣1

𝑤𝑚

𝑣𝑛

𝒢 ෩

෤

𝑝( ሚ

𝒢|𝒢)

variables

constraints

𝑤1

𝑤2

෤ 𝑣

𝑤𝑚

𝑣𝑛

[mask]

𝐆𝐍𝐍𝝓

𝛍

𝝓

𝜮𝝓

𝒛𝑤1

𝒛𝑤2

𝒛𝑣1

𝒛𝑤𝑚

𝒛𝑣𝑛

𝒛𝑣𝑖

𝒛𝑤𝑗

𝒉𝑤1

𝒉෤ 𝑣

𝒉𝑤𝑚

𝒉𝑣𝑛

[mask]

𝒉෤ 𝑣

𝒉𝑤𝑗

𝐆𝐍𝐍𝜽



𝑝𝜽(𝑏෤ 𝑣

| ሚ

𝒢, 𝒁)

𝐌𝐋𝐏𝜽

𝐛𝐢𝐚𝐬

෠

𝑏෤ 𝑣



Degree Predictor

𝑝𝜽(𝑑෤ 𝑣

| ሚ

𝒢, 𝒁)

𝐌𝐋𝐏 𝜽

𝐝𝐞𝐠

መ

𝑑෤ 𝑣



Logits Predictor

𝑝𝜽(𝛿෤ 𝑣,𝑢

| ሚ

𝒢, 𝒁, 𝑑

෤ 𝑣

)

𝐌𝐋𝐏 𝜽

𝐥𝐨𝐠𝐢𝐭𝐬

መ

𝛿෤ 𝑣,𝑢



Weights Predictor

𝑝𝜽(𝑒෤ 𝑣,𝑢

| ሚ

𝒢, 𝒁, 𝑑

෤ 𝑣

)

𝐌𝐋𝐏 𝜽

𝐰𝐞𝐢𝐠𝐡𝐭𝐬

𝑒෤ 𝑣,𝑢

෠

𝑏෤ 𝑣

෠

𝑏෤ 𝑣

𝑒෤ 𝑣,𝑤 1

𝑒෤ 𝑣,𝑤 2

෠

𝑏෤ 𝑣

1

1

0

(a)

(b) Encoder

𝑞𝝓(𝒁|𝒢)

(c) Decoder

𝑝𝜽(𝒢| ሚ

𝒢, 𝒁)

edge features e ij , respectively. Such a representation contains all information of the original MILP instance [18]. We use the off-the-shelf observation function provided by Ecole [45] to build the bipartite graphs from MILP instances. We then apply a graph neural network (GNN) to obtain the node representations h G v i and h G w j , also denoted as h v i and h w j for simplicity. More details on the data representation can be found in Appendix A.1.

3.2 Masked VAE Paradigm

We then introduce our proposed masked VAE paradigm. For the ease of understanding, we provide an intuitive explanation here, and delay the mathematical derivation to Appendix A.2.

Given a graph G drawn from a dataset D , we corrupt it through a masking process, denoted by ˜ G ∼ ˜ p ( ˜ G|G ) . We aim to build a parameterized generator p θ ( ˆ G| ˜ G ) that can generate new instances ˆ G from the corrupted graph ˜ G . We train the generator by maximizing the log-likelihood log p θ ( G| ˜ G ) = log p θ ( ˆ G = G| ˜ G ) of reconstructing G given ˜ G . Therefore, the optimization objective is:

To model the randomnesss in the generation process and produce diverse instances, we follow the standard VAE framework [22, 23] to introduce a latent variable Z = ( z v 1 , · · · , z v m , z w 1 , · · · , z w n ) , which contains the latent vectors for all vertices. During training, the latent vectors are sampled from a posterior distribution given by a parameterized encoder q ϕ , while during inference, they are independently sampled from a prior distribution such as a standard Gaussian distribution. The decoder p θ in the masked VAE framework generates new instances from the masked graph ˜ G together with the sampled latent variable Z .

Ƹ

Ƹ

The evidence lower bound (ELBO), also known as the variational lower bound, is a lower bound estimator of the log-likelihood, and is what we actually optimize during training, because it is more tractable. We can derive the ELBO as:

where p θ ( Z ) is the prior distribution of Z and is usually taken as the standard Gaussian N ( 0 , I ) , and D KL [ · ∥ · ] denotes the KL divergence. Therefore, we formulate the loss function as:

In the formula: (1) the first term L rec is the reconstruction loss, which urges the decoder to rebuild the input data according to the masked data and the latent variables. (2) The second term L prior is used to regularize the posterior distribution in the latent space to approach a standard Gaussian distribution, so that we can sample Z from the distribution when inference. (3) β is a hyperparameter to control the weight of regularization, which is critical in training a V AE model [46].

3.3 Model Implementation

To implement Equation 4, we need to instantiate the masking process ˜ p ( ˜ G|G ) , the encoder q ϕ ( Z |G ) , and the decoder p θ ( G| ˜ G , Z ) , respectively.

Masking Process For simplicity, we uniformly sample a constraint vertex ˜ v ∼ U ( V ) and mask it and its adjacent edges, while keeping the variable vertices unchanged. Specifically, we label the vertex ˜ v with a special [mask] token, and add virtual edges that link ˜ v with each variable vertex. The vertex ˜ v and the virtual edges are assigned special embeddings to distinguish them from the others. We further discuss on the masking scheme in Appendix C.2.

Encoder The encoder q ϕ ( Z |G ) is implemented as:

where h G u is the node representation of u obtained by a GNN ϕ , N denotes the Gaussian distribution, and µ and Σ output the mean and the log variance, respectively.

Decoder The decode p θ aims to reconstruct G during training. We apply a GNN θ to obtain the node representations h ˜ G u , denoted as h u for simplicity. To rebuild the masked constraint vertex ˜ v , the decoder sequentially determines: ① the bias b ˜ v (i.e., the right-hand side of the constraint), ② the degree d ˜ v of ˜ v (i.e., the number of variables involved in the constraint), ③ the logits δ ˜ v,u for all variable vertices u to indicate whether they are connected with ˜ v (i.e., whether the variables are in the constraint), and ④ the weights e ˜ v,u of the edges (i.e., the coefficients of the variables in the constraint). The decoder is then formulated as:

Therefore, we implement the decoder as four cooperative modules: ① Bias Predictor, ② Degree Predictor, ③ Logits Predictor, and ④ Weights Predictor.

① Bias Predictor For effective prediction, we incorporate the prior of simple statistics of the dataset-the minimum b and the maximum b of the bias terms that occur in the dataset-into the predictor. Specifically, we normalize the bias b ˜ v to [0 , 1] via b ∗ ˜ v = ( b ˜ v -b ) / ( b -b ) . To predict b ∗ ˜ v , we use one MLP that takes the node representation h ˜ v and the latent vector z ˜ v of ˜ v as inputs:

where σ ( · ) is the sigmoid function used to restrict the outputs. We use the mean squared error (MSE) loss to train the predictor. At inference time, we apply the inverse transformation to obtain the predicted bias values: ˆ b ˜ v = b +( b -b ) · ˆ b ∗ ˜ v . 1

② Degree Predictor We find that the constraint degrees are crucial to the graph structures and significantly affect the combinatorial properties. Therefore, we use the Degree Predictor to determine coarse-grained degree structure, and then use the Logits Predictor to determine the fine-grained connection details. Similarly to Bias Predictor, we normalize the degree d ˜ v to d ∗ ˜ v = ( d ˜ v -d ) / ( d -d ) , where d and d are the minimum and maximum degrees in the dataset, respectively. We use one MLP to predict d ∗ ˜ v :

We use MSE loss for training, and round the predicted degree to the nearest integer ˆ d ˜ v for inference.

③ Logits Predictor To predict the logits δ ˜ v,u indicating whether a variable vertex u ∈ W is connected with ˜ v , we use one MLP that takes the representation h u and the latent vector z u of u as inputs:

We use binary cross-entropy (BCE) loss to train the logistical regression module. As positive samples (i.e., variables connected with a constraint) are often scarce, we use one negative sample for each positive sample during training. The loss function is:

where p pos and p neg denote the distributions over positive and negative samples, respectively. At inference time, we connect ˆ d ˜ v variable vertices with the top logits to ˜ v ., i.e.,

④ Weights Predictor Finally, we use one MLP to predict the normalized weights e ∗ ˜ v,u for nodes u that are connected with ˜ v :

The training and inference procedures are similar to those of Bias Predictor.

3.4 Training and Inference

During training, we use the original graph G to provide supervision signals for the decoder, guiding it to reconstruct G from the masked ˜ G and the encoded Z . As described above, the decoder involves four modules, each optimized by a prediction task. The first term in Equation 4, i.e., the reconstruction loss, is written as

where L i ( θ , ϕ |G , ˜ G ) ( i = 1 , 2 , 3 , 4 ) are loss functions for the four prediction tasks, respectively, and α i are hyperparameters.

During inference, we discard the encoder and sample Z from a standard Gaussian distribution, which introduces randomness to enable the model to generate novel instances. We iteratively mask one constraint vertex in the bipartite graph and replace it with a generated one. We define a hyperparameter η to adjust the ratio of iterations to the number of constraints, i.e., N iters = η · |V| . Naturally, a larger value of η results in instances that are more novel, while a smaller value of η yields instances that exhibit better similarity to the training set. For further details of training and inference procedures, please refer to Appendix A.3.

4 Experiments

4.1 Setup

We conduct extensive experiments to demonstrate the effectiveness of our model. More experimental details can be found in Appendix B. Additional results are in Appendix C.

Benchmarking To evaluate the quality of the generated MILP instances, we design three benchmarks so as to answer the following research questions. (1) How well can the generated instances preserve the graph structures of the training set? (2) How closely do the generated instances resemble the computational hardness of real-world instances? (3) How effectively do they facilitate downstream tasks to improve solver performance?

I. Structural Distributional Similarity We consider 11 classical statistics to represent features of the instances [17, 47], including coefficient density, node degrees, graph clustering, graph modularity, etc. In line with a widely used graph generation benchmark [48], we compute the Jensen-Shannon (JS) divergence [49] for each statistic to measure the distributional similarity between the generated instances and the training set. We then standardize the metrics into similarity scores that range from 0 to 1 . The computing details can be found in Appendix B.3.

II. Computational Hardness The computational hardness is another critical metric to assess the quality of the generated instances. We draw an analogy from the SAT generation community, where though many progresses achieved, it is widely acknowledged that the generated SAT instances differs significantly from real-world ones in the computational hardness [25], and this issue remains inadequately addressed. In our work, we make efforts to mitigate this problem, even in the context of MILP generation, a more challenging task. To this end, we leverage the state-of-the-art solver Gurobi [12] to solve the instances, and we report the solving time and the numbers of branching nodes during the solving process, which can directly reflect the computational hardness of instances [19].

III. Downstream Task We consider two downstream tasks to examine the the potential benefits of the generated instances in practical applications. We employ G2MILP to generate new instances and augment the original datasets, and then evaluate whether the enriched datasets can improve the performance of the downstream tasks. The considered tasks include predicting the optimal values of the MILP problem, as discussed in Chen et al. [18], and applying a predict-and-search framework for solving MILPs, as proposed by Han et al. [31].

Datasets We consider four different datasets of various sizes. (1) Large datasets. We evaluate the model's capability of learning data distributions using two well-known synthetic MILP datasets: Maximum Independent Set (MIS) [50] and Set Covering [15]. We follow previous works [8, 9] to artificially generate 1000 instances for each of them. (2) Medium dataset. Mixed-integer Knapsack (MIK) is a widely used dataset [34], which consists of 80 training instances and 10 test instances. We use this dataset to evaluate the model's performance both on the distribution learning benchmarks and the downstream task. (3) Small dataset. We construct a small subset of MIPLIB 2017 [10] by collecting a group of problems called Nurse Scheduling problems. This dataset comes from real-world scenarios and consists of only 4 instances, 2 for training and 2 for test, respectively. Since the statistics are meaningless for such an extremely small dataset, we use it only to demonstrate the effectiveness of generated instances in facilitating downstream tasks.

Baselines G2MILP is the first deep learning generative framework for MILP isntances, and thus, we do not have any learning-based models for comparison purpose. Therefore, we compare G2MILP with a heuristic MILP instance generator, namely Bowly [16]. Bowly can create feasible and bounded MILP instances while controlling some specific statistical features such as coefficient density and coefficient mean. We set all the controllable parameters to match the corresponding statistics of the training set, allowing Bowly to imitate the training set to some extent. We also consider a useful baseline, namely Random, to demonstrate the effectiveness of deep neural networks in G2MILP. Random employs the same generation procedure as G2MILP, but replaces all neural networks in the decoder with random generators. We set the masking ratio η for Random and G2MILP to 0 . 01 , 0 . 05 , and 0 . 1 to show how this hyperparameter helps balance the novelty and similarity.

4.2 Quantitative Results

I. Structural Distributional Similarity We present the structural distributional similarity scores between each pair of datasets in Table 1. The results indicate that our designed metric is reasonable in the sense that datasets obtain high scores with themselves and low

±

±

±

scores with different domains. Table 2 shows the similarity scores between generated instances and the corresponding training sets. We generate 1000 instances for each dataset to compute the similarity scores. The results suggest that G2MILP closely fits the data distribution, while Bowly, which relies on heuristic rules to control the statistical features, falls short of our expectations. Furthermore, we observe that G2MILP outperforms Random, indicating that deep learning contributes to the model's performance. As expected, a higher masking ratio η results in generating more novel instances but reduces their similarity to the training sets.

II. Computational Hardness We report the average solving time and numbers of branching

nodes in Table 3 and Table 4, respectively. The results indicate that instances generated by Bowly are relatively easy, and the hardness of those generated by Random is inconclusive. In contrast, G2MILP is capable of preserving the computational hardness of the original training sets. Notably, even without imposing rules to guarantee the feasibility and boundedness of generated instances, G2MILP automatically learns from the data and produces feasible and bounded instances.

III. Downstream Task First, we follow Chen et al. [18] to construct a GNN model for predicting the optimal values of MILP problems. We train a predictive GNN model on the training set. After that, we employ 20 generated instances to augment the training data, and then train another predictive

Training Set

Random

G2MILP

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Relative MSE

MIK

-

20%

0%

20%

40%

60%

80%

100%

Improvement

Training Set

Bowly

G2MILP

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Relative MSE

Nurse Scheduling

-

20%

0%

20%

40%

60%

80%

100%

Improvement

Training Set

Training Set

G2MILP

G2MILP (

η

= 0

.

01)

G2MILP (

η

= 0

.

05)

G2MILP (

η

= 0

.

1)

model using the enriched dataset. We use the prediction mean squared error (MSE) to assess the resulting models, and we present the MSE relative to the default model trained on the original training sets in Figure 2. For the MIK dataset, instances generated by Bowly introduce numerical issues so that Ecole and SCIP fail to read them. For the Nurse Scheduling dataset, Random fails to generate feasible instances. Notably, G2MILP is the only method that demonstrates performance improvement on both datasets, reducing the MSE by 73 . 7% and 24 . 3% , respectively. The detailed results are in Table 8 in Appendix B.4. Then, we conduct experiments on the neural solver, i.e., the predict-and-search framework proposed by Han et al. [31], which employs a model to predict a solution and then uses solvers to search for the optimal solutions in a trust region. The results are in Table 9 in Appendix B.4.

4.3 Analysis

Masking Process We conduct extensive comparative experiments on different implementations of the masking process. First, we implement different versions of G2MILP, which enable us to mask and modify either constraints, variables, or both. Second, we investigate different orders of masking constraints, including uniformly sampling and sampling according to the vertex indices. Third, we analyze the effect of the masking ratio η on similarity scores and downstream task performance improvements. The experimental results are in Appendix C.2.

Size of Dataset We conduct experiments on different sizes of the original datasets and different ratio of generated instances to original ones on MIS. Results are in Table 15 in Appendix C.4. The results show that G2MILP yields performance improvements across datasets of varying sizes.

Visualization We visualize the instance representations for MIK in Figure 3. Specifically, we use the G2MILP encoder to obtain the instance representations, and then apply t-SNE dimensionality reduction [52] for visualization. We observe that the generated instances, while closely resembling the training set, contribute to a broader and more continuous exploration of the problem space, thereby enhancing model robustness and generalization. Additionally, by increasing the masking ratio η , we can effectively explore a wider problem space beyond the confines of the training sets. For comparison with the baseline, we present the visualization of instances generated by Random in Figure 5 in Appendix C.5.

5 Limitations, Future Avenues, and Conclusions

Limitations In this paper, we develop G2MILP by iteratively corrupting and replacing the constraints vertices. We also investigate different implementations of the masking process. However, more versatile masking schemes should be explored. Moreover, employing more sophisticated designs would enable us to control critical properties such as feasibility of the instances. We intend to develop a more versatile and powerful generator in our future work.

Future Avenues We open up new avenues for research on DL-based MILP instance generative models. In addition to producing new instances to enrich the datasets, this research has many other promising technical implications [35]. (1) Such a generator will assist researchers to gain insights into different data domains and the explored space of MILP instances. (2) Based on a generative model, we can establish an adversarial framework, where the generator aims to identify challenging cases for the solver, thus automatically enhancing the solver's ability to handle complex scenarios. (3) Training a generative model involves learning the data distribution and deriving representations through unsupervised learning. Consequently, it is possible to develop a pre-trained model based on a generative model, which can benefit downstream tasks across various domains. We believe that this paper serves as an entrance for the aforementioned routes, and we expect further efforts in this field.

Conclusions In this paper, we propose G2MILP, which to the best of our knowledge is the first deep generative framework for MILP instances. It can learn to generate MILP instances without prior expert-designed formulations, while preserving the structures and computational hardness, simultaneously. Thus the generated instances can enhance MILP solvers under limited data availability. This work opens up new avenues for research on DL-based MILP instance generative models.

References

John A Muckstadt and Richard C Wilson. An application of mixed-integer programming duality to scheduling thermal generating systems. IEEE Transactions on Power Apparatus and Systems , (12), 1968.

Yves Pochet and Laurence A Wolsey. Production planning by mixed integer programming , volume 149. Springer, 2006.

Rodrigo Moreno, Roberto Moreira, and Goran Strbac. A milp model for optimising multiservice portfolios of distributed energy storage. Applied Energy , 137:554-566, 2015.

Robert E Bixby, Mary Fenelon, Zonghao Gu, Ed Rothberg, and Roland Wunderling. Mixedinteger programming: A progress report. In The sharpest cut: the impact of Manfred Padberg and his work , pages 309-325. SIAM, 2004.

Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: a methodological tour d'horizon. European Journal of Operational Research , 290(2):405-421, 2021.

Jiayi Zhang, Chang Liu, Xijun Li, Hui-Ling Zhen, Mingxuan Yuan, Yawen Li, and Junchi Yan. A survey for solving mixed integer programming via machine learning. Neurocomputing , 519: 205-217, 2023.

He He, Hal Daume III, and Jason M Eisner. Learning to search in branch and bound algorithms. Advances in neural information processing systems , 27, 2014.

Maxime Gasse, Didier Chételat, Nicola Ferroni, Laurent Charlin, and Andrea Lodi. Exact combinatorial optimization with graph convolutional neural networks. Advances in neural information processing systems , 32, 2019.

Zhihai Wang, Xijun Li, Jie Wang, Yufei Kuang, Mingxuan Yuan, Jia Zeng, Yongdong Zhang, and Feng Wu. Learning cut selection for mixed-integer linear programming via hierarchical sequence model. In The Eleventh International Conference on Learning Representations , 2023.

Ambros Gleixner, Gregor Hendel, Gerald Gamrath, Tobias Achterberg, Michael Bastubbe, Timo Berthold, Philipp Christophel, Kati Jarck, Thorsten Koch, Jeff Linderoth, et al. Miplib 2017: data-driven compilation of the 6th mixed-integer programming library. Mathematical Programming Computation , 13(3):443-490, 2021.

Jun Sakuma and Shigenobu Kobayashi. A genetic algorithm for privacy preserving combinatorial optimization. In Annual Conference on Genetic and Evolutionary Computation , 2007.

LLC Gurobi Optimization. Gurobi optimizer. URL http://www. gurobi. com , 2021.

Han Lu, Zenan Li, Runzhong Wang, Qibing Ren, Xijun Li, Mingxuan Yuan, Jia Zeng, Xiaokang Yang, and Junchi Yan. Roco: A general framework for evaluating robustness of combinatorial optimization solvers on graphs. In The Eleventh International Conference on Learning Representations , 2023.

Russ J Vander Wiel and Nikolaos V Sahinidis. Heuristic bounds and test problem generation for the time-dependent traveling salesman problem. Transportation Science , 29(2):167-183, 1995.

Egon Balas and Andrew Ho. Set covering algorithms using cutting planes, heuristics, and subgradient optimization: A computational study , pages 37-60. Springer Berlin Heidelberg, Berlin, Heidelberg, 1980. ISBN 978-3-642-00802-3. doi: 10.1007/BFb0120886. URL https://doi.org/10.1007/BFb0120886 .

Simon Andrew Bowly. Stress testing mixed integer programming solvers through new test instance generation methods . PhD thesis, School of Mathematical Sciences, Monash University, 2019.

Jiaxuan You, Haoze Wu, Clark Barrett, Raghuram Ramanujan, and Jure Leskovec. G2sat: learning to generate sat formulas. Advances in neural information processing systems , 32, 2019.

Ziang Chen, Jialin Liu, Xinshang Wang, and Wotao Yin. On representing mixed-integer linear programs by graph neural networks. In The Eleventh International Conference on Learning Representations , 2023.

Yang Li, Xinyan Chen, Wenxuan Guo, Xijun Li, Wanqian Luo, Junhua Huang, Hui-Ling Zhen, Mingxuan Yuan, and Junchi Yan. Hardsatgen: Understanding the difficulty of hard sat formula generation and a strong structure-hardness-aware baseline. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , 2023.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Omar Mahmood, Elman Mansimov, Richard Bonneau, and Kyunghyun Cho. Masked graph modeling for molecule generation. Nature communications , 12(1):3156, 2021.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 , 2016.

Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS central science , 4(2):268-276, 2018.

Tomáš Balyo, Nils Froleyks, Marijn JH Heule, Markus Iser, Matti Järvisalo, and Martin Suda. Proceedings of sat competition 2020: Solver and benchmark descriptions. 2020.

Yang Li, Jinpei Guo, Runzhong Wang, and Junchi Yan. From distribution learning in training to gradient search in testing for combinatorial optimization. In Advances in Neural Information Processing Systems , 2023.

Xijun Li, Qingyu Qu, Fangzhou Zhu, Mingxuan Yuan, Jia Zeng, and Jie Wang. Accelerating linear programming solving by exploiting the performance variability via reinforcement learning. 2023.

Yufei Kuang, Xijun Li, Jie Wang, Fangzhou Zhu, Meng Lu, Zhihai Wang, Jia Zeng, Houqiang Li, Yongdong Zhang, and Feng Wu. Accelerate presolve in large-scale linear programming via reinforcement learning. arXiv preprint arXiv:2310.11845 , 2023.

Vinod Nair, Sergey Bartunov, Felix Gimeno, Ingrid Von Glehn, Pawel Lichocki, Ivan Lobov, Brendan O'Donoghue, Nicolas Sonnerat, Christian Tjandraatmadja, Pengming Wang, et al. Solving mixed integer programs using neural networks. arXiv preprint arXiv:2012.13349 , 2020.

Elias B Khalil, Christopher Morris, and Andrea Lodi. Mip-gnn: A data-driven framework for guiding combinatorial solvers. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 10219-10227, 2022.

Qingyu Han, Linxin Yang, Qian Chen, Xiang Zhou, Dong Zhang, Akang Wang, Ruoyu Sun, and Xiaodong Luo. A gnn-guided predict-and-search framework for mixed-integer linear programming. arXiv preprint arXiv:2302.05636 , 2023.

Radu Baltean-Lugojan, Pierre Bonami, Ruth Misener, and Andrea Tramontani. Scoring positive semidefinite cutting planes for quadratic optimization via trained neural networks. preprint: http://www. optimization-online. org/DB_ HTML/2018/11/6943. html , 2019.

Qingyu Qu, Xijun Li, Yunfan Zhou, Jia Zeng, Mingxuan Yuan, Jie Wang, Jinhu Lv, Kexin Liu, and Kun Mao. An improved reinforcement learning algorithm for learning to branch. arXiv preprint arXiv:2201.06213 , 2022.

Alper Atamtürk. On the facets of the mixed-integer knapsack polyhedron. Mathematical Programming , 98(1-3):145-175, 2003.

Jie Wang, Zijie Geng, Xijun Li, Jianye Hao, Yongdong Zhang, and Feng Wu. G2milp: Learning to generate mixed-integer linear programming instances for milp solvers. nov 2023. doi: 10. 36227/techrxiv.24566554.v1. URL http://dx.doi.org/10.36227/techrxiv.24566554. v1 .

Xiaojie Guo and Liang Zhao. A systematic survey on deep generative models for graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Rocío Mercado, Tobias Rastemo, Edvard Lindelöf, Günter Klambauer, Ola Engkvist, Hongming Chen, and Esben Jannik Bjerrum. Graph networks for molecular design. Machine Learning: Science and Technology , 2(2):025023, 2021.

Wenqi Fan, Chengyi Liu, Yunqing Liu, Jiatong Li, Hang Li, Hui Liu, Jiliang Tang, and Qing Li. Generative diffusion models on graphs: Methods and applications. arXiv preprint arXiv:2302.02591 , 2023.

Yanqiao Zhu, Yuanqi Du, Yinkai Wang, Yichen Xu, Jieyu Zhang, Qiang Liu, and Shu Wu. A survey on deep graph generation: Methods and applications. arXiv preprint arXiv:2203.06714 , 2022.

Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference on machine learning , pages 2323-2332. PMLR, 2018.

Zijie Geng, Shufang Xie, Yingce Xia, Lijun Wu, Tao Qin, Jie Wang, Yongdong Zhang, Feng Wu, and Tie-Yan Liu. De novo molecular generation via connection-aware motif mining. In The Eleventh International Conference on Learning Representations , 2023.

Duncan J Watts and Steven H Strogatz. Collective dynamics of 'small-world'networks. nature , 393(6684):440-442, 1998.

Jure Leskovec, Deepayan Chakrabarti, Jon Kleinberg, Christos Faloutsos, and Zoubin Ghahramani. Kronecker graphs: an approach to modeling networks. Journal of Machine Learning Research , 11(2), 2010.

Iván Garzón, Pablo Mesejo, and Jesús Giráldez-Cru. On the performance of deep generative models of realistic sat instances. In 25th International Conference on Theory and Applications of Satisfiability Testing (SAT 2022) . Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2022.

Antoine Prouvost, Justin Dumouchelle, Lara Scavuzzo, Maxime Gasse, Didier Chételat, and Andrea Lodi. Ecole: A gym-like library for machine learning in combinatorial optimization solvers. In Learning Meets Combinatorial Algorithms at NeurIPS2020 , 2020. URL https: //openreview.net/forum?id=IVc9hqgibyB .

Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349 , 2015.

Frank Hutter, Lin Xu, Holger H Hoos, and Kevin Leyton-Brown. Algorithm runtime prediction: Methods & evaluation. Artificial Intelligence , 206:79-111, 2014.

Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling , 59(3):1096-1108, 2019.

Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory , 37(1):145-151, 1991.

David Bergman, Andre A Cire, Willem-Jan Van Hoeve, and John Hooker. Decision diagrams for optimization , volume 1. Springer, 2016.

Tobias Achterberg. Scip: solving constraint integer programs. Mathematical Programming Computation , 1:1-41, 2009.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

A Implementation Details

A.1 Data Representation

As described in the main paper, we represent each MILP instance as a weighted bipartite graph G = ( V ∪ W , E ) , where V represents the constraint vertex set, W represents the variable vertex set, and E represents the edge set, respectively. The graph is equipped with a tuple of feature matrices ( V , W , E ) , and the description of these features can be found in Table 5.

To ensure consistency, we standardize each instance to the form of Equation 1. However, we do not perform data normalization in order to preserve the potential information related to the problem domain in the original formulation. When extracting the bipartite graph, we utilize the readily available observation function provided by Ecole. For additional details on the observation function, readers can consult the following link: https://doc.ecole.ai/py/en/stable/reference/ observations.html#ecole.observation.MilpBipartite .

A.2 The Derivation of Masked Variational Auto-Encoder

We consider a variable with a distribution p ( x ) . We draw samples from this distribution and apply a masking process to transform each sample x into ˜ x through a given probability ˜ p (˜ x | x ) . Our objective is to construct a parameterized generator p θ ( x | ˜ x ) to produce new data based on the the masked data ˜ x . We assume that the generation process involves an unobserved continuous random variable z that is independent of ˜ x , i.e., z ⊥ ˜ x . Consequently, we obtain the following equation:

We introduce a probabilistic encoder q ϕ ( z | x ) for approximating the intractable latent variable distribution. We can then derive the follows:

In the formula, the term -L ( θ , ϕ | x , ˜ x ) is referred to as the evidence lower bound (ELBO), or the variational lower bound. It can be expressed as:

Consequently, the loss function can be formulated as follows:

where

In the formula, the first term L rec is referred to as the reconstruction loss, as it urges the decoder to reconstruct the input data x . The second term L prior is referred to as the prior loss, as it regularizes the posterior distribution q ϕ ( z | x ) of the latent variable to approximate the prior distribution p θ ( z ) . In practice, the prior distribution p θ ( z ) is commonly taken as N ( 0 , I ) , and a hyperparameter is often introduced as the coefficient for the prior loss. Consequently, the loss function can be expressed as:

where

In G2MILP, the loss function is instantiated as:

A.3 G2MILP Implementation

A.3.1 Encoder

The encoder implements q ϕ ( Z |G ) in Equation 22. Given a bipartite graph G = ( V ∪ W , E ) equipped with the feature metrices ( V , W , E ) , we employ a GNN structure with parameters ϕ to extract the representations. Specifically, we utilize MLPs as embedding layers to obtain the initial embeddings h (0) v i , h (0) w j , and h e ij , given by:

Next, we perform K graph convolution layers, with each layer in the form of two interleaved half-convolutions. The convolution layer is defined as follows:

where

The convolution layer is followed by two GraphNorm layers, one for constraint vertices and the other for variable vertices. We employ a concatenation Jumping Knowledge layer to aggregate information from all K layers and obtain the node representations:

The obtained representations contain information about the instances. Subsequently, we use two MLPs to output the mean and log variance, and then sample the latent vectors for each vertex from a Gussian distribution as follows:

A.3.2 Decoder

The decoder implements p θ ( G| Z , ˜ G ) in Equation 22. It utilizes a GNN structure to obtain the representations, which has the same structure as the encoder GNN, but is with parameters θ instead of ϕ . To encode the masked graph, we assign a special [mask] token to the masked vertex ˜ v . Its initial embedding h (0) ˜ v is initialized as a special embedding h [mask] . We mask all edges between ˜ v and the variable vertices and add virtual edges. In each convolution layer, we apply a special update rule for ˜ v :

This updating is performed after each convolution layer, allowing ˜ v to aggregate and propagate the information from the entire graph.

The obtained representations are used for the four networks-Bias Predictor, Degree Predictor, Logits Predictor, and Weights Predictor-to determine the generated graph. The details of these networks have been described in the main paper. Here we provide the losses for the four prediction tasks. In the following context, the node features, e.g., h ˜ v , refer to those from ˜ G obtained by the decoder GNN.

① Bias Prediction Loss:

② Degree Prediction Loss:

③ Logits Prediction Loss:

④ Weights Prediction Loss:

With these four prediction tasks, the reconstruction loss in Equation 22 is instantiated as:

A.3.3 Training and Inference

We describe the training and inference procedures in Algorithm 1 and Algorithm 2, respectively.

B Experimental Details

B.1 Dataset

The three commonly used datasets, namely MIS, SetCover, and MIK, are the same as those used in [9]. Nurse Scheduling contains a group of 4 instances from MIPLIB 2017: nursesched-medium04 and nursesched-sprint-hidden09 for training, and nursesched-sprint02 and nursesched-sprint-late03 for test. Table 6 summarizes some statistics of these datasets.

B.2 Hyperparameters

We report some important hyperparameters in this section. Further details can be found in our code once the paper is accepted to be published.

We run our model on a single GeForce RTX 3090 GPU. The hidden dimension and the embedding dimension are set to 16 . The depth of the GNNs is 6 . Each MLP has one hidden layer and uses ReLU() as the activation function.

In this work, we simply set all α i to 1 . We find that the choice of β significantly impacts the model performance. For MIS, we set β to 0 . 00045 . For SetCover, MIK and Nurse Scheduling, we apply a sigmoid schedule [46] to let β to reach 0 . 0005 , 0 . 001 , and 0 . 001 , respectively. We employ the Adam optimizer, train the model for 20 , 000 steps, and choose the best checkpoint based on the average error in solving time and the number of branching nodes. The learning rate is initialized to 0 . 001 and decays exponentially. For MIS, SetCover, and MIK, we set the batch size to 30 . Specifically, to provide more challenging prediction tasks in each batch, we sample 15 graphs and use each graph to derive 2 masked ones for training. For Nurse Scheduling, we set the batch size as 1 due to the large size of each graph.

B.3 Structural Distributional Similarity

Table 7 presents the 11 statistics that we use to measure the structural distributional similarity. First, we calculate the statistics for each instance. We then compute the JS divergence D JS ,i between the generated samples and the training set for each descriptor i ∈ { 1 , · · · , 11 } . We estimate the distributions using the histogram function in numpy and the cross entropy using the entropy function in scipy. The JS divergence falls in the range [0 , log 2] , so we standardize it to a score s i via:

Then we compute the mean of the 11 scores for the descriptors to obtain the final score s :

Hence the final score ranges from 0 to 1 , with a higher score indicating better similarity.

We use the training set to train a G2MILP model for each dataset and generate 1000 instances to compute the similarity scores. For MIK, which has only 80 training instances, we estimated the score using put-back sampling.

±

±

±

±

Notice that for a fair comparison, we exclude statistics that remain constant in our approach, such as problem size and objective coefficients. We implement another version of metric that involves more statistics, and the results are in Appendix C.3.

B.4 Downstream Tasks

The generated instances have the potential to enrich dataset in any downstream task. In this work, we demonstrate this potential through two application scenarios, i.e., the optimal value prediction task and the predict-and-search framework.

Optimal Value Prediction Two datasets, MIK and Nurse Scheduling, are considered, with medium and extremely small sizes, respectively. Following [18], we employ a GNN as a predictive model. The GNN structure is similar to the GNNs in G2MIL. We obtain the graph representation using mean pooling over all vertices, followed by a two-layer MLP to predict the optimal values of the instances.

For each dataset, we train a GNN predictive model on the training set. Specifically, for MIK, we use 80% of instances for training, 20% of instances for validating, and train for 1000 epochs to select the best checkpoint based on validation MSE. For Nurse Scheduling, we use both instances to train the model for 80 epochs. We use the generative models, Bowly, Random, and G2MILP, to generate 20 instances similar to the training sets. For Random and G2MILP, we mix together the instances generated by setting the masking ratio η to 0 . 01 and 0 . 05 , respectively. Next, we use the generated instances to enrich the original training sets, and use the enriched data to train another predictive model. We test all the trained model on previously unseen test data. Table 8 presents the predictive MSE on the test sets of the models trained on different training sets. As the absolute values of MSE are less meaningful than the relative values, we report the performance improvements brought by different generative technique. The improvement of Model 2 relative to Model 1 is calculate as follows:

On MIK, Bowly results in numerical issues as some generated coefficients are excessively large. G2MILP significant improves the performance and outperforms Random. On Nurse Scheduling, Random fails to generate feasible instances, and Bowly yields a minor improvement. Notably, G2MILP allows for the training of the model with even minimal data.

Predict-and-Search We conduct experiments on a neural solver, i.e., the predict-and-search framework proposed by Han et al. [31] Specifically, they propose a framework that first predicts a solution and then uses solvers to search for the optimal solutions in a trust region. We consider using generated instances to enhance the predictive model. We first train the predictive model on 100 MIS instances, and then use the generative models to generate 100 new instances to augment the dataset. The results

are in Table 9 . Bowly generates low-quality data that disturbs the model training, so that there is no optimal solution in the trust region around the predicted solution. Though both Random and G2MILP can enhance the solving framework to reduce solving time, we can see G2MILP significantly outperforms Random.

Discussions These two downstream tasks, despite their simplicity, possess characteristics that make them representative problems that could benefit from generative models. Specifically, we identify the following features.

More is better. We want as many data instances as possible. This condition is satisfied when we can obtain precise labels using existing methods, e.g., prediction-based neural solvers [31], or when unlabeled data is required for RL model training, e.g., RL for cut selection [9].

More similar is better. We want independent identically distributed (i.i.d.) data instances for training. Thus

More diverse is better. We want the data to be diverse, despite being i.i.d., so that the trained model can generalize better.

Our experimental results demonstrate the potential of G2MILP in facilitating downstream tasks with these characteristics, thus enhancing the MILP solvers. We intend to explore additional application scenarios in future research.

C Additional Results

C.1 Comparison with G2SAT

We conduct an additional experiment that transfers G2SAT to a special MILP dataset, MIS, in which all coefficients are 1 . 0 and thus the instances can be modeled as homogeneous bipartite graphs. We apply G2SAT to learn to generate new graphs and convert them to MILPs. Results are in Table 10. The results show that G2MILP significantly outperforms G2SAT on the special cases.

C.2 Masking Process

Masking Variables In the mainbody, for simplicity, we define the masking process of uniformly sampling a constraint vertex ˜ v ∼ U ( V ) to mask, while keeping the variable vertices unchanged. We implement different versions of G2MILP that allow masking and modifying either constraints, variables, or both. The results are in Table 11.

Ablation on Masking Ratio We have conduct ablation studies on the effect of the masking ratio η on MIK. The results are in Figure 4. The experimental settings are the same with those in Table 2 and Figure 2. From the results we have the following conclusions. (1) though empirically a smaller leads to a relatively better performance, G2MILP maintains a high similarity performance even when η is large. (2) The downstream task performance does not drops significantly. This makes sense because smaller η leads to more similar instances, while larger η leads to more diverse (but still similar) instances, both of which can benefit downstream tasks. (3) G2MILP always outperforms Random,

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Mask Ratio

η

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Similarity Score

G2MILP

Random

(a) Distributional Similarity

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Mask Ratio

η

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Relative MSE

G2MILP

Random

(b) Relative MSE

which demonstrates that the learning paradigm helps maintain the performance. (4) Bowly fails on this dataset because its generated instances lead to numerical issues and cannot be read by Gurobi or SCIP. Moreover, in real applications, it is reasonable and flexible to adjust the hyperparameter to achieve good performances in different scenarios.

Orders of Masked Constraints We also investigate different orders of masking constraint vertices, including uniformly sampling and sampling according to the vertex indices. Results are in Table 12. We find that uniformly sampling achieves the best performance. Sampling according to indices leads to a performance decrease, maybe because near constraints are relevant and lead to error accumulation. We think these results are interesting, and will study it in the future work.

C.3 Structural Distributional Similarity

In the mainbody, for a fair comparison, we exclude statistics that remain constant in our approach, such as problem size and objective coefficients. However, these statistics are also important features for MILPs. In this section, we incorporate three additional statistics in the computing of similarity scores: (1) mean of objective coefficients c , (2) std of objective coefficients c , and (3) the ratio of continuous variables. With these additional metrics, we recompute the structural similarity scores and updated the results in both Table 2 and Table 11. The new results are in Table 13 and Table 14,

respectively. From the results, we can still conclude that G2MILP outperforms all baselines, further supporting the effectiveness of our proposed method.

C.4 Sizes of Datasets

We conduct experiments on different sizes of the original datasets, as well as the ratio of generated instances to original ones, on MIS. The results are in Table 15. The results show that G2MILP can bring performance improvements on varying sizes of datasets.

C.5 Visualization

The t-SNE visualization for baselines are in Figure 5. G2MILP generates diverse instances around the training set, while instances generated by Random are more biased from the realistic ones.

G2MILP (

η

= 0

.

01)

G2MILP (

η

= 0

.

05)

G2MILP (

η

= 0

.

1)

Random (

η

= 0

.

01)

Training Set

Random

Random (

η

= 0

.

05)

Random (

η

= 0

.

1)

Peer reviews:
comments: SUMMARY:
The paper discusses the recent surge in using machine learning techniques for solving combinatorial optimization problems, particularly mixed-integer linear programs (MILPs). However, the limited availability of real-world instances often hinders optimal decision-making and unbiased solver assessments. To address this issue, the proposed solution is G2MILP, a deep generative framework for MILP instances. G2MILP represents MILP instances as bipartite graphs and employs a masked variational autoencoder to corrupt and replace parts of the graphs iteratively, generating new instances. This approach learns to generate realistic MILP instances without relying on expert-designed formulations, while preserving the structures and computational complexity of real-world datasets. The generated instances can assist in improving MILP solvers when data availability is limited. A set of benchmarks is designed to evaluate the quality of the generated MILP instances, and experimental results demonstrate their resemblance to real-world datasets in terms of both structure and computational difficulty.

STRENGTHS:
1. The paper takes a traditionally hard problem and proposes a novel solution to it.
2. Experiment results seem very compellingly, however the lack of good baseline makes hard to judge how good it actually is.
3. Presentation is good and clear.

WEAKNESSES:
some minor issue might be lack of proper baseline, it's quite hard for the readers and reviewers to access how good this method actually is.

QUESTIONS:
Is it possible to provide some "fair" comparisons on downstream proxy tasks with other popular methods? I must confess that I am not very familiar with the specific subdomain.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a deep generative framework for MILP(mixed integer programs) instances. The work has applications in enhancing MILP solvers under limited data availability.  MILP instances are represented as weighted bipartite graphs.  The authors propose a masked  variational autoencoder (VAE) method for graph generation. \n\nThe problem is relevant and interesting. The experiments show that the proposed approach obtains significantly better results than existing methods.\n\nSTRENGTHS:\n1.  Important problem to tackle. The work has applications.\n\n2. Paper is easy to understand.\n\n3. The authors motivate different components of encoder decoder architecture nicely in the methodology section.\n\n4. The experimental section consists of Structural similarity metrics, hardness comparison of generated MILP instances, application on downstream task.\n\n5. Comparison on variety of datasets( MIS, Set cover, MIK(knapsack), including standard deviation, multiple runs.\n\n6. Visualization of generated instances( through TSNE plots).\n\nWEAKNESSES:\n1.  Downstream Task( learning to solve MIP)- learning to branch etc. :\n\nRecently many neural MIP solvers have come up( Eg: -Gasse et al.[A]). The authors have also cited them. Is it possible for the authors to show whether they can improve the performance of such solvers by data augmentation( on any dataset) under low data availability.\n\nI agree the authors have shown improvement on predicting optimal value task. However, the above mentioned task is also important. It can make the paper stronger.\n\n\n2. Presentation :\nTable 8 and Fig 2 not clear.\n\n Request the authors to add Standard deviation in Table 8 Appendix ( especially for Random). Also I see \"Random\" in Table 8 and not in Fig.2 ( Downstream task).. \n\nAlso is there any difference between Table 8 and Fig 2?\n\n\n3. Code is not provided by authors.\n\n\n\n[A] Maxime Gasse, Didier Chételat, Nicola Ferroni, Laurent Charlin, and Andrea Lodi. Exact354\ncombinatorial optimization with graph convolutional neural networks. Advances in neural355\ninformation processing systems, 32, 2019\n\nQUESTIONS:\nRequest the authors to answer the questions in weakness section. \n1. Downstream task: Augmenting neural mip solvers with generated data. ( Especially under low data availability).\n\n2. Presentation clarification( Fig 2 and Table 8).\n\nLIMITATIONS:\nThe authors have addressed it nicely.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper generates MIP instances with the help of VAEs. Experiment results show that the generated samples remain the characters of the training dataset from several aspects (e.g. the graph statistical value, the solving time of the generated samples, and the number of branching nodes of the generated samples), The evaluation shows that the generation helps the diversity, and preserves the difficulty. The samples may further help the NNs in downstream learning for MIP training, the authors introduces a task that predicts the optimal objective value, models trained on G2MILP shows obvious improvement on the testing dataset.

STRENGTHS:
The task is highly valuable.
The idea of generating instances via deep neural networks is novel and sound. The whole framework provides an approach that generates new instances without the need to start form scratch.
Experiment results prove the effectiveness of the proposed method across various datasets (MIS, MIK, set covering and a nurse scheduling problem from MIPLIB2017).

WEAKNESSES:
N/A

QUESTIONS:
In the experiment results, the ratio $\eta$ is controlled as $0.01, 0.05$ and $0.1$. At most only 10% of the nodes in the original V-C bipartite graph are changed. 

1.	Why doesn’t the generator set  larger $\eta$ for a complete comparison? (As the other baselines may generate the instances from scratch.) From the table, the effectiveness of the method quickly drops as $\eta$ increases.

LIMITATIONS:
As far as I could understand, the generator could only output the instances with the same problem scale.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of limited availability of real-world mixed-integer linear programs (MILPs) instances for machine learning techniques employed in combinatorial optimization problems. The scarcity of these instances often leads to sub-optimal decisions and biased solver assessments. Existing solutions either depend heavily on expert-designed formulations or fail to capture the intricate features of real-world instances.\n\nThe authors propose G2MILP, the first deep generative framework for creating MILP instances. G2MILP treats MILP instances as bipartite graphs and uses a masked variational autoencoder to iteratively corrupt and replace parts of the original graphs, generating new ones. This approach allows G2MILP to learn to generate novel and realistic MILP instances without the need for expert-designed formulations, while simultaneously preserving the structures and computational hardness of real-world datasets. The generated instances can help improve MILP solvers when data availability is limited.\n\nThe authors also present a set of benchmarks to evaluate the quality of the generated MILP instances. The experimental results show that G2MILP is capable of producing instances that closely mimic real-world datasets in terms of both structures and computational hardness.\n\nSTRENGTHS:\n1. This is the first deep generative model for MILP instance generation based on the well known variant-constraint bipartite representation.\n\n2. My favorite part of this work is the masking-based pre-training scheme, making me recall the masked language modeling objective in large language modeling. My feeling is that this direction could probably lead to large optimization model pre-training in the L2O field and this could be a very good start.\n\n3. The empirical performance is good.\n\nWEAKNESSES:\n1. I have the feeling that the problem similarity metric is too privileged for the proposed method, as it essentially modifies existing problem instances.\n\n2. The effectiveness seems to drop quickly with higher ratio of variable/constraint alternations.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is focused on using machine learning techniques to address combinatorial optimization problems, particularly mixed-integer linear programs (MILPs). Particularly, this paper proposed G2MILP, which is a generative model for generating realistic MILP instances. Without prior expert knowledge, G2MILP can generate expressive instances preserving computational hardness of real-world datasets. To achieve this, G2MILP adopts a masked VAE approach to iteratively mask original graphs to generate new ones. Extensive evaluations are made to demonstrate the effectiveness of the proposed approach.  This paper shows how to adopt the well-established concepts (e.g., masked VAE) to the new data type (i.e., MILP instances).\n\nSTRENGTHS:\nThe main strength of this paper is its originality in my eyes. It introduces a generative framework for generating realistic MILP instances. Though different generative models have been established for various types of data, little attention is paid for using generative models to generate MILP instances for improving the solvers.  To better develop training-based MILP solver, adequate training data is needed, and the proposed model can help.\n\nWEAKNESSES:\n1. \n\nThe contribution of the paper is unclear.  Though the paper introduces a new research direction of generating MILP instances, its technical contributions are unclear.  Representing MILP using bipartite graph and capturing features using GNN have been proposed by Gasse et al. The major technical contribution of this work is using masked VAE for expressive and realistic MILP instance generation, which I think is heuristic and lacks novelty. \n\n2. \n\nThe methodology is not easy to follow. Adopting the concept of masked VAE to generate expressive MILP instances seems a cute idea. But the masking process is confusing to me. \n\n2.1. How do you generate the masked graph G_tilde? What is the distribution of p_tilde(G_tilde|G)? From lines 165-168, the masking is performed by randomly selecting a constraint vertex. Does this mean p_tilde(G_tilde|G) is just a uniform distribution or something like that? \n\n2.2. What’s the intuition of making the masked graph as a conditional variable for the decoder?  In masked VAE, the masked images are the input of the encoder. More explanations about the difference between the original masked VAE and the proposed pipeline for MILP instance generation are expected. \n\n3.\t\n\nThe evaluations are insufficient to fully validate the proposed framework.\n\n3.1. There is a lack of ablation study to show that the masking scheme helps improve complex graph generation as claimed in lines 58 – 60. Though there are evaluations considering different masking ratios (0.01, 0.05, 0.1), they are insufficient to help understand the proposed masking scheme. Does the order of masking constraint vertices matter in the generation process?  \n\n3.2. What’s the reconstruction for each graph component (i.e., bias, degree, etc)? How the graph features affect the new instance generation (e.g., the density of the graph)? I guess the new instance generation will be harder if the original graph is dense and complex? \n\n\n3.3. The downstream task evaluation is important to demonstrate the benefits of the proposed method. However, this evaluation is inadequate. What is the mask ratio for the downstream task evaluation (figure 2)? I guess the performance will decrease if the mask ratio is high? Can you report the std for performance improvement given different sets of 20 generated instances? What about the performance on larger datasets? To which extend (e.g., the scarcity of the original dataset), the generated instances help improve the performance?\n\n4.\t\n\nThe significance of the proposed method is unclear. Since the proposed method only mask constraint vertices, I think it is more like a graph generation pipeline. I am not fully convinced that we need to make this many efforts to generate new graphs. Furthermore, the benefits of new instances are not well demonstrated (as I question above). \n\n5.\t\n\nPresentations can be improved. The position of Table 2 needs to be re-arranged. Notation wise, I think it is a bit confusing to use the same G for both the input and the output of the model. Typo: there are two ‘and’s in line 67.\n\nQUESTIONS:\nI would like to adjust my rating if the authors can help clarify my confusions listed in the weakness section. Besides, I have other questions listed below:\n\n1. Generating instances is not a new topic and G2SAT has explored this. Why it is non-trivial to adapt G2SAT to MILP instance generation? How does the consideration of the high-precision numerical prediction affect the design of the generative framework?\n\n2. The generation of a new MILP instance is an iterative process. What is the computation cost for generating one MILP instance?\n\nLIMITATIONS:\nThe limitations are discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a deep generative framework G2MILP based on a masked variational autoencoder (VAE) for mixed-integer linear programs (MILP) instances. The proposed framework G2MILP corrupts and replaces constraints to generate new MILP instances while keeping the variables unchanged. Experiments show that G2MILP can produce instances that closely resemble real-world datasets in terms of both structures and computational hardness. Meanwhile, the generated instances from G2MILP can be used to augment the small training dataset to improve the prediction performance.\n\nSTRENGTHS:\n1. The paper is well motivated by the limited data issue of MILP in the real world and the shortage of a flexible generative framework that can generate high-quality MILP instances.\n2. The proposed decoder consisting of four predictive modules seems quite tailor-designed for the problem.\n\nWEAKNESSES:\n1. The motivation of this work is similar to those Graph generation works and G2SAT, though targeting a different specific application domain. The main technique of masked VAE has already been well studied in the literature. So, this work is not that novel and interesting. \n2. The proposed G2MILP can only modify the constraints of MILP beyond the training samples, which limits the contribution of this work. Although the authors claimed that their framework can be extended to change the variables of MILP, it is expected such flexibility can be achieved in this work. \n3. Eq. (9) and Eq. (11) are inconsistent with Fig. 1. The logits/weights predictor shown in Fig. 1 is conditioned on $d_\\tilde{v}$ but Eq. (9) and (11) are not. Moreover, from my understanding, the logits/weights prediction of $\\delta_{\\tilde{v}, u}$/$e_{u, \\tilde{v}}$ should also be conditioned on $h_\\tilde{v}$ and $z_{\\tilde{v}}$.\n4. The writing needs to be further improved. See the questions below.\n\nQUESTIONS:\n1. How does G2MILP deal with different n and m?  Different instances would have different n and m.\n2. For the proposed normalization, would it be problematic when there is outlier in the dataset?\n3. In Section 4.1, about structural distributional similarity, statistics that remain constant are excluded. Does it mean the similarity is only calculated in terms of the constraints?\n4. For the Bowly baseline, can it change variables? If can, the similarity comparison may be unfair since the Bowly baseline introduces more diversity to the generated instances.\n5. Good generation should introduce novelty compared to the training set, implying that the generated samples should differ from the training samples. So, it is tricky to use the structural distribution similarity to denote a good generation. I expect a discussion on this. t-SNE visualization for the baselines should be included.\n\nLIMITATIONS:\nThe authors have discussed the limitations of their work, which however I think should be a part of this work (point 2 in the weakness).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-17"}