SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} _ {\text {l o g i t s}} = - \mathbb {E} _ {(\tilde {v}, u) \sim p _ {\text {p o s}}} \left[ \log \left(\hat {\delta} _ {\tilde {v}, u} ^ {\prime}\right) \right] - \mathbb {E} _ {(\tilde {v}, u) \sim p _ {\text {n e g}}} \left[ \log \left(1 - \hat {\delta} _ {\tilde {v}, u} ^ {\prime}\right) \right], \tag {10}
$$
where  $p_{\mathrm{pos}}$  and  $p_{\mathrm{neg}}$  denote the distributions over positive and negative samples, respectively. At inference time, we connect  $\hat{d}_{\tilde{v}}$  variable vertices with the top logits to  $\tilde{v}$ , i.e.,
$$
\hat {\delta} _ {\tilde {v}, u} = \left\{ \begin{array}{l} 1, u \in \arg \operatorname {T o p K} \left(\left\{\hat {\delta} _ {\tilde {v}, u} ^ {\prime} \mid u \in \mathcal {W} \right\}, \hat {d} _ {\tilde {v}}\right), \\ 0, \text {o t h e r w i s e .} \end{array} \right. \tag {11}
$$
$④$  Weights Predictor Finally, we use one MLP to predict the normalized weights  $e_{\tilde{v},u}^{*}$  for nodes  $u$  that are connected with  $\tilde{v}$ :
$$
\hat {e} _ {\tilde {v}, u} ^ {*} = \sigma \left(\mathrm {M L P} _ {\theta} ^ {\text {w e i g h t s}} \left(\left[ \boldsymbol {h} _ {u}, \boldsymbol {z} _ {u} \right]\right)\right). \tag {12}
$$
The training and inference procedures are similar to those of Bias Predictor.
3.4 Training and Inference
During training, we use the original graph  $\mathcal{G}$  to provide supervision signals for the decoder, guiding it to reconstruct  $\mathcal{G}$  from the masked  $\tilde{\mathcal{G}}$  and the encoded  $\mathbf{Z}$ . As described above, the decoder involves four modules, each optimized by a prediction task. The first term in Equation 4, i.e., the reconstruction loss, is written as
$$
\mathcal {L} _ {\text {r e c}} = \mathbb {E} _ {\mathcal {G} \sim \mathcal {D}, \tilde {\mathcal {G}} \sim \tilde {p} (\tilde {\mathcal {G}} | \mathcal {G})} \left[ \sum_ {i = 1} ^ {4} \alpha_ {i} \cdot \mathcal {L} _ {i} (\boldsymbol {\theta}, \phi | \mathcal {G}, \tilde {\mathcal {G}}) \right], \tag {13}
$$
where  $\mathcal{L}_i(\pmb {\theta},\phi |\mathcal{G},\tilde{\mathcal{G}})$ $(i = 1,2,3,4)$  are loss functions for the four prediction tasks, respectively, and  $\alpha_{i}$  are hyperparameters.
During inference, we discard the encoder and sample  $\mathbf{Z}$  from a standard Gaussian distribution, which introduces randomness to enable the model to generate novel instances. We iteratively mask one constraint vertex in the bipartite graph and replace it with a generated one. We define a hyperparameter  $\eta$  to adjust the ratio of iterations to the number of constraints, i.e.,  $N_{\mathrm{iters}} = \eta \cdot |\mathcal{V}|$ . Naturally, a larger value of  $\eta$  results in instances that are more novel, while a smaller value of  $\eta$  yields instances that exhibit better similarity to the training set. For further details of training and inference procedures, please refer to Appendix A.3.
4.1 Setup
We conduct extensive experiments to demonstrate the effectiveness of our model. More experimental details can be found in Appendix B. Additional results are in Appendix C.
Benchmarking To evaluate the quality of the generated MILP instances, we design three benchmarks so as to answer the following research questions. (1) How well can the generated instances preserve the graph structures of the training set? (2) How closely do the generated instances resemble the computational hardness of real-world instances? (3) How effectively do they facilitate downstream tasks to improve solver performance?
I. Structural Distributional Similarity We consider 11 classical statistics to represent features of the instances [17, 47], including coefficient density, node degrees, graph clustering, graph modularity, etc. In line with a widely used graph generation benchmark [48], we compute the Jensen-Shannon (JS) divergence [49] for each statistic to measure the distributional similarity between the generated instances and the training set. We then standardize the metrics into similarity scores that range from 0 to 1. The computing details can be found in Appendix B.3.
II. Computational Hardness The computational hardness is another critical metric to assess the quality of the generated instances. We draw an analogy from the SAT generation community, where though many progresses achieved, it is widely acknowledged that the generated SAT instances differs significantly from real-world ones in the computational hardness [25], and this issue remains inadequately addressed. In our work, we make efforts to mitigate this problem, even in the context of MILP generation, a more challenging task. To this end, we leverage the state-of-the-art solver Gurobi [12] to solve the instances, and we report the solving time and the numbers of branching nodes during the solving process, which can directly reflect the computational hardness of instances [19].
III. Downstream Task We consider two downstream tasks to examine the the potential benefits of the generated instances in practical applications. We employ G2MILP to generate new instances and augment the original datasets, and then evaluate whether the enriched datasets can improve the performance of the downstream tasks. The considered tasks include predicting the optimal values of the MILP problem, as discussed in Chen et al. [18], and applying a predict-and-search framework for solving MILPs, as proposed by Han et al. [31].
Datasets We consider four different datasets of various sizes. (1) Large datasets. We evaluate the model's capability of learning data distributions using two well-known synthetic MILP datasets: Maximum Independent Set (MIS) [50] and Set Covering [15]. We follow previous works [8, 9] to artificially generate 1000 instances for each of them. (2) Medium dataset. Mixed-integer Knapsack (MIK) is a widely used dataset [34], which consists of 80 training instances and 10 test instances. We use this dataset to evaluate the model's performance both on the distribution learning benchmarks and the downstream task. (3) Small dataset. We construct a small subset of MIPLIB 2017 [10] by collecting a group of problems called Nurse Scheduling problems. This dataset comes from real-world scenarios and consists of only 4 instances, 2 for training and 2 for test, respectively. Since the statistics are meaningless for such an extremely small dataset, we use it only to demonstrate the effectiveness of generated instances in facilitating downstream tasks.
Baselines G2MILP is the first deep learning generative framework for MILP instances, and thus, we do not have any learning-based models for comparison purpose. Therefore, we compare G2MILP with a heuristic MILP instance generator, namely Bowly [16]. Bowly can create feasible and bounded MILP instances while controlling some specific statistical features such as coefficient density and coefficient mean. We set all the controllable parameters to match the corresponding statistics of the training set, allowing Bowly to imitate the training set to some extent. We also consider a useful baseline, namely Random, to demonstrate the effectiveness of deep neural networks in G2MILP. Random employs the same generation procedure as G2MILP, but replaces all neural networks in the decoder with random generators. We set the masking ratio  $\eta$  for Random and G2MILP to 0.01, 0.05, and 0.1 to show how this hyperparameter helps balance the novelty and similarity.
4.2 Quantitative Results
I. Structural Distributional Similarity We present the structural distributional similarity scores between each pair of datasets in Table 1. The results indicate that our designed metric is reasonable in the sense that datasets obtain high scores with themselves and low
Table 1: Structural similarity scores between each pair of datasets. Higher is better.
MIS	SetCover	MIK
MIS	0.998	0.182	0.042
SetCover	-	1.000	0.128
MIK	-	-	0.997
Table 3: Average solving time (s) of instances solved by Gurobi (mean ± std).  $\eta$  is the masking ratio. Numbers in the parentheses are relative errors with respect to the training sets (lower is better).
MIS	SetCover	MIK
Training Set	0.349 ± 0.05	2.344± 0.13	0.198± 0.04
Bowly	0.007± 0.00 (97.9%)	0.048± 0.00 (97.9%)	0.001± 0.00 (99.8%)
η = 0.01	Random	0.311± 0.05 (10.8%)	2.044± 0.19 (12.8%)	0.008± 0.00 (96.1%)
G2MILP	0.354± 0.06 (1.5%)	2.360± 0.18 (0.8%)	0.169± 0.07 (14.7%)
η = 0.05	Random	0.569± 0.09 (63.0%)	2.010± 0.11 (14.3%)	0.004± 0.00 (97.9%)
G2MILP	0.292± 0.07 (16.3%)	2.533± 0.15 (8.1%)	0.129± 0.05 (35.1%)
η = 0.1	Random	2.367± 0.35 (578.2%)	1.988± 0.17 (15.2%)	0.005± 0.00 (97.6%)
G2MILP	0.214± 0.05 (38.7%)	2.108± 0.21 (10.0%)	0.072± 0.02 (63.9%)
Table 4: Average numbers of branching nodes of instances solved by Gurobi.  $\eta$  is the masking ratio. Numbers in the parentheses are relative errors with respect to the training sets (lower is better).
MIS	SetCover	MIK
Training Set	16.09	838.56	175.35
Bowly	0.00 (100.0%)	0.00 (100.0%)	0.00 (100.0%)
η = 0.01	Random	20.60 (28.1%)	838.51 (0.0%)	0.81 (99.5%)
G2MILP	15.03 (6.6%)	876.09 (4.4%)	262.25 (14.7%)
η = 0.05	Random	76.22 (373.7%)	765.30 (8.7%)	0.00 (100%)
G2MILP	10.58 (34.2%)	874.46 (4.3%)	235.35 (34.2%)
η = 0.1	Random	484.47 (2911.2%)	731.14 (12.8%)	0.00 (100%)
G2MILP	4.61 (71.3%)	876.92 (4.6%)	140.06 (20.1%)
scores with different domains. Table 2 shows the similarity scores between generated instances and the corresponding training sets. We generate 1000 instances for each dataset to compute the similarity scores. The results suggest that G2MILP closely fits the data distribution, while Bowly, which relies on heuristic rules to control the statistical features, falls short of our expectations. Furthermore, we observe that G2MILP outperforms Random, indicating that deep learning contributes to the model's performance. As expected, a higher masking ratio  $\eta$  results in generating more novel instances but reduces their similarity to the training sets.
II. Computational Hardness We report the average solving time and numbers of branching
nodes in Table 3 and Table 4, respectively. The results indicate that instances generated by Bowly are relatively easy, and the hardness of those generated by Random is inconclusive. In contrast, G2MILP is capable of preserving the computational hardness of the original training sets. Notably, even without imposing rules to guarantee the feasibility and boundedness of generated instances, G2MILP automatically learns from the data and produces feasible and bounded instances.
Table 2: Structural distributional similarity scores between the generated instances with the training datasets. Higher is better.  $\eta$  is the masking ratio. We do not report the results of Bowly on MIK because Ecole [45] and SCIP [51] fail to read the generated instances due to large numerical values.
MIS	SetCover	MIK
Bowly	0.184	0.197	-
η = 0.01	Random	0.651	0.735	0.969
G2MILP	0.997	0.835	0.991
η = 0.05	Random	0.580	0.613	0.840
G2MILP	0.940	0.782	0.953
η = 0.1	Random	0.512	0.556	0.700
G2MILP	0.895	0.782	0.918
III. Downstream Task First, we follow Chen et al. [18] to construct a GNN model for predicting the optimal values of MILP problems. We train a predictive GNN model on the training set. After that, we employ 20 generated instances to augment the training data, and then train another predictive
Figure 2: Results of the optimal value prediction task. Bars indicate the relative MSE to the model trained on the original training sets, and lines represent the relative performance improvement.
Figure 3: The t-SNE visualization of MILP instance representations for MIK. Each point represents an instance. Red points are from the training set and blue points are instances generated by G2MILP.
model using the enriched dataset. We use the prediction mean squared error (MSE) to assess the resulting models, and we present the MSE relative to the default model trained on the original training sets in Figure 2. For the MIK dataset, instances generated by Bowly introduce numerical issues so that Ecole and SCIP fail to read them. For the Nurse Scheduling dataset, Random fails to generate feasible instances. Notably, G2MILP is the only method that demonstrates performance improvement on both datasets, reducing the MSE by  $73.7\%$  and  $24.3\%$ , respectively. The detailed results are in Table 8 in Appendix B.4. Then, we conduct experiments on the neural solver, i.e., the predict-and-search framework proposed by Han et al. [31], which employs a model to predict a solution and then uses solvers to search for the optimal solutions in a trust region. The results are in Table 9 in Appendix B.4.
4.3 Analysis
Masking Process We conduct extensive comparative experiments on different implementations of the masking process. First, we implement different versions of G2MILP, which enable us to mask and modify either constraints, variables, or both. Second, we investigate different orders of masking constraints, including uniformly sampling and sampling according to the vertex indices. Third, we analyze the effect of the masking ratio  $\eta$  on similarity scores and downstream task performance improvements. The experimental results are in Appendix C.2.
Size of Dataset We conduct experiments on different sizes of the original datasets and different ratio of generated instances to original ones on MIS. Results are in Table 15 in Appendix C.4. The results show that G2MILP yields performance improvements across datasets of varying sizes.
Visualization We visualize the instance representations for MIK in Figure 3. Specifically, we use the G2MILP encoder to obtain the instance representations, and then apply t-SNE dimensionality reduction [52] for visualization. We observe that the generated instances, while closely resembling the training set, contribute to a broader and more continuous exploration of the problem space, thereby enhancing model robustness and generalization. Additionally, by increasing the masking ratio  $\eta$ , we can effectively explore a wider problem space beyond the confines of the training sets. For comparison with the baseline, we present the visualization of instances generated by Random in Figure 5 in Appendix C.5.
5 Limitations, Future Avenues, and Conclusions
Limitations In this paper, we develop G2MILP by iteratively corrupting and replacing the constraints vertices. We also investigate different implementations of the masking process. However, more versatile masking schemes should be explored. Moreover, employing more sophisticated designs would enable us to control critical properties such as feasibility of the instances. We intend to develop a more versatile and powerful generator in our future work.
Future Avenues We open up new avenues for research on DL-based MILP instance generative models. In addition to producing new instances to enrich the datasets, this research has many other promising technical implications [35]. (1) Such a generator will assist researchers to gain insights into different data domains and the explored space of MILP instances. (2) Based on a generative model, we can establish an adversarial framework, where the generator aims to identify challenging cases for the solver, thus automatically enhancing the solver's ability to handle complex scenarios. (3) Training a generative model involves learning the data distribution and deriving representations through unsupervised learning. Consequently, it is possible to develop a pre-trained model based on a generative model, which can benefit downstream tasks across various domains. We believe that this paper serves as an entrance for the aforementioned routes, and we expect further efforts in this field.
Conclusions In this paper, we propose G2MILP, which to the best of our knowledge is the first deep generative framework for MILP instances. It can learn to generate MILP instances without prior expert-designed formulations, while preserving the structures and computational hardness, simultaneously. Thus the generated instances can enhance MILP solvers under limited data availability. This work opens up new avenues for research on DL-based MILP instance generative models.
The authors would like to thank all the anonymous reviewers for their insightful comments. This work was supported in part by National Key R&D Program of China under contract 2022ZD0119801, National Nature Science Foundations of China grants U19B2026, U19B2044, 61836011, 62021001, and 61836006.
[1] John A Muckstadt and Richard C Wilson. An application of mixed-integer programming duality to scheduling thermal generating systems. IEEE Transactions on Power Apparatus and Systems, (12), 1968.
[2] Yves Pochet and Laurence A Wolsey. Production planning by mixed integer programming, volume 149. Springer, 2006.
[3] Rodrigo Moreno, Roberto Moreira, and Goran Strbac. A milp model for optimising multiservice portfolios of distributed energy storage. Applied Energy, 137:554-566, 2015.
[4] Robert E Bixby, Mary Fenelon, Zonghao Gu, Ed Rothberg, and Roland Wunderling. Mixed-integer programming: A progress report. In The sharpest cut: the impact of Manfred Padberg and his work, pages 309–325. SIAM, 2004.
[5] Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: a methodological tour d'horizon. European Journal of Operational Research, 290(2):405-421, 2021.
[6] Jiayi Zhang, Chang Liu, Xijun Li, Hui-Ling Zhen, Mingxuan Yuan, Yawen Li, and Junchi Yan. A survey for solving mixed integer programming via machine learning. Neurocomputing, 519: 205-217, 2023.
[7] He He, Hal Daume III, and Jason M Eisner. Learning to search in branch and bound algorithms. Advances in neural information processing systems, 27, 2014.
[42] Duncan J Watts and Steven H Strogatz. Collective dynamics of 'small-world'networks. nature, 393(6684):440-442, 1998.
[43] Jure Leskovec, Deepayan Chakrabarti, Jon Kleinberg, Christos Faloutsos, and Zoubin Ghahrami. Kronecker graphs: an approach to modeling networks. Journal of Machine Learning Research, 11(2), 2010.
[44] Ivan Garzon, Pablo Mesejo, and Jesús Giráldez-Cru. On the performance of deep generative models of realistic sat instances. In 25th International Conference on Theory and Applications of Satisfiability Testing (SAT 2022). Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2022.
[45] Antoine Prouvost, Justin Dumouchelle, Lara Scavuzzo, Maxime Gasse, Didier Chételat, and Andrea Lodi. Ecole: A gym-like library for machine learning in combinatorial optimization solvers. In Learning Meets Combinatorial Algorithms at NeurIPS2020, 2020. URL https://openreview.net/forum?id=IVc9hqgibyB.
[46] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349, 2015.
[47] Frank Hutter, Lin Xu, Holger H Hoos, and Kevin Leyton-Brown. Algorithm runtime prediction: Methods & evaluation. Artificial Intelligence, 206:79-111, 2014.
[48] Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling, 59(3):1096-1108, 2019.
[49] Jianhua Lin. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145-151, 1991.
[50] David Bergman, Andre A Cire, Willem-Jan Van Hoeve, and John Hooker. Decision diagrams for optimization, volume 1. Springer, 2016.
[51] Tobias Achterberg. Scip: solving constraint integer programs. Mathematical Programming Computation, 1:1-41, 2009.
[52] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
A Implementation Details
A.1 Data Representation
As described in the main paper, we represent each MILP instance as a weighted bipartite graph  $\mathcal{G} = (\mathcal{V} \cup \mathcal{W}, \mathcal{E})$ , where  $\mathcal{V}$  represents the constraint vertex set,  $\mathcal{W}$  represents the variable vertex set, and  $\mathcal{E}$  represents the edge set, respectively. The graph is equipped with a tuple of feature matrices  $(\mathbf{V}, \mathbf{W}, \mathbf{E})$ , and the description of these features can be found in Table 5.
Table 5: Description of the constraint, variable, and edge features in our bipartite graph representation.
Tensor	Feature	Description
V	bias	The bias valuebi.
W	type	Variable type (binary, continuous, integer, implicit integer) as a 4-dimensional one-hot encoding.
objective	Objective coefficientcj.
has_lower_bound	Lower bound indicator.
has_upper_bound	Upper bound indicator.
lower_bound	Lower bound valuelj.
upper_bound	Upper bound valueuj.
E	coef	Constraint coefficientaij.
To ensure consistency, we standardize each instance to the form of Equation 1. However, we do not perform data normalization in order to preserve the potential information related to the problem domain in the original formulation. When extracting the bipartite graph, we utilize the readily available observation function provided by Ecole. For additional details on the observation function, readers can consult the following link: https://doc.ecole.ai/py/en/stable/reference/observations.html#ecoleobservation.MilpBipartite.
A.2 The Derivation of Masked Variational Auto-Encoder
We consider a variable with a distribution  $p(\pmb{x})$ . We draw samples from this distribution and apply a masking process to transform each sample  $\pmb{x}$  into  $\tilde{\pmb{x}}$  through a given probability  $\tilde{p}(\tilde{\pmb{x}}|\pmb{x})$ . Our objective is to construct a parameterized generator  $p_{\theta}(\pmb{x}|\tilde{\pmb{x}})$  to produce new data based on the the masked data  $\tilde{\pmb{x}}$ . We assume that the generation process involves an unobserved continuous random variable  $z$  that is independent of  $\tilde{\pmb{x}}$ , i.e.,  $z \perp \tilde{\pmb{x}}$ . Consequently, we obtain the following equation:
$$
p _ {\theta} (\boldsymbol {x} | \tilde {\boldsymbol {x}}) = \frac {p _ {\theta} (\boldsymbol {x} | \boldsymbol {z} , \tilde {\boldsymbol {x}}) p _ {\theta} (\boldsymbol {z} | \tilde {\boldsymbol {x}})}{p _ {\theta} (\boldsymbol {z} | \boldsymbol {x} , \tilde {\boldsymbol {x}})} = \frac {p _ {\theta} (\boldsymbol {x} | \boldsymbol {z} , \tilde {\boldsymbol {x}}) p _ {\theta} (\boldsymbol {z})}{p _ {\theta} (\boldsymbol {z} | \boldsymbol {x} , \tilde {\boldsymbol {x}})}. \tag {14}
$$
We introduce a probabilistic encoder  $q_{\phi}(z|\boldsymbol{x})$  for approximating the intractable latent variable distribution. We can then derive the follows:
$$
\begin{array}{l} \log p _ {\boldsymbol {\theta}} (\boldsymbol {x} | \tilde {\boldsymbol {x}}) = \mathbb {E} _ {\boldsymbol {z} \sim q _ {\phi} (\boldsymbol {z} | \boldsymbol {x})} [ \log p _ {\boldsymbol {\theta}} (\boldsymbol {x} | \tilde {\boldsymbol {x}}) ] \\ = \mathbb {E} _ {\boldsymbol {z} \sim q _ {\phi} (\boldsymbol {z} | \boldsymbol {x})} \left[ \log \frac {p _ {\boldsymbol {\theta}} (\boldsymbol {x} | \boldsymbol {z} , \tilde {\boldsymbol {x}}) p _ {\boldsymbol {\theta}} (\boldsymbol {z})}{q _ {\phi} (\boldsymbol {z} | \boldsymbol {x})} \frac {q _ {\phi} (\boldsymbol {z} | \boldsymbol {x})}{p _ {\boldsymbol {\theta}} (\boldsymbol {z} | \boldsymbol {x} , \tilde {\boldsymbol {x}})} \right] \\ = \mathbb {E} _ {\boldsymbol {z} \sim q _ {\phi} (\boldsymbol {z} | \boldsymbol {x})} \left[ \log \frac {p _ {\boldsymbol {\theta}} (\boldsymbol {x} | \boldsymbol {z} , \tilde {\boldsymbol {x}}) p _ {\boldsymbol {\theta}} (\boldsymbol {z})}{q _ {\phi} (\boldsymbol {z} | \boldsymbol {x})} \right] + \mathbb {E} _ {\boldsymbol {z} \sim q _ {\phi} (\boldsymbol {z} | \boldsymbol {x})} \left[ \log \left(\frac {q _ {\phi} (\boldsymbol {z} | \boldsymbol {x})}{p _ {\boldsymbol {\theta}} (\boldsymbol {z} | \boldsymbol {x} , \tilde {\boldsymbol {x}})}\right) \right] \\ = - \mathcal {L} (\boldsymbol {\theta}, \phi | \boldsymbol {x}, \tilde {\boldsymbol {x}}) + D _ {\mathrm {K L}} \left[ q _ {\phi} (\boldsymbol {z} | \boldsymbol {x}) \| p _ {\boldsymbol {\theta}} (\boldsymbol {z} | \boldsymbol {x}, \tilde {\boldsymbol {x}}) \right] \\ \geq - \mathcal {L} (\boldsymbol {\theta}, \phi | \boldsymbol {x}, \tilde {\boldsymbol {x}}). \tag {15} \\ \end{array}
$$
In the formula, the term  $-\mathcal{L}(\theta, \phi | x, \tilde{x})$  is referred to as the evidence lower bound (ELBO), or the variational lower bound. It can be expressed as: