Towards Foundation Models for Mixed Integer Linear Programming
Authors: Sirui Li
Janardhan Kulkarni
Ishai Menache
Cathy Wu
Beibin Li
Venue: ICLR 2025 Poster (2025)
Keywords: Mixed Integer Linear Programming
Large Language Models
Foundation Models
Contrastive Learning
Graph Neural Networks
Primary area: optimization

Abstract: Mixed Integer Linear Programming (MILP) is essential for modeling complex decision-making problems but faces challenges in computational tractability and interpretability. Current deep learning approaches for MILP focus on specific problem classes and do not generalize to unseen classes. To address this shortcoming, we take a foundation model training approach, where we train a single deep learning model on a diverse set of MILP problems to generalize across problem classes. As existing datasets for MILP lack diversity and volume, we introduce MILP-Evolve, a novel LLM-based evolutionary framework that is capable of generating a large set of diverse MILP classes with an unlimited amount of instances. We study our methodology on three key learning tasks that capture diverse aspects of MILP: (1) integrality gap prediction, (2) learning to branch, and (3) a new task of aligning MILP instances with natural language descriptions. Our empirical results show that models trained on the data generated by MILP-Evolve achieve significant improvements on unseen problems, including MIPLIB benchmarks. Our work highlights the potential of moving towards a foundation model approach for MILP that can generalize to a broad range of MILP problem classes. Our code and data are publicly available at https://github.com/microsoft/OptiGuide.

Published as a conference paper at ICLR 2025
TOWARDS FOUNDATION MODELS FOR MIXED INTE- GER LINEAR PROGRAMMING
Sirui Li Janardhan Kulkarni", Ishai Menache", Cathy Beibin Li?
MIT {si ruil, cathywu] @mit.edu
2 Microsoff Research, (jakul, ishai, beibin. li h jeiccrosott.com
ABSTRACT
Mixed Integer Linear Programming (MILP) is essential for modeling complex decision decision-making problems but faces challenges in computational tractability and interpretability. Current deep learning approaches for MILP focus on specific problem classes and do not generalize to unseen classes. To address this shortcom- ing, we take foundation model training approach, where we train single deep learning model on diverse set MILP problems to generalize across problem classes. As existing datasets for MILP lack diversity and volume, we introduce MILP-Evolve, a novel LLM-based evolutionary framework that can generate a large set of diverse MILP classes with unlimited number of instances. We study our methodology on three key learning tasks that capture diverse aspects of MILP: (1) integrality gap prediction, (2) learning to branch, and (3) new task of aligning MILP instances with natural language descriptions. Our empirical results show that models trained on the data generated by MILP-Evolve achieve significant im- provements on unseen problems, including MIPLIB benchmarks. Our work high- lights the potential of moving towards foundation model approach for MILP that can generalize to broad range of MILP problem classes. Our code and data are publicly available at /ettps:/qqiten com/ wom.//icrooooffoptiiiiioo.c
INTRODUCTION
Mixed Integer Linear Programming (MILP) is a versatile mathematical framework widely used model complex decision-making problems across various domains, including healthcare (Eriskin et al. 2024), supply chain management (Kaya & Urek 2016), energy systems (Wouters et al. 2015). and finance (Mansini et al. 2015). Its ability to represent intricate combinatorial structures and constraints makes it an indispensable tool in both academic research and industry applications.
Despite its widespread applicability, MILP faces two fundamental limitations. First, tractability is major concern; solving MILP problems is computationally intensive and time-consuming, particu- larly for large-scale instances, due to the inherent NP-hardness. This complexity often results in long solve times, posing challenges for time-sensitive applications. Second, understanding the MILP for- mulation of an optimization problem requires significant expertise in mathematical modeling and optimization, which limits accessibility for non-experts.
To address these challenges, researchers have attempted to leverage machine learning (MI.) to en- hance MILP solvers, such learning heuristics to accelerate the solving process (Gasse al. 2019 Khalil et al. 2016). The underlying hypothesis for why ML may help in MILP is that in most real- world applications, the instances coming from unknown distributions, which, although hard to represent analytically, can be learned by deep neural networks. Despite this plausibility, the of the-art ML approaches have achieved limited generalizability due to difficulties in adapting learned models to unseen problems and handling distributional shifts. Most existing methods (Prouvost al. 2020 Gasse et al. 2022) focus on specific problem classes such as Set Cover, Capacitated Fa- cility Location, or Maximum Independent Set, and train class specific models. Unfortunately, these problem specific models fail to perform well on different or more complex MILP instances (Huang et al. 2024). For example, model trained to help solve Set Cover instances may not generalize
WWrk done during the author's internship at Microsoft Research

---

Published as a conference paper at ICLR 2025
100
SOTA
I
Ours
100
GIS
NF
80
50
60
Depth
Integrality Gap Pradicton (Cor))
Language Align- nen (Ace.)
Learingg Branch (Sal. %)
500
100
(b) Visualization classes (T-SNE of code embedding): or- ange points with annotated labels represent seed data, while blue points represent data generated by MILP-Evolve with hrightness corresponding to different depths.
(a) Comparison of performance across tasks for the SOTA baseline models and ours.
Figure 1: Comparison of baseline and our model's performance across tasks, and visualization of MILP-Evvlve generated MILP classes and seed MILP classes from prior work.
to instances of slightly modified problem formulations (c.g., more constrained versions) as well as other problem classes, such as Facility Location. This lack of generalization hinders practical adop- tion in real-world applications where problem characteristics vary widely. A concurrent work goes beyond the problem specific training approach (Huang et al. 2022), but the authors consider training a joint model on a small number of selected classes (five), which limits its general applicability.
The state of ML adoption for MILP P is in sharp contrast to areas like computer vision and NLP, where the fields have moved away from training problem or task specific models to general purpose foun- dation models trained on diverse and large-scale datasets that are capable of generalizing
wide range of tasks. There are several challenges towards building such general purpose foundation mod- els for MILP, the chief among them being the lack of diverse and large-scale training data. Reliance on limited datasets like MIPLIB (Gleixner et al. 2021) is insufficient to capture the vast diversity of MILP problems encountered in practice. More significantly, the distribution of the combinatorial structures of optimal solutions for MILP instances more complex than the distribution of images or language, challenging the efficacy and development of general purpose foundation models.
Our work is motivated by the following questions:
Do principles of foundation model training extend to the MILP modality? Can a single model trained on diverse MILP problem classes generalize unseen MILP classes?
The main contributions of this work are highlighted as follows.
A Foundation Model Approach for Efficient Multi-Class MILP Learning. This work is the first to propose a foundation model training approach for MILP learning and demonstrate that, a single model, trained on sufficiently diverse MILP problems, can effectively generalize to a variety of unseen MILP classes. We develop a comprehensive framework that integrates Large Language Models (LLMs) (Achiam al. 2023) Dubey et al., 2024 Brown al. 2020) for generating larger and richer training data with Graph Neural Networks (GNNs). which have proven effective in rep- resenting MILP instances (Gasse al. 2019 Scavuzzo al. 2022| ZZhng al., 2024a). Unlike prior work that trains GNNs on a limited set of MILP classes, we significantly extend the scope by learning joint model on broader range of MILP problem classes.
MILP-Evolve: A Scalable Framework for Generating Diverse MILP Data. As alluded ear- lier, unlike text and image modality. existing public MILP datasets lack diversity and volume. To address this limitation, we introduce MiLP-EEvolve, novel L.LM-based data generation pipeline, which leverages frontier models to generate diverse MILP classes and instances from few example seed classes. MILP-Evolve combines diverse prompting tailored to the MILP domain, along with parameter search and filtering, generate wide range of MILP classes resembling various real- world optimization scenarios. Figure 16 shows that MILP problems generated by our framework have rich diversity. Quantitatively, different MILP problem classes, much more than any publicly available dataset.
approach has enabled us to generate more than thousand
Comprehensive Framework Evaluation. We rigorously evaluate our framework across three chal- lenging learning tasks that test different facets of understanding and solving MILP instances in
2

---

Published as a conference paper at ICLR 2025
MILP Data Synthesis
Sead Classes
LL Based Code Evolution
N Evoived MILP Classes
Evolved MIL Insrances (M Closs)
Learning Tasks
Level- By--Lvell Evelve
Formst Seed Code
LL LLM-Prompting Typea
Neudy Evoloed Coda
Integrolity Gap Prediction
Jncter
Add 3 zert
Gressever Cockitia
Dert
Mas''s L ko.
Optinibation
New 22 Ralowen
Mutato 33 Parmoett
Leaminn to Branch
Delite 1
O Invisote Strong Branching
Filtering & Param, Search
Fext Ei Eircoda
LanguageeMMLL Contrastive Learning
Syntax
Paramnter Saurch
Solution
(Muti Proceos)
Filter
Filter
Figure 2: Overview: Left- SERt-MILP-Evooo pipeline. We design an evolutionary pipeline to leverage Large Language Model to generate a diverse set of MILP classes. Right-the learning tasks. We design three learning tasks capturing different aspects of MILPs for multi-class learning.
multi-class learning setting. First, we consider the problem of estimating the integrality gap (IG) of MILPs, defined as the difference between the optimal value of the MILP and its linear program- ming relaxation. Our second task, learning to branch, is sequential decision decision-making task aimed at accelerating MILP solvers by learning efficient branching strategies, a core technique for solving MIIPs. While previous works have explored similar tasks (Chen et al., 2023 Geng et al. 2023 Gasse et al., 2019; Zhang et al., (2024a), they primarily focused on training problem-specific models that could only generalize small set of problem classes. In contrast, we train single model that can solve each task across broad range of MILP classes and problems.
We further introduce a new learning task designed to improve the interpretability of MILP instances by mapping each MILP instance in mathematical form to one of several available natural language descriptions. This task tackles a key challenge MILP accessibility: many open-source MILP datasets and business scenarios provide only raw constraint and variable values, making them dif- ficult to interpret without domain expertise. To address this, we develop contrastive learning approach that aligns MILP instance embeddings with natural language descriptions, which signifi- cantly outperforms direct interpretation attempts by LLMs like GPT-4o. This task complements the tractability tasks by lowering the entry barrier for non-experts and potentially also aiding experts by deepening their understanding of the optimization problems.
Substantial Multi-Class Learning Gains with Broader Impacts. Our extensive experiments demonstrate that MILP-Evolve combined with our GNN-based learning framework highly ef- fective in improving multi-class MILP learning. Key results from the held-out test set include a 5.8x correlation improvement for Integrality Gap Prediction, a 1.92x accuracy improvement for MILP alignment, and a 1.42x improvement in the fraction of problems solved to optimum within the time limit for Learning to Branch (see Figure Tap. We further observe improved transfer learning to a separate MILP- --volvv test set based on unseen problem classes, as well as on the MIPLIB test set when pre-training on the enriched MILP-Evolve dataset. These results highlight the substantial per- formance gains of our models trained on diverse problem classes generated by MILP-Evolve. Our findings reveal key insights for advancing MILP learning: the diversity of training data has substan- rially greater impact on model performance than data quantity alone. These insights, along with our study's 's broader contributions, represent significant progress towards foundation models for MILPs.
FORMAL DESCRIPTIONS OF MILP AND LEARNING TASKS
2.1 MIXED INTERGER LINEAR PROGRAMMING (MILPS)
MILP involves optimization problems where some decision variables are constrained to be integers, and relationships among variables are linear in the form of constraints and objective function. A
3

---

Published as a conference paper at ICLR 2025
MILP problem is formulated as:
(I) where æ E IK" is the vector decision variables, c I2" the cost vector, E R and b E R define the linear constraints, and I (l,...,n? indicates the indices of variables required to be integer-valued. MILP is extensively used to model complex decision-making problems involving both discrete choices and continuous variables, capturing applications in supply chain optimization, scheduling, network design, and resource allocation (Duong & Bui, 2018 Floudas & Lin 2005) Rieck al.] 20T2). However, solving MILP problems is challenging due their NP-hardness, leading to computational intractability for large-scale instances.
FiLP = arg min(c=x: Ar S b, E EVj E
2.2
LEARNING TASKS
We give formal definitions of the three tasks considered in this work, each captures a different but interconnected aspect of understanding and solving MILPs.
2.2.1 INTEGRALITY GAP PREDICTION
Integrality gap (IG) quantifies how close the linear programming (LP) relaxation of a MILP instance is to its integer optimum. A smaller gap indicates that the LP relaxation is good approximation, while a larger gap suggests a more challenging problem requiring more computational resources. When the integrality gap is small, to obtain integral solutions, which is a foundational technique in the design of approximation algorithms (Raghavan & Tompson 1987 Williamson Shmoys 2011).
can the optimal solution to the LP relaxation and round
For each MILP instance we compute the integrality gap as: g² = [=(p(())={p(a)) where -(pp(=)
20 (x)x)) is the LP relaxation value at the root node, and (x))(x)) is the optimal solution value of the MILP (see Appendix 1 for details). Accurately predicting g (æ)) allows us estimate the diffi- culty of solving the MILP before actually solving it. When the integrality gap is small, a good ap- proximation of (xx) can quickly provide good estimate for the MILP's optimal value, as LPs can be efficiently solved in practice; it further suggests that MILP solvers may converge more quickly to optimal solutions, making a potential indicator of faster solve times for the corresponding MILPs.
2.2.2 LEARNING TO BRANCII
Branching decisions in the Branch-and-Bound (B&B) algorithm (Land & Doig 2010) significantly impact the efficiency of solving MILPs. Selecting the right variable to branch on can reduce the size of the search tree and, consequently, the computation time. Traditional strategies like strong branching (Applegate et al. 1995 Achterberg 2007) are effective but computationally expensive.
By learning branching policy through deep networks, the goal is to approximate the performance of strong branching at a small fraction of the computational cost. Specifically, at each B&B node (s, A(s)), where is the MILP representation the node and A(s) is the set of candidate variables to branch on, the learning task is to imitate the strong branching expert, which selects the action at E A(s). We minimize the cross-entropy loss N E(sar) log log fo(a") (s) of predicting the expert action, given the network fe(.) (see Appendix A.2.2 for details). Building on the work of (aasse al. 2019), this sequential decision-making task aims to enhance solver efficiency, making it more practical for large-scale or time- sensitive applications.
2.2.3 LANGUAGE-MILE CONTRASTIVE LEARNING
Understanding and interpreting MII.P instances, given by the numerical A, b, e values, inherently challenging due to their abstract and mathematical nature. To make MILP problems more accessible and to facilitate interaction with optimization problems as a new modality, we propose a contrastive learning task that aligns MILP instances with their corresponding natural language descriptions.
We design and employ a contrastive learning framework inspired by the Contrastive Language- Image Pretraining (CLIP) framework (Radford et al. (021). Specifically, we minimize symmetric cross-entropy loss over our dataset D, which encourages the model to associate each MILP instance with its corresponding text description and vice versa.
4

---

Published as a conference paper at ICLR 2025
Combinatorial Auction
Exclusive Auction
Auctions within Facilities
Add
Cross Over
Facility Allocation
Figure 3: An evolution chain for an auction problem consisting of two iterations. In the first iteration of the evolution, we add constraint. The second iteration involves crossover with the facility allocation problem. The details of code generated by LLM is in Appendix Figure
Formally, let D- {(M, Ti) denote a dataset comprising N pairs MILP instances M, and their corresponding textual descriptions T,. Our objective is learn embedding functions fas M - I2d and fr T - Rd that map MILP instances and text descriptions into a shared d-dimensional latent space. The goal is ensure that the embeddings of matching pairs (M, T,) are closer to each other than those of non-matching pairs, effectively capturing the semantic rela- tionships between MILPs and their descriptions. Details are provided in Appendix A.2.3
By optimizing the contrastive loss, the model learns to project MILP instances and their textual de- scriptions into common embedding space where semantically related pairs close together. This learning task can serve an important stepping-stons towards the more challenging task of directly generating natural language descriptions from MILP instances. This in similar spirit to how the CLIP framework (Radford et al. (021) paved the way for text to image generative models such as DALL-E (Ramesh et al. 2021) by training decoder model on CLIP embeddings.
Enhancing MILP applicability. The three learning tasks are complementary and collectively es- sential for enhancing the applicability of MILP through understanding, predicting, and accelerat- ing. In particular, the Language-MILP task helps understand the structure and properties of MILP instances, aiding non-experts in problem comprehension and may further assist experts by deepen- ing their understanding of the problems; the Integrality Gap Prediction task focuses analyzing solution properties of the MILP instance, potentially allowing instances with tight gaps to be solved via LP relaxation, coupled with rounding algorithms, without fully solving the MILP; the learning to Branch task enhances MILP solving efficiency through more effective branching, which can have huge time and cost savings in industrial applications.
MILP-EVOLVE FOR GENERATING DIVERSE MILP CLASSES
The diversity of MILP problems a crucial aspect for using a foundation model approach for MIIPP Existing studies, however, often focus small number of problem classes, limiting the model's ability to generalize. We propose MILP-Evolve, an LLM-based pipeline designed to generate a di- verse set of MILP classes. By starting from small of seed classes and systematically generating new ones, we aim to capture a broader spectrum of optimization problems.
Class representation. We represent each class of MILP problem with modularized optimization code script, comprising three primary functions: (1) Data, responsible for generating the necessary data to construct the objective and constraint coefficients, often from uniform distributions or com- mon graph structures; (2) Optimization Modeling, which utilizes this data to define the decision variables, constraints, and the objective function of the optimization problem; and (3) Parameters, which outlines the problem-specific parameters for the data distribution, such the number of nodes or graph density in graph-related problems.
Seed classes. We curate a set of eight MILP classes commonly used in prior literature (see Ap- pendix A.I.I and use them as the starting point to generate a more diverse set of MILP classes.
MILD-Evolve LLM-generrted classes. Generating diverse MILP classes is non-trivial due to potential pitfalls such as generating incorrect code, infeasible instances, or lack of diversity. To overcome these challenges, our pipeline, as illustrated in Fig. (left), consists of two key steps.
1. Class Generation. Given the seed class representations, we use LLMs to generate new MILP classes by applying several transformations. These transformations are represented by 10 chain-of- thought (Wei et al.) 2022) prompt operators, which are described in detail in Appendix A.1.2 At each evoive iteration, M-L-P-Evolve selects random subset of these operators and invokes them
5

---

Published as a conference paper at ICLR 2025
quentially, where each prompt operator is applied to the previously evolved MILP class. We add each MILP class generated by this process to our MIL.P-Evolve dataset. An example of a -ooiieeatioon evolution is shown in Figure E Intuitively, MILP classes that are generated in early iterations of the framework are closer to the seed class, whereas those towards the end are further away.
At high level, the prompt operators include adding or deleting constraints, variables, the asso- ciated data; crossing over classes; mutaring constraints and/or objectives; and creating entirely new classes. Each prompt consists of three main modules that the LLM should follow step-by-step: (1) summarize the given MILP class, (2) describe how to modify the MILP class based on the specific operator type, (3) generate the new MILP class that follows the class representation.
To encourage diversity and realism of the generated classes, we guide the LLM to integrate MILP problems into real-world contexts and generate classes tailored to specific industry needs. This requirement is achieved by either prompting the LLM to base the MILP on real-world applications or directly providing realistic topies, such as "pharmaceutical company planning cross-country vaccine distribution with storage, transport, and demand constraints," " when generating the MILP class.
We further prompt the LLM to apply commonly used and advanced MILP formulation techniques, such as Big M (Wolsey) 2020), Special Order Sets (Beale & Forrest 1976). and Symmetry Break- ing (Margot 2009), to make the generated instances technically challenging. By doing so, we guide the LLMs to produce wide range of MILP formulations reflecting real-world scenarios.
Filtering. After generating new MILP classes, we perform filtering and parameter adjustment to ensure the usefulness and mathematical feasibility of the generated instances. Our modular class representation allows us to easily identify and adjust key parameters to achieve desired properties such as feasibility, appropriate solve times, and reasonable problem sizes (see Appendix A.1.44)
Outcome. Using our pipeline, with GPT-4o as the LLM, we create a dataset of MILP classes significantly more diverse than those used in prior work; Figure 1b Examples of the generated classes and implementtionn details of the MILP-Evolve pipeline can be found in Appendix A.
4 TRAINING PIPELINE
Our training pipeline (Fig. consists of two main components: a) The MILP-Evolve system de- scribed above to obtain diverse dataset; b) The learning architecture, which combines GCNs with an attention module, to effectively learn in a multi-class setting. We provide more details below.
Input. We represent each MILP instance as bipartite graph connecting variable nodes V and constraint nodes C (Gasse et al. 2019), capturing the structural relationships within the MILP. This graph serves input to our neural network architecture. For Language-MIIL Contrastive Learning, the text description inputs are embedded via WV-Embed-vl (Lee et al. 2024). an open-source text embedding model based on Mistral 7B (Jiang et al. 2023).
Architecture. For Integrality Gap Prediction and the MILP encoder for Language- MILP Contrastive Learning, our architecture consists of the following main components: 1) We lutional Network (GCN) (Kipf & Welling, 2017 Gasse et al., 2019 Paulus et al. 2022 Scavuzzo ct al. 2024) to embed the variable and constraint nodes, allowing information to flow between them and capturing local structural patterns. 2) To capture global dependencies while reducing computa- tional overhead, we use the attention mechanism (Vaswani 2017) over a sub-sampled set of variable and constraint nodes. 3) We include three additional nodes for the attention: two representing the aggregated variable and constraint node embeddings (obtained via mean pooling) and a summary node used to extract a global representation of the MILP instance. 4) The attention module outputs an updated embedding of the summary node, which is then passed through final linear layer to produce the final output. Empirically we find that incorporating the attention layers improves per- formance, especially for transfer learning to the MIPLIB dataset (Sec. 5.3). This result is expected as attention mechanisms allow for better global understanding of MILP instances.
Graph Convo-
For the learning to branch task, we directly use the variable embeddings produced by the GCN to predict the variables selected for branching, building upon similar architecture employed in Gasse ct al.] (2019). As the model needs to be invoked for each Branch-and und-Bound node, we omit the attention layer to reduce the computational overhead.

---

Published as a conference paper at ICLR 2025
Table 1: Out-Domain Performance on the MILP-Evolve held-out test set. Ours outperforms the baseline methods in all three learning tasks.
(a) Integrality Gap Prediction.
(b) Language-NMe MIL. Contrastive Learning.
Deviation (
Corr: (f)
4- W-y Acc. (f)
10-Way Acc. (1)
Mean 33.07%
0.00
Seed
37.21%
18.73%
Seed
32.96%
0.10
Seed Param.
39.68%
20.35%
Seed Param.
32.77%
0.07
Seed VAE
33.67%
15.58%
Seed VAE
30.27%
0.26
Ours Attn.
70.41%
52.76%
Ours Attn.
20.82%
0.57
Ours
70.54 %
54.17% %
Ours
20.14%
0.58
(c) Learning Branch. For each method, we report the proportion of instances solved to optimal within 200s time limit (including network overhead); among those instances, we report the 1-shifted geometric mean of time improvement over SCIP Default including and excluding (Ex.) the network overhead (See Appendix A.2.2)
Time Improv. (1)
Time Improv. (Ex) (1)
% Instances Solved to Optimal (1)
Sced (GCN)
-30.65%
-16.60%
49.59%
Seed + Param. (GCN)
-3.70%
5.89%
64.52%
Seed VAE (GCN)
-8%
-13500%
49.06 99.06%
Ours (GCN)
15.49%
31.02%
70.90% %
EXPERIMENTS
To evaluate the effectiveness and generalization capabilities of our approach, we conduct compre- hensive experiments focusing out-of-domain settings where models are tested on MILP classes unseen during training. For each learning task, we train a single model on the diverse MILP classes generated by Evolve. The evaluation is performed two fronts: first, held-out set of MILP classes from MILP-Evolve, and second, through transfer learning on instances from MIPLIB, widely recognized MILP benchmark dataset.
5.1 EXPERIMENTAL SETUP
More than thousand MILP problem classes are used, with the exact number varying by task. For IG prediction, we excluded instances where the optimal solution was not found within a time limit of 200s. For the learning to branch task, our training data consists collecting strong branching expert examples that are obtained by solving MILP instances up to the same time limit of 200s (see details in Appendix 1.5.)
ffate-of-the-atee Baselines. To assess the effectiveness of training the diverse MILP classes generated by MILP-Evolve, we compare our model against several methods. We collected a Seed dataset containing all problem classes used in recent state-of-the-ar (SOTA) studies (Gasse al., 2019 [Scavuzzo et al.) 2022 Labassi et al.) [2022), totaling 16 sets with two parameters for each of the eight existing classes. We also create a competitive baseline, Seed Param., which expands the Seed dataset by including 89 additional parameters identified through a parameter search procedure similar that used in MILP-Evolve. The third approach, Seed VAE (Guo al. 2024), employs Variational Autoencoder (VAE)-based instance generation method to augment the seed class in- stances, which further incorporates constraint grouping to improve upon the seminal work of (Geng et al. 2023) Details of these baselines can also be found in Appendix A.1.5.. This comparison emphasizes the distinction between our MILPP class augmentation approach and existing instance augmentation methods. For integrality gap prediction, we also include Mean bascline, which, for all MILP instances, predicts the same constant value given by the mean of all the training set labels. For Language-MILP Alignment, we include further comparison with GPT-4o in Appendix A.3.7
For simplicity, we refer to the model trained with data generated from MILP-EvoNs as Ours. Addi- tionally, to evaluate the impact of our architectural choices, include comparison with model trained without the attention layer, referred to as Ours Attn. Details on architecture, training, met- rics, and additional ablation results are provided in Appendices A.2 and A.3

---

Published as a conference paper at ICLR 2025
# Classes
200
400
600
Branching Acc, (%)
35
80
Deviation (%)
80
30
Acc. (%)
60
60
40
40
20
20
20
10,000
20, .00000
30, 000
40 .000
10
10
103
10
10a
10a
# Classes
Classes
Instances
4-way Acc. (b) Learning Branch. We keep the total number training instances constant and vary the number of training MILP classes. We plot the prediction accuracy with respect to strong branch- ing expert.
Acc.
Top-5 Ace.
10-way Ace.
(a) Integrality Gap Prediction. Purple: Different number of in- stances per class, same number of classes. Gray: Different num- ber of classes, same number of instances per class.
(c) Language-MILP Contrastive learning. We control the to- tal number of training instances to be the same (1000 instances) across experiments and change the number training MILP classes.
Figure 4: Effect of Scaling Training Classes on Test Performance: Increasing training classes im- proves test performance across all learning tasks.
Table Transfer Learning Results on a MILP-E-vee Test Set on Six Unseen Seed Classes. Initial- izing with the Ours pre-trained model yields the best performance (see Appendix A.3.3 for details).
Integrality Gap
Learning to Branch
Language-MIIL
Deviation (I)
Correlation (1)
Acc. (1)
Top Acc. (1)
Way Acc. (1)
10 Way Acc. (1)
Train From Seratch
21.41%
0.65
28.93%
69.70%
72.37%
46.50%
Seed
21.25 21.25%
0.65
23.11%
56.82%
72.20%
42.45%
Seed Param.
25.61%
0.52
27.87%
68.32%
75.17%
42.66%
Seed VAE
23 23.40%
0.58
25.25%
60.81%
72.90%
44.61%
Ours
17.98%
0.68
30.71 %
70.33%
77.62%
53.99%
Metrics. For the integrality gap task, we compute the mean absolute error (deviation) and Pearson approach, we measure the proportion of
correlation coefficient. To evaluate the learning-to-brant instances optimally solved within the time limit, as well as the percentage improvement in solve time compared the default SCIP solver. The time improvement presented both with and without accounting for the GPU time required by the deep learning model. In assessing the contrastive learning task, we report both 4-way and 10-way accuracy, where the model must identify the correct text description corresponding given MILP instance from set of 4 or 10 options, respectively.
5.2 MILP-Evolve TEST SETS
We partition the set of MILP problem classes generated by MILP-Evolve into roughly a 7:1:2 split for training, validation, and test subsets which we denote by B B'oolie and Bbvolve For each learning task, we train our model on instances from Arain Bvvolv referred to as Ours, and evaluate its
Xal
fest
performance on the test instances X.Fvollee This setup helps us to assess the impact of class diversity and instance variety on the model performance.
Table I showcases the performance on the MILP-Evolve held-out test set across the three learning tasks. The Ours approach consistently outperforms baselines trained on fewer classes, highlight- ing the critical role of class diversity in enhancing model generalization. Furthermore, the inclusion of the attention mechanism significantly boosts performance, underscoring its importance cap- turing global information within MILP instances. The (Seed + Param) baseline outperformed the dearning-base data augmentation method (Seed + VAE) in contrastive learning and branching, sug- gesting that the VAE approach might not be readily extensible. Moreover, the learning to branch task remains particularly challenging in a multi-class setting, suggesting an area for future improvement and research (see Appendix A.3.2 for details).
8

---

Published as a conference paper at ICLR 2025
Table 3: Transfer Learning Results on MIPLIB. Initializing with the Ours pre-trained model yields the best performance on the MIPLIB test set.
Integrality Gap Prediction
Language-MILP Contrastive Learning
Deviation (!) Correlation (1)
Way Ace. (1)
10 Way Ace. (*)
Train From Scratch
28.21%
0.43
73.28%
65.29%
Seed
26.97%
0.44
79.69%
72.71%
Seed Param.
23.30%
0.54
76.41 %
71.74%
Seed VAE
24.76%
0.50
79.92%
70.99%
Ours
21.56%
0.59
82.08% %
75.57%
10-way Aee. (%
Deviation (%)
Seed
Scratch
Ours
Seed
Seratch
Ours
50
-10
60
330
41
20
20
0
200
400
600
800
1,000
200
400
600
800
1,000
Num. Gradient Steps
Epochs
(a) Integrality Gap Prediction.
(b) Language-MILP Contrastive Learning
Figure 5: Convergence Rate on the MIPLIB Dataset. We plot the test performance for models at different stages of the training process for transfer learning to MIPLIB. Initializing with Ours pre- trained model leads to significantly faster convergence than other baselines.
We also find that scaling up the number of training classes is much more important than increasing the number of training instances, emphasizing the importance of diverse MILP classes for training. As illustrated in Figure 4a reducing the number of classes while maintaining the same total number of instances significantly affects performance, whereas reducing the number of instances per class has minimal effect. Similarly, we observe in Figures 4b and 4c an upward trend in performance as the number classes increases, while keeping the total number of training instances constant. These results reinforce the value of MILP-Evole in generating a diverse set of MILP classes.
To further ensure that our held-our test set is representative and can generalize to unseen datasets, We collect another test set of 50 MILP classes, by rumning MILP-Evolve six unseen Seed classes different from those used in Table ] We split the dataset into &tram and X fine-tune the pretrained models from Table on Rrew and evaluate the performance on X In Table 2 see that initializing with Ours enables the best transfer learning performance on this new test set.
5.3 MIPLIB TRANSFER LEARNING
Finally, we evaluate the transfer learning performance of our model on MIPLIB GGeiinnr et al., 2021], commonly used heterogeneous benchmark dataset which is never used in our pretraining process. We filter MIPLIB to include instances with known optimal solution (for gap prediction) or with meaningful description (for contrastive learning), and split them into training and test sets, XMIPLBB We fine-tune pretrained model from & on MMILII and subsequently MMPLIIB (see Appendix 1. details).
Auran and Aes
evaluate its performance on Aest
Table presents the results of transfer learning on the MIPLIB dataset for the IG and CL tasks. First, we observe that pretraining models on diverse dataset is beneficial compared to training from scratch (first row the table). Further, our model trained on data from MIL-P-vvolve achieves superior performance compared to all basclines. Instance augmentation methods like VAE and parameter search provide some incremental gains, but are outperformed by our approach. Figure further illustrates the convergence behavior during fine-tuning on MIPLIB. Models pretrained with MILP-Evolve not only converge faster, but also achieve better final performance after fine-tuning.
We hypothesize that the enhanced performance during fine-tuning stems from the broad diversity of MILP classes generated by MILP-Evolve which captures a wide range of problem distributions, constraint types, and variable interactions (see Fig. 11 Appendix A.3. for details). This extensive

---

Published as a conference paper at ICLR 2025
coverage enables the model to adapt more effectively to the varied structures present in MIPLIB, compared to models trained on more homogeneous or limited datasets.
We omit learning to branch task on MIPLIB dataset because find that a vast majority of MIPLIB instances take either too little or too much time to solve; in fact, only 13 instances can be solved to optimality with a solve time between 20s and 300s, making the evaluation set too small. We believe that more comprehensive pre- treetrainnng with more compute are critical for learning to branch in datasets similar to MIPLIB; this is an interesting direction for future work.
RELATED WORK
Machine Learning for MILP. There has been a surge of research using machine learning techniques
to improve MILP solvers. Studies cover various aspects of MILP solving, including presolving (Liu
et al.
2024a). variable selection for branching (Zhang et al.
2024a Huang et al. 2024]
Scavuzzo
itt al.
2022; Gupta et al., 2020), node selection for exploration (Labassi et al. 2022 Song et al.
2018
He et al., 2014), generating cutting planes (Wang et al. 2023 Paulus et al. [022 Tang al.
2020,
configuring solver parameters (Li et al. 2023b Balcan et al. 202 Xu et al. 2011). and
scheduling primal heuristics (Chmiela al. [02 Hendel et al. 2019 Khali al. (017))
Despite these advancements, a significant challenge remains in the generalizability of learned mod- els across different tasks and problem classes. Most existing ML methods for MILP focus specific problem classes (Prouvost et al. [2200] Gasse et al. (0022) and struggle to generalize to unseen types of problems. This limitation hinders practical adoption, real-world applications often involve wide variety of MILP formulations.
Large Language Models for MILP. Recent research has explored prompting LLMs for MILP and combinatorial optimization tasks, such as optimization modeling (Ahmadi'Tesnnin et al.. 2024).
(024).. and automatic heuristic what-if analysis (Li Liettl al.) 2023a), infeasibility diagnosis (Chen et al.
design (Liu et al. [2024b] Romera-Pareeee
et al.] 2024 Yang et al. 2024). The majority of these
works are data-agnostic, usually taking code or human language as input and performing analytical
tasks without accessing the critical A, b, matrices. While many studies have aligned images, voices, code, and genomics modalities for LLMs (Liu al. 2024c [Barrult al. al.] [023 Roziere et al. 2023
Abdine et al. 2024), few have integrated the MILP modality with text.
MILP Datasets. For common benchmark MILP classes (Prouvost et al. 2020), the objective and
constraint coefficients are typically generated randomly within specified ranges, often represent- ing weights or budgets of variables. For graph-based MILPs (c.g., Set Cover), the constraints and variables b, depend the underlying graph structure and generated from distributions like
Erdös-Rényi (Erdös & Rényi 1959) or Barabási-Albert (Albert & Barabási 2002).
Recent studies use heuristics (Drugan 2013 Bowly| 2019). Variational Autoencoders Guo al., 2024 Geng et al., 2023). or Diffusion Models (Zhang et al. 2024b) to generate MILP instances from given set. These works focus on instance-based MILP generation: given limited number of instances within a specific MIIP class (e.g., Set Cover), they generate similar instances to augment the dataset, rather than more diverse ones. In contrast, aim to generate diverse problem classes, including more complex or entirely different problems not found in existing public datasets.
CONCLUSION
This paper takes a first step towards a foundation model training approach for MILP. We address three key learning tasks and train single model for each task to generalize across MILP problem classes. We introduce MIll-Evvolve a general MIIP class generation method that leverages LLMS to enable larger, richer, and more diverse datasets for training. Our experiments demonstrate that our framework, trained with SIL.--Evolve-eeeeee
data, significantly outperforms previous works.
While we advanced the state-of-the-arr by using a single model across MILP classes instead of multiple class-specific models, we acknowledge that this work still trains separate models for each learning task. An important future direction is to unify these tasks into single "foundation" model which would potentially extend to more tasks, such as generating cutting planes, learning solver parameters, and formulating reliable optimization problems from text descriptions.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGEMENT
The authors would like to thank Luke Marshall, Konstantina Mellou, Marco Molinaro, and Yining Ma for insightful discussions. The authors further thank the anonymous reviewers for their valuable feedback.
REFERENCES
Hadi Abdine, Michail Chatzianastasis, Costas Bouyioukos, and Michalis Vazirgiannis. Prot2text: Multimodal protein's 's function generation with gnns and transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 10757-10765, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint ar arXiv:2303.88774 2023.
Tobias Achterberg. Constraint integer programming. Phl) thesis, 2007.
Ali AhmadiTeshnizi, Wenzhi Gao, and Madeleine Udell. Optimus: Scalable optimization modeling with (mi) Ip solvers and large language models. arXiv preprint arXiv:2402.1017. 2024.
Réka Albert and Alsert-LászIó Barabási. Statistical mechanics of complex networks. Reviews of modern physics, 74(1):47, 2002.
David Applegate, Robert Bixby, Vašck Chvátal, and William Cook. Finding cuts in the TSP (A preliminary report), volume 95. Citeseer, 1995.
Egon Balas and Andrew Ho. Set covering algorithms using cutting planes, heuristics, and subgradi- ent optimization: computational study. Combinatorial Optimization, pp. 37-60, 1980.
Maria-Florina Balcan, Dan DeBlasio, Travis Dick, Carl Kingsford, Tuomas Sandholm, and Bllen Vitercik. How much data is sufficient to learn high-performing algorithms? generalization guar- antees for data-driven algorithm design. In Proceedings of the 53rd Annual ACM SIGACT Sym- posium on Theory of Computing, pp. 919-932, 2021.
Loic Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. Seamlessm4-- massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308. 11596, 2023.
EML Beale and John Forrest. Global optimization using special ordered sets. Mathematical Programming, 10:52-69, 1976.
Ramón Béjar, Alba Cabiscol, Felip Manyå, and Jordi Planes. Generating hard instances for maxsat. In 2009 39th International Symposium on Multiple-Vaie Valued Logic, pp. 191-195. IEEE, 2009.
David Bergman, Andre A Cire, Willem-Jan Van Hoeve, and John Hooker. Decision diagrams for optimization, volume 1. Springer, 2016.
Ksenia Bestuzheva, Mathieu Besançon, Wei-Kun Chen, Antonia Chmiela, Tim Donkiewicz, Jasper van Doormmalen, Leon Eifler. Oliver Gaul, Gerald Gamrath, Ambros Gleixner, et al. The scip optimization suite 8.0. arXiv preprint arXiv:2112.08872 2021.
Simon Bowly. Stress testing mixed integer programming solvers through new test instance genera- tion methods. PhD thesis, University of Melbourne, Parkville, Victoria, Australia, 2019,
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems,

---

Published as a conference paper at ICLR 2025
volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL http:://prooeeeepnng....//////oe neurips.cc/saper /ttp////////////////555005525510 142f64a-Papex.pde
Hao Chen, Gonzalo E Constante-Floreee and Can Li. Diagnosing infeasible optimization problems using large language models. INFOR: Information Systems and Operational Research, pp. 1-15, 2024.
W-H Chen and J-M Thizy, Analysis relaxations for the multi-item capacitated lot-sizing problem. Annals of operations Research, 26( 26(1):29-72, 1990.
Ziang Chen, Jialin Liu, Xinshang Wang. and Wotao Yin. On representing mixed-integer linear programs by graph neural networks. In The Eleventh International Conference on Learning Rep- resentations, 2023. URL
net http::/www.......ooooooooooccce.
Antonia Chmiela, Elias Khalil, Ambros Gleixner, Andrea Lodi, and Sebastian Pokutta. Learning to schedule heuristics in branch and bound. Advances in Neural Information Processing Systems, 02224235-24244 2021.
Marco Colombi, Renata Mansini, and Martin Savelsbergh. The generalized independent set prob- lem: Polyhedral analysis and solution approaches. European Journal of Operational Research, 260(1):41-55, 2017.
Gérard Cornuéjols, Ranjani Sridharan, and Jean-Michel Thizy. A comparison of heuristics and relaxations for the capacitated plant location problem. European journal of operational research, 50(3):280-277, 1991.
Mädälina M Drugan. Instance generator for the quadratic assignment problem with additively de- composable cost function. In 2013 IEEE Congress on Evolutionary Computation, pp. 2086-2093. IEEE, 2013.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Vo Hung Duong and Nguyen Hung Bui. A mixed-integer linear formulation for capacitated facility location problem in supply chain network design. International Journal of Operational Research, 33(1):32-54, 2018.
Paul Erdös and Alfréd Rényi. On random graphs i. Publicationes Mathematicae Debrecen, 6(290- 297):18, 1959.
Levent Eriskin, Mumtaz Karatas, and Yu-Jun Zheng. A robust multi-objective model for healtheare resource management and location planning during pandemics. Annals of Operations Research, 131(3):4471-15188 2024.
Christodoulos A Floudas and Xiaoxia Lin. Mixed integer linear programming in process scheduling: Modeling, algorithms, and applications. Annals of Operations Research, 139:131-11 2005.
Maxime Gasse, Didier Chételat, Nicola Ferroni, Laurent Charlin, and Andrea Lodi. Exact combi- natorial optimization with graph convolutional neural networks. Advances in neural information processing systems, 32, 2019.
Maxime Gasse, Simon Bowly, Quentin Cappart, Jonas Charfreitag, Laurent Charlin, Didier Chételat, Antonia Chmiela, Justin Dumouchelle, Ambros Gleixner, Aleksandr M Kazachkov, et al. The machine learning for combinatorial optimization competition (ml4co): Results and insights. In NeurlPS 2021 Competitions and Demonstrutions Track, pp. 220-231. PMLR, 2022.
Zijie Geng, Xijun Li, Jie Wang, Xiao Li. Yongdong Zhang, and Feng Wu. A deep instance genera- tive framework for milp solvers under limited data availability. Advances in Neural Information Processing Systems, 36:26025-26047, 2023.
12

---

Published as a conference paper at ICLR 2025
Ambros Gleixner, Gregor Hendel, Gerald Gamrath, Tobias Achterberg, Michael Bastubbe, Timo Berthold, Philipp Christophel, Kati Jarek, Thorsten Koch, Jeff Linderoth, et al. Miplib 2017: data- driven compilation of the 6th mixed-integer programming library. Mathematical Programming Computation, 13(3):443-49)) 4900, 2021.
Ziao Guo, Yang Li, Chang Liu, Wenli Ouyang, and Junchi Yan. Acm-milp: Adaptive constraint modification via grouping and selection for hardness-preserving milp instance generation. In Forty-first International Conference Machine Learning. 2024.
Prateek Gupta, Maxime Gasse, Elias Khalil, Pawan Mudigonda, Andrea Lodi, and Yoshua Bengio. Hybrid models for learning to branch. Advances in neural information processing systems, 33: 18087-18097, 2020.
Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023. URL http::tps///// //wp: www. gu robi coom
He He, Hal Daume III, and Jason M Eisner. Learning to search in branch and bound algorithms.
Advances in neural information processing systems, 27, 2014.
Gregor Hendel, Matthias Miltenberger, and Jakob Witzig. Adaptive algorithmic behavior for solving mixed integer programs using bandit algorithms. In Operations Research Proceedings 2018: Se- lected Papers of the Annual International Conference of the German Operations Research Society (GOR), Brussels, Belgium, September 12-14, 2018, pp. 513-519. Springer, 2019.
Mike Hewitt, George Nemhauser, and Martin WP Savelsbergh. Combining exact and heuristic approaches for the capacitated fixed-charge network flow problem. INFORMS Journal on Com- puting, 22(2):314-3555 2010.
Weimin Huang, Taoan Huang, Aaron M Ferber, and Bistra Dilkina. Distributional miplib: a multi- domain library for advancing ml-guided milp methods. arXiv preprint arXiv:2406.0666 06954, 2024.
Peter Huber. Robust estimation of a location parameter. Breakthroughs statistics: Methodol- ogy and distribution, pp. 492-518. Springer, 1992.
Tommy R Jensen and Bjarne Toft. Graph coloring problems. John Wiley & Sons, 2011.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 2023.
Onur Kaya and Busra Urek. A mixed integer nonlinear programming model and heuristic solu- tions for location, inventory and pricing decisions in a closed loop supply chain. Computers & Operations Research, 65:93-1088 2016.
Elias Khalil, Pierre Le Bodic, Le Song. George Nemhauser, and Bistra Dilkina. Learning branch in mixed integer programming. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Elias B Khalil, Bistra Dilkina, George L Nemhauser, Shabbir Ahmed, and Yufen Shao. Learning to run heuristics in tree search. In Ijcai, pp. 659-666, 2017.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations, 2017.
Abdel Ghani Labassi, Didier Chételat, and Andrea Lodi. Learning to compare nodes in branch and bound with graph neural networks. arXiv preprint arXiv:2210.1668 2022.
Ailsa H Land and Alison G Doig. An automatic method for solving discrete programming problems. Springer, 2010.
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi. Bryan Catan- zaro, and Wei Ping. Ny-embed: Improved techniques for training llms as generalist embedding models. arXiv preprint arXiv:2405.I 7428, 2024.
13

---

Published as a conference paper at ICLR 2025
Kevin Leyton-Browyy Mark Pearson, and Yoav Shoham. Towards a universal test suite for combi- natorial auction algorithms. In Proceedings of the 2nd ACM conference on Electronic commerce, pp. 66-76, 2000.
Beibin Li, Konstantina Mellou, Bo Zhang, Jeevan Pathuri, and Ishai Menache. Large language models for supply chain optimization. arXiv preprint arXiv:2307.038% 2023a.
Sirui Li, Wenbin Ouyang, Max B. Paulus, and Cathy Wu. Learning to configure separators in branch-and-cut. In Thirty-seventh Conference on Neural Information Processing Systems, 2023b. URL
httpilewwweeeeteeeeeeeeeeeeeeeee forum i gx5xXV085p
Chang Liu, Zhichen Dong, Haobo Ma, Weilin Luo, Xijun Li, Bowen Pang, Jia Zeng, and Junchi Yan. L2p-mip: Learning to presolve for mixed integer programming, In The Twelfth International Conference on Learning Representations, 2024a.
Fei Liu, Tong Xialiang, Mingxuan Yuan, Xi Lin, Fu Luo, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang. Evolution of heuristics: Towards efficient automatic algorithm design using large language model. In Forty-first International Conference on Machine Learning. 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024c.
Renata Mansini, WL. odzimierz Ogryezak, M Grazia Speranza, and EURO: The Association of Eu- ropean Operational Research Societies. Linear and mixed integer programming for portfolio optimization, volume 21. Springer, 2015.
François Margot. Symmetry in integer linear programming. 50 Years of Integer Programming 1958- 2008: From the Early Years the state-of-the-Are pp. 647-686, 2009.
Max B Paulus, Giulia Zarpellon, Andreas Krause, Laurent Charlin, and Chris Maddison. Learning to cut by looking ahead: Cutting plane selection via imitation learning. In International conference on machine learning, pp. 17584-17600. PMLR, 2022.
David Pisinger. An exact algorithm for large multiple knapsack problems. European Journal of Operational Research, 114(3):528-544 1999.
Antoine Prouvost, Justin Dumouchelle, Lara Scavuzzo, Maxime Gasse, Didier Chételat, and Andrea Lodi. Ecole: A gym-like library for machine learning in combinatorial optimization solvers. In Learming Meets Combinatorial Algorithms at NeurlPS2020, 2020. URL htttps:/
ew. wtw.ee/e/oorummmndn1.e.oe..eeaee
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Prabhakar Raghavan and Clark D Tompson. Randomized rounding: a technique for provably good algorithms and algorithmic proofs. Combinatorica, 7(4):365-33) 374, 1987.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821-8831. Pmlr, 2021.
Julia Rieck, Juergen Zimmermann, and Thorsten Gather. Mixed-integer linear programming for resource leveling problems. European Journal of Operational Research, 221(1):27-37, 2012.
Bernardino Romera-Paredes, Mohammadanin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468 475, 2024.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi. Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code laaa:: Open foundation models for code. arXiv preprint 2308v:2308.1 /2950, 2023.
14

---

Published as a conference paper at ICLR 2025
Lara Scavuzzo, Feng Chen, Didier Chételat, Maxime Gasse, Andrea Lodi, Neil Yorke-Smith, and Karen Aardal. Learning to branch with tree mdps. Advances in Neural Information Processing Systems, 35:18514-18522 2022.
Lara Scavuzzo, Karen Aardal, Andrea Lodi, and Neil Yorke-Smith. Machine learning augmented branch and bound for mixed integer linear programming. Mathematical Programming, pp. 1-44, 2024.
Jialin Song, Ravi Lanka, Albert Zhao, Aadyot Bhatnagar, Yisong Yue, and Masahiro Ono. Learning to search via retrospective imitation. arXiv preprint arXiv: 18X::0884 2018.
Yunhao Tang, Shipra Agrawal, and Yuri Faenza. Reinforcement learning for integer programming: Learning to cut. In International conference on machine learning, pp- 9367-9376. PMLR, 2020.
JA Tomlin and IS Welch. Mathematical programming systems. Handbooks in Operations Research and Management Science, 3:561-601, 1992.
Vaswani. Attention all you need. Advances Neural Information Processing Systems, 2017.
Zhihai Wang, Xijun Li, Jie Wang, Yufei Kuang, Mingxuan Yuan, Jia Zeng, Yongdong Zhang, and Feng Wu. Learning cut selection for mixed-integee linear programming via hierarchical sequence model. In The Eleventh International Conference on Learning Representations, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
H Paul Williams. Model building in mathematical programming. John Wiley & Sons, 2013.
David P Williamson and David B Shmoys. The design of approximation algorithms. Cambridge university press, 2011.
Laurence A Wolsey, Integer programming. John Wiley & Sons, 2020.
Carmen Wouters, Eric S Fraga, and Adrian M James. An energy integrated, multi-microgrid, milp (mixed-integee linear programming) approach for residential distributed energy system planning- south australian case-study. Energy, 85:30 44, 2015.
Hegen Xiong, Shuangyuan Shi, Danni Ren, and Jinjin Hu. A survey of job shop scheduling problem: The types and models. Computers & Operations Research, 142: 105731, 2022.
Lin Xu, Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Hydra-mip: Automated algorithm configuration and selection for mixed integer programming. In RCRA workshop on experimental evaluation of algorithms for solving problems with combinatorial explosion at the international joint conference on artificial intelligence (IJCAI), pp. 16-30, 2011.
Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers. In The Twelfth International Conference on Learning Representations, 2024. /ttps::////opeeeeeee
net / forum? iid L
Changwen Zhang, Wenli Ouyang, Hao Yuan, Liming Gong, Yong Sun, Ziao Guo, Zhichen Dong, and Junchi Yan. Towards imitation learning branch for mip: A hybrid reinforcement learn- ing based sample augmentation approach. In The Twelfth International Conference on Learning Representations, 2024a.
Yahong Zhang, Chenchen Fan, Donghui Chen, Congrui Li, Wenli Ouyang. Mingda Zhu, and Junchi Yan. Milp-fbgen: Lp/milp instance generation with feasiblityyboumdddnes. In Forty-first Inter- national Conference on Machine Learning, 2024b.
15

---

Published as a conference paper at ICLR 2025
A
APPENDIX
CONTENTS
GPT- Based Evolution
17
MILP Classes Details
18
A. 1.2
Prompt Types and High Level Descriptions
19
A.1.3
Generating MILP Problem Topics
20
A.1.4
Parameter Adjustment and Filtering Details
21
A. A.1.5
MILP Instance and Learning Dataset Collection Details
................................
21
A.1.6
MIPLIB Dataset Collection Details
23
A.2
MILP Learning Details
24
A.2.1
Integrality Gap Prediction
24
................................
A.2.2
Learning to Branch
................................
27
A.2.3
MILP- MILP-Language Contrastive Learning
................................
29
A.3
Additional Experimental Results and Discussions
A.3.1
MILP Instances Structural Analysis
31
A.3.2
Learning to Branch: Performance per class.
32
A.3.3
A New MII Set based on Six Unseen Seed Classes.
33
A.3.4
Effects of Different Seed Classes
33
A.3.5
Mixing different fractions of seed v.s. MILP- Evolve generated data.
34
A.3.6
Practical Application of Language-Mor MILP Contrastive Learning Task
35
A.3.7
GPT only baseline for Language-MILL Contrastive Learning.
35
A.4
Prompt Details
39
A.4.1
MILP Code Syntax
39
A.4.2
Example Prompts
41
A.4.3
Prompt-Type Specific Requirements
46
A.4.4
General Requirements
49
16

---

Published as a conference paper at ICLR 2025
Combinatorial Auction
Exclusive Auction
Auction in Facilities
thenh "Lxxclln in paiss" 1C)
[sython] parametors
:python
"nLexc_ paire"; left
pacanaters MaLiton"":
"rubids": 75C, ...l
Ercciitt__nAn_ccun 300,
Spociityymox_cconee 18751
'python!
Ipythen;
values np ap.zandon, wrw.iomn....rr......
[python]
compate rp..eadon.
"alos', etc. left
bide
randon
"bida" zame left
bide_por__te ... candon
r_tasility ...
instance ("bids"; bids, "bide_psr_itenn" bids_pes_itoo
usw awxclusive" var Lo acoze confil
operation_coste I...]
capasity [...] ... instance ("bids"; bids. "exclusive": exclesive)
for range (ruexeliu www.leeeeeeee....... pel ral bidl, bid2 sandon, sandon,choion choioes exclosive. (bid), bid2))
nodel Model ("Comstcction")
honl
Add
Cross
boolean decisior werlable bid_vars
instance ("bida"; bida, Mexclusive"s exclusive
Over
sinilas Le left [python]
model vaddas (vrype=*B"))
Koze decision variables
in rançe (bid111))1)
Ipythen]
Roee cunalzaints for facilily: t capacity, workload, shipping. and other
-arr, facility
+ Objectives max:mize cotal price nedel. s
for i, (bundle, price) in cnuncrate (iisstanccc"baae 1)11
paickkumippiis (id_vacs(d]
similar
... skip in
t CCjject live considers facility facility costs
f additional constraint for ibidl, exciesive: adacoona (biid bid_vara [bid2] 11
Constz: bundle par item. fer indices in pics_per_iton Atem: model laddCone coum iiddd (a)
model se:oojeativei....
fael faellity lity.nosta penalsiax)
in indices)
Facility Allocation Problem
Figure 6: An evolution chain for a combinatorial auction problem. In the first evolution, an addi- tional "mutually exclusive" constraint is introduced for different bids. The second evolution involves a crossover with a facility planning problem, where bids are placed at different facilities, each with its own auction capacity. For each problem class, the components are highlighted in different colors: parameters (gray), data (orange), and solver (blue). For brevity, most of the code is omitted.
A.1 GPT-BASED EVOLUTION
MILP- Evolvevolve, as depicted in Fig. 2 generates MILP code (classes) level by level, combining LLM- based code generation with a gost-generation parameter search and filtering each level. The process begins with 8 seed classes taken from previous literature. The code of each seed class reformatted into standard, modular code structure including data, optimization modeling, and parameters, detailed in Appendix
At each level, up to K 108 pairs of (MILP code, prompt type) are sampled to create a new batch of prompts for the LLM. For each pair, the MILP code is sampled uniformly at random from the suc- cessfully generated MILP code from the previous level (or seed classes at the first level). A prompt type is then randomly sampled to prompt the LLM, which generates type is selected with weights within a set of 10 prompt subcategories across five broad categories (Add, Cross-Over, Mutate, New, Delete), with further details provided in Appendix A.1.2 We use OpenAl GPT-4o Achiam et al. (203) as the LLM for the MILP class generation.
MILP code. The prompt
Since L.L--generted code often be buggy, infeasible, trivially solved within short time, we implement an (automated) post-generation parameter search and filtering procedure. This in- cludes: removing all buggy generated MILP code, (2) identifying parameters for each remaining MILP code, and (3) conducting a parameter grid search by solving the MILP with various parame- ter values. MILP code is considered "successful" if at least one parameter combination meets set of intuitive criteria for problem size, solve time, and branch-and-boum tree size; otherwise, the code discarded. Details on the parameter grid-search ranges and filtering criteria are provided in Appendix A.1.4
Computation Requirements. We conduct 92 levels of MILP class evolution, with total of 9, 044 prompts submitted to GPT-4o. This results in 2, 006 successfully generated MILP classes that meet the filtering criteria. Each evolution level takes approximately 3 hours average, with total of
17

---

Published as a conference paper at ICLR 2025
mat Bal Chantoalnatiott www.ereettetteeaaaoaaooo misttoo)
LLM Generated
mettuae a in wis atte Grome coeflwedd ood (eweeeeeee
ceee Tell
senn aanu Crtpae by "aare war (or) Tar ragglt H4840)
mase vadeund pnlie
soleestto contal) vortd raggt roced)
ar ume areectooeeii for (orpoo.cet))
rager
for renedl ootoo
coraciir aucrttuction
-25
75
120
3enerdon
Seed
cell wationa aratee roae acriv
dragtiia sor occorutivr cama/rodd) for rode 11 (o30).
aalelaaiiaaee airaaae vark/) carrilnaca sant..
PPoordhaaee grane 013
Bottom Left: an example of seed class' optimization modeling component. Right: an example of a MILP-Evolve generated class' optimization modeling component.
Figure 7: Top Left: The same T-SNE visualization in Fig. la
10 days with 60 parallel running threads to generate all levels; the majority of the time is spent on parameter search and filtering, it involves solving large number of MILP instances. For the learning experiments, we use the first 1, 5,000 generated MILP classes, which were produced in the first 42 levels over approximately 5 days. We plan to release all LLM-generated classes upon paper acceptance, and we believe this dataset will be valuable resource for future multi-class MILP learning tasks.
A.1.1 MILP CLASSES DETAILS
Seed Classes. The evolution pipeline starts with 8 benchmark MILP classes commonly used in previous literature. Table provides provides list abbreviations and references of the seed classes. We reformat the code from (Scavuzzo et al. 2022) to generate IS, SC, CA, CF, and KS instances, and from (Labassi et al. 2022) to generate GIS, NF and SAT instances.
Table 4: Abbreviations, Full Names, and References for the 8 Seed Classes.
Abbreviation
Full Name
Reference
IS
Maximum Independent Set
Bergman al. 2016)
SC
Set Cover
Balas & Ho 1980)
CA
Combinatorial Auction
Leyton-Brown al.
(2000)
CF
Capacitated Facility Location
Cornuéjols et al. 1991
KS
Multiple Knapsack
Pisinger (1999)
GIS
Generalized Independent Set
Colombi et al.
(201))
NF
Multicommodity Network Flow
Hewitt al.
2010
SAT
Max Satisfiability
Bejar et al.
2009
Code Syntax. We formulate each seed class into modular code structure, and we also prompt each --gnerrrd class to follow a similar structure. detailed example of an LLM -----eeeertted class
chedulieeigggg provided in Appendix(A44. Each MILP class is imple- mented as a Python class with a descriptive name, with three main components:
1. Data: Inside the generate instance ance function, necessary data for the constraint and objec- tive coefficients generated. In the conference room scheduling example, the data corresponds to the meeting schedules, room availability and capacities. For many graph based problems (e.g. Set Cover, Independent Set), the data includes a graph generated by common distributions such
18

---

Published as a conference paper at ICLR 2025
Erdos-Renyi or Barabasi-Albes The data generation typically uses random functions with parameters to control data properties (e.g. the range of room capacity or the graph density).
Optimization Modeling: Inside the solve function, variables, constraints and the objective function of the MILP defined. For example, the conference room scheduling example maxi- mizes the number of meetings scheduled, subject to the constraints that each room can host only one meeting at given time and each meeting should be scheduled in most one room. The data generated by the
instancce function is used to generate these constraints.
3. Parameters: The parameters . dictionary specifies parameters for data generation to complete the code for the MILP class.
The code for each MILP class is self-contained and complete; When executed, can generate data, model the optimization, and solve the corresponding MILP problem.
When prompting the LLM to generate new MILP code, we pre-process each file by marking the data, optimization, and parameters blocks with comments including ### given instance data code ends here, ### new instance data code ends here (data) ### given constraints and variables and ob- jective code ends here, #iII new constraints and variables and objective code ends here (optimization modeling), and #ilff given parameter code ends here, ### new parameter code ends here (parameters) at the end of each function.
A.1.2 PROMPT TYPES AND HIGH LEVEL DESCRIPTIONS
Level-by-Level Prompt Generation At each level, randomly sample most K = 108 pairs of (MILP code, prompt type) to form the new hatch of prompts for the L.M.. The MILP code are sampled uniformly random from the successful MILP code generated from the previous level. For an existing code, we select prompt type by randomly sampling based on the default weights ("Formulation :ddd:: 1, mulation MMutate"" 0.8, "Mutate ceuttateredudddancy" where we lower the weight of certain prompts to balance the amount of prompts from different categories (Add, Mutate, Cross-Over. New, Delete). If the MILP instance associated with the ex- isting code has long solve time (> 150s), we increase the weight of the Delete prompt to 0.8 and decrease the weight of all Add prompts to half of the default weights.
Add": 0.5, "comv add": "conv_add": 0.5, "Cross Over": 1, "Mutate": "For-
0.8. "Topic_news _ne__:eww" 1, "New": 0.8, "Delete": 0.5),
We provide a detailed description of each prompt type as follows.
Prompt Structure We use chain-of- chain-of-thought prompting technique to prompt the LLM to generate a new MILP code from a given MILP code. The prompts generally contain the following main components:
1. Summarize the given MILP code.
Describe how to modify the MILP code based on the specific prompt type.
Generate the new MILP code, stt-b following a prompt-type specific requirements and general requirement.
In addition, we give specific requirement of the generation output format to follow the Data, Optimization Modeling, and Parameters modular structure, and provide the LLM the given MILP code. In the second component, we ask the LLM to describe necessary changes in the new MILP based on different prompt subcategories. Specifically, we have
We randomly select three formulation methods within the following list of commonly used MILP formulation methods: Knapsack Constraints (Set Packing, Set Covering, Partitioning). Clique Inequalities, Big M Formulation. Convex Hull Formulation, Logical Conditions, Piecewise Linear Functions, Symmetry Breaking, Special Ordered Sets, Indicator Constraints, CemiiContinuous Variables, Stochastic and Robust Optimization, Network Flow Models. We ask the LLM to select formulation method and describe how this method can be applied to the given MILP enhancing the complexity and realism of the MILP formulation.
Formulation Add:
Topic Add: We randomly select a topic from a large pool --M-geeerated optimization topics covering various real-world applications and optimization methodologies (see Appendix A.1.3
19

---

Published as a conference paper at ICLR 2025
below). We then prompt the LLM describe how it can retain the original MILP's structure while adding complexity by incorporating the specific topic into the MILP
3. Conversation Add: We randomly select topic similar to topic add, but we ask the LLM to generate a dialogue between an expert (assistant) and a novice (user) in the MILP domain around the given topic. We then ask the LLM to summarize the dialogue into new MILP.
Given two existing MILPs, we ask the LLM to generate a new MILP by incorpo- rating information from the second MILP to the second MILP. Specifically, we ask the LLM to embed the two MILPs within the same specific real-world application, explain similarities and differences between the two MILPs, and incorporate the second MILP code to the first code.
Cross-Over:
5. General Mutate: We ask the LLM to describe the MILP under different real-world scenario and discuss how to the given MILP code can be slightly modified to suite the different real-world application while maintaining similar level of complexity as the given MILP.
Formulation Mutate: Similar to Formulation Add, we provide the LLM with three randomly selected formulation methods and ask the LLM to choose MILP formulation and explain how it can replace some of the existing constraints in the given MILP code by new constraints using the chosen MILP formulation method in the real-world context.
7. Mutate Remove Redundancy: We ask the LLM to identify and remove redundancies from the given MILP code and further introduce novel, more diverse component to create the new MILP.
New: We ask the LLM to describe how the new MILP can follow a similar python syntax and structure as the given MILP but models completely different optimization problem with a dif- ferent real-world application.
9. Topic New: Similar to Topic Add, we randomly select a topic from the topic pool and ask the LLM provide detailed description of new MILP code to suite different optimization problem under the selected topic with specific real world application.
10. Delete: We ask the LLM identify less important components from the given MILP and explain how the new MILP can remove the less important components.
The complete prompts of Formulation Add, Cross-Over, Mutate, Topic New, Delete can be found in Appendix AA.42
For the third component, the prompt-type-specinn requirements guide the generation (or removal, in the case of deletion) of the data, optimization models, and parameter blocks in the MILP code, with slight wording variations depending on the type. The general requirement outlines instructions on the completeness, modularity, executability, novelty, and difficulty of the generated code. The type- specific requirements are detailed in Appendix A.4.3 while the general requirement is provided in Appendix A.4.4
A.1.3 GENERATING MILP PROBLEM TOPICS
To generate MILP topics, begin with the generation of two distinct sets of topics with GPT-4, The first set, referred to as the application topics, focuses on different sectors of the real world. These sectors include logistics, supply chain, healtheare, environmental planning, among others. Each application topic is accompanied by a detailed description, a list of notable companies, associated sub-domains, and various challenges faced within that sector. The second set, methodology topics, consists a variety of mathematical models and optimization techniques used in operations research. These include topics like linear programming, Just-In In-Time (JIT) models, lean systems, Markov processes, and stochastic programming. Each methodology represents a distinct approach or toolset that can be applied to the application domains to address optimization problems.
Then, we create MILP topics set cross product of the above two sets, where one element from the application set is paired with one element from the methodology set. This process results in the generation of potential MILP topics, each representing combination of an application sector and methodology, forming the basis for more specific research or study. In general, we applied GPT-4o to generate 90 companies (from 23 industries) application topics and 108 methodology topics, which results in 9720 (seed) MILP topics, and then we randomly selected 5000 of them for future tasks.
20

---

Published as a conference paper at ICLR 2025
Application Topics
Methodology Topics
MILP Topics
Figure 8: MILP Topic Generation Process.
A.1.4 PARAMETER ADJUSTMENT AND FILTERING DETAILS
Parameter Grid Search Ranges For each parameter in the independently sample its value uniformly at random from the following ranges:
For an integer parameter v, we define the search space as int 1,2,3,,, 9, 15)).
For floating point parameter with the current parameter value 2 1, we define the search space as 0.75, 2, 5, 7,9, 10, 15).
For a floating point parameter with the current parameter value v < 1, we define the search space the linear space between 0.1 and 0.8 with 11 equally distance values: [0.1, 0.17, 0.24, 0.31, 0.38, 0.45, 0.52, 0.59, 0.66, 0.73, 0.8).
For a boolean parameter, we define the search space as [True, False].
Filtering Criteria: given set of parameter values from grid search, we declare success for the parameter if the following hold
Solve time E [foow Liight [20s, 180s]
Presolve Time: tpre [tpre: tpre
[0s, 15s), and [fractou frac [0,0.2]
pre
Total Var nor E [moow Rhigh [50, 5 10a]
Total Binary and Integer Var: phin_int_vee var
[ntow
bin bsn_int_voa bin int_ear [50, 2 10a) high
Total Cons; n cons
ponss [ntow
nonss hiigh
[50, 10*|
Number of branch and bound nodes na bmb E [n low Righh
bnb
= 5000)
Integrality Gap: gap [gaptows gaphighh [0, 300%)
The above information be parsed and computed from the MILP solving log file for the MILP instance associated with the MILP class, given the default seed parameter.
A.1.5 MILP INSTANCE AND LEARNING DATASET COLLECTION DETAILS
MILP-Evolve. We take the first 800 MILP classes generated by MILP-Evolve and generate mul- tiple MILP instances per class using different random seeds, following standard practice from pre- vious studies (Prouvost et al. 2020 Scavuzzo al. 2022). Details of the instance generation and dataset collection are as follows:
Integrality Gap Prediction: split the first 800 MILP classes into 643 classes for training/val- idation and 157 classes for testing ( ~ 8:2 ratio). We generate 100 instances per class, and further split all training/validation instances with 8:2 ratio into separate training and validation set.
To collect the Integrality Gap data, we solve each instance with a time limit 200s and exclude any instance not solved to optimal within the time limit. Among all the optimally solved instances, we clip the integrality gap to the range (0%, 100%], where we set 100% to be the upper bound and represents instances with particularly loose LP relaxations. This results in set of 38, 256 training instances, 9, 564 validation instances and 11, 584 test instances. Data collection with 50 CPUs takes around 66 hours (2. 75 days).
21

---

Published as a conference paper at ICLR 2025
Learning to Branch: We split the first 800 MILP classes with roughtly 7:1:2 ratio into 579 for training, 59 for validation and 162 for testing. We then collect 50, 10 and 30 instances per class for training, validation, and testing. Following the same data collection procedure in Gasse et al. (2019), each B&B node has a probability of 0.95 to apply a Beeudocost-branchine exploration and 0.05 to use the Strong Branching expert to collect the training data. Up to 50 data per instance are collected.
strategy for
We set solve time limit of 200s collect the strong branching data for each instance, excluding instances solved optimally at the root node by default SCIP. This results in a set of 26, 502 MILP instances for training, 512 instances for validation and 4, 756 instances for testing. Data collection using 50 CPUs takes around 34 hours (1.4 days). Additional instances are collected for baseline models match the total training instances in Fig. 4
Language-MILP Contrastive Learning: We generated 10 instances for each of the 1 1,260 classes, except in the ablation experiment shown in Figure 4c] where we generated more instances for the selected classes. Extracting numerical information from the MPS file only takes negligi- ble amount of time, and most of the data generation time occurs in querying LLMs. We used 80% of the classes for training and the rest for testing. During training, 10% of the training instances were held out for validation.
Seed, Seed+ and Seed+VAE.
Seed. For each of the 8 seed classes, we manually select two parameters, resulting in total of 16 seed parameters. We then collect the learning datasets to train the Seed model Table
(a) Integrality Gap Prediction: We solve 100 MILP instances for each parameter, with a time limit of 200s per instance. We then split the optimally solved instances into a set of 1208 training instances and a set of 303 validation instances (~> 8:2 ratio).
(b) Learning to branch: We solve 500 MILP instances per parameter, using the same 200- second time limit, resulting in 7252 training instances. For validation, we use the same MILP-Evolve set without further splitting the seed instances.
(c) Language-MII
Contrastive Learning: We generated 1, 446 instances and used 20% in- stances as a validation set to select the parameters (learning rate, dropout, etc.); then, we used the entire dataset for training.
Seed + Param. For each of the 8 seed classes, we use our parameter search and filtering procedure (Fig. 2 Appendix(A.1. (A. to generate additional valid parameters. We sample 240 parameters per seed class, from which we obtain 89 new parameters that satisfy the filtering criteria. Then we collect the learning dataset for the new parameters to augment the dataset of the Seed Classes, which we use to train the Seed Param. model in Table [ Specifically,
(a) Integrality Gap Prediction:
We generate 100 MILP instances for each new parameter, sulting in 7462 augmented MILP instances with Integrality Gap values. These are combined with the seed dataset, resulting in a final set of 7177 training instances and 1796 validation instances.
(b) Learning to Branch:
We further collect the strong branching data for 100 instances per parameter, excluding instances solved optimally at the root node by SCIP Default. This yields a total of 13831 training instances. We similarly use the same MILP-Evolve set for validation.
(c) Language- Contrastive Learning: We generated 7, 389 instances and used 20% in- stances validation set to select the parameters (learning rate, dropout, etc.); then, we used the entire dataset for training.
Seed + VAE. We adopt the state-of-the-at instance generation method from Guo et al. 2024 to augment the seed MILP instances. Specifically, we first train Variational Autoencoder (VAE)- based model on the combined set of Seed training and validation instances. The trained VAE generates 21, 000 } (the fraction of constraints modified). We then collect datasets for the Seed + VAE model in T We note that find many of the generated instances are infeasible and
instances, with 12000, 6000, 3000 instances generated for mask ratios of
? = {0.01, 0.05, 0.1)
cannot be used for training.
(a) Integrality Gap Prediction: We obtain 8316 valid instances. Combined with the seed dataset, this results in final set of 7860 training and 1967 validation instances.
22

---

Published as a conference paper at ICLR 2025
(b) Learning to Branch:
We collect strong branching data for 7552 instances. Combined with the seed dataset, this provides a total of 14, 804 training instances. We again use the same MILP-Evolve set for validation.
(c) Language-MILP Contrastive Learning: We generated 8, 305 instances and used 20% in- stances as validation set to select the parameters (learning rate, dropout, etc.); then, we used the entire dataset for training.
Language-Mil Contrastive Learning: Description Collection. To generate meaningful de- scription of an MILP problem from its MILP instance file (where we use the MPS format (Tomlin & Welch 1992) to save the instances), we combine both the characteristics extracted from the source code and the values from the MPS file to query a LLM to produce an accurate textual description that can capture both the high-level problem information and data information from the A/B matri- ces. Generating Problem Characteristics: The process begins by using a Large Language Model (LLM) extract specific characteristics of the problem from the solver's Python source code (which can be written using SCIP. Pyomo, or Gurobi). The LLM analyzes the solver code, identifying key attributes such as the MPS format details, formulation techniques (e.g., "big M" or inequality for- mulations), problem domain (e.g.. "bin packing" or "set cover"), as well as important details about the objective function, constraints (linear or non-linear), and variable types (integer, binary, con- tinuous). Extracting MPS Information: After identifying problem characteristics, rule-based algorithm parses the MPS file into its key sections: ROWS, COLUMNS, RHS, and BOUNDS. Each of these sections corresponds to different aspects of the MILP problem. For example, ROWS de- fines the objective function and constraints, COLUMNS identifies the coefficients of the decision variables, RHS contains the right-hand side values for the constraints, and BOUNDS specifies the variable bounds.
LLM
solver.py
Characteristics
Dacorintion
LLM
Dacociaionn
Description
dump
me EIL.
Downod A Tafo.
Parser
MAD Lul.E.l.
Domond Tacka
MPS File
Parsed Info
Figure 9: Process for generating descriptive text from MILP problems using source code and MILP instance files (MPS format). The generated text is then used for the language-MILP contrastive learning task.
A.1.6 MIPLIB DATASET COLLECTION DETAILS
widely used benchmark dataset for MILP, featuring diverse range of instances from various application domains. We study transfer learning on this heteroge- neous dataset for Integrality Gap Prediction and Language- -MILP Contrastive Learning.
MIPLIB GGliinner et al.
2021)
Integrality Gap Prediction A majority of MIPLIB instances (specifically, 914 of them) have their optimal solution values posted on the website. We curate this set of instances and compute the integrality gap based the posted optimal solution. We split the set into 614 instances for training (fine-tuning) and 300 instances for testing for the MIPLIB experiments in Table
Language-MILP Contrastive Learning. Each MIPLIB instance given as the A, matrices rather than the optimization problem formulation (e.g. constraints and variables that can be used subset these
to generate descriptions). However, most instances include a brief description. descriptions are informative and suitable for the language task. For example, the description of the 'map10' instance is: 'Land parcel selection problems motivated by Red-Cockaded Woodpecker con- servation problem Imported from MIPLIB201 In contrast, other descriptions are less informative
https::/mipliii zib http://mi.pp.bbiii...eeeaatteenp 1s_mapllppppppplp wwp.ppppp. html
23

---

Published as a conference paper at ICLR 2025
and not useful for the language task. For instance, the meeos-1582420 instance is described simply as "Collection of anonymous submissions to the NEOS Server for Optimization
To ensure that only relevant and informative instances model (LLM) to filter out instances with descriptions deemed uninformative. As a result, the dataset was reduced from initial set of 914 instances for Integrality Gap Prediction to a final count of 303 instances. We then divided the dataset with a 8:2 ratio into a training (fine-tuning) set of 242 instances and and test set of 61 instances. This division provided a sufficient balance between training the model and retaining a subset for unbiased evaluation.
retained, applied large language
Given the relatively small size of the filtered dataset, we conducted the testing process 10 times and averaged the results to ensure robustness. For Figure 4c, where only 1,000 instances were used for training, we repeated the training process four times with different randomly selected data, performing 10 rounds of testing each time, and reported the average. This repetition mitigated any potential variability that could arise from the limited number of instances.
Data filtering details. Examples of the descriptions that to undisclosed industrial applications from companies like Google, instances imported from earlier MIPLIB submissions, or those collected from forums such as the Gurobi forum with unknown ap- plications. Some instances were also removed because they were marked as infeasible by optimiza- tion solvers such as ParaSCIP, taking an extended time solve, or because they were anonymous submissions to optimization servers like NEOS. Other filtered instances originated from MiniZine Challenges between 2012 and 2016 or were randomly generated integer and binary programming instances. Descriptions related to railway line planning and other irrelevant application contexts were also excluded.
filtered out include those related
We report the MPS-to-Text accuracies in results because only fine-tuned the GNN model and froze the text embedding model. Interestingly, we observed that the Text-to-MPS accuracy was consistently about 1-2% higher than the text accuracy, suggesting that the text encoder in our CLIP model was better trained than the graph neural network (GNN) encoder used for the Mixed-Integer Programming (MIP) instances.
A.2 MILP LEARNING DETAILS
A.2.1 INTEGRALITY GAP PREDICTION
Training and Evaluation Setup. We train, validate and test all methods on distributed cluster using nodes equipped with 80 Intel(R) Xeon(R) Silver 4316 CPU and A single Nvidia Volta A100 GPU. Training takes less than 24 hours for each model.
Training Loss. At training time, we use Huber Loss (Huber 1992) to minimize the deviation of predicted integrality gap from the label. Specifically, given ground truth label g (xx) and a predicted integrality gap fthera (x) for a MILP instance x, the Huber Loss is defined as
1(0;x) = ((lo())-g(<)))) if fff(()) g²)) S (fo(z)) gx(m) otherwise.
(2)
Evaluation Metric. At test time, we report the absolute deviation Neaal Exe (f(()) - g" (x) across a set of multi-class test instances & E Neett- We further evaluate the Pearson Correlation of the prediction and the ground truth as the second test metric.
Input Features We use the variable and constraint features provided in Paulus al. (2022). A list of the features are provided in Table 5 As we only need to extract the state information after the first root-node LP relaxation, we remove subset of irrelevant constraint features related cutting planes from the original set.
Note: We use the Paulus et al. (022 implementation instead of Ecole (Gasse et al. 2019 because the customized PySCIPOpt interface by Paulus et al. (2022) allows us to extract input features after
neone 15824200... htm]
24

---

Published as a conference paper at ICLR 2025
Emboldiing
V-CC
C&Y (Subsample) Attention
Final Embedding
Graph Convolation
mnd,
C
Attention
score
1x1
V
Subb sample
Linear
Multi head Attention
Variable nodes
Constraint nodes
V
C
Figure 10: Input Graph Representation and Learning Architecture for Integrality Gap Prediction. For Ours Attn., the attention layer is replaced with global pooling on the hidden embeddings of all constraints and variables to obtain the global summary vector sp.
each LP relaxation, whereas that by Gasse et al. 2019) only allows extracting input features each branching decisions, and the first branching decision happens later than the first LP relaxation.
Table 5: Integrality Gap Prediction and Language MILL Contrastive Learning: MILP instance input features for variable and constraint nodes (Paulus et (2022)).
Node Type Feature
Description
norm coef type has lb has ub
Objective coefficient, normalized objective norm
Type (binary, integer, impl. integer, continuous) one-hol
Lower bound indicator
Upper bound indicator
norm redcost solval solfrac sol is at lb
Reduced cost, normalized by objective norm Solution value
Vars
Solution value fractionality
Solution value equals lower bound Solution value equals upper bound LP age, normalized by total number of solved LPs Simplex basis status (lower, basic, upper, zero) one-hot
sol_is_ai_ut norm age basestal
rank Korm_nnzrr bias
Rank row Fraction of nonzero entries Unshifted side normalized by norm Row value equals left hand side Row value equals right hand side Dual LP solution of row, normalized by and objective
row row_is_a ____s_____________ Ihs row jww_is__m at rhs dualsol basestat norm_age
Basis status of a row in the LP solution, one-hot Age of row, normalized by total number of solved LPs LPs since the row has been created, normalized
Cons
norm_nlp_cn crcattio
worm_intcon is integral
Fraction of integral columns in the row
Activity of the row is always integral in feasible solution
removable
Row is removable from the
is is_in_p obj_par
Row is member of current LP
Objective parallelism score of
Architecture and Training Hyperparameters. Our network, illustrated in Fig. 10 first em- beds E 1x mmx29 (where n, m are the number variables and constraints) into hidden representations of dimension dridden 64 with BatchNorm followed by two (Linear, ReLU)
25

---

Published as a conference paper at ICLR 2025
Table 6:
Architecture hyperparameters for Integrality
Gap Prediction.
Table 7: Training hyperparameters.
Input dimension dn
R
GCN Message V-+C C-V
Optimizer Adam
x da
R
Passing Order
Learning rate
0.001
Num. GCN
Output dimension
Layers
Batch size
32
Embedding
Attention
Num. of Gradient Steps
30000
dimension dasdden Num. sampled Cons.
Num. Hcads
Atlention
512
0.6
Vars. Seubaomple
Dropout
blocks. The hidden embeddings are fed into a Graph Convolution module Kipf & Welling (2017), with message passing, following the direction of V-C and C-+V, with final (LayerNorm, Lin- ear, ReLU, Linear) block that maintains the dimension dhidden- Then, for each MILP instance, we randomly sample a subset of Seubsample a (Ssubeamppe + 3) dhidden = 515 x 64 matrix, where the first Ssubeample dimensions are the subsampled constraint and variable hidden embedding, and the last three dimensions are the mean constraint embedding, mean variable embedding, and special summary node where we extract the global information to. Lastly, we feed this matrix into Transformer Encoder Layer Vaswani (2017) with Weedd = 8 heads and dropout rate of 0.6, and we take the attention output embedding of the all-zero input vector and finally use a (Linear, ReLU, Linear) block to map the vector into a sealar output.
512 constraint and variable nodes and construct
We train with Adam optimizer with a learning rate of 0.001 and a batch size of 32 with a total of 30000 gradient steps. All hyperparameters selected on the validation set and frozen before evaluating on the test set. Table and provides list of hyperparameters.
26

---

Published as a conference paper at ICLR 2025
A.2.2
LEARNING TO BRANCH
B&B background and common branching strategies. Exact MILP solvers (Bestuzheva et al.
2021 Gurobi Optimization, LLC 2023) typically employ the Branch-and-Bout (B&B) algorithm, which systematically explores a search tree to find the optimal solution. At each node the tree, a Linear Programming (LP) relaxation is solved, where the integrality constraints on the variables are relaxed. the LP solution satisfies the integrality constraints, it is feasible for the MILP, and the process can backtrack. Otherwise, the algorithm selects variable to branch on, creating two child nodes with additional constraints (e.g., 23 & | and xj 2 [a3])
The efficiency of the B&B algorithm heavily depends on the branching strategy used to select vari- ables for branching (Achterberg. 2007). Common strategies include:
Most Infeasible Branching: Selecting the variable with the value closest to fractional (i.e., x3 closest to 0.5).
Strong Branching:
Evaluating the potential impact of branching on each candidate variable by temporarily branching and estimating the resulting lower bounds.
Pseudo-Cost Branching: Estimating the effect of branching based on historical information gath- ered during the search.
However, these heuristics may not be optimal for all problem instances, and designing effective branching strategies remains an area of active research. In this work, we extend the setup of (Gasse et al. 2019) to imitate the Strong Branching expert to the multi-class learning context.
Training and Evaluation Setup. We train, validate and test all methods on distributed cluster using nodes equipped with 80 Intel(R) Xeon(R) Silver 4316 CPU and A single Nvidia Volta A100 GPU. Training for each model takes less than 24 hours. During data collection and testing, use a single CPU to solve each MILP instance. Following Gasse et al. (2019), we disable presolving and cutting plane separation focus on the impact of learning for B&B. A 200s time limit set for solving each instance.
Training Loss. Each training data corresponds to a state-action pair (s, a²) a B&B node, where s represents the MILP subproblem at the node and a" is the strong branching expert action. Given a learned policy fo(.) that outputs the probability of selecting each variable from candidate set, we minimize the cross-entropy loss N ( log log (a (s) of predicting the expert action.
Evaluation Metric. At test time, we compare the solve time of each learned method with SCIP Default for each test instance, with a 200s solve time limit per instance. For each method, we report the proportion instances solved optimal, optimal, the I-shifted geometric mean of time improvement over SCIP Default, and the 1-shifted geometric mean time improvement excluding neural network overhead (Ex.). Time improvements are calculated for instances where the learned method solves to optimal within the time limit.
report the proportion of instances solved to
We follow the Ecole (aasse et al. 2019) implementation for the learning to branch experiments. As listed Table g the input features are similar to those in Paulus et al. (2022) with a slightly larger set of variable features and smaller set of constraint features.
Input Features
Architecture and Training Hyperparameters. We adopt similar GCN architecture as in Gasse al.) (2019). Specifically, given the Ecole input feature with V 1 19 and C E R m the model includes the same embedding layer as in Fig. 10 followed by three layers of the V-C & C-V Graph Convolution. We increase the number of layers as it improves the prediction accuracy without adding significant overhead. After the GCN block, each variable node's 's hidden embedding is projected through an MLP; this results in a predicted score for each variable, which we use to select the variable. We set the same hidden dimension of Ahidden 64 and train each model for 100 epochs. The remaining hyperparameters are identical to those in Table(6land]]
Notably, exclude the attention layer from the architecture due to its computational overhead: since we predict a score for each variable rather than for the whole graph, subsampled attention from Fig. 10 cannot be applied, and using full attention for all variables would increase the solve
27

---

Published as a conference paper at ICLR 2025
Table Learning Branch: MILP instance input features for variable and constraint nodes (Gasse et al.
2019).
Node Type
Feature
Description
norm coef type
Objective coefficient, normalized by objective norm
Type (binary, integer, impl. integer, continuous) one-hot Lower hound indicator
has
has ub norm redcost solval solfrac at col_is_at_ut
Upper bound indicator
Reduced cost, normalized by objective norm
Vars
Solution value
Solution value fractionality
Solution value equals lower bound Solution value equals upper bound LP age, normalized by total number of solved LPs The objective value of the current best solution he mean of all incumbent values found far Simplex basis status (lower, basic, upper, zero) one-hot
norm_ag
incumbent value avg avg_incuu
incumbent value
basestat
bias
Unshifted side normalized by row norm Cosine Similarity of the row with the objective
abj_cosiie___e sim js_ighht
Cons
Row value equals right hand side Dual I.P solution row, normalized by row and objective Age row, normalized by total number of solved LPs
dualsol
norm age
time (our main evaluation criterion). We recognize developing an efficient attention mechanism to enhance multi-class branching as an important direction for future work.
28

---

Published as a conference paper at ICLR 2025
A.2.3 MILP-LANGUAGE CONTRASTIVE LEARNING
Loss Details. Let D- {(M,, denote a dataset comprising N pairs of MILP instances Mj and their corresponding textual descriptions T, Our objective is to learn embedding functions far M - Rd and fr T F that map MILP instances and descriptions into shared d-dimensional latent space.
For batch of K MII.P-text pairs, we compute the embeddings:
hM fM(M) E RR
(3)
hy = fT( E Rxd
where hM and by are the embeddings of the MILP instances and text descriptions in the batch, respectively. We then normalize these embeddings to have unit length:
hv normalize(harr,
(4)
hy normalize(hr),
where the normalization is performed along the embedding dimension, applied row-wise for each example in the batch:
h;
normalize(h.) ||h,l|2
(5)
Next, we compute the similarity scores (logits) between all pairs in the batch using the dot product of the normalized embeddings:
ZM = hhahh E R
(6)
ZT-M = hr E R
Here, (M--r(i..) represents the cosine similarity between the i-th MILP instance and the j-th text description.
We define the labels for contrastive learning as:
y 00, 10.1,
indicating the correct matching pairs in the batch.
then compute the cross-entropy losses for both MILP-to-text and text-to-MILP directions:
LM-TT 1
CrossEntropy [ZM-sr(i.
K
(8)
K
LT-M =
CrossEntropy (Zr (i.
K
The total loss is the average of the two losses:
1
C = (CM-T LT-MM).
Training and Evaluation Setup. We train, validate and test all methods distributed cluster using nodes equipped with 80 Intel(R) Xeon(R) Silver 4316 CPU and A single Nvidia Volta A100 GPU. Training takes less than 24hr for each model.
29

---

Published as a conference paper at ICLR 2025
Engineering Tricks. We used a caching mechanism for the text embedding model because it was not fine-tuned during our experiments. Hence, we only needed to utilize the 7B embedding model once per text description, which largely improved the efficiency of the training process.
Evaluation Metric. At test time, given a MILP instance a and a set natural language descriptions f y& } for different MILP classes, then perform k-way classification to distinguish the correct natural language description for each MILP instance from the set of options. We report the 4-way and 10-way accuracy on the test (k and 10).
Input Features For the MIIP, we use the same input feature in Table 5 to embed each MILP instance x; that is, the input graph is constructed after the first root-node LP solved, which is easy to obtain as the root-node L. is typically fast to solve. The text description is fed directly into language encoder.
Architecture and Training Hyperparameters. For the MILP encoder, we adopt the same archi- tecture as Integrality Gap Prediction (Fig. [0] T with change in the final layer (head), instead of projecting into single score, we project into embedding vector with dimension notpuu 4096 so that we can project the modality between MILP and text (NV-Embed-v) F with temperature 0). We the model with dropout ratio 0.5 and train the model with learning rate 5 10 with Adam Optimizer for 100 epochs.
30

---

Published as a conference paper at ICLR 2025
Freq, (x109)
Consr Ox (x100%)
Constr (x10%)
Conat (x10%)
Comar. 10
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
0.5
Vars. (x10%)
Vans. (x10%)
Vars. (x10%)
Vars. (x10%)
Vars. (x 10
0.3
0.3
Rd. Freq &
0.3
0.3
0.3
Rel Freu,
Rdl Frej
0.2
Rd. Preq
0.1
0.5
0.0
0.6
0.6
0.5
Integrality Gap
Innogrality Cap
Integrality Cap
Integrulity Gap
Integrality Gapp
(a) MILPLIB Train
(b) Ours
(c) Seed + Param.
(d) Seed + VAE
(e) Seed
Figure 11: MILP Instance Statistics from Different Data Sources. We plot the problem size distri-
bution (top) and Integrality Gap training label distribution (bottom). Instances generated by MILP- Evolve(Ours) exhibits a closer problem size label distribution to MIPLIB than baseline methods.
See Appendix A.3.1 for details.
A.3 ADDITIONAL EXPERIMENTAL RESULTS AND DISCUSSIONS
A.3.1 MILP INSTANCES STRUCTURAL ANALYSIS
In Table 9 we present structural statistics to compare the different data sources (MILP-Evolyy Seed, Vu et al. 2024), Param. Search). We also include statistics from the MIPLIB training set (Sec. 5.3) to understand why Ours outperforms methods trained on Seed instances or their aug- mented versions in the transfer learning setting. The statistics are categorized into four groups, each representing different aspect of MILP structure. We visualize subset of these statisties in Fig.
1. Class:
We compare the optimization problem formulations of classes generated by MILP-Evolve with the 8 seed classes. We extract the number of model. addVar and model. addCons statements in each code, which correspond to different types of variables and constraints. As seen in Fig MILP classes generated by MILP-Evolve include a wider variety of variable and constraint types. We note that this aligns with real-world optimization problems, which typically involve many types of variables and constraints capture the complexity of the problem.
2. Instances: We analyze the structural properties of MILP instances generated from different data sources, including instance size (number of variables and constraints). the ratio of continuous variables, and constraint coefficient density (proportion of non-zero entries in the A matrix). While the continuous variable ratio and constraint densities are similar across sources, instances from MILP-Evolte have more variables and constraints than Seed, VAE, and Param. This is expected, as MILP-Evolve generates more diverse MILP formulations. As shown in Fig. [T] the MiLp-Evolve produces instances with diverse range of sizes, similar as the MIPLIB dataset; in contrast, VAE and Param. can only perturb instances slightly from the Seed instances.
3. Solving:
We measure the distribution of solve time and the number of B&B nodes across different data sources. Fig. 12 further shows the solve time distribution for Seed, VAE, Param., and MILP-Evolve. We observe that instances generated by MILP-Evolve have similar solve times to Seed. In contrast, VAE instances have shorter solve times, which highlights the challenge of preserving solve times in learning-based instance generation. We note that the parameter search and filtering in MILP-E is key to maintaining similar solve times to Seed; without it, generated instances typically solve under seconds. This demonstrates the importance of post-filtering to maintain desired problem properties for LLLM-based MIIP class generation.
Gap similarity with MIPLIB Test: We compute how similar each gap distribution from different data sources are to MIPLIB Test. We find that the gap distribution from MILP- Evolve is closer to MIPLIB Test than the baseline methods (Seed, VAE, and Param.). ). As MIPLIB contains set of heterogeneous instances with diverse set of gap distribution. this highlights the benefit of MILP-Evolv in increasing training instance label diversities, which we see in Table 3 leads to improved transfer learning performance.
Integrality
31

---

Published as a conference paper at ICLR 2025
Table 9: Instance Statistics from Different Data Sources.
MIPLIB Train
MILP-Evolve
VAE
Param.
Seed
# Var
0.67
1.58
0.67
Class
# model.addCons
N/A
3.46
1.84 0.87
1.84 0.87
1.84 0.87
# Variables Ratio of Continuous
15666 37561
16039 21235
3443 3579
5350 2 9336
3592 5064
Instance
0.37
0.44
0.71 0.44
0.83 0.37
0.75 0.43
# Constraints
14514 41000
12367 22461
2534 3402
6522 11886
3087 5064
Constraint
Cocfficient Density
0.040 0.145
0.013 0.082
0,014 0.020
0.041 0.07
0.017 0.022
Number of podes
926 + 4668
869
1454 4965
1847 = 5765
Solving
Pre-solve Time (8)
N/A
1.82 = 6.80
0.14 0.27
0.68 0.89
0.22 0,43
Solve Time (8)
N/A
56.78 a: 49.19
27.88 35.66
59.92 1: 45.58
59.02 a: 48.74
Integrality Gap
Correlation (?)
0.98
0,92
0.86
0.56
0.65
Similarity
Intersection (f)
0.67
0.56
0.51
MIPLIB
Chi-Square Dist. (.)
0.07
0.13
0.39
0.52
0.64
Bhattacharyya dist. (L)
0.02
0.04
0.15
0.21
0.28
Relabive Frequency
0.15
Relative Frequency
0.15
Relative Frequancy
0.15
Relatvv Freqquecc
0.15
0.12
0.12
0.12
0.12
0.06
0.06
0.08
0.06
0.08
6.03
50
100
150
700
0.00
150
200
0.00
50
100
150
0,00
50
100
150 200
Solve Time (s)
Salve lime (s)
Solve Lime (s)
Solve Time (s)
(a) Ours (MILP-Evolve
(b) VAE
(c) Param.
(d) Sced
Figure 12: Solve Time Distribution from Different Data Sources. The solve times for Ours (MILP- Evolve), Seed, and Param. Search similar, while VAE-augmented instances have slightly shorter solve times.
Similarity Metric Details. For each data source (including MIPLIB Test), we first construct a histogram with values between [0, 1] using Réina 30. The histogram for different data sources are visualized in Fig. I the MIPLIB Test distribution (not plotted) is similar to MIPLIB Train. We then calculate the similarity of the histogram of each data source and the MIPLIB Test histogram using similarity measures (i) Correlation (higher the better), which computes the Pearson correlation
Shiie min(hist(]j) hist22))))
of two histograms; (ii) Intersection (higher the better), calculated as "hamm (hux(hittlll,,ksstt (iii)
Chi-Square Distance (lower the better), computed as Emhne
(histl kist2(()
+hist22 (iv) Bhattacharyya
Distance (lower the better), calculated log( Σ
/histl hist2
A.3.2 LEARNING TO BRANCH: PERFORMANCE PER CLASS.
In Fig. 13 show the branching performance each test class for Seed and Ours. We caleu- late the T-shitted geometric mean of time improvement over SCIP Default for all instances within each MILP class, accounting for neural network overhead. Using the same T-SNE embedding from 7 each test class is represented as a circle. A blue circle indicates the method improves solve time over SCIP Default, while red circle indicates a slower solve time (with darker shades repre- senting greater improvement or degradation). Seed is trained only on the seed classes, plotted as orange stars, while Ours is trained on both the seed classes and the MILP-Evolve generated classes, shown as light gray stars. We observe that Ours improves solve time for significantly more classes than Seed. though there is still a subset where Ours underperforms compared to SCIP Default. This suggests future work enhancing multi-class learning performance for these classes, developing a meta- model to select between SCIP Default and a learned method based on problem characteris- tics.
32

---

Published as a conference paper at ICLR 2025
Seed
Ours
30
MF
SAI
Dimension 2
50
Dimension
25
0-
0
-25
-25
-50
-50
-20
-20
-75
-75
-100
-75
-50
-25
25
50
75
-100
-75
-50
-25
25
50
Dimension
Dimension 1
Figure 13: Learning to Branch Solve Time Improvement in Each MILP Class. Using the same T-SNE embedding as Fig. 7 we visualize branching performance on individual test classes. Each class is represented by circle: blue indicates improved solve time over SCIP Default, red indicates slower solve time, with darker shades showing greater improvement or degradation. Seed trained on seed classes (orange stars), and Ours is trained on both seed and MILP-Evolve-ee Evolve-generated classes (light gray stars).
A.3.3 A NEW MILP-Evolve SET BASED ON SIX UNSEEN SEED CLASSES.
Six Unseen Seed Classes. We take different set of 6 unseen seed classes, consisting of Graph Coloring, Job-Shop Scheduling, Protein Folding, Multi-Iten Lot Sizing, Bin Packing, and Max Cut. We SHILP-Evolve slightly expand the test set total of 50 classes. We perform transfer learning of different models to this unseen test set, where we use 40% classes for fine-tuning and set aside 60% classes for testing.
Table 10: Abbreviations, Full Names, and References for another set Unseen Seed Classes.
Abbreviation
Full Name
Reference
GC
Graph Coloring
Jensen & Toft 201
JS
Job-Shop Scheduling
Xiong 22022
PF
ProteinFolding
Willims 2013
LS
Multi-Item Lot Sizing
Chen & hizy
(190
BP
Bin Packing
Tang ct al.
2020
MC
Max Cut
Tang et al.
2020
Language-MII Contrastive Learning: Expert Curate/ Verified Language Labels. To ensure the quality of language descriptions, we manually verified and modified the linguistic description and made sure the descriptions matches the optimization problem in the testing set.
We observed that GPT-generated descriptions are generally accurate, correctly reflecting the opti- mization problem and highlighting potential real-world applications of the optimization class (e.g.. "This type of scheduling is essential in manufacturing and project management, where minimiz- ing total completion time across multiple tasks is critical."", Some descriptions included irrelevant details (e.g., "PySCIPOpt is used to solve the optimization problem"), which we manually removed.
A.3.4 EFFECTS OF DIFFERENT SEED CLASSES.
We include ablation study to analyze the effect of different seed classes the performance.
Statistics. Table shows the proportion of generated classes from each seed class 4 see that
some seed classes can lead more generated classes (e.g. Combinatorial Auction (CA)) than others
Note that for cross-over prompts, only tracked the trace of the first MILP class, so the number here he slightly noisy.
33

---

Published as a conference paper at ICLR 2025
(e.g. Knapsack (KS)). One potential reason could be that our filtering and parameter adjustment procedures can find good solutions for certain classes more easily than others.
Table 11: Proportion of MIlvP-Evvlve generated classes from each of the seed class.
IS
CA
KS
GIS
NF
SC
SAT
CF
Proportion 10.8%
27.3%
4.3%
22.4%
7.5%
6.5%
11.9%
9.3%
Impact of different classes on the learning performance. For each of the eight seed classes, we train separate models on instances (I) from all evolved classes based only on this seed class (One Seed), (2) with weighted sampling. 70% from all evolved classes based on this seed class, and 30% based on other seed classes (Weighted). We fix the number of train and validation instances, and compare the test performance on MILP-Evolve held out test set (Table 1 and the transfer learning performance to MIPLIB (Table
Results: We report our findings Table 12 We see that
1. Table I One Seed: Learning on classes based one single seed class has limited performance.
2. Table 1 Weighted: Classes with higher proportion in the MILP-Evolve dataset (CA, GIS), when given higher weight when sampling training set, typically lead to a better test performance on the --volve held out set; exception is Capacitated Facility Location (CF), which has a lower ratio than CA and GIS, but its learned model achieves the best test performance among all models.
3. MIPLIB, Weighted: The transfer learning performance when initializing with the different mod- els seem be similar on the MIPLIB test set, and is worse than Ours performance when trained on instances from all MILP classes.
Finally, these results give more evidence to the primary hypothesis of this paper: the importance of having diverse of MILP classes from different seed classes improve the generalization performance.
Table 12: Generalization performance on the MILP-Evolle hold out set (Table I and MIPLIB (Table 3) when learning on (1) One Seed: classes originating from single seed, and (2) Weighted: 70% classes originating from single seed, 30% classes from other seeds. We fix the total number of training and validation instances across all settings.
Table
One Seed
Table
Weighted
MIPLIB, Weighted
(Q)
Corr. (*)
Dev. ())
Corr. (1)
Dev. (L)
Corr: (f)
Seed 0: Indep Set (IS)
32.66%
0.26
25.29%
0.47
25.41%
0.47
Seed 1: Comb. Auction (CA)
30.01%
0.34
21.40%
0.53
23.44%
0.55
Seed 2: Multiple Knapsack (KS)
33.84 %
0.09
24.22%
0.49
23.60%
0.53
Seed 3: Generalized Indep. Set (GIS)
31.74%
0.19
22.64 %
0.51
25.61%
0.49
Seed 4: Multi-Comm. Network Flow (NF)
36.49 %
0.22
20%
0.41
24.08%
0.52
Seed 5: Set Cover (SC)
34.45 %
0.11
24.80%
0.45
26.13%
0.47
Seed 6: Max Satisfiability (SAT)
43.07 %
0.20
23.52%
0.49
23.79%
0.52
Seed 7: Cap. Fac. Location (CF)
33.00%
0.09
20.39%
0.58
25.24%
0.51
Ours (Full Dataset)
20.14%
0.58
20.14%
0.58
21.56%
0.59
A.3.5 MIXING DIFFERENT FRACTIONS OF SEED V.S. MILP-Evolve GENERATED DATA.
We study the effect of mixing different fractions of Seed v.s. MILP-Evolve generated MILP instances on the performance for the Integrality Gap Prediction task. Specifically, we construct different training sets by varying the ratio of seed and instances generated from MILP-Evolve classes. We fix the total number of training and validation instances as 1200 and 300 instances, respectively. We then train model on each of these training sets and test on the MILP-Evolve hold out test set (Table l). From the result table below, we see that including more MILP instances from MILP- Evolve (i.e. more diverse MILP classes) improve the performance.
34

---

Published as a conference paper at ICLR 2025
Table 13: Integrality Gap Prediction: Performance when mixing different fractions of Seed v.s. MILP-Evolve generated MILP instances training and validation set, fixing the total number of training and validation instance constant. We test on the MILP-Evole held out test set Table T
Seed
Seed 80%
Seed 60%
Seed 40%
Seed 20% Evolve 80%
100%
+ Evolve 20%
+ Evolve 40%
Evolve 60%
Deviation (4)
32.96
25.66
23.57
21.67
21.32
Correlation (t)
0.10
0.41
0.49
0.55
0.57
A.3.6 PRACTICAL APPLICATION OF LANGUAGE-MILE CONTRASTIVE LEARNING TASK
We answer this question on two fronts: the utility of this task from the perspective of understanding MILPs and the potential of contrastive learning technique itself.
Many open source MILP datasets such as MIPLIB and in many business scenarios, the MILP in- stance files contain only constraints and variables (the raw A, b, values in the optimization), which are typically hard to understand and massive in size. Most of these MILP files lack descriptions of the underlying optimization problem and/or not sufficiently meaningful. Moreover, we cannot directly feed them into LLMS to interpret and generate language descriptions of the MILP formula- tions due to the context length limit, as seen from Table 14 and discussed in the next section.
Hence, this work takes first step with a contrastive learning approach to align GNN embedding of MILP instances with the text embeddings, aiming provide meaningful interpretations when giving the MILP instances as input. Our results indicate that our approach holds lot of promise.
Given the abstract nature of the MILP instances, believe any assistance in helping users' under- standing of them is crucial. This can help nonexperts to understand the problem and also identify the incorrect formulations. This task also complements our other two tasks which concerned with solving MILPs rather than understanding. We believe that a foundation model for MILPs that aims democratize solving MILPs should also have the ability help users understand them.
A.3.7 GPT ONLY BASELINE FOR LANGUAGE-MII MILP CONTRASTIVE LEARNING.
Table 14: Language-MILP Contrastive Learning: Performance Comparison of GPT-4o on the (sub- sampled) MILP instance files and the GNN-LLM contrastive model results.
GPT-4o Train From Scratch
Seed
Seed Param Search
Seed + VAE
Ours
4- Way Ace. (t)
47.79%
72.37% 46.50%
72.20%
75.17% 42.66%
72.90% 44.61%
77.62% 53.99%
Way Acc (f)
16.81% %
42.45%
For the Language-MILL Contrastive Learning task, an interesting question is, whether we can use large language models (LLMs) directly interpret the MILP instance select the language align- ment, hence bypassing the need to embed the MILP instance with the Graph Neural Network (GNN).
One of the bottlenecks with LLM directly interpreting MILP instances is that, the MILP instance files are typically huge as they contain the raw numerical values of the variables and constraints, substantially surpassing the context length of LLMs that's why in this work, we focus on using GNNs to embed MILP instances, which can handle large MILP instances, to contrastive with the text embeddings from a language model.
We perform the experiment by subsampling the rows of the instance files up to context length. Specifically, we preprocess the MPS content for GPT-4o by anonymizing identifiers (e.g.. problem names, variable names) and selecting 150 representative lines: 50 from the header, 50 randomly chosen from the middle, and 50 from the tail. This approach ensures sufficient context about the MPS file while staying within content length limits. We then construct multiple-choic question and prompt GPT-4o to provide a step-by-stee guess.
th-huugh analysis before making an educated
35

---

Published as a conference paper at ICLR 2025
We observe the significant performance gap between GP and the GNN-LLM aligmment methods as in the paper. This shows it is insufficient to use LLMs directly the MILP- MinguLan Contrastive Learning task.
Table 14 presents the results the same MILP-Evolve test set as in Table 2
An example GPT-4o's prompt and answer can be found next.
An example prompt for GPT-4o to Perform the MILP- L-Lnguageee Contrastive Learning Task
You provided with an MPS file representing a mixed integer programming (MIP) prob- lem. Based on the content of this file, determine which the following problems it is most likely associated with.
MPS FILEE CONTENT
SCIP STATISTICS
Variables: 3440 13328 binary. integer, implicit integer. 112 contiauous)
Constraints:
OBJSENSE
MIN
ROWS
OBj
L Z_0
L Z_1
Z_2
150
lines in total here. Skipped LaTeX for conciseness
L C_96
C_94
C_102
C_105
X_5
C_94
C_73
-1
C_83
C_89
ENDATA
CHOICES
This optimization model addiesses job-shop schodaling problems that incorperates muchine and resounee constraints, as well penco- dence and piirity requirements smoog jobs. Esch job has a madoally assigned processing time, machine assignment, and resource requirement, The model also enforces group based precedence constraints, dictating the order which certain groups of jobs must be complesed. Additonilly, jobs assigned posest model the inclusion machine-ppciife capacity machine do not exceed its capacity The objective function seeks to minimize the makespun, or the tocal time to complete all jobs, while also factoring in joh affinitis creflectngg job prioray) and resomree unilization. This combiiied objective is designed to promote efficient machine usage. balance workload, respect prierit allocations high-demand scheduling envconments..
machine constrinne sequencing requirements
everiap. ensaring that cunulative resource roquicements for johs on
B:
The max cut oprimication problem involves dividing the nodes of graph into two groups maximize the sum of weights on edges that have endpoint in each group. Imagine you have setweek where modes are connacted by weighed edges, and you want to split the into
the highest possible sotal weight. In this code, a gaph is penersted with raddom weights olges, and hinary variable resigned to each node to indicate its group. Constmints are adied to ename each edge cither connests modes in the same set or across sats. The objective finetion then maximizes the wei gitt edges across the cut, solving the problem using optinization techniques. This approsch useful network design, where maximizing the sepomation losd betwee groups desired.
them edges connecting nodes different
C: This multi-item lat-sizing problem models semmrio where a company noeds to plan production and inventory for multiple products over number time periods minimize peoduce, how much inventury cost (for proparing machines or equipment). holding cost ffor stoeing any leflover inventory) ) limits on how much can be produced periud due to resource consiraints. The model ensures that all customer demand is met while keeping vosts low by balancing production quamiities, sctup times, and imvenorry levels periods. This sprooch widely used in mamu faclurnn and suppyy chiin management optimize production schedales, reduce
For each period, the company faces specific demands each product and must decide production decision
whecher produce how
associated setup
officienly minage
D: This optimization model is designed to identify optimal locations for electric vehicle (EV) charging stations within a given netwoek of polential nodes. The model operates
gnph structure representing the network, where each node signifies potential station cosss, energy availability
location each edge denotes connectnity betwees locations. considers virious factors, such zoning compatibiiiy. gri substutions, and tniif impacts. Each node his binary decision variables indicating station placement. resource allocation, and compliance with zoning regulations. The objective function veeks musimixe connecivity and substation resources while lund
taiff impaces. Additinna constraints varable large consiant (Big M) enforces priority comsraints specific nodes. solition for station depioyment that maximizes netwock aecessibiiity and manamizes opentional costs while meeting regulatory and resource limitations.
that eneryy demands
avaiabilin, and outcome identifies balanced
INSTRUCTIONS
Think through the structure, variables, and constraints provided in the MPS file. Evaluate their alignment with the listed problem types. Justify your reasoning step by step before selecting the final answer.
Finally, provide only the single letter (A-D) that corresponds to your chosen answer in the format below. The last letter in your response will be used for evaluation. FINAL ANSWER: [A/B/C/D]
36

---

Published as a conference paper at ICLR 2025
example GPT-4o Response to the MILP- LanguLan Contrastive Learning Task
ANALYSIS OF THE MPS FILE
To identify which problem the provided MPS file most likely associated with, let's analyze the structure and characteristics listed in the file and match them with the given choices.
1. VARIABLES
The MPS file contains a total of 3440 variables, with 3328 binary variables and 112 continuous variables.
Binary variables typically indicate decisions of whether or engage in specific actions (e.g., opening facilities, assigning specific nodes to a group, etc.).
2. CONSTRAINTS
There are 6320 constraints present in the problem, a significant number which sug- gests complex interactions, common with problems involving combinatorial signment and network design.
3. OBJECTIVE
The objective is one of minimization (OBJSENSE MIN), which often implies cost minimization minimizing resource usage.
MATCHING WITH CHOICES
Given these observations, let's match the structure to the choices:
OPTION A (JOB-SHOP SCHEDULING)
This problem typically involves assignment and sequencing constraints, requiring integer and binary variables for ordering and timing decisions. While binary variables are used, may not fit the minimization objective the large number of binary variables if detailed resource constraints, as described, are absent.
OPTION B (MAX CUT PROBLEM)
This involves dividing nodes into two groups, which aligns well with binary variable defi- nitions. However, the problem typically emphasizes maximization of cut weights, not mini- mization.
OPTION C (LOT-SIZING PROBLEM)
This problem focuses on production planning, inventory levels, and setup costs. While involves minimization, the variable structure typically includes more continuous and integer variables, and it is less binary-heavy compared to the structure of the given MPS file.
OPTION (OPTIMAL LOCATION FOR EV CHARGING STATIONS)
This problem involves deciding locations, which aligns well with a binary variable for place- ment decisions. The large number of constraints fits with logistical, regulatory, and resource allocation concerns. Additionally, the focus on minimizing land costs matches the mini- mization objective.
CONCIUUSION
Taking into consideration the large number of binary variables, minimization objective, and extensive constraints, Option seems to be the best fit. This type of problem typically has numerous binary variables for potential location decisions and many constraints related to logistical, regulatory, and resource allocation concerns, aligning well with the characteristics of the given MPS file.
FINAL ANSWER
FINAL INSWEE D
37

---

Published as a conference paper at ICLR 2025
Example Description from the Test Set
This optimization model addresses complex job-shop scheduling problem with various constraints and objectives aimed at minimizing overall operational costs and delays. Each job has processing time, tariff impact (costs affected by international tariffs), setup time between jobs on the same machine, and energy consumption rate. The model incorporates precedence constraints to ensure certain jobs are completed before others and assigns jobs to specific machines while managing limited machine capacity. Additional complexity is in- troduced through a machine breakdown risk managed by auxiliary variables, with penalties applied if breakdowns occur. The objective function minimizes the makespan (completion time of the last job), risk and energy costs, setup times between jobs, and potential break- down penalties. This model enables efficient scheduling by balancing energy consumption, setup times, and breakdown management, ultimately supporting cost-effective job sequenc- ing and machine utilization.
Another Example Description from the Test Set
This model focuses on optimizing the placement and phased deployment of electric vehi- cle (EV) charging stations across graph-represented network. Each potential station site (node) is evaluated based on land costs, energy availability, zoning compatibility, and tariff' impacts, with connections between nodes representing possible network benefits. Placement decisions involve constraints on energy resources and land use compatibility. The model in- cludes deployment timing variables enforce phased installation sequence, ensuring sta- tions are deployed in specified order. The objective function seeks to maximize network accessibility- enthancing connections while considering node-specific weights-whil min- imizing associated land costs and tariff impacts. By accounting for setup costs, regional resource limits, and sequential deployment, the model provides strategy for an efficient, cost-effective expansion of EV charging infrastructure.
Example Description of the Seed Problem
The provided MPS file outlines mathematical model for combinatorial auction problem, structured for a mixed integer programming formulation. The model aims to maximize the total price of accepted bids, employing binary decision variables is accepted. Each item is constrained to be included in at most one accepted bid, with inequality constraints set on the items ensuring this limit. The objective function involves maximization with coefficients provided for the bids, while RHS values impose constraints such that certain items may appear at most once. This structured information allows one to understand how the model strategically enforces constraints and aims to achieve the highest possible total price for the accepted bids while ensuring no item overlaps. Additionally, performance details and model-specifet techniques noted to highlight optimization and study aspects within the combinatorial auction domain.
denote whether bid
Example Description for MILPLIB Dataset
A problem in wireless networks. The objective is to select minimum number of relay nodes so that any nonadjacent nodes can communicate by way of the chosen relay nodes in at most hops, where is problem input. The 2-hop case of this problem can be formulated as a set cover/hitting set problem with n binary variables and n² constraints:
Σ KENG)NNG)
Z6.
for nonadjacent node pairs {i.j}.
Despite the formulation's simplicity, instances with as few as 120 variables are left unsolved after one hour using Gurobi 7.0.2.
38

---

Published as a conference paper at ICLR 2025
A.4 PROMPT DETAILS
A.4.1 MILP CODE SYNTAX
import candom
2 Import time
inport nungy 48 up
Erem pyscipopt import Model, quicksun.
class ConfereenceRoonScceell....
def _init_tteell (self, parameters, seed-None);
for key, value in parameters. itemmt setatti (self, key, valuel
10
11
12
13
self.seed seed if self.seed:
14
rendom, rp.random.eee seed (seed)
16
17
18
Generation tRREсtEсснtсссс
19
def ( (sel
Instance
20
assert self.min_capacity
self.max_capacity www..minnccaaciiee
21
room capacities
23
coos_capscities self.min_capecigy (expeiixx.capacity
selp.iin
24
random. rand oew.:...meee.....ooo..
25
26
meetings
27
28
# Creete meeting schedules
renge www.err_of_neeeinneeeeeeeeeeeeee
30
requi red_napaciity self n_capacity + faelf.max__nee....eoooemm mae_pcccyy
31
self.min..spacitt)) + rp.rraddm.randly
32
start_time randon.randint (0, self self.max_tan .max__tmee self..metiiggddraat
33
end_time start_time + wew.. ing
34
35
meetings. append (requuired start_time, end_timel)
37
roon_aval [I] for room in range homberrrumbrrrrooeoo _of_rromss]]]]__
38
for i, required__oppciine start_time, end_time) in enumerate (mestings): (self wumber_of_rromeeeee if con__capclllee [100n] >- requu_eed_cppcclly
39
40
woos_evailabilityyeeeee apppedd
42
1P8 given instance dota cade ends here
44
449 new Instance datá code ends here
45
46
47
"meetings": neetings, "zoon_evailebilie" scom zcom_aveilabilityy "room_aapacities" room_cspacities
49
51
52
#8164684e11144118 Optinization ion Modeling (PYSCTPOPt) #*6/368с/45ясн44
53
def (self, inslance);
54
meetings instance l'meetinge"]
rocm_evailebility instance "coomovvillailitty)
56
model Model PoonfereneeRoomscheediiinn
58
59
Decision varisbles
60
schedule_vare (1,, nedel. addVar (vLype-"B", name=E"Room Meeeeeetggggi for in range (len (meetings)
61
for in range wtw.eeef..nmbbooofeceo
63
à object/ maximize ze the number of meet' ngs acheduled il for in zange (aageeeeeenneeeeeeee for in range (len (meetings)) if in coor_eveilabilltyyell
db objective_expr quiqquuuutssseeee hattssumttsheddleeeaa.a.
65
66
67
reom can host only one meeting a given time for in range (self, (self.nnmben wwwmber_of_ree rcoms))
Constraints:
68
range (len (mostings) 15 i1 not in room_awat.abilltyyy: continue
70
71
72
for 12 in range
len (meetings)!:
73
if 12 not. in rcom_availabilittyyft
74
75
if meet meotings [nnga[ii](] (1] x meotings [12] [2] and maetings [1] mectings(11) [2]:
model addoona (schodule, ( 11] + peooduleevvra_ee 12,
39

---

Published as a conference paper at ICLR 2025
78
79
Constraints: Each meeting ahould be acheduled most one zoom
80
for in (len (meetings));
81
edddons ggsickssm(schecnleee.. FF, i1 for = in range (ww.s:eeeeee...eee.eeee rfrooms rooms]
82
if i in room_ovsslbbiittyyyn <= I"M
83
modal, sedel.setobjectivvv (objective_expp. "moximize")
85
143 given constraints and variables and objective code ends here
87
### constraints and variables and objective code ends here
88
89
start_time time.time end_time time.time!)
90
92
93
zeturn model. geteel.getStatur(l end start_time
94
95
1f
main nome
96
97
#
99
parameLers
30,
"nunber_of_meetinn 125, mnin_capacity' : 30,
max__apacity'; 300,
104
20,
'max_time':
106
#84 given parameter code ends here ##t peraneter coda here
108
scheduler Confereeeeeoommeeeeuul instance = sckedulez. gewerazee status, sclve_time scheduler. soive (instance)
(paramelers, seed)
111
schedbles.geaersee gearrsseinntttteeee instance ()
113
115
print Staatvs wwwlvosaaattlleeeee print (f*Solve, Times, [solve_cieer aeconds")
116
40

---

Published as a conference paper at ICLR 2025
A.4.2 EXAMPLE PROMPTS
Formulation Add Prompt
Follow these step-by-step instructions to generate diverse and realistic MILP by adding complexity to given MILP using a specific MILP formulation method:
Describe the Given MILP in Detail:
Place the MILP within a specific real-world context.
Clearly outline the context, constraints, objectives, and variables, emphasizing its prac- tical significance.
2. Choose MILP Formulation Method
from the following options:
[three_random random formulation }
Explain how this method applies the given MILP in real-world context.
Discuss the relevance of the chosen method enhancing the complexity and realism of the MILP formulation.
Summarize and Formulate a New MILP:
Retain original MILP's structure but add complexity by incorporating the specific MILP formulation method.
Diseuss how introduce constraints, variables, data, change the objective func- tion in the context of the real-world application.
Ensure new constraints or objectives are linear. Use tricks to linearize if necessary.
4. Generate New MILP Code Step-by-Step as follows:
[Add Requirements, see Appendix A.4.3) (We set prompt_opp opt imptimereeeecc the gi ven MILP formulation method" " for formulation add prompt.)
[General Requirements, see Appendix A.4.4
5. Output Format:
Description of the given MILP:
(text to explain of the given MILP and its real-world application) )
Description of the chosen MILP formulation method and its application to the given MILP:
(text explain the chosen MILP formulation method, its relevance to the given MILP in the context of the real-world application
Description of New Constraints, Variables, Objectives, Data, Parameters:
(text to describe how to incorporate the specific formulation method to the given MILP by proposing new constraints, variables, objectives, data, or parameters)
(New (cosrrantkutabbooooceivaaaaaaeee I: description,
New coostraiffurabbboboecrvvdduaaeer 2: description.
...)
New complete MILP code:
"python
(complete code that incorporates modified constraints, data generation, and param- eters)
6. Here the given MILP Code: (code)
41

---

Published as a conference paper at ICLR 2025
Cross-Over Prompt
Follow these step-by-step instructions to generate diverse and realistic MILP by incorpo- rating information from another MILP to add complexity to given MILP:
1. Provide a Detailed Description of the given MILP code:
Embed the MILP within specific real-world application.
Clearly deseribe the context, constraints, objective, and variables in way that high- lights its practical importance.
Provide a Detailed Description of the second MILP code that you should incorporate into the given MILP code:
Embed the MILP within the same specific real-world application as the given MILP code.
Clearly describe the context, constraints, objective, and variables way that high- lights its practical importance.
Explain the similarities and differences between the given MILP and the second MILP.
3. Explain the Modifications to the given MILP code and Their Impact:
Discuss how incorporating constraints, variables, changing the objective function from the second MILP code the given MILP code the given real-world scenario.
The new MILP should retain the majority of the given MILP's structure, but make significant addition based the second MILP codebase to make it more complex and challenging.
Ensure new constraints or objectives are linear. Use tricks to linearize if necessary.
4. Generate New MILP Code Step-by-Step as follows
[Add Requirements, see Appendix A.4.3) (We set prompt_ wot.ooii combination of both MILP code for cross-over prompt.)
[General Requirements, see Appendix A.4.4
5. Output Format:
Description of the given MILP:
(text to explain of the given MILP and its real-world application
Description of the second MILP:
(text to explain of the second MILP in terms of the similarities and differences between the given MILP and the second MILP in wolldworld application context
Description New Constraints, Variables, Objectives, Data, or Parameters:
(text to describe how incorporate the second MILP the given MILP by proposing new constraints, variables, objectives, data, or parameters
(New [onttran/vaiibbocccccccduaaaennn 1: description,
New [anntran//saiacccccccciveaaaaner 2: description,
...)
New complete MILP code:
" "python
(complete code that incorporates modified constraints, data generation, and parame- ters)
Here is the given MILP Code: (code)
Here is the second MILP code: (code2)
42

---

Published as a conference paper at ICLR 2025
Mutate Prompt
Follow these step-by-step instructions to generate a diverse and realistic MILP by slightly modifyying the given MILP code:
1. Provide Detailed Description of the given MILP code:
Embed the MILP within a specific real-world application.
Clearly describe the context, constraints, objective, and variables in way that high- lights its practical importance.
2. Provide a Detailed Description of new MILP code to suite different real world applica- tion.
Discuss the novelty of the modifying constraints, variables, or changing the objective function from the given MILP by embedding it in different real-world scenario.
Explain the relevance of these changes, detailing the similarity and differences between the original and new MILP.
The names of the new constraints, variables, data should be full words starting with one of the letter (five_random_letterss)
3. Generate the new MILP code by following these procedures step-by-stey
[Mutate Requirements, see Appendix AA..3) (We set prompt_ oot im= mmn=reeeect the new real world appl
for the general mulate prompt.)
[General Requirements, see Appendix A.4.4
4. Output Format:
Description of the given MILP:
(text to explain of the given MILP and its real-world application
Description of the new MILP:
Names: (The full words the names cunstraiioaraaaacheeeevvv/aaaeee which starts with the capital letter selected from above.
(text to explain the new MILP and its real-world application! )
New complete MILP code: "python
(complete code that incorporates modified constraints, data generation, and parame- ters)
5. Here is the given MILP Code: (code)
43

---

Published as a conference paper at ICLR 2025
Topic New Prompt
Follow these step-by-step instructions to generate diverse and realistic MILP following the given MILP code structure:
1. Provide a Detailed Description of the python coding style of the given MILP code.
The description should only focus on the code style, but not the MILP details.
Provide a Detailed Description new MILP code to suite a different optimization prob- lem under the topic (topic) with specific real world application.
Discuss the novelty of the new MILP code in terms of constraints, variables, objective function, data and parameters and how they align with the given topic and the associ- ated real-world scenario.
The names of the new constraints, variables, data should be full words starting with one of the letter |five_raiie vandom_letters
Generate the new MILP code by following the coding style of the given MILP code. For example, it must contain the following components
[New Requirements, see Appendix A.4.3 (We set
the ven topic and the associated real world application " for topic new prompt.)
[General Requirements, see Appendix A.4.4
4. Output Format:
Description the python coding style of the given MILP: (text to explain of the python coding style
Description the new MILP:
Names: Thhe full words the names conssraii/mrabbeeoojcccdddddaaee which starts with the capital letter selected from above.)
(text to describe how to incorporate the specific topic to generate the new MILP) )
New complete MILP code: "python
(complete code that incorporates modified constraints, data generation, and parameters
Here the given MILP Code: (code)
44

---

Published as a conference paper at ICLR 2025
Delete Prompt
Follow these step-by-step instructions to generate diverse and realistic MII.P by removing less important components from the given MILP code:
1. Provide a Detailed Description of the given MILP code.
2. Provide Detailed Description of the new MILP code that removes the less important and components.
Explain how the new MILP removes the less important components from the given
MILP.
You may modify the constraints, variables, or objectives, data generation or parameters to make the resulting MILP coherent.
The resulting MILP should still be complex and challenging to solve.
Generate the new MILP code by following these procedures step-by-step:
[Delete Requirements, see Appendix A.4.3
[General Requirements, see Appendix A.4.4
Output Format:
Description of the given MILP: (text describe in the given MILP)
Description the new MILP: (text to explain what need to changed to remove the less important components from the given MILP)
New complete MILP code:
"'python
(complete code that incorporates modified constraints, data generation, and parame- ters)".
5. Here is the given MILP code (code)
45

---

Published as a conference paper at ICLR 2025
A.4.3 PROMPT-TYPE SPECIFIC REQUIREMENTS
Add Cross-Over Requirements
Add Constraints, Variables, or Objectives:
Introduce new elements to increase problem complexity while ensuring feasibility.
Place these in the solve function between ### given constraints and variables and objective code ends here and ### new constraints and variables and objective code ends here to @rompp_optimi
Modify Data Generation:
Usee functions
like random. rand or random. randint or or or similar functions to create diverse datasets.
np random. normal np. random. gamma nx erdos arddos_renyi_gaae
nx
Insert new data in the get_ instance function between #### given instance data code ends here and ### new instance data code ends here to support the added constraints, variables, or objectives.
Update the Parameters:
Define new parameters in if _name____ rameter code ends here and ### new parameter code ends here to support the data generation for the optimization.
ma between ### given pa-
The value of each parameter should be a constant (e.g. integer, float, boolean, string). That is, if there a tuple value, you should break down the tuple into individual parame- ters with constant values. If it is a more complicated data structure (a list, dictionary, set, or function), please put the data structure the get_inst ier_iinstte ance function and only the required constants to construct the data structure in the parameters dictionary. Ensure parameters are adjustable to scale the problem's 's complexity.
Mutate Requirements
Modify Realistic Constraints, Variables, or Objectives:
-Modify the 'solve" function by updating the constraints, variables, or objective to [prompt_optim).
Ensure the modifications challenge the solver and vary difficulty, contributing to diverse solving times and gap improvements.
Modify the Data Generation Procedure:
-Modify the get_ instane function by updating the res dictionary to support the modified constraints, variables, objectives.
-
Use functions like random. rand or random. randint or np rwndom.doo.mmm normal or np. random. gamma nx erdos_ Berdos_reeyi_grrreee or
nx nx. www.araaasi... a be aeer__rrape or similar functions to create datasets of varying sizes.
Modify the Parameters:
Modify the parameters within the if name port the data generation for the optimization.
ma m__ block to sup-
The value of each parameter should be constant (e.g. integer, float, boolean, string). That is, if there is tuple value, you should break down the tuple into individual parame- ters with constant values. it is a more complicated data structure (a list, dictionary, set, or function), please put the data structure in the get_ins inrt-iistaneee function and only put the required constants to construct the data structure the parameters dictionary.
Ensure parameters can be modified easily to scale the problem or to alter its com- plexity.
46

---

Published as a conference paper at ICLR 2025
New Requirements
Generate Realistic Constraints, Variables, or Objectives:
Inside the solve function, define the constraints, variables, objective
optim). (prompt_optim).
-Ensuee the new optimization problem challenge the solver and vary in difficulty, con- tributing to diverse solving times and gap improvements.
Generate the Data Generation Procedure:
Inside the get set_instance function, define the data generation and the dictionary to support the new constraints, variables or objective.
Use functions
like random. rand or random. randint or np rwndom. normal or np. random. gamma nx. erdos_rreeeet ardos__reyyiggrap or nx. wwx.barabas.. a www.aaleettrapppe or similar functions to create datasets of varying sizes.
Ensure the new generated data support the generated constraints, variables, or objec- tives.
Generate the Parameters:
-nder the if name == main block, generate the parameters to sup-
port the data generation for the optimization.
'Te value each parameter should be a constant (e.g. integer, float, boolean, string). That is, if there is tuple value, you should break down the tuple into individual parame- ters with constant values. it is a more complicated data structure (a list, dictionary, set, function), please put the data structure in the get_inst nee function and only put the required constants to construct the data structure the parameters dictionary.
Ensure parameters can be modified easily to scale up the problem or to alter its com- plexity.
Avoid Repeating Existing Code. Make the modified constraints, variables, objectives, data, and parameters distinct from the original.
47

---

Published as a conference paper at ICLR 2025
Delete Requirements
Modify Realistic Constraints, Variables, or Objectives:
-Modify the selve function by removing less important constraints variables, simplifying the objectives.
Modify the Data Generation Procedure:
Modify the
function by updating the res dictionary to support the modified optimization modeling.
Use funetions like random. rand
random. randint np. random.. normal or np. random. gamma or nx. exx.erdo__renee erdos_reny__grapp or ww..brabaasi. a gerpgggrpphh similar functions create datasets of varying sizes.
Ensure the modified data support the modified constraints, variables, or objectives.
Modify the Parameters:
Modify the parameters within the if name
ma _ block to sup-
port the data generation.
The value of each parameter should be constant (e.g. integer, float, boolean, string). That is, if there is a tuple value, you should break down the tuple into individual parame- ters with constant values. it a more complicated data structure (a list, dictionary, set, or function), please put the data structure in the get_ instancee function and only put the required constants to construct the data structure the parameters dictionary.
Ensure parameters can be modified easily to scale up the problem or to alter its com-
plexity.
48

---

Published as a conference paper at ICLR 2025
A.4.4 GENERAL REQUIREMENTS
General Requirements
Ensure the Following:
Code completeness:
Provide the COMPLETE, EXECUTABLE code, including all necessary library im- ports, the MILP class, and the parameters and code call the MILP class.
Do not omit any part of the provided MILP code with '.. (same as before) ..., even it is not modified. Do not inherit from a previously defined class; instead, provide the entire codebase.
Novelty and Increased difficulty for the new MILP while maintaining feasibility and ducing redundacy:
The new constraints, variables, objectives, data, and parameters types should be diverse, such having different constraint types, and creative data generation schemes with correct syntax.
The complexity of the new MILP can be achieved by more complicated constraint, objective, data generation scheme or larger parameter values.
Avoid adding redundant constraints, variables, or objectives that do not contribute to the problem's complexity, but do provide clear, concise, and complete executable MILP code.
Modularity and executability of the new MILP code:
Maintain function integrity by ensuring that no references to out-of-scope variables are used.
Define any new helper functions within the get jet_instance solve functions, suring they are correctly scoped and called.
Use clear, descriptive names for new parameters and variables, ensuring they align with the real- wolld context and adhere correct syntax.
Ensure the code remains modular and can easily scale to larger or different MILP prob- lems by simply adjusting parameters, without altering functions.
Do not include ### given constraints and variables and objective code ends here, ### new constraints and variables and objective code ends here, #### given instance data code ends here, ### new instance data code ends here, ### given parameter code ends here and #### new parameter code ends here in your final code.
49


Reviews:
[{"review_id": "QXCdP8Dogi", "reviewer": "Reviewer_zQjg", "summary": "This paper considers a novel dataset generation method for learning to solve mixed integer linear programming (MILP), leveraging the large language model (LLM). Given an input MILP instance, this method combines the evolution algorithm and parameter search to compute diversified new instances. The authors consider three tasks: (1) predicting the integrality gap. (2) learning to branch and (3) Aligning MILP problems with natural language to help non-experts.\n\nThe authors then tested their method on a dataset called SEED, gathered from the recent popular deep learning for MILP papers. The results showed that their method outperformed all other baselines. Moreover, the attention used on the variables can further improve the transferability.", "questions": "1. See the weaknesses (1)\n\n2. One more interesting experiment is to fix the number of training data for your method and the baselines. To be more specific, let N be the number of instances of SEED. Then, we randomly take N / 10 data and use Evolve to generate N instances and call them dataset B. Then, training directly on SEED and this B can further show the power of your model.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Employing LLM to generate diversified MILP instances is novel and helpful for training a foundation model for MILP. The entire MILP space is too huge, so datasets created by humans can only cover a part of it. So, leveraging the power of LLM is a good direction.\n\n2. The authors' commitment to open-source the entire framework is valuable for the entire community.", "weaknesses": "1. The dataset test seems to be not that \"unseen.\" You mentioned that you collected MILP problems from eight classes. But you randomly split them after the augmentation. Then, the trained model still learned from all these eight classes. So it would be great if you only use six classes for training, 1 for validation, and 1 for testing. Then this can further show the power of your method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AReDjfmPRC", "reviewer": "Reviewer_fRvL", "summary": "The paper explores the potential of foundation models for Mixed Integer Linear Programming (MILP), introducing a novel framework called MILP-Evolve that leverages large language models (LLMs) to generate diverse MILP instances. The authors apply this framework to three distinct tasks: (1) integrality gap prediction, (2) learning to branch, and (3) a new task of aligning MILP instances with natural language descriptions. While promising empirical results are shown, especially in generalizing across different MILP classes, some aspects—particularly the Language-MILP Contrastive Learning task—require further clarification regarding its practical significance and feasibility.", "questions": "1. Could the authors further clarify the real-world impact of the Language-MILP task? Specifically, how does aligning natural language descriptions with MILPs help non-experts understand and solve optimization problems?\n2. Given that the language samples are generated by LLMs from solver code, how do the authors ensure the quality of these samples? Would human-generated descriptions lead to better learning outcomes in this task?\n3. How does the Language-MILP Contrastive Learning task connect to the other tasks in the paper, such as integrality gap prediction and learning to branch? Could the authors provide more insights into the overall coherence of the tasks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- **Diversity of MILP Generation**: The MILP-Evolve framework introduces an innovative approach to generating diverse MILP problem classes, which has the potential to enhance generalization in ML-based MILP solvers.\n- **Empirical Performance**: The paper demonstrates strong performance improvements on the integrality gap prediction and learning to branch tasks, providing evidence that the proposed approach can generalize to unseen MILP classes.\n- **Extensive Experimental Work**: The paper presents a substantial amount of experimental results across various tasks, demonstrating the significant effort in evaluating the proposed methods. The authors cover a wide range of experiments, which showcases the robustness of their approach.", "weaknesses": "1. **Practical Application of Language-MILP Contrastive Learning**: The Language-MILP Contrastive Learning task is positioned as a way to assist non-experts in understanding and formulating MILPs. However, the generated natural language descriptions tend to emphasize technical details (e.g., linear constraints, integer variables), and it is not entirely clear how this helps users grasp the real-world significance of MILP problems. It would be helpful if the authors could provide more clarification on how aligning these mathematical descriptions with natural language assists in bridging the gap between abstract optimization models and their practical applications. Including more concrete examples or case studies could further reinforce this task’s practical relevance.\n\n2. **Language Quality**: The natural language descriptions used in the Language-MILP Contrastive Learning task are generated by LLMs from solver code (e.g., SCIP, Pyomo, Gurobi). There may be some concerns regarding the quality of these descriptions. If the language samples were curated by human experts, this task could capture valuable domain-specific insights. However, relying solely on LLM-generated descriptions raises questions about the meaningfulness of the alignment. It might be worth considering how this task compares to having an LLM directly interpret a new MILP, as it is not immediately clear whether the proposed approach would outperform such a method. Clarifying the value added by this task would strengthen the paper.\n\n3. **Disconnect Between Tasks**: While the paper introduces multiple tasks, the connection between them could be better articulated. For instance, the relationship between the Language-MILP Contrastive Learning task and the Multi-Class Learning task is not immediately clear, which might make the paper seem somewhat disjointed. A clearer explanation of how these tasks fit together within the broader scope of MILP optimization, particularly how Language-MILP Contrastive Learning complements the other optimization tasks, would improve the cohesion of the work.\n\n4. **Comparative Experiments**: The comparison between the proposed method and works like ACM-MILP in the experiments might benefit from some adjustments. The current experimental setup involves:\n   - Using problems generated by MILP-Evolve based on 8 seed MILP classes to create a large number of new problem types.\n   - Using problems generated by ACM-MILP, which also learns and generates problems based on the same 8 seed MILP classes.\n\n   Both sets of problems are then used as training data for a downstream ML-based MILP optimization framework, and the models are tested on other MILP classes generated by MILP-Evolve. However, existing MILP generation frameworks typically aim to enhance performance within a specific type of MILP class. Therefore, I suggest the following alternative comparative experiments:\n\n   **Experiment 1:** Compare the models trained on problems generated by:\n   - MILP-Evolve, which generates a large number of new problem types based on the 8 seed MILP classes.\n   - ACM-MILP, which learns and generates problems based on the same 8 seed MILP classes.\n\n   Then, test the trained MILP optimization frameworks on the same 8 seed MILP classes used by both MILP-Evolve and ACM-MILP. This would allow for a more direct comparison within the shared MILP classes.\n\n   **Experiment 2:** Select a set of MILP problems generated by MILP-Evolve or from MIPLIB as seed MILP classes. Compare the models trained on problems generated by:\n   - Problems generated by MILP-Evolve based on the selected seed MILP classes.\n   - Problems generated by ACM-MILP based on the same selected seed MILP classes.\n\n   After training, test both frameworks on the selected seed MILP classes to directly compare their performance.\n\n   These alternative experimental designs would provide a more balanced comparison, as they ensure that both approaches have access to similar training data. This could help avoid potential biases in the current experimental setup, where ACM-MILP might be disadvantaged by the absence of instances from the test problem classes in its training set.\n\n5. **Overclaim in Contribution**: The paper states that it achieves “Substantial Multi-Class Learning Gains Across All Tasks,” but the results presented primarily focus on integrality gap prediction and learning to branch. Since there are no substantial results or experiments demonstrating gains in the Language-MILP task, this claim could be seen as somewhat overstated. The authors could either provide additional results for the Language-MILP task or rephrase the contribution to more accurately reflect the scope of the work.\n\n6. **Impact of Seed Class Selection**: The choice of seed classes in the MILP-Evolve framework likely has an important influence on the distribution and diversity of the generated MILP classes. However, the paper does not delve deeply into this aspect or provide an experimental evaluation of how seed class selection affects the generated instances. Including an analysis of how different seed classes influence the diversity and quality of the generated MILP instances, and whether certain seed classes lead to better generalization in the optimization tasks, would help strengthen the paper’s claims regarding the versatility of MILP-Evolve.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EVuPkzqcLP", "reviewer": "Reviewer_dtCu", "summary": "This paper introduces MILP-Evolve, a novel LLM-based evolutionary framework designed to generate a large and diverse set of MILP instances. This paper evaluates the proposed method on three key learning tasks relevant to MILP: integrality gap prediction, learning to branch, and a new task of aligning MILP instances with natural language descriptions.", "questions": "1. The fairness of experiments comparing the proposed data generation method with others is unclear. The paper mentions a 7:1:2 split of generated MILP problem classes into training, validation, and test subsets, raising concerns that the test data distribution may resemble that of the training data. In contrast, other methods likely produce differently distributed training data, which could skew comparisons and inflate the performance of MILP-Evolve.\n2. For the task of aligning MILP instances with natural language descriptions, what are the specific formats for both the MILP instances and the textual descriptions? What are the sources of these instances and descriptions? Will all elements in each set be matched one-to-one, and does this task hold practical significance?\n3. How should Figure 1b be interpreted?\n4. The meaning and context of the “Mean” baseline used in the experiment is unclear.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The proposed data generation method is validated on three MILP-related learning tasks.\n2. The paper is well-structured and presented clearly.", "weaknesses": "1. The methodological contribution is somewhat limited, primarily offering a data augmentation approach that employs LLMs to generate diverse MILP instances.\n2. There is a mismatch between the content and title of the paper. The title “Towards Foundation Models for Mixed Integer Linear Programming” suggests a broader scope, while the paper mainly discusses a data generation method for MILP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b1YEILMEg4", "reviewer": "Reviewer_3Jap", "summary": "This paper takes an early step to leverage foundation models for solving Mixed Integer Linear Programming (MILP), which plays an important role in real-world applications. Specifically, it studies three important tasks, including two typical tasks integrality gap prediction and learning to branch, and one proposed task of aligning MILP instances with natural language descriptions. Compared with previous works, it emphasizes generalization performance across problem classes, and proposes an LLM-based data augmentation framework named MILP-Evolve to generate diverse problem classes for the training of models. Experimental results demonstrate that the proposed MILP-Evolve can generate diverse data, and improve the overall performance of pretrained models on the three studied tasks.", "questions": "1. What about the cost of data generation in the experiments? As the running of such a LLM-based data generation process may be very expensive, will the generated problem classes be collected and open-sourced together? \n\n2. How do you envision the practical applications of the newly proposed task of aligning MILP instances with natural language descriptions in the MILP solving process? Can you discuss more on it?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Even that there have been a variety of works on leveraging LLMs to solve complex decision-making problems, to the best of my knowledge, this is the first work that focus on LLM-based training data augmentation in the field of learning to solve MILP. In experiments, the proposed MILP-Evolve show great capacity to generate diverse problems and improve the generalization performance of the trained models.\n2. This paper is well-written, with rich technical details of the proposed MILP-Evolve. I am convinced that such a new open-source and powerful data argumentation method can benefit the community of learning to solve MILP.", "weaknesses": "1. As the idea of the proposed MILP-Evolve, which prompts the LLMs to generate diverse data under an evolution framework, is straightforward and not new, I am concerned that the technical insights of this paper are limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)