SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We fully integrate L2Dive into the open-source solver SCIP 7.0.2 [16]. This solver exposes a plug-in for diving heuristics that implements an optimized version of Algorithm 1 in the programming language C. We extend the solver's Python interface [32] to include this plug-in and use it to realize L2Dive. This integration facilitates direct comparison to all standard divers implemented in SCIP (Section 5.1) and makes it easy to include L2Dive in SCIP for use in branch and bound (Section 5.2). Importantly, we call our generative model only once at the initiation of a dive to predict a variable assignment. While predictions may potentially improve with additional calls at deeper depths, this limits the in-service overhead of our method. It also simplifies the collection of training data and produced good results in our experiments (Section 5).
Nair et al. [33] propose a method that learns to tighten a subset of the variable bounds. It spawns a smaller sub-integer program which is then solved with an off-the-shelf branch and bound solver to find feasible solutions for the original program. Sonnerat et al. [41] improve this approach using imitation learning. Others explore reinforcement learning [49] or hybrids [40], but only focus on improving primal performance. All of these methods are variants of large neighborhood search [39, 2, 36], where a neighborhood for local search is not proposed heuristically, but learnt instead. In contrast, our approach L2Dive does not propose a fixed neighborhood and it does not require access to a branch and bound solver to run. Instead, we use our model's predictions to iteratively modify and solve linear programs instead of sub-integer programs. In practice, linear programs tend to solve significantly faster which makes L2Dive more applicable. Khalil et al. [26] propose a method to learn variable assignments from good feasible solutions, but combine their model predictions with a heuristic rule for node selection, whereas we consider diving.
Overall, there is vivid interest in exploring the use of machine learning for integer programming [5, 52, 31]. With regard to branch and bound, several works learn models for variable selection in branching [25, 3, 17, 19, 42, 51]. Others focus on node selection in the search tree [20, 50] or deal with cutting plane management [35, 22, 43, 9, 45]. Further, related work includes both general [23, 24, 46] and specific [12, 9, 8] attempts of learning to configure the solver. To the best of our knowledge, we are the first to propose learning to dive to improve the performance of branch and bound solvers.
The goal of our work is to learn application-specific diving heuristics to improve on existing diving heuristics. We view other primal methods (Section 4) as complementary, and accordingly compare primarily to other diving heuristics. We evaluated the effectiveness of L2Dive in two different experiments and on a total of six datasets. The first set of experiments (Section 5.1) was designed to study the diving performance of L2Dive in isolation and compare it against existing diving heuristics. On a benchmark of four synthetic combinatorial optimization problems from previous work [17], we performed single dives with each diver and measured the average primal gap. We found that L2Dive outperformed all existing divers on every dataset and produced the best solutions amongst all divers. The second set of experiments (Section 5.2) directly included L2Dive into the branch and bound process of the open-source solver SCIP. The solver called L2Dive in place of existing diving heuristics and our goal was to improve overall performance on real-world mixed integer linear programs. We considered instances from neural network verification [33] and server load balancing in distributed computing [18]. We measured performance with L2Dive against the default configuration and a highly challenging baseline that tuned the solver's diving ensemble. We found that L2Dive improved the average primal-dual integral by  $7\%$  ( $35\%$ ) on load balancing and improved average solving time by  $20\%$  ( $29\%$ ) on neural network verification over the tuned (default) solver.
We collected data for training and validation: In all experiments, we extracted a bipartite graph input representation of each instance's root node. On all but two datasets, we chose  $\mathcal{X} = \{x^{\dagger}\}$  where  $x^{\dagger}$  is the best solution the solver finds within a given time limit  $T$ . For set cover and maximum independent set only, we observed strong symmetries and used the solution augmentation described in Appendix C. We trained separate models for each dataset. We trained each model with ADAM [27] for 100 epochs in the first set of experiments and for 10 epochs in the second set of experiments. We individually tuned the learning rate from a grid of  $[10^{-2}, 10^{-3}, 10^{-4}]$ . For each dataset, we chose the batch size to exhaust the memory of a single NVIDIA GeForce GTX 1080 Ti device. We validated after every epoch and chose the model that achieved the best validation loss. In all experiments, we use the mode prediction of the generative model and only perform a single dive from a given node. We do not attempt multiple dives in parallel and did not use any accelerators at test time. In all experiments, we only call L2Dive's generative model once at the beginning of a dive to limit the in-service overhead from serving the graph neural network.
5.1 Diving with L2Dive
With this first set of experiments, we studied the diving performance of L2Dive and compared it against existing diving heuristics. We used the same benchmark as previous work [17]. This benchmark consists of four different classes of combinatorial optimization problems, including set covering, combinatorial auctions, capacitated facility location and maximum independent sets. For each class, we used 2000 instances in total; we trained on 1000 instances and validated and tested on 500 instances respectively. We resolved all instances before diving, but did not branch and disabled cutting planes and other primal heuristics as our interest is solely in diving. We compared L2Dive against all other standard diving heuristics that are implemented in the open-source solver SCIP and do not require an incumbent solution. This includes coefficient, fractional, linesearch, pseudocost, distributional, vectorlength [6] and Farkas diving [48]. We briefly describe these baseline divers in Appendix A. In addition, we considered three trivial divers that respectively fix integer variables to their lower (lower) or upper limit (upper) or either with equal probability (random). All divers ran with the same diving budget ( $d_{max} = 100$ ) and their execution was directly triggered by the user after resolving the root node. We ignore the few test instance that were solved directly at root by SCIP before we could initiate a dive.
L2Dive outperformed all other standard divers (Table 1). It achieved the lowest average test primal gap on each of the four problem classes. The improvements over the best heuristic diver ranged from roughly  $15\%$  for combinatorial auctions to more than  $70\%$  for independent sets. The trivial divers only found solutions that are significantly worse, which indicates that L2Dive learnt to exploit more subtle patterns in the problem instances to find better feasible solutions. Some baseline divers (e.g., linesearch, distributional) failed to consistently outperform the trivial divers across all problem classes and best heuristic diver varied (pseudocost diving for combinatorial auctions, Farkas diving for facility location, vectorlength diving for set cover, independent set). This confirms that in practice most diving heuristics tend to be specialized and work particularly well for particular problem classes.
Table 1: L2Dive finds better feasible solution on all four problem classes than existing diving heuristics. Average primal gap with standard error on test set. Best diver is in **bold** and best heuristic is in **italics**.
SET COVER	COMB. AUCTION	FAC. LOCATION	IND. SET
L2Dive	55 (3)	222 (7)	160 (10)	5 (1)
Best heuristic	95 (3)	256 (8)	484 (7)	18 (2)
Coefficient	3,700 (55)	671 (11)	762 (9)	246 (4)
Distributional	3,900 (50)	1,504 (12)	760 (9)	196 (3)
Farkas	105 (3)	476 (9)	484 (7)	-
Fractional	3,726 (57)	672 (10)	1,058 (11)	232 (4)
Linesearch	1,269 (24)	467 (10)	1,036 (15)	77 (1)
Pseudocost	195 (9)	256 (8)	505 (11)	32 (2)
Vectorlength	95 (3)	832 (20)	840 (19)	18 (1)
Random	416 (13)	704 (12)	902 (14)	78 (2)
Lower	2,918 (63)	1,587 (11)	623 (8)	171 (5)
Upper	239 (6)	611 (11)	828 (14)	62 (2)
L2Dive is a generic recipe to design effective divers for any specific application. Finally, we found that the mode predictions of our learnt models were rarely feasible (e.g., set cover, combinatorial auctions) or yielded poor solutions (e.g., independent set). This highlights that learning a generative model for diving may be a more promising approach than trying to predict feasible solutions directly.
In order to validate the dual reasoning rule proposed in subsection 3.2, we paired L2Dive with two alternative rules for variable selection in capacitated facility location. The first ablative rule chooses a variable  $j \in \mathcal{C}$  uniformly at random, i.e.,  $s_j^{rand} \sim \mathcal{U}_{[0,1]}$ . The second ablative rule simply uses model confidence, i.e.,  $s_j^{conf} = q_\theta(\hat{x}_j)$  and unlike dual reasoning does not treat variables  $j \in J(\hat{x}_{\mathcal{C}})$  preferentially. We found that even when variables were selected uniformly at random (where the model prediction is used only for bound tightening), L2Dive outperformed the best standard diver (Table 2). However, selecting variables whose model predictions are more certain significantly improved performance by a large margin, while additionally employing dual reasoning tended to improve performance further for capacitated facility location. The effectiveness of dual reasoning is likely problem-specific, as dual reasoning will collapse to the model confidence rule, if the set  $J \cap \mathcal{C}$  is empty. To test the generalization performance of L2Dive to larger instances, we performed an additional ablation study and report the results in Appendix E.
Table 2: Dual reasoning in L2Dive tends to improve diving performance on capacitated facility location. Even selecting variables for diving uniformly at random outperforms the best heuristic diver, but using model confidence (and dual reasoning) facilitates significant improvements.
FAC. LOCATION
L2Dive	160 (10)
L2Dive (with sjconf)	164 (10)
L2Dive (with sjrand)	335 (14)
Best heuristic	484 (7)
5.2 L2Dive in branch and bound
With this second set of experiments, our goal was to use L2Dive within branch and bound to improve solver performance on real-world mixed integer linear programs. To this end, we included L2Dive into the open-source solver SCIP. We disabled all other diving heuristics in SCIP and dive with L2Dive from the root node. We found this to work well, but results for L2Dive may likely improve with a more subtle schedule for L2Dive or by integrating L2Dive into the solver's diving ensemble.
We considered two strongly contrasting applications from previous work. The first application deals with safely balancing workloads across servers in a large-scale distributed compute cluster. This problem is an instance of bin-packing with apportionment and can be formulated as a mixed integer linear program. We used the dataset from Gasse et al. [18] which contains 9900 instances for training and 100 instances for validation and testing respectively. Solving these instances to optimality is prohibitively hard<sup>2</sup> and we therefore set a time limit of  $T_{\mathrm{limit}} = 900$  seconds, both for data collection and test time evaluations. The second application deals with verifying the robustness of neural networks. This problem can be formulated as a mixed integer linear program [11, 44]. We considered the instances from Nair et al. [33], but used the same subset as Paulus et al. [35] who disregard trivial and numerically unstable instances. This dataset contains 2384 instances for training, 519 instances for validation and 545 instances for testing. These instances are challenging, but can mostly be solved within a reasonable time. We set a limit of  $T_{\mathrm{limit}} = 3600$  seconds, both for data collection and test time evaluations.
To assess solver performance, we measure solving time  $T$  for neural network verification and the primal-dual integral  $\Gamma_{pd}(T_{\mathrm{limit}})$  for server load balancing. Both measures fully account for the entire in-service overhead of L2Dive (e.g., computing the bipartite graph representation from the tree node, forward-propagating the generative model, diving etc.), because the L2Dive diver is directly included into SCIP and called by the solver during the branch and bound process. Our experiments were conducted on a shared distributed compute cluster. To reduce measurement variability, we ran all test time evaluations repeatedly on machines equipped with the same Intel Xeon Gold 5118 CPU 2.3 GHz processors for three different seedings of the solver. We batched evaluations randomly across instances and methods to be processed sequentially on the same machine. We report test set means and standard errors over the three different random seeds.
Table 3: L2Dive improves the performance of the branch and bound solver SCIP on real-world applications. When using L2Dive instead of standard divers, the average primal-dual integral for load balancing improves by  $7\%$  (35%) and solving time on neural network verification shrinks by  $20\%$  (29%) against the tuned (default) solver.
LOAD BALANCING	NEURAL NETWORK VERIF.
Primal-dual Integral	Wins	Solving Time	Wins
SCIP
Default	4,407 (34)	0 (0)	55.8 (2.3)	54 (5)
No diving	4,221 (21)	0 (0)	53.7 (0.6)	40 (4)
Tuned	3,067 (10)	7 (3)	49.9 (2.8)	164 (3)
L2Dive	2,863 (13)	93 (3)	39.8 (2.3)	287 (5)
L2Dive improved solver performance ad-hoc (Table 3, L2Dive). On load balancing, L2Dive improved the average primal-dual integral by over  $30\%$  from the solver at default settings (Table 3, Default). On neural network verification, L2Dive reduced the average solving time from approximately 56 seconds to less than 40 seconds  $(35\%)$ . As a control, we also ran SCIP without any diving and surprisingly found small improvements on both datasets (Table 3, No diving). The solver's default setting are calibrated on a general purpose set of mixed integer programs and are typically a challenging baseline to beat. However, our results suggest that SCIP's divers are either ineffective or may be poorly calibrated for these two applications. For this reason, we decided to tune the diving heuristics of the solver to seek an even more challenging baseline for comparison. We leveraged expert knowledge and random search to find strong diving ensembles in the vicinity of the default configuration. Then, we selected the best tuned solver configuration on a validation set using the same budget of solver calls that L2Dive expended for data collection. Details are in Appendix F. Our tuned solver baseline (Table 3, Tuned) significantly improved performance over Default, but was still outperformed by L2Dive. This highlights that our approach to learn specific divers may be more promising than fitting ensembles of generic diving heuristics to a particular application. Overall, L2Dive achieved the best average performance on 93 (out of 100) test instances for load balancing, and achieved the best
2Using a Xeon Gold 5118 CPU processor with  $2.3\mathrm{GHz}$  and 8 GB of RAM, none of the instances could be solved with SCIP 7.0.2 at default settings within an hour.
average performance on 287 (out of 545) test instances for neural network verification, more than the three SCIP configurations collectively.
We presented L2Dive to learn application-specific diving heuristics for branch and bound. Our approach combines ideas from generative modeling and relational learning with a profound understanding of integer programs and their solvers. We tested L2Dive on a range of applications including combinatorial optimization problems, workload apportionment and neural network verification. It found better feasible solutions than existing diving heuristics and facilitated improvements in overall solver performance. We view our work as yet another example that demonstrates the fruitful symbiosis of learning and search to design powerful algorithms.
This work presents a method to use machine learning for improving the performance of branch and bound solvers. Branch and bound is a powerful general-purpose method for solving mixed integer linear programs which appear frequently across business, science and technology. Therefore, we expect the impact of this work to be overwhelmingly beneficial. However, in cases where integer programming is exploited with ill intentions, our work may potentially have a harmful societal impact.
There are limitations in learning diving heuristics for specific applications. For example, in some cases the set of training instances may be small or collecting feasible solutions could be prohibitively expensive. In such cases, it may be desirable to transfer models from other applications or to utilize self-supervised representations that require fewer labelled examples for training [14]. This is a natural direction to explore in the future, for this and other work at the intersection of machine learning and integer programming. Alternatively, one may attempt to learn a universal diving heuristic using a diverse set of instances from a variety of applications. However, the extent to which machine learning can prove effective in this setting, for diving or other sub-routines, remains an open question.
MBP gratefully acknowledges support from the Max Planck ETH Center for Learning Systems. Resources used in preparing this research were provided, in part, by the Sustainable Chemical Processes through Catalysis (Suchcat) National Center of Competence in Research (NCCR).
[1] Tobias Achterberg, Timo Berthold, and Gregor Hendel. Rounding and propagation heuristics for mixed integer programming. In Operations research proceedings 2011, pages 71-76. Springer, 2012.
[2] Ravindra K Ahuja, James B Orlin, and Dushyant Sharma. New neighborhood search structures for the capacitated minimum spanning tree problem. 1998.
[3] Alejandro Marcos Alvarez, Quentin Louveaux, and Louis Wehenkel. A machine learning-based approximation of strong branching. INFORMS Journal on Computing, 29(1):185-195, 2017.
[4] Egon Balas, Stefan Schmieta, and Christopher Wallace. Pivot and shift—a mixed integer programming heuristic. *Discrete Optimization*, 1(1):3-12, 2004.
[5] Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: A methodological tour d'horizon. European Journal of Operational Research, 290(2):405-421, 2021. ISSN 0377-2217. doi: https://doi.org/10.1016/j.ejor.2020.07.063. URL https://www.sciencedirect.com/science/article/pii/S0377221720306895.
[6] Timo Berthold. Primal heuristics for mixed integer programs. 2006.
[7] Timo Berthold. Rens-relaxation enforced neighborhood search. 2007.
[8] Timo Berthold and Gregor Hendel. Learning to scale mixed-integer programs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3661–3668, 2021.
[49] Yaoxin Wu, Wen Song, Zhiguang Cao, and Jie Zhang. Learning large neighborhood search policy for integer programming. Advances in Neural Information Processing Systems, 34: 30075-30087, 2021.
[50] Kaan Yilmaz and Neil Yorke-Smith. Learning efficient search approximation in mixed integer branch and bound. arXiv preprint arXiv:2007.03948, 2020.
[51] Giulia Zarpellon, Jason Jo, Andrea Lodi, and Yoshua Bengio. Parameterizing branch-and-bound search trees to learn branching policies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3931-3939, 2021.
[52] Jiayi Zhang, Chang Liu, Xijun Li, Hui-Ling Zhen, Mingxuan Yuan, Yawen Li, and Junchi Yan. A survey for solving mixed integer programming via machine learning. Neurocomputing, 519: 205-217, 2023.
A Diving Heuristics
In Table 4, we briefly describe existing diving heuristics. In addition, SCIP's diving ensemble includes adaptive diving, conflict diving, objective pseudocost diving and guided diving. We did not compare against these heuristics in Section 5.1, because they either choose from the other heuristics (adaptive), were ineffective (conflict), require a feasible solution (guided) or do not use the generic diving algorithm (objective pseudocost). However, these divers are active for the baselines in Section 5.2. Berthold [6] gives a more detailed illustration of some of the divers.
Table 4: Overview of standard diving heuristics.
Diver	Description
Coefficient	Selects the variable with the minimal number of (positive) up-locks or down-locks and bounds it in the corresponding direction; ties are broken using Fractional diving.
Distribution	Selects a variable based on the solution density following [37].
Farkas	Attempts to construct a Farkas proof, Farkas diving bounds a variable in the direction that improves the objective value and selects the variable for which the improvement in the objective is largest.
Fractional	Selects the variable with the lowest fractionality |xj* - [xj* + 0.5]| in the current LP solution x* and bounds it in the corresponding direction.
Linesearch	Considers the ray originating at the LP solution of the root and passing through the current node's LP solution x*, selects the variable j whose coordinate hyperplane xj = [xj*] or xj = [xj*] is first intersected by the ray.
Pseudocost	Selects and bounds a variable based on its branching pseudocosts, its fractionality and the LP solutions at the root and the current node.
Vectorlength	Inspired by set partition constraints, selects a variable for which the quotient between the objective cost from bounding it and the number of constraint it appears in is smallest.
B Bipartite Graph and Graph Neural Network for L2Dive
B.1 Bipartite Graph
In our experiments, we represent a mixed integer linear program as a bipartite graph of variables and constraints. This idea is taken from Gasse et al. [17] and has been adopted by many others since. Each variable  $x_{j}$  and each constraint  $A_{i}$ . in the program gives rise to a node in a bipartite graph. For every non-zero coefficient  $A_{ij}$  in a linear constraint of the program the nodes of the corresponding variable  $j$  and constraint  $i$  are joined by an undirected edge. Each node is associated with a vector of features. We use the same features as Paulus et al. [35] but exclude cut-specific ones, such that variable nodes use 36 features and constraint nodes use 69 features in total.
Figure 2: Our model uses a bipartite graph representation to represent a mixed integer linear program as in [17]. Both variables and constraints of the program represent nodes in a bipartite graph. Edges in the graph correspond to non-zero coefficients in the linear constraint of the program. We use the same features for variables and constraints as in [35], but exclude cut-specific ones.
B.2 Graph Neural Network