SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Learning to Dive in Branch and Bound
Primal heuristics are important for solving mixed integer linear programs, because they find feasible solutions that facilitate branch and bound search. A prominent group of primal heuristics are diving heuristics. They iteratively modify and resolve linear programs to conduct a depth-first search from any node in the search tree. Existing divers rely on generic decision rules that fail to exploit structural commonality between similar problem instances that often arise in practice. Therefore, we propose L2Dive to learn specific diving heuristics with graph neural networks: We train generative models to predict variable assignments and leverage the duality of linear programs to make diving decisions based on the model's predictions. L2Dive is fully integrated into the open-source solver SCIP. We find that L2Dive outperforms standard divers to find better feasible solutions on a range of combinatorial optimization problems. For real-world applications from server load balancing and neural network verification, L2Dive improves the primal-dual integral by up to  $7\%$  ( $35\%$ ) on average over a tuned (default) solver baseline and reduces average solving time by  $20\%$  ( $29\%$ ).
Mixed integer linear programming problems are optimization problems in which some decision variables represent indivisible choices and thus must assume integer values. They arise in numerous industrial applications, e.g., workload apportionment to balance server loads. They can be used to solve combinatorial problems or to verify the robustness of neural networks [11, 44]. We write a mixed integer linear program  $M^{\dagger} \coloneqq (c, \bar{P}^{\dagger})$ , with  $A \in \mathbb{R}^{m \times n}$ ,  $b \in \mathbb{R}^{m}$ ,  $c, x, \in \mathbb{R}^{n}$ ,  $\underline{\pi}, \overline{\pi} \in \mathbb{R}_{\infty}^{n}$  and  $\mathcal{I} \subseteq \{1, \ldots, n\}$  indexing the variables restricted to be integrals, as
$$
z ^ {\dagger} := \min  _ {x \in P ^ {\dagger}} c ^ {\intercal} x, \quad P ^ {\dagger} = \left\{x \in \mathbb {R} ^ {n} \mid A x = b, \underline {{\pi}} \leq x \leq \bar {\pi}, x _ {j} \in \mathbb {Z} \forall j \in \mathcal {I} \right\} \tag {1}
$$
Typically, mixed integer linear programs are solved with a variant of branch and bound search [29, B&B]. This approach recursively builds a search tree, whose root represents the original problem in (1). Child nodes are created by introducing additional constraints (branching) to partition the set of feasible solutions. B&B uses bounds on the optimal solution to prune the tree and direct the search. To obtain strong upper bounds for  $z^{\dagger}$ , modern solvers typically rely on an array of primal heuristics. These are methods designed to quickly find good feasible solutions or an optimal solution  $x^{\dagger} \in \mathcal{X} := \{x \in P^{\dagger} \mid c^{\intercal} x = z^{\dagger}\}$ . Primal heuristics include problem-specific methods [e.g., 34, 30], variants of large neighborhood search [15, 13, 7, 38], rounding procedures [47, 4] or diving heuristics [see e.g., 6, 48].
Diving heuristics are a prominent group of primal heuristics. They are based on the linear program (LP) relaxation  $M^{*} \coloneqq (c, P^{*})$  of the problem in (1) given by
$$
z ^ {*} := \min  _ {x \in P ^ {*}} c ^ {\intercal} x, \quad P ^ {*} = \left\{x \in \mathbb {R} ^ {n} \mid A x = b, \underline {{\pi}} \leq x \leq \bar {\pi} \right\} \tag {2}
$$
(a) Traditional approach: Diving heuristics in branch and bound
Manual Design
Solver Integration
Collect  $\mathcal{X}\subseteq P^{\dagger}$
(b) Our approach: L2Dive
LP Duality
Figure 1: Traditional diving heuristics are based on generic manual heuristics and integrated efficiently into the branch and bound solver. In contrast, our approach L2Dive learns application-specific diving heuristics by collecting feasible solutions for a set of training instances to train a generative model. At test time, L2Dive uses the model predictions and leverages the duality of linear programs for diving. Finally, we integrate L2Dive into an open-source branch and bound solver.
Diving heuristics attempt to drive the LP solution  $x^{*} \in \mathcal{X}^{*} \coloneqq \arg \min c^{\top}x$  s.t.  $x \in P^{*}$  towards integrality. For this purpose, they conduct a depth-first search from any node in the search tree by repeatedly modifying and resolving linear programs. Diving heuristics are popular, because linear programs can typically be solved relatively fast and hence a number of diving heuristics have been proposed. However, standard divers rely on generic rules that fail to exploit problem-specific characteristics. This is particularly severe in applications, where similar problem instances are solved repeatedly and structural commonality exists. In this setting, learning is a promising alternative to design effective divers with the ultimate goal of improving solver performance.
We propose L2Dive to learn such application-specific diving heuristics. L2Dive collects good feasible solutions for some instances of a particular application and trains a generative model to minimize a variational objective on them. The model is a graph neural network that, for a given mixed integer linear program, predicts an assignment of the integer variables. At test time, L2Dive uses the model prediction and leverages the duality of linear programs to select variables for diving and tighten their bounds. We fully integrate L2Dive into the open-source solver SCIP and demonstrate the effectiveness of our approach in two sets of experiments. Our approach is illustrated in First, we compare L2Dive against existing diving heuristics on a common benchmark of combinatorial optimization problems and show that it finds better feasible solutions with the same diving budget. Second, we use L2Dive within branch and bound and test it on two real-world applications where we find that L2Dive improves overall solver performance. For server load balancing, it improves the average primal-dual integral by up to  $7\%$  ( $35\%$ ) over a tuned (default) solver baseline and in neural network verification it reduces average solving time by  $20\%$  ( $29\%$ ).
2.1 Diving Heuristics
Diving heuristics conduct a depth-first search in the branch and bound tree to explore a single root-leaf path. They iteratively modify and solve linear programs to find feasible solutions. Algorithm 1 illustrates a generic diving heuristic. A dive can be initiated from any node in the branch and bound tree and will return a (possibly empty) set of feasible solutions  $\mathcal{X}$  for the original mixed integer linear program in (1). Typically, diving heuristics alternate between tightening the bound of a single candidate variable  $x_{j}$  with  $j\in \mathcal{C}\subseteq \mathcal{I}$  (line 5) and resolving the modified linear program (line 7), possibly after propagating domain changes [1]. The resultant solution may be integral  $(x_{j}\in \mathbb{Z},\forall j\in \mathcal{I})$  or admit an integral solution via rounding [line 9, 1]. Eventually, the procedure is guaranteed to result in at least one primal-feasible solution (line 10) or an infeasible program (line 6). However, in practice solvers may prematurely abort a dive to curb its computational costs,
for example by imposing a maximal diving depth (line 2), an iteration limit on the LP solver or a bound on the objective value.
Diving heuristics feature prominently in integer programming. In contrast to large neighborhood search, they only require repeatedly solving linear programs (in complexity class P) instead of small integer programs (NP-hard). As a result, they tend to be considerably faster and more applicable in practice. Diving heuristics may also leverage sophisticated rounding methods in each iteration which makes them more likely to be successful. Various diving heuristics have been proposed. We briefly list and describe the most common ones in Appendix A. They mostly differ in how they select variables for bound tightening (line 3). For example, fractional diving selects the variable with the lowest fractionality  $|x_j^* - \lfloor x_j^* + 0.5 \rfloor|$  in the LP solution, while linesearch diving computes a score  $s_j$  for each variable based on the LP solutions at the current and the root node. All of these diving heuristics choose integer variables for diving from the same set of candidates  $\mathcal{C}$ . This set of divisible variables typically excludes integer variables that are slack variables [10, Chapter 1.1] or that have already been fixed ( $\pi = \overline{\pi}$ ). In general, diving heuristics apply to any mixed integer linear program. Empirically, however, standard divers often tend to be only effective for certain problems. For example, we found Farkas diving to deliver good results on a class of facility location problems, but to be ineffective for instances of maximum independent set (Section 5.1). As a result, modern solvers use an array of different diving heuristics during branch and bound. Selectively tuning this diving ensemble can be effective in improving solver performance on particular applications (Section 5). In contrast, our approach is to directly learn problem-specific diving heuristics for particular applications and demonstrate in our experiments that this yields improvements in solver performance over using a (tuned) ensemble of divers as is common practice.
Algorithm 1: Generic Diving Heuristic
Input:  $M^{\dagger}$  with relaxation  $M^{*}$ , maximal depth  $d_{\mathrm{max}}$   
Output:  $\mathcal{X}$ , a (possibly empty) set of feasible solutions  
Require:  $s$ , a scoring function to select variables for bound tightening
1  $d = 1$    
2 while  $d\leq d_{max}$  do   
3  $j = \arg \max_{j\in \mathcal{C}}s_j$    
4 Either  $\underline{\pi}_j\gets \lceil x_j^*\rceil$  or  $\overline{\pi}_j\gets \lfloor x_j^*\rfloor$    
5  $P^{*}\gets P^{*}\cap \{\underline{\pi}_{j}\leq x_{j}\leq \overline{\pi}_{j}\}$    
6 if  $P^{*}$  is infeasible then break;   
7  $x^{*} = \arg \min \{c^{\intercal}x\mid x\in P^{*}\}$    
8 if  $x^{*}$  is roundable then   
9  $\tilde{x} = \mathrm{round}(x^{*})$    
10  $\mathcal{X}\leftarrow \mathcal{X}\cup \{\tilde{x}\}$    
11 end   
12  $d\gets d + 1$    
13 Possibly update candidates C   
14 end
2.2 Solver Performance and Primal Performance
The most intuitive way to assess the performance of a solver for mixed integer linear programs is by measuring solving time, i.e., the average time it takes to solve a set of instances. However, realistic instances are often not solved to completion, because this may be prohibitively expensive or a particular application only requires bounds on the optimal solution, which the solver readily yields. In these cases, it is common to consider the primal-dual gap<sup>1</sup>:
$$
\gamma_ {p d} \left(\tilde {z}, \tilde {z} ^ {*}\right) = \left\{ \begin{array}{l l} \frac {\tilde {z} - \tilde {z} ^ {*}}{\operatorname* {m a x} \left(| \tilde {z} | , | \tilde {z} ^ {*} |\right)} & \text {i f} 0 <   \tilde {z} \tilde {z} ^ {*} <   \infty \\ 1 & \text {e l s e} \end{array} \right. \tag {3}
$$
<sup>1</sup>Note that we use a definition of the primal-dual gap that is used in SCIP 7.0.2 for the computation of the integral and do not quantify the gap or the integral in per cent.
Here,  $\tilde{z} \coloneqq c^{\mathsf{T}}\tilde{x}$  is the upper (also known as primal) bound given by the feasible solution  $\tilde{x}$  and  $\tilde{z}^{*}$  is a global lower (also known as dual) bound on the optimal solution  $z^{\dagger}$ . Performance can be measured by solving instances with a fixed cutoff time  $T$  and then computing the primal-dual gap  $\gamma_{pd}(\tilde{z}_T,\tilde{z}_T^*)$  where  $\tilde{z}_t^*$  and  $\tilde{z}_t$  denote the solver's best lower and upper bounds at time  $t$  (if non-existent, then  $-\infty$  and  $+\infty$  respectively).
Primal-dual integral Unfortunately, measuring the primal-dual gap at time  $T$  is susceptible to the particular choice of cutoff time. This is particularly troublesome, because the lower and upper bounds of branch and bound solvers tend to improve in a stepwise fashion. In order to alleviate this issue, it is common to integrate the primal-dual gap over the solving time and measure the primal-dual integral:
$$
\Gamma_ {p d} (T) = \int_ {t = 0} ^ {T} \gamma_ {p d} (\tilde {z} _ {t}, \check {z} _ {t} ^ {*}) d t \tag {4}
$$
Primal gap When directly comparing diving heuristics with each other, it can be useful to consider primal performance instead of solver performance. Primal performance assesses the quality of the feasible solution  $\tilde{x}$  a heuristic may find and can be measured by the primal gap:
$$
\gamma_ {p} (\tilde {z}) = \tilde {z} - z ^ {\dagger} \tag {5}
$$
Sometimes, the primal gap is normalized by  $|z^{\dagger}|$  which can be useful when  $\gamma_{p}$  is averaged across disparate instances. We do not normalize  $\gamma_{p}$  in this work.
3 Learning to Dive
We propose L2Dive to learn application-specific diving heuristics with graph neural networks. L2Dive uses a generative model to predict an assignment for the integer variables of a given mixed integer linear program. This model is learnt from a distribution of good feasible solutions collected initially for a set of training instances of a particular application. The model is a graph neural network closely related to the model in Gasse et al. [17]. It is trained to minimize a variational objective. At test time, L2Dive leverages insights from the duality of linear programs to select variables and tighten their bounds based on the model's predictions. We fully integrate L2Dive into the open-source solver SCIP.
3.1 Learning from feasible solutions
We propose to learn a generative model for good feasible solutions of a given instance  $M^{\dagger}$ . For this purpose, we first pose a conditional probability distribution over the variables  $x$ :
$$
\log p _ {\tau} \left(x \mid M ^ {\dagger}\right): \propto \left\{ \begin{array}{l l} - c ^ {\intercal} x / \tau & \text {i f} x \in \mathcal {X} \\ - \infty & \text {e l s e} \end{array} \right. \tag {6}
$$
The distribution  $p_{\tau}(x|M^{\dagger})$  is defined with respect to a set of good feasible solutions  $\mathcal{X}$  for the given instance. Solutions with a better objective are given larger mass as regulated by the temperature  $\tau$ , while solutions that are not known to be feasible or are not good  $(x \notin \mathcal{X})$  are given no probability mass. In practice, a model for diving will only need to make predictions on the divisible variables  $x_{\mathcal{C}}$  that are integral, non-fixed and not slack (Section 2). Hence, our model will target the marginal distribution  $p_{\tau}^{\mathcal{C}}(x_{\mathcal{C}}|M^{\dagger}) := \sum_{\tilde{x} \in \mathcal{X}} \mathbb{1}\{x_{\mathcal{C}} \in \tilde{x}\} p_{\tau}(x|M^{\dagger})$ .
Our goal is to learn a generative model  $q_{\theta}$  that closely approximates the distribution  $p_{\tau}^{\mathcal{C}}$ . The model  $q_{\theta}$  will be used to make predictions on unseen test instances for which  $p_{\tau}$  and  $\mathcal{X}$  are unknown. To learn a good generative model, our objective is to minimize the Kullback-Leibler divergence between  $p_{\tau}^{\mathcal{C}}$  and  $q_{\theta}$ ,
$$
\mathrm {K L} \left(p _ {\tau} ^ {\mathcal {C}} \mid \mid q _ {\theta}\right) = \sum p _ {\tau} ^ {\mathcal {C}} \left(x _ {\mathcal {C}} \mid M ^ {\dagger}\right) \log \left(\frac {p _ {\tau} ^ {\mathcal {C}} \left(x _ {\mathcal {C}} \mid M ^ {\dagger}\right)}{q _ {\theta} \left(x _ {\mathcal {C}} \mid M ^ {\dagger}\right)}\right) \tag {7}
$$
jointly over all training instances. The sum in (7) can be evaluated exactly, because the number of good feasible solutions in  $\mathcal{X}$  tends to be small. Our model for  $q_{\theta}$  is a variant of the graph neural
network described in Gasse et al. [17] and we give details in Appendix B. This model represents a mixed integer linear program as a bipartite graph of variables and constraints (Figure 3 in Appendix B) and its core are two bi-partite graph convolutions between variables and constraints (Figure 2 in Appendix B). Our variant uses some of the variable and constraint features from Paulus et al. [35] as well as batch normalization. It makes conditionally independent predictions for each integer variable, such that  $q_{\theta}(x_{\mathcal{C}}|M^{\dagger}) \coloneqq \prod_{j \in I} q_{\theta}(x_j|M^{\dagger})$ . For binary variables,  $q_{\theta}(x_j|M^{\dagger})$  is a Bernoulli distribution and the model outputs the mean parameter  $\theta$ . For general integer variables,  $q_{\theta}(x_j|M^{\dagger})$  is a sequence of Bernoulli distributions over the bitwise representation of the variable's integer domain and the model outputs a mean for each. Although conditionally independent predictions limit our model to unimodal distributions, this parameterization delivered strong empirical performance in our experiments (Section 5). It is possible to choose more delicate models for  $q_{\theta}$ , such as autoregressive models, but those will typically impose a larger cost for evaluations.
Solution augmentation At train time, we rely on the availability of a set of good feasible solutions  $\mathcal{X}$  for each instance. This is required to define  $p_{\theta}$  in (6) and evaluate the objective in (7) on all training instances. Several choices for  $\mathcal{X}$  are possible, and the effectiveness of our approach may depend on them. For example, if  $\mathcal{X}$  only contains poor feasible solutions, we cannot reasonably hope to learn any good integer variable assignments. The most obvious choice perhaps is to let  $\mathcal{X} = \{x^{\dagger}\}$ , where  $x^{\dagger}$  is the best solution the solver finds within a given time limit  $T$ . However, solvers are typically configured to not only store  $x^{\dagger}$ , but also a set number of its predecessors. Thus, alternatively some or all of the solutions in store could be used to define  $\mathcal{X}$  at no additional expense. Lastly, many instances of combinatorial optimization (e.g., set cover, independent set) exhibit strong symmetries and multiple solutions with the same objective value may exist as a result. These can be identified by using a standard solver to enumerate the solutions of an additional auxiliary mixed integer linear program as described in Appendix C. We used this method to augment solutions in  $\mathcal{X}$  for some of our experiments. This technique may be of independent interest, because the problem of handling symmetries is ubiquitous in machine learning for combinatorial optimization [28]. While it can be expensive to collect feasible solutions for each training instance regardless of the choice of  $\mathcal{X}$ , this cost may be curbed, because we do not require  $\mathcal{X}$  to contain the optimal solution. Moreover, the cost is incurred only once and ahead of training, such that all solver calls are embarassingly parallelizable across instances. In some cases, a practitioner may be able to draw on previous solver logs and not be required to expend additional budget for data collection. Finally, any training expense will be ultimately amortized in test time service.
3.2 Using a generative model for diving
At test time, we use our generative model  $q_{\theta}$  to predict an assignment for the divisible integer variables  $x_{\mathcal{C}}$  of a given instance. Typically, we will choose  $\hat{x}_{\mathcal{C}} = \arg \max q_{\theta}(x_{\mathcal{C}}|M^{\dagger})$  to predict an assignment, but assignments can also be sampled from the model, if multiple dives are attempted in parallel. To use the prediction for diving, we need to decide which variable to select (line 3 in Algorithm 1) and how to tighten its bounds (line 4). Ideally, our decision rules will admit a feasible solution at shallow depths, i.e., only a few bounds must be tightened to result in an integral or roundable solution to the diving linear program. Which variables should we tighten for this purpose and how? Compellingly, the theory of linear programming affords some insight:
Proposition 1. Let  $\tilde{x}$  be a feasible solution for  $M^{\dagger}$  as in (1). For the linear program  $M^{*}$ , its dual linear program  $M_D^*$  is defined in (11) in Appendix D. Let  $y^{*} \coloneqq (y_{b}^{*}, y_{\underline{\pi}}^{*}, y_{\overline{\pi}}^{*})$  be an optimal solution for  $M_D^*$ . Let  $\underline{J}(\tilde{x})$  and  $\overline{J}(\tilde{x})$  index the set of variables that violate complementary slackness (12) between  $\tilde{x}$  and  $y^{*}$ , such that
$$
\underline {{J}} (\tilde {x}) := \left\{j \mid \left(\tilde {x} _ {j} - \underline {{\pi}} _ {j}\right) y _ {\underline {{\pi}} _ {j}} ^ {*} > 0 \right\}
$$
$$
\overline {{J}} (\tilde {x}) := \left\{j \mid \left(\tilde {x} _ {j} - \bar {\pi} _ {j}\right) y _ {\bar {\pi} _ {j}} ^ {*} > 0 \right\}
$$
and define  $J(\tilde{x}) \coloneqq \underline{J}(\tilde{x}) \cup \overline{J}(\tilde{x})$ . Let  $M_J^* \coloneqq (c, P_J^*)$  be the linear program, where the bounds of all variables indexed by  $J(\tilde{x})$  are tightened, such that
$$
P _ {J} ^ {*} = P ^ {*} \cap \left\{x \in \mathbb {R} ^ {n} \mid x _ {j} \geq \tilde {x} _ {j} \forall j \in \underline {{J}} (\tilde {x}), x _ {j} \leq \tilde {x} _ {j} \forall j \in \overline {{J}} (\tilde {x}) \right\}
$$
Then,  $\tilde{x}$  is an optimal solution to the linear program  $M_J^*$ , i.e.,  $\tilde{x} \in \arg \min_{x \in P_J^*} c^\top x$ .
Proof.  $\tilde{x}$  is clearly a feasible solution for  $M_J^*$ .  $y^*$  is a feasible solution for the dual linear program of  $M_J^*$ , because it is feasible for  $M_D^*$ .  $\tilde{x}$  and  $y^*$  satisfy complementary slackness, hence  $\tilde{x}$  is optimal.
This suggests that for a prediction  $\hat{x}_{\mathcal{C}}$ , the bounds of variables in  $J(\hat{x}_{\mathcal{C}})$  should be tightened to restore complementary slackness. If the integer variable assignment  $\hat{x}_{\mathcal{C}}$  is feasible and the candidate set includes all integer variables, this will yield a diving linear program for which the assignment is optimal and that may be detected by the LP solver. Unfortunately, this is not guaranteed in the presence of slack variables (where typically  $\mathcal{C} \subset \mathcal{I}$ ) or if multiple optimal solutions exist (some of which may not be integer feasible). In practice, it may thus be necessary to tighten additional variables in  $\mathcal{C}$ . Inspired by Proposition 1, we propose the following dual reasoning rule to select to select a variable  $j^{*} \in \mathcal{C}$  for tightening
$$
j ^ {*} = \arg \max  _ {j \in \mathcal {C}} s _ {j} := q _ {\theta} (\hat {x} _ {j}) + \mathbb {1} \{j \in J (\hat {x} _ {\mathcal {C}}) \} \tag {8}
$$
This rule will select any variables in  $J(\hat{x}_{\mathcal{C}})$  before considering other variables for tightening. The score  $s$  breaks ties by selecting the variable in whose predictions the model is most confident in. Conveniently, the set  $J(\hat{x})$  can be easily computed on the fly from the dual values  $y^{*}$ , which standard LP solvers readily emit on every call at no additional expense. We tighten  $\underline{\pi}_j = \hat{x}_j$  if  $\hat{x}_j \leq x_j^*$  and we tighten  $\overline{\pi}_j = \hat{x}_j$  if  $\hat{x}_j \geq x_j^*$  to replace line 4 in Algorithm 1. We update the candidate set  $\mathcal{C}$  in line 13 to exclude variables whose value has been fixed, i.e.,  $\underline{\pi}_j = \overline{\pi}_j$  as usual. We validate dual reasoning in an ablation study in section 5.1.
3.3 Deployment