SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sirui Li*
siruil@mit.edu
Wenbin Ouyang*
oywenbin@mit.edu
Max B. Paulus
max.paulus@inf.ethz.ch
Cathy Wu
cathywu@mit.edu
Cutting planes are crucial in solving mixed integer linear programs (MILP) as they facilitate bound improvements on the optimal solution. Modern MILP solvers rely on a variety of separators to generate a diverse set of cutting planes by invoking the separators frequently during the solving process. This work identifies that MILP solvers can be drastically accelerated by appropriately selecting separators to activate. As the combinatorial separator selection space imposes challenges for machine learning, we learn to separate by proposing a novel data-driven strategy to restrict the selection space and a learning-guided algorithm on the restricted space. Our method predicts instance-aware separator configurations which can dynamically adapt during the solve, effectively accelerating the open source MILP solver SCIP by improving the relative solve time up to  $72\%$  and  $37\%$  on synthetic and real-world MILP benchmarks. Our work complements recent work on learning to select cutting planes and highlights the importance of separator management.
Mixed Integer Linear Programs (MILP) have been widely used in logistics [15], management [12], and production planning [17]. Modern MILP solvers typically employ a Branch-and-Cut (B&C) framework that utilizes a Branch-and-Bound (B&B) tree search procedure to partition the search space. As illustrated in Fig. 1, cutting plane algorithms are applied within each node of the B&B tree, tightening the Linear Programming (LP) relaxation of the node and improving the lower bound.
This paper presents a machine learning approach to accelerate MILP solvers. Modern MILP solvers implement various cutting plane algorithms, also referred to as separators, to generate cutting planes that tighten the LP solutions. Different separators have varying performance and execution times depending on the specific MILP instance. Typical solvers use simple heuristics to select separators, which can limit the ability to exploit commonalities across problem instances. While there is a growing body of work considering the 'branch' and 'cut' aspects of B&C [18, 35, 51, 42], profiling the open-source academic MILP solver SCIP [8], we find generating cutting planes through separators is a major contributor to the total solve time, and deactivating unused separators leads to faster solves and fewer B&B tree nodes. That is, a well-configured separator setup allows the selected cutting planes to more effectively tighten the LP solution, leading to fewer nodes in the B&B tree.
To our knowledge, the problem of how to leverage machine learning for this critical task of separator configuration, namely the selection of separators to activate and deactivate during the MILP solving process has not been considered. Therefore, the goal of this paper is to explore the extent to which
Figure 1: Separator Configuration in Branch-and-Cut (B&C). Modern MILP solvers perform Branch-and-Bound (B&B) tree search to solve MILPs. At each node of the B&B tree, cuts are added to tighten the Linear Programming (LP) relaxation of the MILP. To generate these cuts, a set of separators (e.g. Gomory) are invoked to first generate cuts into the cutpool  $\mathcal{C}^k$ . A subset of these cuts  $\mathcal{P}^k \subseteq \mathcal{C}^k$  are then selected and added to the LP. The process is repeated for several separation rounds at each node. While previous works study cut selection from a pre-determined cutpool, this work focuses on the upstream task of separator configuration to generate a high quality cutpool efficiently.
tailoring the separator configuration to the MILP instance in a data-driven manner can accelerate MILP solvers. The central challenge comes from the high dimensionality of the configuration search space (induced by the large number of separators and configuration steps), which we address by introducing a data-driven search space restriction strategy that balances model fitting and generalization. We further propose a learning-guided algorithm, which is cast into the framework of neural contextual bandit, as an effective means of optimizing configurations within the reduced search space.
Our contributions can be summarized as
- We identify separator management as a crucial component in B&C, and introduce the Separator Configuration task for selecting separators to accelerate solving MILPs.
- To overcome the high dimensionality of the configuration task, we propose a data-driven strategy, directly informed by theoretical analysis, to restrict the search space. We further design a learning method to tailor instance-aware configurations within the restricted space.
- Extensive computational experiments demonstrate that our method achieves significant speedup over the competitive MILP solver SCIP on a variety of benchmark MILP datasets and objectives. Our method further accelerates the state-of-the-art MILP solver Gurobi and uncovers known facts from literature regarding separator efficacy for different MILP classes.
The utilization of machine learning in MILP solvers has recently gained considerable attention. Various components in the B&B algorithm have been explored, including node selection [25, 49, 35], variable selection [18, 23, 58], branching rule [30, 23, 58, 46], scheduling primal heuristics [31, 26, 13], and deciding whether to apply Dantzig-Wolfe decomposition [34].
Our work is closely related to cutting plane selection, which can be achieved through heuristics [55, 2] or machine learning [51, 42]. The key difference, as shown in Figure 1 in Sec. 4 is that these works focus on selecting cutting planes from a pre-given cutpool generated by the available separators. That is, they consider the 'how to cut' question, whereas we focus on the equally crucial, but much less explored 'when (and what separators should we use) to cut' question [14, 16, 7]. For example, Wesselmann and Stuhl [55] state that they do not use any additional scheme to deactivate specific separators. In contrast, our work configures separators to generate a high-quality cutpool.
Another closely related line of work is on algorithm selection and parameter configurations [56, 57, 4, 5, 27, 28]. The most relevant works [57, 5] consider portfolio-based algorithm selection by first choosing a subset of algorithm parameter settings, and then selecting a parameter setting for each problem instance from the portfolio. We specialize and extend the general framework to separator configuration, by proposing a novel data-driven subspace restriction strategy, followed by a learning method, to configure separators for multiple separation rounds. We further present a theoretical analysis that directly informs our subspace restriction strategy, whereas the generalization guarantees from the prior work [5] is not informative for designing the portfolio-construction procedure.
It is common to restrict combinatorial space to improve the quality of solutions in discrete optimization. Previous research focuses primarily on decomposing large-scale problems, including heuristic works on Bender decomposition [44] and column generation [6], and recent learning-based works [50, 36] that train networks to select among a set of random or heuristic decomposition strategies. Our data-driven action space restriction strategy is general and could be of interest for a broader set of combinatorial optimization tasks, as well as other applications such as recommendation systems.
3 MILP Background
Mixed Integer Linear Programming (MILP). A MILP can be written as  $x^{*} = \arg \min \{c^{\mathsf{T}}x : Ax \leq b, x_{j} \in \mathbb{Z} \forall j \in I\}$ , where  $x \in \mathbb{R}^n$  is the set of  $n$  decision variables,  $A \in \mathbb{R}^{m \times n}$  and  $b \in \mathbb{R}^m$  formulate the set of  $m$  constraints, and  $c \in \mathbb{R}^n$  formulates the linear objective function.  $I \subseteq \{1,\dots,n\}$  defines the integer variables.  $x^{*} \in \mathbb{R}^{n}$  denotes the optimal solution to the MILP with an optimal objective value  $z^{*}$ .
Branch-and-Cut. State-of-the-art MILP solvers perform branch-and-cut (B&C) to solve MILPs, where a branch-and-bound (B&B) procedure is used to recursively partition the search space into a tree. Within each node of the B&B tree, linear programming (LP) relaxations of the MILP are solved to obtain lower bounds. B&C further invoke Cutting plane algorithms to tighten the LP relaxation.
Cutting Plane Separation. When the optimal solution  $x_{LP}^{*}$  to the LP relaxation is not a feasible solution to the original MILP, the cutting plane methods aim to find valid linear inequalities  $\nu^{\mathrm{T}}x\leq \omega$  (cuts) that separate  $x_{LP}^{*}$  from the convex hull of all feasible solutions of the MILP. Cutting plane separation happens in rounds, where each round  $k$  consists of the following steps (1) solving the current LP relaxation, (2) calling different separators to generate a set of cuts and add them to the cutpool  $C_k$ , (3) select a subset of cuts  $\mathcal{P}_k\subseteq \mathcal{C}_k$  and update the LP with the selected cuts. Detailed background information on separators in the B&C framework can be found in Appendix A.1.
4 Problem Formulation
Different separators are designed to exploit different structures of the solution polytope defined by the MILP instance. The solution polytope also varies at different separation rounds, as changes to the constraints (e.g. after a branch) lead to different structures and thus different effective separators. Moreover, multiple separators can combine to exploit more sophisticated structures. The inherently combinatorial nature of the problem hence presents a challenge in assigning the appropriate separators to each MILP instances. This work aims to enhance the MILP solving process via intelligent separator configuration. We formally introduce the separator configuration task as follows.
Definition 1 (Separator Configuration). Suppose the MILP solver implements  $M$  different separator algorithms. Given a set  $\mathcal{X}$  of  $N$  MILP instances (where  $|\mathcal{X}| = N$ ), and a maximum number of separation rounds  $R$  in a MILP solving process, we want to select a configuration  $s_{x,n} \in \{0,1\}^{M}$  for each instance  $x \in \mathcal{X}$  and separation round  $1 \leq n \leq R$ , where the  $w^{th}$  entry of  $s_{x,n}$  equaling one means we activate the  $w^{th}$  separator in separation round  $n$ , and equaling zero means we deactivate the  $w^{th}$  separator in the corresponding round.
Figure 1 illustrates the separator configuration task and highlights the difference between our task and the downstream cutting plane selection task in previous works [51, 42].
We measure the success of an algorithm for the separator configuration task by the relative time improvement from SCIP's default configuration. Denote a proposed configuration policy as  $\pi : \mathcal{X} \to \prod_{n=1}^{R} \{0,1\}^{M}$ , where for each MILP instance  $x \in \mathcal{X}$ , we have  $\pi(x) = \{s_{x,1}, \ldots, s_{x,R}\}$  as the proposed configurations. Let  $t_{\pi}(x)$  be the solve time of instance  $x$  using the configuration sequence  $\pi(x)$  and  $t_{0}(x)$  be the solve time using the default SCIP configuration (both to optimality or a fixed gap). We evaluate the effectiveness of  $\pi$  by the relative time improvement
$$
\Delta (\pi) := \mathbb {E} _ {x \in \mathcal {X}} [ \delta (\pi (x), x) ] \text {w h e r e} \delta (\pi (x), x) := \left(t _ {0} (x) - t _ {\pi} (x)\right) / t _ {0} (x) \tag {1}
$$
The search space for the separator configuration task is enormous, with a size of  $N \times 2^{M \times R}$ . SCIP contains  $M = 17$  separators, and a typical solve run yields  $R \geq 30$ , making the task highly challenging. In the next section, we discuss our data-driven approach to finding high quality configurations.
5 Learning to Separate
Two sources of high dimensionality in the search space come from (1) combinatorial number  $|O| = 2^M$  of configurations, where each element of  $O \coloneqq \{0,1\}^M$  is a combination of separators (e.g., Gomory, Clique) to activate, and (2) a large number of configuration updates that results in the  $|O|^R$  factor. We address the first challenge in Sec. 5.1 by restricting the number of configuration options, and the second challenge in Sec. 5.2 by reducing the frequency of configuration updates. The resulting restricted search space allows efficient learning in Sec. 5.3 to find high quality customized configurations for each MILP instance, which we term as instance-aware configurations.
5.1 Configuration space restriction
For simplicity, we first consider a single configuration update such that we apply the same configuration for all separation rounds, and our goal is to learn an instance-aware configuration predictor  $\tilde{f}:\mathcal{X}\to O$ . That is, we set  $s_{x,1} = \ldots = s_{x,R} = \tilde{f} (x)$  for each  $x\in \mathcal{X}$ ; Sec. 5.2 discusses extensions to multiple configuration updates. To address the challenge of learning the predictor in the high dimensional space  $O$ , we constrain the predictor  $\tilde{f}_A$  to select from a subset  $A\subseteq O$  of configurations with  $|A|$  reasonably small, i.e.  $\tilde{f}_A(x)\in A\forall x\in \mathcal{X}$ . We design a data-driven strategy, supported by theoretical rationale, to identify a subspace  $A$  for  $\tilde{f}_A$  to achieve high performance.
Preliminary definitions. Let  $\mathcal{X}$  be a class of MILP instances, and  $\mathcal{K} = \{x_1,\dots,x_K\} \subseteq \mathcal{X}$  be a given training set where we can acquire the time improvement  $\{\delta (s,x_i)\}_{i = 1..K;s\in O}$ . The true performance of  $\tilde{f}_A$  on  $\mathcal{X}$  is  $\Delta (\tilde{f}_A) = \mathbb{E}_{x\in \mathcal{X}}[\delta (\tilde{f}_A(x),x)]$ , and the empirical counterpart on  $\mathcal{K}$  is  $\hat{\Delta} (\tilde{f}_A) = \frac{1}{K}\sum_{i = 1}^{K}\delta (\tilde{f}_A(x_i),x_i)$ . We further denote the true instance-agnostic performance of applying a single configuration  $s\in \{0,1\} ^M$  to all MILP instances as  $\bar{\delta} (s) = \mathbb{E}_{x\in \mathcal{X}}[\delta (s,x)]$ , and the empirical counterpart as  $\hat{\bar{\delta}} (s) = \frac{1}{K}\sum_{i = 1}^{K}\delta (s,x_i)$ . Appendix A.2.1 details all relevant definitions.
Restriction algorithm. To find a subspace  $A$  that optimizes the true performance  $\Delta(\tilde{f}_A)$  for the predictor  $\tilde{f}_A$ , we employ the following training performance v.s. generalization decomposition:
$$
\Delta \left(\tilde {f} _ {A}\right) = \underbrace {\hat {\Delta} \left(\tilde {f} _ {A}\right)} _ {\text {t r a i n i n g p e r f .}} - \underbrace {\left(\hat {\Delta} \left(\tilde {f} _ {A}\right) - \Delta \left(\tilde {f} _ {A}\right)\right)} _ {\text {g e n e r a l i z a t i o n}} \tag {2}
$$
The first term measures how well  $\tilde{f}_A$  performs on the training set  $\mathcal{K}$ , while the second term reflects the generalization gap of  $\tilde{f}_A$  to the entire distribution  $\mathcal{X}$ . Notably, a similar trade-off exists in standard supervised learning [47], where regularizations are used to balance fitting and generalization by implicitly restricting the hypothesis class. Relatedly, in this problem, we can balance the two terms by explicitly restricting the output space of the predictor. Intuitively, a larger subspace  $A$  can improve training performance (more configuration options to leverage), but hurt generalization (more options that could perform poorly on unseen instances). This intuition is formalized next.
First, since the second term in Eq. (2) is unobserved, the following proposition imposes assumptions that allow us to restrict the configuration space. A detailed proof can be found in Appendix A.2.2.
Proposition 1. Assume the predictor  $\tilde{f}_A$ , when evaluated on the entire distribution  $\mathcal{X}$ , achieves perfect generalization (i.e., zero generalization gap) with probability  $1 - \alpha$ ; with probability  $\alpha$ , the predictor makes mistake and outputs a configuration  $s \in A$  uniformly at random. Then, the trainset performance v.s. generalization decomposition can be written as  $\Delta(\tilde{f}_A) = (1 - \alpha)\hat{\Delta}(\tilde{f}_A) + \alpha\frac{1}{|A|}\sum_{s \in A}\bar{\delta}(s)$ .
As  $\bar{\delta}(s)$  is also unobservable, we further rely on its empirical counterpart  $\hat{\bar{\delta}}_t(s)$  (see Appendix A.2.2 for a discussion of the reduction) and select the subspace  $A$  based on the following objective:
$$
(1 - \alpha) \hat {\Delta} (\tilde {f} _ {A}) + \alpha \frac {1}{| A |} \sum_ {s \in A} \hat {\delta} (s) \tag {3}
$$
The impact of the subspace  $A$  on these two terms further depends on the nature of  $\tilde{f}_A$ ; we assume that the predictor  $\tilde{f}_A$  uses empirical risk minimization (ERM) and performs optimally on the training set  $\mathcal{K}$ , i.e.  $\tilde{f}_A^{ERM}(x) = \arg \max_{s \in A} \delta(s, x) \forall x \in \mathcal{K}$ , hence bypassing the need to train any predictor for constructing  $A$ . The discussion of the ERM assumption's validity and the extension to predictors with training error are provided in Appendix A.2.4 (See Lemma 3 for the extension).
Eq. (3) then sheds light on how to construct a good  $A$  under the ERM assumption: an ideal subset  $A$  allows  $\tilde{f}_A^{ERM}$  to have (1) high training performance  $\hat{\Delta}(\tilde{f}_A^{ERM})$ , obtained when some configuration in  $A$  achieves good performance for any MILP instance in a training set, and (2) low generalization gap, achieved when each configuration in  $A$  has good performance across MILP instances in a test set, which we approximate with the average instance-agnostic performance on the training set  $\frac{1}{|A|} \sum_{s \in A} \hat{\delta}(s)$ . In fact, a larger or more diverse subspace  $A$  results in better  $\hat{\Delta}(\tilde{f}_A^{ERM})$ , as the ERM predictor can leverage more configuration options to improve the training set performance. Meanwhile, it may also lower  $\frac{1}{|A|} \sum_{s \in A} \hat{\delta}(s)$  which harms generalization, as we may include some configurations that perform poorly on most MILP instances but well on a small subset. The following proposition (proven in Appendix A.2.3) formalizes the diminishing marginal returns of  $\tilde{f}_A^{ERM}$ 's training performance with respect to  $A$ , which enables an efficient algorithm to construct  $A$ :
Proposition 2. The empirical performance of the ERM predictor  $\hat{\Delta} (\tilde{f}_A^{ERM})$  is monotone submodular, and a greedy strategy where we include the configuration that achieves the greatest marginal improvement  $\arg \max_{s\in \{0,1\} ^M\setminus A}\hat{\Delta} (\tilde{f}_{A\cup \{s\}}^{ERM}) - \hat{\Delta} (\tilde{f}_{A}^{ERM})$  at each iteration is a  $(1 - 1 / e)$ -approximation algorithm for constructing the subspace  $A$  that optimizes  $\hat{\Delta} (\tilde{f}_A^{ERM})$ .
To balance the two terms in Eq. (3), we couple the greedy selection strategy with a filtering criterion that eliminates configurations with poor instance-agnostic performance to construct the subspace  $A$ . Due to the high computational cost of calculating the marginal improvement for all  $2^{M}$  configurations, we first sample a large set  $S$  of configurations, which we use to construct the subspace  $A$ . Then, at each iteration, we expand the current set  $A$  with the configuration that produces the best marginal improvement in training performance, but only considering configurations  $s \in S$  whose empirical instance-agnostic performance is greater than a threshold, i.e.  $\hat{\delta}(s) > b$ . The extra filtering procedure enables us to improve the second term with small concessions in the first term. We continue the process while monitoring the two opposing terms, and terminate with a reasonably small  $A$  that balances the trade-off. The detailed algorithm and discussions of the filtering and termination procedure are provided in Appendix A.3.
5.2 Configuration update restriction
Learning to update configurations at each separation round is challenging due to cascading errors from a large number of updates. Instead, we periodically update the configuration at a few intermediate rounds and hold it fixed between updates: we perform  $k \ll R$  updates at rounds  $\{n_j\}_{j=1}^k$  with  $1 \leq n_j \leq R$ , and set  $s_{x,n_j} = \ldots = s_{x,n_{j+1}-1}$  for each  $1 \leq j \leq k$  and  $x \in \mathcal{X}$ . Fig. 7 (Left) in Appendix A.4 shows an example of the configuration update restriction with  $k = 2$  and  $R = 6$ , where we also discuss the trade-off of  $k$  in approximation v.s. estimation. We empirically find a small number of updates can already yield a decent time improvement (we set  $k = 2$  in Experiment Sec. 6).
We use a forward training algorithm [45] to learn the configuration policy  $\tilde{\pi}^{(k)}:\mathcal{X}\to \prod_{i = 1}^{k}\{0,1\} ^{M}$ . The algorithm decomposes the sequential task into  $k$  single configuration update tasks  $\tilde{\pi}^{(k)} = \{\tilde{f}_{\theta^T}^m\}_{m = 1}^k$ , where each  $\tilde{f}_{\theta^T}^j:\mathcal{X}\rightarrow \{0,1\} ^M$  is a separate network for the  $j$ -th configuration update. As illustrated in Fig.7 (Right) of Appendix A.4, at each iteration, we fix the weights of the trained networks for earlier updates  $\{\tilde{f}_{\theta T}^m\}_{m = 1}^{j - 1}$ , and train the network  $\tilde{f}_{\theta}^{j}$  for the  $j^{th}$  update. The detailed algorithm is provided in Alg. 2 of Appendix A.4. We incorporate the configuration space restriction in Sec. 5.1 by constraining each network  $\tilde{f}_{\theta}^{j}$  to select configurations from a subset  $A\subseteq O$ , such that  $\tilde{f}_{\theta}^{j}(x)\in A$  for all  $x\in \mathcal{X}$ . This reduces the search space from  $N\times 2^{M\times R}$  to  $N\times |A|\times k$ , significantly easing the learning process. Notably, we construct the subspace  $A$  once at the initial update for computational efficiency benefits, as it yields comparable performances to constructing a new subspace for each update. Further details and discussions can be found in Appendix A.7.2.
5.3 Neural UCB algorithm
Given the restricted configuration space  $A$ , we frame each configuration update as a contextual bandit problem with  $A$  arms (configurations). Conditional on the context (a MILP instance  $x \in \mathcal{X}$ ), each arm  $s \in A$  has a reward (time improvement  $\delta(s, x)$ ). We employ the neural UCB algorithm [59] to efficiently train a network  $\tilde{f}_{\theta^0}^j(x, s): \mathcal{X} \times A \to \mathbb{R}$  to estimate the reward, where the confidence
Figure 2: (a) Our triplet graph encoding of the MILP instance (the context) and the separator configuration (the arm / action). (b) Our neural architecture  $\tilde{f}_{\theta}$ . It involves three graph convolutions, an attention block for the separator nodes, and global poolings to extract the final score for reward prediction. We show the dimensionality of a tensor in gray if it is different from the previous size.