Title: Learning to Configure Separators in Branch-and-Cut
Venue: NeurIPS 2023
Accepted: yes
Keywords: Combinatorial Optimization

Branch-and-Cut

Learning Guided Optimization

Deep Learning
Abstract: Cutting planes are crucial in solving mixed integer linear programs (MILP) as they facilitate bound improvements on the optimal solution. Modern MILP solvers rely on a variety of separators to generate a diverse set of cutting planes by invoking the separators frequently during the solving process. This work identifies that MILP solvers can be drastically accelerated by appropriately selecting separators to activate. As the combinatorial separator selection space imposes challenges for machine learning, we *learn to separate* by proposing a novel data-driven strategy to restrict the selection space and a learning-guided algorithm on the restricted space. Our method predicts instance-aware separator configurations which can dynamically adapt during the solve, effectively accelerating the open source MILP solver SCIP by improving the relative solve time up to 72% and 37% on synthetic and real-world MILP benchmarks. Our work complements recent work on learning to select cutting planes and highlights the importance of separator management.

Learning to Configure Separators in Branch-and-Cut

Abstract

Cutting planes are crucial in solving mixed integer linear programs (MILP) as they facilitate bound improvements on the optimal solution. Modern MILP solvers rely on a variety of separators to generate a diverse set of cutting planes by invoking the separators frequently during the solving process. This work identifies that MILP solvers can be drastically accelerated by appropriately selecting separators to activate. As the combinatorial separator selection space imposes challenges for machine learning, we learn to separate by proposing a novel data-driven strategy to restrict the selection space and a learning-guided algorithm on the restricted space. Our method predicts instance-aware separator configurations which can dynamically adapt during the solve, effectively accelerating the open source MILP solver SCIP by improving the relative solve time up to 72% and 37% on synthetic and real-world MILP benchmarks. Our work complements recent work on learning to select cutting planes and highlights the importance of separator management.

1 Introduction

Mixed Integer Linear Programs (MILP) have been widely used in logistics [15], management [12], and production planning [17]. Modern MILP solvers typically employ a Branch-and-Cut (B&C) framework that utilizes a Branch-and-Bound (B&B) tree search procedure to partition the search space. As illustrated in Fig. 1, cutting plane algorithms are applied within each node of the B&B tree, tightening the Linear Programming (LP) relaxation of the node and improving the lower bound.

This paper presents a machine learning approach to accelerate MILP solvers. Modern MILP solvers implement various cutting plane algorithms, also referred to as separators , to generate cutting planes that tighten the LP solutions. Different separators have varying performance and execution times depending on the specific MILP instance. Typical solvers use simple heuristics to select separators, which can limit the ability to exploit commonalities across problem instances. While there is a growing body of work considering the 'branch' and 'cut' aspects of B&C [18, 35, 51, 42], profiling the open-source academic MILP solver SCIP [8], we find generating cutting planes through separators is a major contributor to the total solve time, and deactivating unused separators leads to faster solves and fewer B&B tree nodes. That is, a well-configured separator setup allows the selected cutting planes to more effectively tighten the LP solution, leading to fewer nodes in the B&B tree.

To our knowledge, the problem of how to leverage machine learning for this critical task of separator configuration, namely the selection of separators to activate and deactivate during the MILP solving process has not been considered. Therefore, the goal of this paper is to explore the extent to which

This Work:

Branch-and-BoundTree

SeparatorConfiguration

MILP x*=argmin{c'x:Ax≤b;x,∈ZVj ∈I}

CutpoolCk

11

11

Gomory

x≤1

1p

lp

Separators

Generate

Clique

13

x≤

13

->'x

Cuts:a'x≤b

Zero Half

V

s1

x≤∞

×

s1

V

x

sp

tailoring the separator configuration to the MILP instance in a data-driven manner can accelerate MILP solvers. The central challenge comes from the high dimensionality of the configuration search space (induced by the large number of separators and configuration steps), which we address by introducing a data-driven search space restriction strategy that balances model fitting and generalization. We further propose a learning-guided algorithm, which is cast into the framework of neural contextual bandit, as an effective means of optimizing configurations within the reduced search space.

Our contributions can be summarized as

We identify separator management as a crucial component in B&C, and introduce the Separator Configuration task for selecting separators to accelerate solving MILPs.

To overcome the high dimensionality of the configuration task, we propose a data-driven strategy, directly informed by theoretical analysis, to restrict the search space. We further design a learning method to tailor instance-aware configurations within the restricted space.

Extensive computational experiments demonstrate that our method achieves significant speedup over the competitive MILP solver SCIP on a variety of benchmark MILP datasets and objectives. Our method further accelerates the state-of-the-art MILP solver Gurobi and uncovers known facts from literature regarding separator efficacy for different MILP classes.

2 Related Work

The utilization of machine learning in MILP solvers has recently gained considerable attention. Various components in the B&B algorithm have been explored, including node selection [25, 49, 35], variable selection [18, 23, 58], branching rule [30, 23, 58, 46], scheduling primal heuristics [31, 26, 13], and deciding whether to apply Dantzig-Wolfe decomposition [34].

Our work is closely related to cutting plane selection, which can be achieved through heuristics [55, 2] or machine learning [51, 42]. The key difference, as shown in Figure 1 in Sec. 4, is that these works focus on selecting cutting planes from a pre-given cutpool generated by the available separators. That is, they consider the 'how to cut' question, whereas we focus on the equally crucial, but much less explored 'when (and what separators should we use) to cut' question [14, 16, 7]. For example, Wesselmann and Stuhl [55] state that they do not use any additional scheme to deactivate specific separators. In contrast, our work configures separators to generate a high-quality cutpool.

Another closely related line of work is on algorithm selection and parameter configurations [56, 57, 4, 5, 27, 28]. The most relevant works [57, 5] consider portfolio-based algorithm selection by first choosing a subset of algorithm parameter settings, and then selecting a parameter setting for each problem instance from the portfolio. We specialize and extend the general framework to separator configuration, by proposing a novel data-driven subspace restriction strategy, followed by a learning method, to configure separators for multiple separation rounds. We further present a theoretical analysis that directly informs our subspace restriction strategy, whereas the generalization guarantees from the prior work [5] is not informative for designing the portfolio-construction procedure.

It is common to restrict combinatorial space to improve the quality of solutions in discrete optimization. Previous research focuses primarily on decomposing large-scale problems, including heuristic works on Bender decomposition [44] and column generation [6], and recent learning-based works [50, 36] that train networks to select among a set of random or heuristic decomposition strategies. Our data-driven action space restriction strategy is general and could be of interest for a broader set of combinatorial optimization tasks, as well as other applications such as recommendation systems.

3 MILP Background

Mixed Integer Linear Programming (MILP). A MILP can be written as x ∗ = arg min { c ᵀ x : Ax ≤ b, x j ∈ Z ∀ j ∈ I } , where x ∈ R n is the set of n decision variables, A ∈ R m × n and b ∈ R m formulate the set of m constraints, and c ∈ R n formulates the linear objective function. I ⊆ { 1 , ..., n } defines the integer variables. x ∗ ∈ R n denotes the optimal solution to the MILP with an optimal objective value z ∗ .

Branch-and-Cut. State-of-the-art MILP solvers perform branch-and-cut (B&C) to solve MILPs, where a branch-and-bound (B&B) procedure is used to recursively partition the search space into a tree. Within each node of the B&B tree, linear programming (LP) relaxations of the MILP are solved to obtain lower bounds. B&C further invoke Cutting plane algorithms to tighten the LP relaxation.

Cutting Plane Separation. When the optimal solution x ∗ LP to the LP relaxation is not a feasible solution to the original MILP, the cutting plane methods aim to find valid linear inequalities ν ᵀ x ≤ ω (cuts) that separate x ∗ LP from the convex hull of all feasible solutions of the MILP. Cutting plane separation happens in rounds, where each round k consists of the following steps (1) solving the current LP relaxation, (2) calling different separators to generate a set of cuts and add them to the cutpool C k , (3) select a subset of cuts P k ⊆ C k and update the LP with the selected cuts. Detailed background information on separators in the B&C framework can be found in Appendix A.1.

4 Problem Formulation

Different separators are designed to exploit different structures of the solution polytope defined by the MILP instance. The solution polytope also varies at different separation rounds, as changes to the constraints (e.g. after a branch) lead to different structures and thus different effective separators. Moreover, multiple separators can combine to exploit more sophisticated structures. The inherently combinatorial nature of the problem hence presents a challenge in assigning the appropriate separators to each MILP instances. This work aims to enhance the MILP solving process via intelligent separator configuration. We formally introduce the separator configuration task as follows.

Definition 1 (Separator Configuration). Suppose the MILP solver implements M different separator algorithms. Given a set X of N MILP instances (where |X| = N ), and a maximum number of separation rounds R in a MILP solving process, we want to select a configuration s x,n ∈ { 0 , 1 } M for each instance x ∈ X and separation round 1 ≤ n ≤ R , where the w th entry of s x,n equaling one means we activate the w th separator in separation round n , and equaling zero means we deactivate the w th separator in the corresponding round.

Figure 1 illustrates the separator configuration task and highlights the difference between our task and the downstream cutting plane selection task in previous works [51, 42].

We measure the success of an algorithm for the separator configuration task by the relative time improvement from SCIP's default configuration. Denote a proposed configuration policy as π : X → ∏ R n =1 { 0 , 1 } M , where for each MILP instance x ∈ X , we have π ( x ) = { s x, 1 , ..., s x,R } as the proposed configurations. Let t π ( x ) be the solve time of instance x using the configuration sequence π ( x ) and t 0 ( x ) be the solve time using the default SCIP configuration (both to optimality or a fixed gap). We evaluate the effectiveness of π by the relative time improvement

The search space for the separator configuration task is enormous, with a size of N × 2 M × R . SCIP contains M = 17 separators, and a typical solve run yields R ≥ 30 , making the task highly challenging. In the next section, we discuss our data-driven approach to finding high quality configurations.

5 Learning to Separate

Two sources of high dimensionality in the search space come from (1) combinatorial number | O | = 2 M of configurations, where each element of O := { 0 , 1 } M is a combination of separators (e.g., Gomory, Clique) to activate, and (2) a large number of configuration updates that results in the | O | R factor. We address the first challenge in Sec. 5.1 by restricting the number of configuration options, and the second challenge in Sec. 5.2 by reducing the frequency of configuration updates. The resulting restricted search space allows efficient learning in Sec. 5.3 to find high quality customized configurations for each MILP instance, which we term as instance-aware configurations.

5.1 Configuration space restriction

For simplicity, we first consider a single configuration update such that we apply the same configuration for all separation rounds, and our goal is to learn an instance-aware configuration predictor ˜ f : X → O . That is, we set s x, 1 = ... = s x,R = ˜ f ( x ) for each x ∈ X ; Sec. 5.2 discusses extensions to multiple configuration updates. To address the challenge of learning the predictor in the high dimensional space O , we constrain the predictor ˜ f A to select from a subset A ⊆ O of configurations with | A | reasonably small, i.e. ˜ f A ( x ) ∈ A ∀ x ∈ X . We design a data-driven strategy, supported by theoretical rationale, to identify a subspace A for ˜ f A to achieve high performance.

Preliminary definitions. Let X be a class of MILP instances, and K = { x 1 , ..., x K } ⊆ X be a given training set where we can acquire the time improvement { δ ( s, x i ) } i =1 ..K ; s ∈ O . The true performance of ˜ f A on X is ∆ ( ˜ f A ) = E x ∈ X [ δ ( ˜ f A ( x ) , x )] , and the empirical counterpart on K is ˆ ∆ ( ˜ f A ) = 1 K ∑ K i =1 δ ( ˜ f A ( x i ) , x i ) . We further denote the true instance-agnostic performance of applying a single configuration s ∈ { 0 , 1 } M to all MILP instances as ¯ δ ( s ) = E x ∈ X [ δ ( s, x )] , and the empirical counterpart as ˆ ¯ δ ( s ) = 1 K ∑ K i =1 δ ( s, x i ) . Appendix A.2.1 details all relevant definitions.

Restriction algorithm. To find a subspace A that optimizes the true performance ∆ ( ˜ f A ) for the predictor ˜ f A , we employ the following training performance v.s. generalization decomposition:

The first term measures how well ˜ f A performs on the training set K , while the second term reflects the generalization gap of ˜ f A to the entire distribution X . Notably, a similar trade-off exists in standard supervised learning [47], where regularizations are used to balance fitting and generalization by implicitly restricting the hypothesis class. Relatedly, in this problem, we can balance the two terms by explicitly restricting the output space of the predictor. Intuitively, a larger subspace A can improve training performance (more configuration options to leverage), but hurt generalization (more options that could perform poorly on unseen instances). This intuition is formalized next.

First, since the second term in Eq. (2) is unobserved, the following proposition imposes assumptions that allow us to restrict the configuration space. A detailed proof can be found in Appendix A.2.2.

Proposition 1. Assume the predictor ˜ f A , when evaluated on the entire distribution X , achieves perfect generalization (i.e., zero generalization gap) with probability 1 -α ; with probability α , the predictor makes mistake and outputs a configuration s ∈ A uniformly at random. Then, the trainset performance v.s. generalization decomposition can be written as ∆ ( ˜ f A ) = (1 -α ) ˆ ∆ ( ˜ f A ) + α 1 | A | ∑ s ∈ A ¯ δ ( s ) .

As ¯ δ ( s ) is also unobservable, we further rely on its empirical counterpart ˆ ¯ δ t ( s ) (see Appendix A.2.2 for a discussion of the reduction) and select the subspace A based on the following objective:

The impact of the subspace A on these two terms further depends on the nature of ˜ f A ; we assume that the predictor ˜ f A uses empirical risk minimization (ERM) and performs optimally on the training set K , i.e. ˜ f ERM A ( x ) = arg max s ∈ A δ ( s, x ) ∀ x ∈ K , hence bypassing the need to train any predictor for constructing A . The discussion of the ERM assumption's validity and the extension to predictors with training error are provided in Appendix A.2.4 (See Lemma 3 for the extension).

Eq. (3) then sheds light on how to construct a good A under the ERM assumption: an ideal subset A allows ˜ f ERM A to have (1) high training performance ˆ ∆ ( ˜ f ERM A ) , obtained when some configuration in A achieves good performance for any MILP instance in a training set, and (2) low generalization gap, achieved when each configuration in A has good performance across MILP instances in a test set, which we approximate with the average instance-agnostic performance on the training set 1 | A | ∑ s ∈ A ˆ ¯ δ ( s ) . In fact, a larger or more diverse subspace A results in better ˆ ∆ ( ˜ f ERM A ) , as the ERM predictor can leverage more configuration options to improve the training set performance. Meanwhile, it may also lower 1 | A | ∑ s ∈ A ˆ ¯ δ ( s ) which harms generalization, as we may include some configurations that perform poorly on most MILP instances but well on a small subset. The following proposition (proven in Appendix A.2.3) formalizes the diminishing marginal returns of ˜ f ERM A 's training performance with respect to A , which enables an efficient algorithm to construct A :

Proposition 2. The empirical performance of the ERM predictor ˆ ∆ ( ˜ f ERM A ) is monotone submodular, and a greedy strategy where we include the configuration that achieves the greatest marginal improvement arg max s ∈ { 0 , 1 } M \ A ˆ ∆ ( ˜ f ERM A ∪ { s } ) -ˆ ∆ ( ˜ f ERM A ) at each iteration is a (1 -1 /e ) -approximation algorithm for constructing the subspace A that optimizes ˆ ∆ ( ˜ f ERM A ) .

To balance the two terms in Eq. (3), we couple the greedy selection strategy with a filtering criterion that eliminates configurations with poor instance-agnostic performance to construct the subspace A . Due to the high computational cost of calculating the marginal improvement for all 2 M configurations, we first sample a large set S of configurations, which we use to construct the subspace A . Then, at each iteration, we expand the current set A with the configuration that produces the best marginal improvement in training performance, but only considering configurations s ∈ S whose empirical instance-agnostic performance is greater than a threshold, i.e. ˆ ¯ δ ( s ) > b . The extra filtering procedure enables us to improve the second term with small concessions in the first term. We continue the process while monitoring the two opposing terms, and terminate with a reasonably small A that balances the trade-off. The detailed algorithm and discussions of the filtering and termination procedure are provided in Appendix A.3.

5.2 Configuration update restriction

Learning to update configurations at each separation round is challenging due to cascading errors from a large number of updates. Instead, we periodically update the configuration at a few intermediate rounds and hold it fixed between updates: we perform k /lessmuch R updates at rounds { n j } k j =1 with 1 ≤ n j ≤ R , and set s x,n j = ... = s x,n j +1 -1 for each 1 ≤ j ≤ k and x ∈ X . Fig. 7 (Left) in Appendix A.4 shows an example of the configuration update restriction with k = 2 and R = 6 , where we also discuss the trade-off of k in approximation v.s. estimation. We empirically find a small number of updates can already yield a decent time improvement (we set k = 2 in Experiment Sec. 6).

We use a forward training algorithm [45] to learn the configuration policy ˜ π ( k ) : X → ∏ k i =1 { 0 , 1 } M . The algorithm decomposes the sequential task into k single configuration update tasks ˜ π ( k ) = { ˜ f m θ T } k m =1 , where each ˜ f j θ T : X → { 0 , 1 } M is a separate network for the j -th configuration update. As illustrated in Fig.7 (Right) of Appendix A.4, at each iteration, we fix the weights of the trained networks for earlier updates { ˜ f m θ T } j -1 m =1 , and train the network ˜ f j θ for the j th update. The detailed algorithm is provided in Alg. 2 of Appendix A.4. We incorporate the configuration space restriction in Sec. 5.1 by constraining each network ˜ f j θ to select configurations from a subset A ⊆ O , such that ˜ f j θ ( x ) ∈ A for all x ∈ X . This reduces the search space from N × 2 M × R to N × | A | × k , significantly easing the learning process. Notably, we construct the subspace A once at the initial update for computational efficiency benefits, as it yields comparable performances to constructing a new subspace for each update. Further details and discussions can be found in Appendix A.7.2.

5.3 Neural UCB algorithm

Given the restricted configuration space A , we frame each configuration update as a contextual bandit problem with A arms (configurations). Conditional on the context (a MILP instance x ∈ X ), each arm s ∈ A has a reward (time improvement δ ( s, x ) ). We employ the neural UCB algorithm [59] to efficiently train a network ˜ f j θ 0 ( x, s ) : X × A → R to estimate the reward, where the confidence

(a)

(b)

Embedding

Convolution Convolution Convolution

V/C-Graph

S/V-Graph

S/C-Graph

SAttention

Mean-Pool

Concatenate

Embedding

Final

Variable

Constraint

mxd

m

mxd

hidden

Mean-Pool

hidden

×1

nodes

nodes

Co

C

C

pool

C

pxu

Vo

nxd

V1

hidden

V²-→V3

Mean-Pool

d

V

hidden

pool

×1

13d

Z

hidden

pool

×1

score

1×1

Separator

nodesS

sxd

s°→s1

sxd

hidden

S3

Attention

Mean-Pool

d

S

hidden

pool

×1

bound estimation is enabled by the small size of A . We provide the complete training procedure in Alg. 3 of Appendix A.5. At each training epoch t , we randomly sample P instances from X . For each instance, we sample D configurations based on the upper confidence bound ucb ( x, s ) , which combines a reward point estimate ˜ f j θ t -1 ( x, s ) and a confidence bound estimate. The confidence bound estimate incorporates the gradient ∇ θ ˜ f j θ t -1 ( x, s ) and a normalizing matrix Z t -1 only feasible to obtain when the number of arms is small. We run the MILP solver on each of the P × D pairs of instance-configuration and observe the reward labels. Lastly, we add all instance-configurationreward tuples { x, s, r } P × D to the data buffer and retrain the network ˜ f j θ . At test time, we select the configuration with the highest predicted reward s ∗ x,j = arg max s ∈ A ˜ f j θ T ( x, s ) or the highest UCB score s ∗ x,j = arg max s ∈ A ucb ( x, s ) , based on validation performance, at each update step n j . We provide further details on the inference strategy in Appendix A.6.1.

/negationslash

Context encoding. We encode the context for each MILP instance x and separator configuration s as a triplet graph with three types of nodes in the graph: variable nodes V , constraint nodes C , and separator nodes S . The variable and constraint nodes ( V , C ) appear in the previous works [18, 42]. We follow Paulus et al. [42] to use the same input features for V and C , and construct edges between them such that a variable node V i is connected to a constraint node C j if the variable appears in the constraint with the weight corresponds to the coefficient A ij = 0 . The separator nodes S are unique to our problem. We represent each configuration s by M separator nodes; each node S k has M +1 dimensional input features, representing whether the separator is activated (the first dimension), and which separator it is (one-hot M -dimensional vector). We connect each separator node with all variable and constraint nodes, all with a weight of 1 for complete pairwise message passing. We provide detailed descriptions of the input features in Appendix A.5.

Neural architecture ˜ f θ . We extend the architecture in Paulus et al. [42] for our network ˜ f θ ( x, s ) : X × S → R . The architecture, as illustrated in Fig. 2, involves a Graph Convolutional Network (GCN) [33], an attention block on the hidden embeddings of the separator nodes [48], and a global pooling to output a single score for reward prediction. It first embeds C , V , and S input features into hidden representations, and performs message passing following the directions of V → C → V , S → V → S , and S → C → S . Then, the S nodes pass through an attention module to emphasize the task of the separator configuration. Lastly, since we require the model to output a single score (in contrary to Paulus et al. [42] which outputs a score for each cut node), we perform a global mean pooling on each of the C , V , and S hidden embeddings to obtain three embedding vectors, concatenate them into a single vector, and finally use a multilayer perceptron (MLP) to map the vector into a scalar.

Clipped Reward Label. To account for variations in MILP solve time, we perform l MILP solver runs for each configuration-instance pair ( s, x ) and take the average time improvement as the unclipped reward label. Additionally, if a certain configuration s takes significantly longer solve time than SCIP default on a MILP instance x , we terminate the MILP solver run when the relative time improvement is less than a predefined threshold r min /lessmuch 0 to expedite data collection, and assign a clipped reward label of r clip ( s, x ) = ( ∑ i =1 ..l max { δ ( i ) ( s, x ) , r min } ) /l . Reward clipping also simplifies learning by obviating the need to accurately fit the exact value of extreme negative improvements, which may skew the network's prediction. As long as the prediction's sign is right, we will not select such a configuration with a negative predicted value during testing.

Loss function L . We use a L 2 loss between the prediction ˜ f θ ( x, s ) and the clipped reward label r clip :

6 Experiments and Analysis

Wedivide the experiment section into two main parts. First, we evaluate our method on standard MILP benchmarks from Tang et al. [51] and Ecole [43], where the number of variables and constraints range from 60 to 10 , 000 . We conduct detailed ablation studies to validate the design choices made for our method. Second, we examine the efficacy of our method by applying it to large-scale real-world MILP benchmarks, including the MIPLIB [20], NN Verification [40], and Load Balancing in the ML4CO challenges [19], where the number of variables and constraints reaches up to 65 , 000 . We omit certain MILP classes from the benchmarks with excessively short solve times, few generated cutting planes, or small dataset sizes. Appendix A.6.5 provides a detailed description of the datasets.

6.1 Setup

Evaluation Metric. As we aim to accelerate SCIP solving through separator configuration, we evaluate our learned configuration by the relative time improvement from SCIP default, defined in Eq. (1), when both are solved to optimality (for standard instances) or a fixed gap (for large-scale instances) as described in Appendix A.6.4. We report the median and standard deviation across all test instances, and defer mean and interquartile mean to Appendix A.8 as they yield similar results.

MLSetup. We train the networks with ADAM [32] under a learning rate of 10 -3 . The reward label collection is performed via multi-processing with 48 CPU processes. As in previous works [51, 42, 54], we train separate models for each MILP class. By default, we generate a training set K small of 100 instances for configuration space restriction, another training set K large of 800 for predictor network training, a validation set of 100 instances, and a test set of 100 instances for each class Appendix A.6 provides full details of the setup.

Baselines. To our knowledge, our separator configuration task has not been explored in previous research. We design the following baselines to assess the effectiveness of our proposed methods: (1) Default , where we run SCIP with the default parameters; (2) Random , where for each MILP instance x , we randomly sample a configuration s ∈ { 0 , 1 } M ; (3) Prune , where we first run SCIP default on the K small , and then at test time, we deactivate separators whose generated cutting planes are never applied to any instances in K small .

Proposed Methods. We evaluate the performances of our complete method and its sub-components: (1) Ours (L2Sep) , where we perform k = 2 instance-aware configuration updates per MILP instance

(Sec. 5.2). We use forward training to learn predictors via the neural UCB algorithm (Sec. 5.3) within the restricted configuration subspace A (Sec. 5.1). (2) Instance Agnostic Configuration , where we select a single configuration ˜ s with the best instance-agnostic performance ˆ ¯ δ (˜ s ) on K small from the initial large subset S for our space restriction algorithm ( | S | ≈ 2000 ); ˜ s is included in A . (3) Random within Restricted Subspace , where for each MILP instance, we select a random configuration within A . The latter two sub-components assess the quality of the restricted subspace and the benefit of learning instance-aware configurations. Further details can be found in Appendix A.6.1.

6.2 Standard MILP Benchmarks with Detailed Ablations

Performance. Table 1 presents the relative time improvement of different methods over SCIP default, on the datasets of Tang et al. and Ecole. Our method demonstrates a substantial speed up from SCIP default across all MILP classes, with a relative time improvement ranging from 25% to 70%. In contrast, the random baseline performs poorly, demonstrating that separator configuration is a nontrivial task. Meanwhile, although the pruning baseline generally outperforms SCIP default, its time improvement is significantly less than ours, confirming the efficacy of our proposed algorithm. Notably, both of our two heuristic sub-components achieve impressive speed-up from SCIP default, indicating the high quality of our restricted subspace (and a configuration within) to accelerate SCIP; additionally, our complete learning method outperforms the sub-components on all MILP classes, further underscoring the advantages of learning for instance-aware configurations.

We note that the high standard deviation, exhibited in all methods including SCIP default and also observed in the recent studies [54], is reasonable due to instance heterogeneity, as the standard deviation is calculated based on the time improvements across instances within each MILP dataset.

Ablations. In Table 2, we further conduct comprehensive ablation studies to assess the effectiveness of our learning method. The ablations are performed on four representative MILP classes in Ecole and Tang, covering a wide range of problem sizes and solve times. Appendix A.7 provides detailed descriptions as well as additional ablation results. We aim to answer the following questions: (i) Does the restricted config. space improve learning performance? (ii) How does the performance vary with fewer or more updates? (iii) Does the use of neural UCB lead to efficient predictor learning?

(i) Configuration space restriction (Sec. 5.1). We train our configuration predictors to select within a restricted subspace A constructed by a greedy strategy coupled with a filtering criterion. To evaluate the importance of the space restriction in learning high quality predictors, we perform an ablation study where we train the predictors to select within (1) the unrestricted space O = { 0 , 1 } M ( No Restr. ), and (2) a same-sized subspace A ′ constructed solely by the greedy strategy without filtering ( Greedy Restr. ). The restricted search space substantially enhances the learned predictors when compared to No Restr. , improving the median performance and lowering the standard deviation. We also observe the benefit of the filtering criterion when compared to Greedy Restr. . The filtering criterion excludes configurations with subpar instance-agnostic performance from entering the restricted configuration space, improving model generalization as demonstrated in our theoretical analysis.

(ii) Configuration update restriction (Sec. 5.2). We apply the forward training algorithm (Sec. 5.2) to perform two configuration updates ( k = 2 ) for each MILP instance. To examine the impact of fewer or more updates, we conduct an ablation study where we (1) performed a single update at round n 1 = 0 ( k = 1 ), and (2) added an additional third update at a later round n 3 ( k = 3 ). The results show that while a single update yields decent time improvement, adding the second update leads to further time savings. Meanwhile, we observe little improvement from the third update ( k = 3 ). We speculate that this is because the performance improvement primarily occurs during the early stages of a solve, and holding a fixed configuration for longer may be advantageous by making the solve process more stable. We leave further investigation of more configuration updates as a future work.

(iii) Neural UCB algorithm (Sec. 5.3). Our method employs the online neural UCB algorithm to improve training efficiency for configuration predictors. We present the ablation (1) where we train the predictor using an offline regression dataset whose size is four times as ours while training the model until convergence ( Supervise ( × 4 ) ); we conduct an additional ablation (2) where we train the predictor using neural contextual bandit with /epsilon1 -greedy exploration strategy ( /epsilon1 -greedy ). Our model performs comparably to Supervise ( × 4 ) while using significantly fewer data, highlighting the importance of the contextual bandit for improving training efficiency by collecting increasingly higher quality datasets online. The ablation results with /epsilon1 -greedy further confirm the benefit of the confidence bound estimation in neural UCB for more efficient contextual bandit exploration.

6.3 Large-scale Real-world MILP Benchmarks

The real-world datasets of MIPLIB, NN Verification, and Load Balancing present significant challenges due to the vast number of variables and constraints (on the order of 10 4 ), including nonstandard constraint types that MILP separators are not designed to handle. MIPLIB imposes a further challenge of dataset heterogeneity, as it contains a diverse set of instances from various application domains. Prior research [52] struggles to learn effectively on MIPLIB due to this heterogeneity, and a recent study [54] attempts to learn cutting plane selection over two homogeneous subsets (with 20 and 40 instances each). In contrast, we attempt to learn separator configuration across a larger MIPLIB subset that includes 443 of the 1065 instances in the original set, while carefully preserving the heterogeneity of the dataset. We provide our subset curation procedure in Appendix A.6.5.

Main Results. Table 3 presents the relative time improvement of various methods over SCIP default, on the large-scale real-world datasets. Again, our complete method displays a substantial speed up from SCIP default with a relative time improvement ranging from 12% to 37%. Our method also improves from our heuristic sub-components, further indicating the efficacy of our learning component on the challenging datasets. In contrast, the random baseline fails to improve from SCIP default, while the pruning baseline, despite having a reasonable median performance, suffers from a high standard deviation due to poor performance on many instances (See Appendix A.8.1 for IQM and mean results). Our results show the effectiveness of our learning method in improving the efficiency of practical applications that involve large-scale MILP optimization.

Although not a perfect comparison, for reference, we attempt to contextualize our result by examining the time improvement in the most comparable setting we found, which we provide comparison details in Appendix A.8.2: the learning method for cutting plane selection in Paulus et al. [42] achieves a median relative time improvement of 11 . 67% on the NN Verification dataset, and that in Wang et al. [54] obtains a 3% and 1% improvement in the solve time on two small homogeneous MIPLIB

subsets. While the comparison is far from perfect, our learning method for separator configuration achieves much higher time improvements of 37.5% on NN Verification and 12.9% on MIPLIB.

6.4 Interpretation Analysis: L2Sep Recovers Effective Separators from Literature

Bin Packing: It is known that instances with few bins approximate the Knapsack problem (Clique cuts are known to be effective [9]), and that instances with many bins approximate Bipartite Matching (Flowcover cuts can be useful [53]). We analyze the separators activated by L2Sep when we gradually decrease the number of bins, and observe that the prevalence of selected Clique and Flowcover cuts increased and decreased, respectively. This is illustrated in Fig 8 in Appendix A.8.3.

Other MILP Classes: We provide visualizations and interpretations for other MILP classes in Appendix A.8.3. Notably, Clique is known to be effective for Indep. Set [16]; L2Sep recovers this fact by frequently selecting configurations that activate Clique. Meanwhile, L2Sep discovers the instance heterogeneity of MIPLIB, resulting in a more dispersed distribution of selected configurations.

6.5 State-of-the-art MILP Solver Gurobi

We apply our method L2Sep with Gurobi, which contains a larger set of 21 separators. As Gurobi is closed-source, we cannot change configurations after the solving process starts, so we only consider one stage of separator configuration ( k = 1 ). As seen from Table 4, L2Sep achieves significant relative time improvements over the Gurobi default, with gains ranging from 12% to 56%. This result confirms the efficacy of L2Sep as an automatic instance-aware separator configuration method.

6.6 Additional Results

In Appendix A.8.5 and A.8.4, we further demonstrate (1) Separator Configuration has immediate and multi-step effects in the B&C Process. For instance, even though L2Sep does not modify branching, the branching solve time is reduced. (2) L2Sep is effective under an alternative objective, achieving 15%-68% relative gap improvements under fixed time limits.

7 Conclusion

This work identifies the opportunity of managing separators to improve MILP solvers, and further formulates and designs a learning-based method for doing so. We design a data-driven strategy, supported by theoretical analysis, to restrict the combinatorial space of separator configurations, and overall find that our learning method is able to improve the relative solve time (over the default solver) from 12% to 72% across a range of MILP benchmarks. In future work, we plan to apply our algorithm to more challenging MILP problems, particularly those that cannot be solved to optimality. We also aim to learn more fine-grained controls by increasing the frequency of separation configuration updates. Our algorithm is highly versatile, and we plan to investigate its potential to manage aspects of the MILP solvers, and further integrate with previous works on cutting plane selection. Our code is publicly available at https://github.com/mit-wu-lab/learning-to-configure-separators . We believe that our learning framework can be a powerful technique to enhance MILP solvers.

Acknowledgments and Disclosure of Funding

The authors would like to thank Mark Velednitsky and Alexandre Jacquillat for insightful discussions regarding an interpretative analysis of the learned model. This work was supported by a gift from Mathworks, the National Science Foundation (NSF) CAREER award (#2239566), the MIT Amazon Science Hub, and MIT's Research Support Committee. The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources that have contributed to the research results reported within this paper.

References

Tobias Achterberg. Constraint integer programming . PhD thesis, 2007.

Edoardo Amaldi, Stefano Coniglio, and Stefano Gualandi. Coordinated cutting plane generation via multi-objective separation. Mathematical Programming , 143:87-110, 2014.

Egon Balas, Sebastian Ceria, Gérard Cornuéjols, and N Natraj. Gomory cuts revisited. Operations Research Letters , 19(1):1-9, 1996.

Maria-Florina Balcan, Dan DeBlasio, Travis Dick, Carl Kingsford, Tuomas Sandholm, and Ellen Vitercik. How much data is sufficient to learn high-performing algorithms? generalization guarantees for data-driven algorithm design. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 919-932, 2021.

Maria-Florina Balcan, Tuomas Sandholm, and Ellen Vitercik. Generalization in portfoliobased algorithm selection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 12225-12232, 2021.

Cynthia Barnhart, Ellis L Johnson, George L Nemhauser, Martin WP Savelsbergh, and Pamela H Vance. Branch-and-price: Column generation for solving huge integer programs. Operations research , 46(3):316-329, 1998.

Timo Berthold, Matteo Francobaldi, and Gregor Hendel. Learning to use local cuts. arXiv preprint arXiv:2206.11618 , 2022.

Ksenia Bestuzheva, Mathieu Besançon, Wei-Kun Chen, Antonia Chmiela, Tim Donkiewicz, Jasper van Doornmalen, Leon Eifler, Oliver Gaul, Gerald Gamrath, Ambros Gleixner, et al. The scip optimization suite 8.0. arXiv preprint arXiv:2112.08872 , 2021.

Natashia Boland, Andreas Bley, Christopher Fricke, Gary Froyland, and Renata Sotirov. Cliquebased facets for the precedence constrained knapsack problem. Mathematical programming , 133:481-511, 2012.

Endre Boros, Yves Crama, and Peter L. Hammer. Chvátal cuts and odd cycle inequalities in quadratic 0-1 optimization. SIAM Journal on Discrete Mathematics , 5(2):163-177, 1992.

Alberto Caprara and Matteo Fischetti. { 0, 1/2 } -chvátal-gomory cuts. Mathematical Programming , 74:221-235, 1996.

Steven Cheng, Christine W Chan, and Gordon H Huang. An integrated multi-criteria decision analysis and inexact mixed integer linear programming approach for solid waste management. Engineering Applications of Artificial Intelligence , 16(5-6):543-554, 2003.

Antonia Chmiela, Elias Khalil, Ambros Gleixner, Andrea Lodi, and Sebastian Pokutta. Learning to schedule heuristics in branch and bound. Advances in Neural Information Processing Systems , 34:24235-24246, 2021.

Claudio Contardo, Andrea Lodi, and Andrea Tramontani. Cutting planes from the branchand-bound tree: Challenges and opportunities. INFORMS Journal on Computing , 35(1):2-4, 2023.

Eray Demirel, Neslihan Demirel, and Hadi Gökçen. A mixed integer linear programming model to optimize reverse logistics activities of end-of-life vehicles in turkey. Journal of Cleaner Production , 112:2101-2113, 2016.

Santanu S Dey and Marco Molinaro. Theoretical challenges towards cutting-plane selection. Mathematical Programming , 170:237-266, 2018.

Christodoulos A Floudas and Xiaoxia Lin. Mixed integer linear programming in process scheduling: Modeling, algorithms, and applications. Annals of Operations Research , 139: 131-162, 2005.

Maxime Gasse, Didier Chételat, Nicola Ferroni, Laurent Charlin, and Andrea Lodi. Exact combinatorial optimization with graph convolutional neural networks. Advances in neural information processing systems , 32, 2019.

Maxime Gasse, Simon Bowly, Quentin Cappart, Jonas Charfreitag, Laurent Charlin, Didier Chételat, Antonia Chmiela, Justin Dumouchelle, Ambros Gleixner, Aleksandr M Kazachkov, et al. The machine learning for combinatorial optimization competition (ml4co): Results and insights. In NeurIPS 2021 Competitions and Demonstrations Track , pages 220-231. PMLR, 2022.

Ambros Gleixner, Gregor Hendel, Gerald Gamrath, Tobias Achterberg, Michael Bastubbe, Timo Berthold, Philipp Christophel, Kati Jarck, Thorsten Koch, Jeff Linderoth, et al. Miplib 2017: data-driven compilation of the 6th mixed-integer programming library. Mathematical Programming Computation , 13(3):443-490, 2021.

Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv:1810.12715 , 2018.

Zonghao Gu, George L Nemhauser, and Martin WP Savelsbergh. Lifted flow cover inequalities for mixed 0-1 integer programs. Mathematical Programming , 85:439-467, 1999.

Prateek Gupta, Maxime Gasse, Elias Khalil, Pawan Mudigonda, Andrea Lodi, and Yoshua Bengio. Hybrid models for learning to branch. Advances in neural information processing systems , 33:18087-18097, 2020.

Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023. URL https://www. gurobi.com .

He He, Hal Daume III, and Jason M Eisner. Learning to search in branch and bound algorithms. Advances in neural information processing systems , 27, 2014.

Gregor Hendel, Matthias Miltenberger, and Jakob Witzig. Adaptive algorithmic behavior for solving mixed integer programs using bandit algorithms. In Operations Research Proceedings 2018: Selected Papers of the Annual International Conference of the German Operations Research Society (GOR), Brussels, Belgium, September 12-14, 2018 , pages 513-519. Springer, 2019.

Frank Hutter, Holger H Hoos, Kevin Leyton-Brown, and Thomas Stützle. Paramils: an automatic algorithm configuration framework. Journal of Artificial Intelligence Research , 36:267-306, 2009.

Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Learning and Intelligent Optimization: 5th International Conference, LION 5, Rome, Italy, January 17-21, 2011. Selected Papers 5 , pages 507-523. Springer, 2011.

Michael Jünger and Sven Mallach. Exact facetial odd-cycle separation for maximum cut and binary quadratic optimization. INFORMS Journal on Computing , 33(4):1419-1430, 2021.

Elias Khalil, Pierre Le Bodic, Le Song, George Nemhauser, and Bistra Dilkina. Learning to branch in mixed integer programming. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 30, 2016.

Elias B Khalil, Bistra Dilkina, George L Nemhauser, Shabbir Ahmed, and Yufen Shao. Learning to run heuristics in tree search. In Ijcai , pages 659-666, 2017.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations , 2017.

Markus Kruber, Marco E Lübbecke, and Axel Parmentier. Learning when to use a decomposition. In Integration of AI and OR Techniques in Constraint Programming: 14th International Conference, CPAIOR 2017, Padua, Italy, June 5-8, 2017, Proceedings 14 , pages 202-210. Springer, 2017.

Abdel Ghani Labassi, Didier Chételat, and Andrea Lodi. Learning to compare nodes in branch and bound with graph neural networks. arXiv preprint arXiv:2210.16934 , 2022.

Sirui Li, Zhongxia Yan, and Cathy Wu. Learning to delegate for large-scale vehicle routing. Advances in Neural Information Processing Systems , 34:26198-26211, 2021.

Stephen Maher, Matthias Miltenberger, João Pedro Pedroso, Daniel Rehfeldt, Robert Schwarz, and Felipe Serrano. PySCIPOpt: Mathematical programming in python with the SCIP optimization suite. In Mathematical Software - ICMS 2016 , pages 301-307. Springer International Publishing, 2016. doi: 10.1007/978-3-319-42432-3_37.

Alberto Maria Metelli, Flavio Mazzolini, Lorenzo Bisi, Luca Sabbioni, and Marcello Restelli. Control frequency adaptation via action persistence in batch reinforcement learning. In International Conference on Machine Learning , pages 6862-6873. PMLR, 2020.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Vinod Nair, Sergey Bartunov, Felix Gimeno, Ingrid Von Glehn, Pawel Lichocki, Ivan Lobov, Brendan O'Donoghue, Nicolas Sonnerat, Christian Tjandraatmadja, Pengming Wang, et al. Solving mixed integer programs using neural networks. arXiv preprint arXiv:2012.13349 , 2020.

George L Nemhauser, Laurence A Wolsey, and Marshall L Fisher. An analysis of approximations for maximizing submodular set functions-i. Mathematical programming , 14:265-294, 1978.

Max B Paulus, Giulia Zarpellon, Andreas Krause, Laurent Charlin, and Chris Maddison. Learning to cut by looking ahead: Cutting plane selection via imitation learning. In International conference on machine learning , pages 17584-17600. PMLR, 2022.

Antoine Prouvost, Justin Dumouchelle, Lara Scavuzzo, Maxime Gasse, Didier Chételat, and Andrea Lodi. Ecole: A gym-like library for machine learning in combinatorial optimization solvers. In Learning Meets Combinatorial Algorithms at NeurIPS2020 , 2020. URL https: //openreview.net/forum?id=IVc9hqgibyB .

Ragheb Rahmaniani, Teodor Gabriel Crainic, Michel Gendreau, and Walter Rei. The benders decomposition algorithm: A literature review. European Journal of Operational Research , 259 (3):801-817, 2017.

Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 661-668. JMLR Workshop and Conference Proceedings, 2010.

Lara Scavuzzo, Feng Chen, Didier Chételat, Maxime Gasse, Andrea Lodi, Neil Yorke-Smith, and Karen Aardal. Learning to branch with tree mdps. Advances in Neural Information Processing Systems , 35:18514-18526, 2022.

Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjing Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21 , pages 1548-1554, 8 2021.

Jialin Song, Ravi Lanka, Albert Zhao, Aadyot Bhatnagar, Yisong Yue, and Masahiro Ono. Learning to search via retrospective imitation. arXiv preprint arXiv:1804.00846 , 2018.

Jialin Song, Yisong Yue, Bistra Dilkina, et al. A general large neighborhood search framework for solving integer linear programs. Advances in Neural Information Processing Systems , 33: 20012-20023, 2020.

Yunhao Tang, Shipra Agrawal, and Yuri Faenza. Reinforcement learning for integer programming: Learning to cut. In International conference on machine learning , pages 9367-9376. PMLR, 2020.

Mark Turner, Thorsten Koch, Felipe Serrano, and Michael Winkler. Adaptive cut selection in mixed-integer linear programming. arXiv preprint arXiv:2202.10962 , 2022.

Mathieu Van Vyve. Fixed-charge transportation on a path: Linear programming formulations. In International Conference on Integer Programming and Combinatorial Optimization , pages 417-429. Springer, 2011.

Zhihai Wang, Xijun Li, Jie Wang, Yufei Kuang, Mingxuan Yuan, Jia Zeng, Yongdong Zhang, and Feng Wu. Learning cut selection for mixed-integer linear programming via hierarchical sequence model. In The Eleventh International Conference on Learning Representations , 2023.

Franz Wesselmann and Uwe Stuhl. Implementing cutting plane management and selection techniques. In Technical Report . University of Paderborn, 2012.

Lin Xu, Holger Hoos, and Kevin Leyton-Brown. Hydra: Automatically configuring algorithms for portfolio-based selection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 24, pages 210-216, 2010.

Lin Xu, Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Hydra-mip: Automated algorithm configuration and selection for mixed integer programming. In RCRA workshop on experimental evaluation of algorithms for solving problems with combinatorial explosion at the international joint conference on artificial intelligence (IJCAI) , pages 16-30, 2011.

Giulia Zarpellon, Jason Jo, Andrea Lodi, and Yoshua Bengio. Parameterizing branch-and-bound search trees to learn branching policies. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 3931-3939, 2021.

Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural contextual bandits with ucb-based exploration. In International Conference on Machine Learning , pages 11492-11502. PMLR, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors use machine learning to decide how and when to toggle the on/off switches for different cutting plane families provided by SCIP, the fastest open-source MIP solver. Their learning algorithm outperforms the default setting of SCIP on various benchmark sets.\n\nSTRENGTHS:\nThe techniques used to deal with the high dimensional combinatorial space of possible configurations are interesting. The results are promising and yield speedups over SCIP default on established benchmark sets.\n\nThe interactions between different cut families is not an extremely well understood topic. It is nice to know that machine learning can help decide what families to activate, and when.\n\nWEAKNESSES:\nUltimately I don’t think the proposed methodology is too novel, since it boils down to learning how to set on-off toggles for a variety of cut families. I acknowledge that this toggling is a critical component of tuning MIP solvers, but since this paper doesn’t mention what cut families ended up being selected and most useful for the different problem instances, I believe it leaves the most interesting question of what separator families worked best for which problems on the table. So methodologically this paper doesn’t seem too different from an array of previous algorithm configuration papers for MIP (e.g. Hydra-MIP by Xu, Hutter, Hoos, Leyton-Brown) that aim to tune parameters of a MIP solver from past data, but without any deeper principled investigation of what the parameters are actually doing. Hypothetically, it could be the case that turning off SCIP’s Gomory cut generation helps out for a class of MIPS, but maybe if the actual Gomory cut generation was tweaked performance would improve. Learning to toggle does not yield any such insights, as far as I can tell.\n\nQUESTIONS:\nThe title/phrase “learning to separate” slightly misleading. The *separation problem* refers to the specific problem of generating a cut (from some class of cuts) that cuts off the LP optimum. The authors here are not concerned with that problem, rather they are concerned with the on/off toggle for a particular family of cuts that is being generated via SCIP’s separation routines.\n\nThe authors discuss generalization guarantees but do not cite or compare to any prior work on such theoretical guarantees for cutting planes/integer programming/tree search. It might be worth mentioning how this compares to that work and where it is different (ostensibly “learning to toggle” falls into some of the frameworks studied previously).\n\nAs mentioned previously, one of the most interesting questions that is completely missing: what cut families ended up being selected and working well for the different problem instances? To me this would be the most interesting set of conclusions, since it is well known that MIP solvers need to be tuned to yield improved performance. The question of what cuts work best for what problems is much more elusive. Overall, I would be much more in favor of acceptance if there was some discussion about this aspect. Presumably this question is already answered by the experiments the authors ran, and including some observations here would be great.\n\nGiven that the authors are controlling granular on/off parameters (and not fundamentally modifying the underlying algorithms), why not compare to a state-of-the-art solver like Gurobi or CPLEX, which are free for academic use and significantly faster than SCIP?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a pipeline for a learning-enhanced cut separation management in modern MIP solvers (specifically, in the academic solver SCIP). To pick a promising subset of the large number of possible settings, the authors derive a data-driven and theoretically motivated method to focus on specific configurations that are expected to perform well. The actual policy on how cut separators are used during a MIP solving run are then learned over the restricted configuration space. The experimental results show significant solve-time improvements for a variety of benchmark MIP classes and test sets; this holds both when learning within MIP subclasses and when learning within heterogeneous benchmark sets such as MIPLIB. The presented work has the potential to replace current fixed, heuristic solver settings for cutting plane separation by carefully learned policies that often yield large speed-ups, which in turn can increase the size of problem instances that can be solved to optimality, and therefore represents a very notable advancement in data-driven MIP solving techniques.\n\n-- update: I have read and acknowledged all other reviews and the authors rebuttals, see discussion. --\n\nSTRENGTHS:\nBesides the impressive empirical results, the theoretical justification/motiviation for the configuration pre-selection is a strength of this work. Moreover, although the work spans quite many different and subtle aspects of the intricate workings of MIP solvers and employs different machine learning techniques, the authors did an excellent job in condensing their work into an understandable and well-written main paper; the supplementary material, while containing the technical proofs, is largely additional information that well complements the main paper but is mostly not necessary to follow the main paper.\n\nWEAKNESSES:\nIt appears to have become common practice to submit papers to NeurIPS (and ICML) whose actual, main content is put in a separate \"Supplementary Material\" document whose length far exceeds that of the supposed main paper. This paper is only a partial exception -- the main paper provides enough information and details to stand on its own except for the most technical bits, which can then be found in the very long Appendix (supplementary document; three times as long as the main paper) along with a host of additional information. Thus, it may be considered a weakness of the paper to have such a long Appendix, because this format bears the danger of the formally most important parts of the work (proofs; algorithm details and specific setups) not being reviewed thoroughly due to the short review period and high review load of reviewers at these conferences. I cannot exclude myself from this -- I simply did not have the time to rigorously check all the details in the long supplementary document, and therefore cannot give a definitive answer regarding the proofs' correctness beyond \"believing\" everything appears to be well in order. In this regard, I cannot help but wonder if a full journal paper would not be the better way to publish results that simply do not fit into the 9-page limit. But, again, I have seen papers that exploit the main paper/supplementary material split in a much worse way; for the present work, it actually hardly bothered me because the main paper is nearly self-sufficient. Nevertheless, if I have to point to a weakness, this aspect is what stands out to me.\n\nQUESTIONS:\n- there are several (minor) typos throughout paper and supplementary document (e.g., no comma after \"e.g.\", \"c.f.\" or \"v.s.\" instead of \"cf.\" and \"vs.\", \"constraint\" <-> \"constrained\" (caption of Fig.2), \"contrary\" <-> \"contrast\" (l.193), \"included A\" <-> \"included in A\" (l. 239), \"which we provide comparision\" <-> \"for which we provide comparison\" (l. 311), \"U Stuhl\" (ref. [47] -- what's the first name?))\n- some wordings are a bit inaccurate (at least when not having seen the supplementary document yet, i.e., when reading just the first paper):\n  * l.19: \"lower bound\" -- this implies that the problem is a minimization problem, which is not specified here (only in the supplement). Maybe clarify, or use \"dual bound\"?\n  * l. 66 and again later: The product-sign (\\prod , large Pi) should be a *Cartesian* product-sign\n  * Prop. 1 is referred to as \"lemma\" in the paragraph preceding it.\n   * Sect. 3.2: \"network\" kind of drops out of nowhere -- please clarify a bit here that neural networks are used for the discussed task\n   * in Tab. 1 and 3 at least: \"higher the better\" only applies to the median; for standard deviation, lower is better!\n   * is the reported standard deviation given in percent deviation from the default or as percent variation from the median values? how often, if at all, does reconfiguration actually slow down the solver? what are the mean/average deviations from default? \n- it would be good to clarify in the main paper that the separation rounds after which configuration updates occur are specified in advance; this only becomes clear in the supplementary document, so reading the main paper, one wonders how this is decided and/or if it is part of the learning procedure to decide when to update configurations. Also, are there cases in which the MIP solver terminates before the number k of updates has been performed?\n- have you tried reverting to the default configuration as the last update? \n- which LP solver was used by SCIP to solve the relaxations? SoPlex? \n- were all MIPs solved in single-thread mode?\n\nLIMITATIONS:\nLimitations have been appropriately addressed (at least in the supplementary material).", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies learning to manage separators to improve MILP solvers. Specifically, it learns a policy to determine which separators to use to generate cutting planes. The task is well formulated and experiments on many datasets demonstrate the effectiveness of the proposed model.\n\nSTRENGTHS:\n1.\tThe paper is well written. Especially the problem is well formulated.\n2.\tThe paper identifies the opportunity of managing separators to improve MILP solvers.\n3.\tExperiments on many datasets demonstrate the effectiveness.\n\nWEAKNESSES:\nIn general, I think this paper is an OK work, but it is just borderline to the NeurIPS bar, so I only give borderline accept.\n\nThere have been many works that study different parts of branch-and-bound, e.g., cut selection and node selection. Replacing one of the solving stages to a learning method, which leads to improvements compared with heuristic rules, is unsurprising. However, in real applications, it would be impossible to replace every part to learned models because of the memory limitation. Whether the studied part is critical enough in branch-and-bound is an important question. Therefore, separator configuration may not be in a trend of the research community, and thus the impact of the proposed method may be limited.\n\nI would give some suggestions in improving the work. First, the authors can compare the focused task, i.e., separator configuration, with other solving stages to demonstrate its significance. Or the authors can compare the resource usage to show the proposed method is lightweight enough so that it can be used jointly with other methods. Or the authors can try to merge the proposed framework with existing work and conduct enough ablation study. Summarily, the authors should denonstrate the real usefulness of the proposed method beyond only its performance.\n\nQUESTIONS:\nSee Weakness.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses machine learning to decide which families of cutting planes should be applied in each of a finite number of rounds when a discrete mathematical optimization problem is solved by an MILP solver. The authors conduct experiments on the SCIP solver, in which they show that their selection of separators (generators of different families of cutting planes) is able to solve problems faster than the default configurations of SCIP.\n\nSTRENGTHS:\nWithin the mathematical optimization community, it is known that many people have looked at this problem but few obtained good results, which shows in the limited number of direct references shown by the authors.\n\nThe results are certainly motivating, and the authors managed to analyze their work across a representative number of datasets.\n\n*****\n\nFollowing the rebuttal, I still have concerns about the significance of the work, which are similar to the ones presented in reviewer wvgK's review. For that reason, I cannot be more enthusiastic than a borderline accept for this paper.\n\nWEAKNESSES:\nI tend to consider the selection of separators a special case of what was done in prior work, in which the cuts themselves were selected. By using a setting in which their work is not comparable to prior ML studies on this topic, I am left wondering if what is proposed in this paper indeed leads to a better approach than the ones already known.\n\nThe paper explains very little about MILP, to the point that it fells like the application is an afterthought. While the appendix generously makes up for that, a reader unfamiliar with MILP might read the entire paper and not understand much about the application considered. In fact, for a span of approximately 3 pages (Line 56 in Page 2 to Line 175 in Page 5) there is very little that is specific to the application to MILP solvers. In great part that is because of using a language that is very different. I would not say that this is an issue of using ML instead of MILP terminology because I also got lost with the abstractions used, such as when the authors say \"single configuration update\" to mean that the same separators would be used in every round. This abstractness of the language left me with many questions about what exactly was done (see Questions). For someone with a greater interest in MILP as myself, I find it difficult to translate what the authors did to the application. As a consequence, I would have a hard time trying to reproduce their work if I wanted to.\n\nQUESTIONS:\n1) Can you compare your work with prior cut selection approaches?\n\n2) In plain terms, what exactly is the subset of configurations A and how is it pre-selected?\n\n3) What is the relevance of Propositions 1 and 2 in the context of MILP? \n\n4) How exactly are the separator nodes S used in the representation of the instances?\n\n5) How exactly do you consider instances in which neither SCIP nor skip with your separator selection were able to solve?\n\nFigure 1: The letter b is used both as the RHS of the set of constraints as well as the RHS of the next cut generated; also using A and a with different meanings (constraint LHS and cut LHS) does not seem advisable.\n\nEquation 2: Why is the term on the left repeated at the end of the equation? Is this a typo?\n\nLIMITATIONS:\nAlthough the authors do not explicitly acknowledge this, their focus on runtime for solving problems to optimality means that their approach is of little help precisely for the case in which it would be needed the most: problems in those benchmarks for which the provable optimal solution is not known. It is also unclear what happens in the case of the problems that do not finishing running on time: my guess is that timeout is counted equally as bad, although it would make more sense to look for the best solution found or the remaining optimality gap.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}