SlowGuess/NeurIPS_2023_OCR huggingface dataset:
bound estimation is enabled by the small size of  $A$ . We provide the complete training procedure in Alg. 3 of Appendix A.5. At each training epoch  $t$ , we randomly sample  $P$  instances from  $\mathcal{X}$ . For each instance, we sample  $D$  configurations based on the upper confidence bound  $ucb(x,s)$ , which combines a reward point estimate  $\tilde{f}_{\theta^{t-1}}^j(x,s)$  and a confidence bound estimate. The confidence bound estimate incorporates the gradient  $\nabla_{\theta} \tilde{f}_{\theta^{t-1}}^j(x,s)$  and a normalizing matrix  $Z_{t-1}$  only feasible to obtain when the number of arms is small. We run the MILP solver on each of the  $P \times D$  pairs of instance-configuration and observe the reward labels. Lastly, we add all instance-configuration-reward tuples  $\{x,s,r\}_{P \times D}$  to the data buffer and retrain the network  $\tilde{f}_{\theta}^j$ . At test time, we select the configuration with the highest predicted reward  $s_{x,j}^* = \arg \max_{s \in A} \tilde{f}_{\theta^T}^j(x,s)$  or the highest UCB score  $s_{x,j}^* = \arg \max_{s \in A} ucb(x,s)$ , based on validation performance, at each update step  $n_j$ . We provide further details on the inference strategy in Appendix A.6.1.
Context encoding. We encode the context for each MILP instance  $x$  and separator configuration  $s$  as a triplet graph with three types of nodes in the graph: variable nodes  $\mathbf{V}$ , constraint nodes  $\mathbf{C}$ , and separator nodes  $\mathbf{S}$ . The variable and constraint nodes  $(\mathbf{V}, \mathbf{C})$  appear in the previous works [18, 42]. We follow Paulus et al. [42] to use the same input features for  $\mathbf{V}$  and  $\mathbf{C}$ , and construct edges between them such that a variable node  $\mathbf{V}_i$  is connected to a constraint node  $\mathbf{C}_j$  if the variable appears in the constraint with the weight corresponds to the coefficient  $\mathbf{A}_{ij} \neq 0$ . The separator nodes  $\mathbf{S}$  are unique to our problem. We represent each configuration  $s$  by  $M$  separator nodes; each node  $\mathbf{S}_k$  has  $M + 1$  dimensional input features, representing whether the separator is activated (the first dimension), and which separator it is (one-hot  $M$ -dimensional vector). We connect each separator node with all variable and constraint nodes, all with a weight of 1 for complete pairwise message passing. We provide detailed descriptions of the input features in Appendix A.5.
Neural architecture  $\tilde{f}_{\theta}$ . We extend the architecture in Paulus et al. [42] for our network  $\tilde{f}_{\theta}(x,s): \mathcal{X} \times \mathcal{S} \to \mathbb{R}$ . The architecture, as illustrated in Fig. 2, involves a Graph Convolutional Network (GCN) [33], an attention block on the hidden embeddings of the separator nodes [48], and a global pooling to output a single score for reward prediction. It first embeds  $\mathbf{C}$ ,  $\mathbf{V}$ , and  $\mathbf{S}$  input features into hidden representations, and performs message passing following the directions of  $\mathbf{V} \rightarrow \mathbf{C} \rightarrow \mathbf{V}$ ,  $\mathbf{S} \rightarrow \mathbf{V} \rightarrow \mathbf{S}$ , and  $\mathbf{S} \rightarrow \mathbf{C} \rightarrow \mathbf{S}$ . Then, the  $\mathbf{S}$  nodes pass through an attention module to emphasize the task of the separator configuration. Lastly, since we require the model to output a single score (in contrary to Paulus et al. [42] which outputs a score for each cut node), we perform a global mean pooling on each of the  $\mathbf{C}$ ,  $\mathbf{V}$ , and  $\mathbf{S}$  hidden embeddings to obtain three embedding vectors, concatenate them into a single vector, and finally use a multilayer perceptron (MLP) to map the vector into a scalar.
Clipped Reward Label. To account for variations in MILP solve time, we perform  $l$  MILP solver runs for each configuration-instance pair  $(s,x)$  and take the average time improvement as the unclipped reward label. Additionally, if a certain configuration  $s$  takes significantly longer solve time than SCIP default on a MILP instance  $x$ , we terminate the MILP solver run when the relative time improvement is less than a predefined threshold  $r_{\mathrm{min}} \ll 0$  to expedite data collection, and assign a clipped reward label of  $r^{\mathrm{clip}}(s,x) = (\sum_{i=1..l} \max \{\delta^{(i)}(s,x), r_{\mathrm{min}}\}) / l$ . Reward clipping also simplifies learning by obviating the need to accurately fit the exact value of extreme negative improvements, which may skew the network's prediction. As long as the prediction's sign is right, we will not select such a configuration with a negative predicted value during testing.
Table 1: Tang et al. and Ecole. Absolute solve time of SCIP default, and the median (higher the better) and standard deviation (in parentheses) of relative time improvement of different methods.
Method	Tang	Ecole
Bin. Pack.	Max. Cut	Pack.	Comb. Auc.	Indep. Set	Fac. Loc.
Default Time (s)	0.076s(0.131s)	1.77s(0.56s)	8.82s(25.46s)	2.73s(4.43s)	8.21s(114.15s)	61.1s(55.37s)
HeuristicBaselines	Default	0%	0%	0%	0%	0%	0%
Random	-23.4%(153.8%)	-108.4%(168.2%)	-91%(127.8%)	-48.6%(159.0%)	-5%(161.4%)	-33.3%(157.2%)
Prune	13.9%(27.0%)	2.7%(26.2%)	6.6%(45.0%)	12.3%(24.2%)	18.0%(24.2%)	24.7%(47.9%)
OursHeuristicVariants	Inst. AgnosticConfiguration	33.7%(36.6%)	69.8%(10.5%)	20.1%(38.0%)	60.1%(27.6%)	57.8%(29.5%)	11.5%(21.8%)
Random withinRestr. Subspace	26.9%(33.6%)	68.0%(11.0%)	18.8%(38.7%)	58.1%(28.7%)	57.4%(75.8%)	17.7%(33.0%)
OursLearned	L2Sep	42.3%(34.2%)	71.9%(11.3%)	28.5%(39.3%)	66.2%(26.2%)	72.4%(27.8%)	29.4%(39.6%)
Loss function  $\mathcal{L}$ . We use a  $L_{2}$  loss between the prediction  $\tilde{f}_{\theta}(x,s)$  and the clipped reward label  $r^{\mathrm{clip}}$ :
$$
L \left(\tilde {f} _ {\theta} (x, s), r\right) = \left(\tilde {f} _ {\theta} (x, s) - r ^ {\mathrm {c l i p}}\right) ^ {2} \tag {4}
$$
6 Experiments and Analysis
We divide the experiment section into two main parts. First, we evaluate our method on standard MILP benchmarks from Tang et al. [51] and Ecole [43], where the number of variables and constraints range from 60 to 10,000. We conduct detailed ablation studies to validate the design choices made for our method. Second, we examine the efficacy of our method by applying it to large-scale real-world MILP benchmarks, including the MIPLIB [20], NN Verification [40], and Load Balancing in the ML4CO challenges [19], where the number of variables and constraints reaches up to 65,000. We omit certain MILP classes from the benchmarks with excessively short solve times, few generated cutting planes, or small dataset sizes. Appendix A.6.5 provides a detailed description of the datasets.
Evaluation Metric. As we aim to accelerate SCIP solving through separator configuration, we evaluate our learned configuration by the relative time improvement from SCIP default, defined in Eq. (1), when both are solved to optimality (for standard instances) or a fixed gap (for large-scale instances) as described in Appendix A.6.4. We report the median and standard deviation across all test instances, and defer mean and interquartile mean to Appendix A.8 as they yield similar results.
ML Setup. We train the networks with ADAM [32] under a learning rate of  $10^{-3}$ . The reward label collection is performed via multi-processing with 48 CPU processes. As in previous works [51, 42, 54], we train separate models for each MILP class. By default, we generate a training set  $\mathcal{K}_{small}$  of 100 instances for configuration space restriction, another training set  $\mathcal{K}_{large}$  of 800 for predictor network training, a validation set of 100 instances, and a test set of 100 instances for each class Appendix A.6 provides full details of the setup.
Baselines. To our knowledge, our separator configuration task has not been explored in previous research. We design the following baselines to assess the effectiveness of our proposed methods: (1) Default, where we run SCIP with the default parameters; (2) Random, where for each MILP instance  $x$ , we randomly sample a configuration  $s \in \{0, 1\}^M$ ; (3) Prune, where we first run SCIP default on the  $\kappa_{small}$ , and then at test time, we deactivate separators whose generated cutting planes are never applied to any instances in  $\kappa_{small}$ .
Proposed Methods. We evaluate the performances of our complete method and its sub-components: (1) Ours (L2Sep), where we perform  $k = 2$  instance-aware configuration updates per MILP instance
Table 2: Detailed ablations of different components in our L2Sep algorithm. Learning with neural UCB in the restricted config. space and performing  $k = 2$  config. updates achieves the best result.
Ablation Method	Config. Space Restriction	Config. Update Restriction	Neural Contextual Bandit	Ours: L2Sep
No Restr.	Greedy Restr.	k=1	k=3	Supervise (×4)	ε-greedy	w/ Restr. + k=2 + UCB
Bin. Pack.	18.6% (125.3%)	35.8% (35.6%)	40.4% (42.3%)	44.2% (32.4%)	40.2% (19.7%)	36.3% (32.3%)	42.3% (34.2%)
Pack.	19.6% (61.1%)	18.4% (49.8%)	23.8% (38.1%)	27.8% (38.1%)	24.0% (44.1%)	25.1% (44.3%)	28.5% (39.3%)
Indep. Set	38.6% (23.5%)	68.5% (28.1%)	70.2% (38.6%)	69.7% (29.1%)	68.7% (33.9%)	64.1% (48.7%)	72.4% (27.8%)
Fac. Loc.	15.5% (121.2%)	27.1% (38.7%)	20.1% (37.8%)	29.7% (29.8%)	31.0% (41.5%)	28.1% (23.6%)	29.4% (39.6%)
(Sec. 5.2). We use forward training to learn predictors via the neural UCB algorithm (Sec. 5.3) within the restricted configuration subspace  $A$  (Sec. 5.1). (2) Instance Agnostic Configuration, where we select a single configuration  $\tilde{s}$  with the best instance-agnostic performance  $\hat{\tilde{\delta}}(\tilde{s})$  on  $\mathcal{K}_{small}$  from the initial large subset  $S$  for our space restriction algorithm  $(|S| \approx 2000)$ ;  $\tilde{s}$  is included in  $A$ . (3) Random within Restricted Subspace, where for each MILP instance, we select a random configuration within  $A$ . The latter two sub-components assess the quality of the restricted subspace and the benefit of learning instance-aware configurations. Further details can be found in Appendix A.6.1.
6.2 Standard MILP Benchmarks with Detailed Ablations
Performance. Table1 presents the relative time improvement of different methods over SCIP default, on the datasets of Tang et al. and Ecole. Our method demonstrates a substantial speed up from SCIP default across all MILP classes, with a relative time improvement ranging from  $25\%$  to  $70\%$ . In contrast, the random baseline performs poorly, demonstrating that separator configuration is a nontrivial task. Meanwhile, although the pruning baseline generally outperforms SCIP default, its time improvement is significantly less than ours, confirming the efficacy of our proposed algorithm. Notably, both of our two heuristic sub-components achieve impressive speed-up from SCIP default, indicating the high quality of our restricted subspace (and a configuration within) to accelerate SCIP; additionally, our complete learning method outperforms the sub-components on all MILP classes, further underscoring the advantages of learning for instance-aware configurations.
We note that the high standard deviation, exhibited in all methods including SCIP default and also observed in the recent studies [54], is reasonable due to instance heterogeneity, as the standard deviation is calculated based on the time improvements across instances within each MILP dataset.
Ablations. In Table 2, we further conduct comprehensive ablation studies to assess the effectiveness of our learning method. The ablations are performed on four representative MILP classes in Ecole and Tang, covering a wide range of problem sizes and solve times. Appendix A.7 provides detailed descriptions as well as additional ablation results. We aim to answer the following questions: (i) Does the restricted config. space improve learning performance? (ii) How does the performance vary with fewer or more updates? (iii) Does the use of neural UCB lead to efficient predictor learning?
(i) Configuration space restriction (Sec.5.1). We train our configuration predictors to select within a restricted subspace  $A$  constructed by a greedy strategy coupled with a filtering criterion. To evaluate the importance of the space restriction in learning high quality predictors, we perform an ablation study where we train the predictors to select within (1) the unrestricted space  $O = \{0,1\}^{M}$  (No Restr.), and (2) a same-sized subspace  $A'$  constructed solely by the greedy strategy without filtering (Greedy Restr.). The restricted search space substantially enhances the learned predictors when compared to No Restr., improving the median performance and lowering the standard deviation. We also observe the benefit of the filtering criterion when compared to Greedy Restr. The filtering criterion excludes configurations with subpar instance-agnostic performance from entering the restricted configuration space, improving model generalization as demonstrated in our theoretical analysis.
Table 3: Real-world MILPs. Absolute solve time of SCIP default, and the median (higher the better) and standard deviation (in parentheses) of relative time improvement of different methods.
Methods	Default Times (s)	Heuristic Baselines	Ours Heuristic Variants	Ours Learned
Default	Random	Prune	Inst. Agnostic Configuration	Random within Restr. Subspace	L2Sep
MIPLIB	25.08s	0%	-149.1%	4.8%	5.5%	1.9%	12.9%
(57.05s)	(149.7%)	(107.6%)	(71.5%)	(74.9%)	(73.1%)
NN Verification	31.42s	0%	-300.0%	31.5%	31.4%	30.7%	37.5%
(22.44s)	(152.3%)	(36.3%)	(38.3%)	(34.1%)	(33.9%)
Load Balancing	31.86s	0%	-300.1%	21.1%	10.4%	10.0%	21.2%
(7.07s)	(129.5%)	(150.8%)	(8.5%)	(31.5%)	(20.3%)
(ii) Configuration update restriction (Sec. 5.2). We apply the forward training algorithm (Sec. 5.2) to perform two configuration updates ( $k = 2$ ) for each MILP instance. To examine the impact of fewer or more updates, we conduct an ablation study where we (1) performed a single update at round  $n_1 = 0$  ( $\mathbf{k} = \mathbf{1}$ ), and (2) added an additional third update at a later round  $n_3$  ( $\mathbf{k} = \mathbf{3}$ ). The results show that while a single update yields decent time improvement, adding the second update leads to further time savings. Meanwhile, we observe little improvement from the third update ( $\mathbf{k} = \mathbf{3}$ ). We speculate that this is because the performance improvement primarily occurs during the early stages of a solve, and holding a fixed configuration for longer may be advantageous by making the solve process more stable. We leave further investigation of more configuration updates as a future work.
(iii) Neural UCB algorithm (Sec. 5.3). Our method employs the online neural UCB algorithm to improve training efficiency for configuration predictors. We present the ablation (1) where we train the predictor using an offline regression dataset whose size is four times as ours while training the model until convergence (Supervise  $(\times 4)$ ); we conduct an additional ablation (2) where we train the predictor using neural contextual bandit with  $\epsilon$ -greedy exploration strategy ( $\epsilon$ -greedy). Our model performs comparably to Supervise  $(\times 4)$  while using significantly fewer data, highlighting the importance of the contextual bandit for improving training efficiency by collecting increasingly higher quality datasets online. The ablation results with  $\epsilon$ -greedy further confirm the benefit of the confidence bound estimation in neural UCB for more efficient contextual bandit exploration.
6.3 Large-scale Real-world MILP Benchmarks
The real-world datasets of MIPLIB, NN Verification, and Load Balancing present significant challenges due to the vast number of variables and constraints (on the order of  $10^{4}$ ), including nonstandard constraint types that MILP separators are not designed to handle. MIPLIB imposes a further challenge of dataset heterogeneity, as it contains a diverse set of instances from various application domains. Prior research [52] struggles to learn effectively on MIPLIB due to this heterogeneity, and a recent study [54] attempts to learn cutting plane selection over two homogeneous subsets (with 20 and 40 instances each). In contrast, we attempt to learn separator configuration across a larger MIPLIB subset that includes 443 of the 1065 instances in the original set, while carefully preserving the heterogeneity of the dataset. We provide our subset curation procedure in Appendix A.6.5.
Main Results. Table3 presents the relative time improvement of various methods over SCIP default, on the large-scale real-world datasets. Again, our complete method displays a substantial speed up from SCIP default with a relative time improvement ranging from  $12\%$  to  $37\%$ . Our method also improves from our heuristic sub-components, further indicating the efficacy of our learning component on the challenging datasets. In contrast, the random baseline fails to improve from SCIP default, while the pruning baseline, despite having a reasonable median performance, suffers from a high standard deviation due to poor performance on many instances (See Appendix A.8.1 for IQM and mean results). Our results show the effectiveness of our learning method in improving the efficiency of practical applications that involve large-scale MILP optimization.
Although not a perfect comparison, for reference, we attempt to contextualize our result by examining the time improvement in the most comparable setting we found, which we provide comparison details in Appendix A.8.2: the learning method for cutting plane selection in Paulus et al. [42] achieves a median relative time improvement of  $11.67\%$  on the NN Verification dataset, and that in Wang et al. [54] obtains a  $3\%$  and  $1\%$  improvement in the solve time on two small homogeneous MIPLIB
Table 4: Gurobi as the MILP Solver. Absolute solve time of Gurobi default, and the median (higher the better) and standard deviation (in parentheses) of relative time improvement of different methods.
Methods	Default Times (s)	Heuristic Baselines	Ours Heuristic Variants	Ours Learned
Default	Random	Inst. Agnostic Configuration	Random within Restr. Subspace	L2Sep
Max. Cut	0.087s(0.051s)	0%	18.6%(49.0%)	35.1%(35.8%)	37.3%(48.0%)	45.4%(38.4%)
Pack.	4.048s(3.216s)	0%	15.5%(28.2%)	22.9%(39.4%)	24.3%(32.2%)	30.6%(29.6%)
Comb. Auc.	1.687s(3.596s)	0%	-10.7%(69.1%)	3.1%(65.3%)	5.1%(84.2%)	12.6%(63.5%)
Fac. Loc.	27.872s(14.733s)	0%	13.4%(46.0%)	40.6%(48.1%)	40.2%(46.8%)	56.7%(35.7%)
subsets. While the comparison is far from perfect, our learning method for separator configuration achieves much higher time improvements of  $37.5\%$  on NN Verification and  $12.9\%$  on MIPLIB.
6.4 Interpretation Analysis: L2Sep Recovers Effective Separators from Literature
Bin Packing: It is known that instances with few bins approximate the Knapsack problem (Clique cuts are known to be effective [9]), and that instances with many bins approximate Bipartite Matching (Flowcover cuts can be useful [53]). We analyze the separators activated by L2Sep when we gradually decrease the number of bins, and observe that the prevalence of selected Clique and Flowcover cuts increased and decreased, respectively. This is illustrated in Fig 8 in Appendix A.8.3.
Other MILP Classes: We provide visualizations and interpretations for other MILP classes in Appendix A.8.3. Notably, Clique is known to be effective for Indep. Set [16]; L2Sep recovers this fact by frequently selecting configurations that activate Clique. Meanwhile, L2Sep discovers the instance heterogeneity of MIPLIB, resulting in a more dispersed distribution of selected configurations.
6.5 State-of-the-art MILP Solver Gurobi
We apply our method L2Sep with Gurobi, which contains a larger set of 21 separators. As Gurobi is closed-source, we cannot change configurations after the solving process starts, so we only consider one stage of separator configuration ( $k = 1$ ). As seen from Table 4, L2Sep achieves significant relative time improvements over the Gurobi default, with gains ranging from  $12\%$  to  $56\%$ . This result confirms the efficacy of L2Sep as an automatic instance-aware separator configuration method.
6.6 Additional Results
In Appendix A.8.5 and A.8.4, we further demonstrate (1) Separator Configuration has immediate and multi-step effects in the B&C Process. For instance, even though L2Sep does not modify branching, the branching solve time is reduced. (2) L2Sep is effective under an alternative objective, achieving  $15\% - 68\%$  relative gap improvements under fixed time limits.
This work identifies the opportunity of managing separators to improve MILP solvers, and further formulates and designs a learning-based method for doing so. We design a data-driven strategy, supported by theoretical analysis, to restrict the combinatorial space of separator configurations, and overall find that our learning method is able to improve the relative solve time (over the default solver) from  $12\%$  to  $72\%$  across a range of MILP benchmarks. In future work, we plan to apply our algorithm to more challenging MILP problems, particularly those that cannot be solved to optimality. We also aim to learn more fine-grained controls by increasing the frequency of separation configuration updates. Our algorithm is highly versatile, and we plan to investigate its potential to manage aspects of the MILP solvers, and further integrate with previous works on cutting plane selection. Our code is publicly available at https://github.com/mit-wu-lab/learning-to-configure-separators. We believe that our learning framework can be a powerful technique to enhance MILP solvers.
The authors would like to thank Mark Velednitsky and Alexandre Jacquillat for insightful discussions regarding an interpretative analysis of the learned model. This work was supported by a gift from Mathworks, the National Science Foundation (NSF) CAREER award (#2239566), the MIT Amazon Science Hub, and MIT's Research Support Committee. The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources that have contributed to the research results reported within this paper.