Sample Complexity of Tree Search Configuration: Cutting Planes and Beyond

1 Introduction . Integer programming is one of the most broadly-applicable tools in computer science , used to formulate problems from operations research ( such as routing , scheduling , and pricing ) , machine learning ( such as adversarially-robust learning , MAP estimation , and clustering ) , and beyond . Branchand-cut ( B & C ) is the most widely-used algorithm for solving integer programs ( IPs ) . B & C is highly configurable , and with a deft configuration , it can be used to solve computationally challenging problems . Finding a good configuration , however , is a notoriously difficult problem . We study machine learning approaches to configuring policies for selecting cutting planes , which have an enormous impact on B & C ’ s performance . At a high level , B & C works by recursively partitioning the IP ’ s feasible region , searching for the locally optimal solution within each set of the partition , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . until it can verify that it has found the globally optimal solution . An IP ’ s feasible region is defined by a set of linear inequalities Ax ≤ b and integer constraints x ∈ Zn , where n is the number of variables . By dropping the integrality constraints , we obtain the linear programming ( LP ) relaxation of the IP , which can be solved efficiently . A cutting plane is a carefully-chosen linear inequality αTx ≤ β which refines the LP relaxation ’ s feasible region without separating any integral point . Intuitively , a well-chosen cutting plane will remove a large portion of the LP relaxation ’ s feasible region , speeding up the time it takes B & C to find the optimal solution to the original IP . Cutting plane selection is a crucial task , yet it is challenging because many cutting planes and cut-selection policies have tunable parameters , and the best configuration depends intimately on the application domain . We provide the first provable guarantees for learning high-performing cutting planes and cut-selection policies , tailored to the application at hand . We model the application domain via an unknown , application-specific distribution over IPs , as is standard in the literature on using machine learning for integer programming [ e.g. , 21 , 23 , 31 , 36 , 43 ] . For example , this could be a distribution over the routing IPs that a shipping company must solve day after day . The learning algorithm ’ s input is a training set sampled from this distribution . The goal is to use this training set to learn cutting planes and cut-selection policies with strong future performance on problems from the same application but which are not already in the training set—or more formally , strong expected performance . 1.1 Summary of main contributions and overview of techniques . As our first main contribution , we provide sample complexity bounds of the following form : fixing a family of cutting planes , we bound the number of samples sufficient to ensure that for any sequence of cutting planes from the family , the average size of the B & C tree is close to the expected size of the B & C tree . We measure performance in terms of the size of the search tree B & C builds . Our guarantees apply to the parameterized family of Chvátal-Gomory ( CG ) cuts [ 10 , 17 ] , one of the most widely-used families of cutting planes . The overriding challenge is that to provide guarantees , we must analyze how the tree size changes as a function of the cut parameters . This is a sensitive function—slightly shifting the parameters can cause the tree size to shift from constant to exponential in the number of variables . Our key technical insight is that as the parameters vary , the entries of the cut ( i.e. , the vector α and offset β of the cut αTx ≤ β ) are multivariate polynomials of bounded degree . The number of terms defining the polynomials is exponential in the number of parameters , but we show that the polynomials can be embedded in a space with dimension sublinear in the number of parameters . This insight allows us to better understand tree size as a function of the parameters . We then leverage results by Balcan et al . [ 8 ] that show how to use structure exhibited by dual functions ( measuring an algorithm ’ s performance , such as its tree size , as a function of its parameters ) to derive sample complexity bounds . Our second main contribution is a sample complexity bound for learning cut-selection policies , which allow B & C to adaptively select cuts as it solves the input IP . These cut-selection policies assign a number of real-valued scores to a set of cutting planes and then apply the cut that has the maximum weighted sum of scores . Tree size is a volatile function of these weights , though we prove that it is piecewise constant , as illustrated in Figure 1 , which allows us to prove our sample complexity bound . Finally , as our third main contribution , we provide guarantees for tuning weighted combinations of scoring rules for other aspects of tree search beyond cut selection , including node and variable selection . We prove that there is a set of hyperplanes splitting the parameter space into regions such that if tree search uses any configuration from a single region , it will take the same sequence of actions . This structure allows us to prove our sample complexity bound . This is the first paper to provide guarantees for tree search configuration that apply simultaneously to multiple different aspects of the algorithm—prior research was specific to variable selection [ 5 ] . Sample complexity bounds are important because if the parameterized class of cuts or cut-selection policies that we optimize over is highly complex and the training set is too small , the learned cut or cut-selection policy might have great average empirical performance over the training set but terrible future performance . In other words , the parameter configuration procedure may overfit to the training set . The sample complexity bounds we provide are uniform-convergence : we prove that given enough samples , uniformly across all parameter settings , the difference between average and empirical performance is small . In other words , these bounds hold for any procedure one might use to optimize over the training set : manual or automated , optimal or suboptimal . No matter what parameter setting the configuration procedure comes up with , the user can be guaranteed that so long as that parameter setting has good average empirical performance over the training set , it will also have strong future performance . 1.2 Related work . Applied research on tree search configuration . Over the past decade , a substantial literature has developed on the use of machine learning for integer programming and tree search [ e.g. , 2 , 7 , 9 , 13 , 19 , 23–25 , 29 , 31–33 , 35 , 36 , 41–43 ] . This has included research that improves specific aspects of B & C such as variable selection [ 2 , 13 , 24 , 29 , 32 , 41 ] , node selection [ 19 , 35 , 44 ] , and heuristic scheduling [ 25 ] . These papers are applied , whereas we focus on providing theoretical guarantees . With respect to cutting plane selection , the focus of this paper , Sandholm [ 36 ] uses machine learning techniques to customize B & C for combinatorial auction winner determination , including cutting plane selection . Tang et al . [ 37 ] and Huang et al . [ 20 ] study machine learning approaches to cutting plane selection . The former work formulates this problem as a reinforcement learning problem and shows that their approach can outperform human-designed heuristics for a variety of tasks . The latter work studies cutting plane selection in the multiple-instance-learning framework and proposes a neural-network architecture for scoring and ranking cutting planes . Meanwhile , the focus of our paper is to provide the first provable guarantees for cutting plane selection via machine learning . Ferber et al . [ 15 ] study a problem where the IP objective vector c is unknown , but an estimate ĉ can be obtained from data . Their goal is to optimize the quality of the solutions obtained by solving the IP defined by ĉ , with respect to the true vector c. They do so by formulating the IP as a differentiable layer in a neural network . The nonconvex nature of the IP does not allow for straightforward gradient computation for the backward pass , so they obtain a continuous surrogate using cutting planes . Provable guarantees for algorithm configuration . Gupta and Roughgarden [ 18 ] initiated the study of sample complexity bounds for algorithm configuration . In research most related to ours , Balcan et al . [ 5 ] provide sample complexity bounds for learning tree search variable selection policies ( VSPs ) . They prove their bounds by showing that for any IP , hyperplanes partition the VSP parameter space into regions where the B & C tree size is a constant function of the parameters . The analysis in this paper requires new techniques because although we prove that the B & C tree size is a piecewiseconstant function of the CG cutting plane parameters , the boundaries between pieces are far more complex than hyperplanes : they are hypersurfaces defined by multivariate polynomials . Kleinberg et al . [ 26 , 27 ] and Weisz et al . [ 38 , 39 ] design configuration procedures for runtime minimization that come with theoretical guarantees . Their algorithms are designed for the case where there are a finitely-many parameter settings to choose from ( although they are still able to provide guarantees for infinite parameter spaces by running their procedure on a finite sample of configurations ; Balcan et al . [ 5 , 6 ] analyze when discretization approaches can and can not be gainfully employed ) . In contrast , our guarantees are designed for infinite parameter spaces . 2 Problem formulation . In this section we give a more detailed technical overview of branch-and-cut , as well as an overview of the tools from learning theory we use to prove sample complexity guarantees . 2.1 Branch-and-cut . We study integer programs ( IPs ) in canonical form given by max { cTx : Ax ≤ b , x ≥ 0 , x ∈ Zn } , ( 1 ) where A ∈ Zm×n , b ∈ Zm , and c ∈ Rn . Branch-and-cut ( B & C ) works by recursively partitioning the input IP ’ s feasible region , searching for the locally optimal solution within each set of the partition until it can verify that it has found the globally optimal solution . It organizes this partition as a search tree , with the input IP stored at the root . It begins by solving the LP relaxation of the input IP ; we denote the solution as x∗LP ∈ Rn . If x∗LP satisfies the IP ’ s integrality constraints ( x∗LP ∈ Zn ) , then the procedure terminates—x∗LP is the globally optimal solution . Otherwise , it uses a variable selection policy to choose a variable x [ i ] . In the left child of the root , it stores the original IP with the additional constraint that x [ i ] ≤ bx∗LP [ i ] c , and in the right child , with the additional constraint that x [ i ] ≥ dx∗LP [ i ] e. It then uses a node selection policy to select a leaf of the tree and repeats this procedure—solving the LP relaxation and branching on a variable . B & C can fathom a node , meaning that it will stop searching along that branch , if 1 ) the LP relaxation satisfies the IP ’ s integrality constraints , 2 ) the LP relaxation is infeasible , or 3 ) the objective value of the LP relaxation ’ s solution is no better than the best integral solution found thus far . We assume there is a bound κ on the size of the tree we allow B & C to build before we terminate , as is common in prior research [ 5 , 21 , 26 , 27 ] . Cutting planes are a means of ensuring that at each iteration of B & C , the solution to the LP relaxation is as close to the optimal integral solution as possible . Formally , let P = { x ∈ Rn : Ax ≤ b , x ≥ 0 } denote the feasible region obtained by taking the LP relaxation of IP ( 1 ) . Let PI = conv ( P ∩ Zn ) denote the integer hull of P . A valid cutting plane is any hyperplane αTx ≤ β such that if x is in the integer hull ( x ∈ PI ) , then x satisfies the inequality αTx ≤ β . In other words , a valid cut does not remove any integral point from the LP relaxation ’ s feasible region . A valid cutting plane separates x ∈ P \ PI if it does not satisfy the inequality , or in other words , αTx > β . At any node of the search tree , B & C can add valid cutting planes that separate the optimal solution to the node ’ s LP relaxation , thus improving the solution estimates used to prune the search tree . However , adding too many cuts will increase the time it takes to solve the LP relaxation at each node . Therefore , solvers such as SCIP [ 16 ] , the leading open-source solver , bound the number of cuts that will be applied . A famous class of cutting planes is the family of Chvátal-Gomory ( CG ) cuts1 [ 10 , 17 ] , which are parameterized by vectors u ∈ Rm . The CG cut defined by u ∈ Rm is the hyperplane buTAcx ≤ buT bc , which is guaranteed to be valid . Throughout this paper we primarily restrict our attention to u ∈ [ 0 , 1 ) m. This is without loss of generality , since the facets of P ∩ { x ∈ Rn : buTAcx ≤ buT bc ∀u ∈ Rm } can be described by the finitely many u ∈ [ 0 , 1 ) m such that uTA ∈ Zn ( Lemma 5.13 of Conforti et al . [ 11 ] ) . Some IP solvers such as SCIP use scoring rules to select among cutting planes , which are meant to measure the quality of a cut . Some commonly-used scoring rules include efficacy [ 4 ] ( score1 ) , objective parallelism [ 1 ] ( score2 ) , directed cutoff distance [ 16 ] ( score3 ) , and integral support [ 40 ] ( score4 ) ( defined in Appendix A ) . Efficacy measures the distance between the cut αTx ≤ β and x∗LP : score1 ( α Tx ≤ β ) = ( αTx∗LP − β ) / ‖α‖2 , as illustrated in Figure 2a . Objective parallelism measures the angle between the objective c and the cut ’ s normal vector α : score2 ( αTx ≤ β ) =∣∣cTα∣∣ / ( ‖α‖2 ‖c‖2 ) , as illustrated in Figures 2b and 2c . Directed cutoff distance measures the distance between the LP optimal solution and the cut in a more relevant direction than the efficacy scoring rule . Specifically , let x be the incumbent solution , which is the best-known feasible solution to the input IP . The directed cutoff distance is the distance between the hyperplane ( α , β ) and the current LP solution x∗LP along the direction of the incumbent x , as illustrated in Figures 2d and 2e : score3 ( αTx ≤ β ) = ‖x− x∗LP‖2 · ( α Tx∗LP − β ) / ∣∣αT ( x− x∗LP ) ∣∣ . SCIP uses the scoring rule 3 5score1 + 1 10score2 + 1 2score3 + 1 10score4 [ 16 ] . 1The set of CG cuts is equivalent to the set of Gomory ( fractional ) cuts [ 12 ] , another commonly studied family of cutting planes with a slightly different parameterization .

Summary: Linear integer programming is an incredibly important algorithmic tool that is used widely in practice.  In general the problem is NP-hard and so we have to resort to heuristics. These heuristics often use the branch-and-cut framework: we branch on "guesses" on variables and we cut by adding valid inequalities that tightens the linear relaxation of the integer program.  When dealing with such heuristics, an obvious question arises: which cut should I add and on which variable should I branch? The goal here is to limit  the size of the search tree and therefore getting a good running time. There has been a growing amount of experimental work on using ML algorithms to guide this choice.    The paper under submission considers this problem from a more theoretical perspective. The main results bound the sample complexity for finding "good" cuts complementing and generalizing a prior work that studied this question for variable selection.