Sample Complexity of Tree Search Configuration: Cutting Planes and Beyond

1 Introduction . Integer programming is one of the most broadly-applicable tools in computer science , used to formulate problems from operations research ( such as routing , scheduling , and pricing ) , machine learning ( such as adversarially-robust learning , MAP estimation , and clustering ) , and beyond . Branchand-cut ( B & C ) is the most widely-used algorithm for solving integer programs ( IPs ) . B & C is highly configurable , and with a deft configuration , it can be used to solve computationally challenging problems . Finding a good configuration , however , is a notoriously difficult problem . We study machine learning approaches to configuring policies for selecting cutting planes , which have an enormous impact on B & C ’ s performance . At a high level , B & C works by recursively partitioning the IP ’ s feasible region , searching for the locally optimal solution within each set of the partition , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . until it can verify that it has found the globally optimal solution . An IP ’ s feasible region is defined by a set of linear inequalities Ax ≤ b and integer constraints x ∈ Zn , where n is the number of variables . By dropping the integrality constraints , we obtain the linear programming ( LP ) relaxation of the IP , which can be solved efficiently . A cutting plane is a carefully-chosen linear inequality αTx ≤ β which refines the LP relaxation ’ s feasible region without separating any integral point . Intuitively , a well-chosen cutting plane will remove a large portion of the LP relaxation ’ s feasible region , speeding up the time it takes B & C to find the optimal solution to the original IP . Cutting plane selection is a crucial task , yet it is challenging because many cutting planes and cut-selection policies have tunable parameters , and the best configuration depends intimately on the application domain . We provide the first provable guarantees for learning high-performing cutting planes and cut-selection policies , tailored to the application at hand . We model the application domain via an unknown , application-specific distribution over IPs , as is standard in the literature on using machine learning for integer programming [ e.g. , 21 , 23 , 31 , 36 , 43 ] . For example , this could be a distribution over the routing IPs that a shipping company must solve day after day . The learning algorithm ’ s input is a training set sampled from this distribution . The goal is to use this training set to learn cutting planes and cut-selection policies with strong future performance on problems from the same application but which are not already in the training set—or more formally , strong expected performance . 1.1 Summary of main contributions and overview of techniques . As our first main contribution , we provide sample complexity bounds of the following form : fixing a family of cutting planes , we bound the number of samples sufficient to ensure that for any sequence of cutting planes from the family , the average size of the B & C tree is close to the expected size of the B & C tree . We measure performance in terms of the size of the search tree B & C builds . Our guarantees apply to the parameterized family of Chvátal-Gomory ( CG ) cuts [ 10 , 17 ] , one of the most widely-used families of cutting planes . The overriding challenge is that to provide guarantees , we must analyze how the tree size changes as a function of the cut parameters . This is a sensitive function—slightly shifting the parameters can cause the tree size to shift from constant to exponential in the number of variables . Our key technical insight is that as the parameters vary , the entries of the cut ( i.e. , the vector α and offset β of the cut αTx ≤ β ) are multivariate polynomials of bounded degree . The number of terms defining the polynomials is exponential in the number of parameters , but we show that the polynomials can be embedded in a space with dimension sublinear in the number of parameters . This insight allows us to better understand tree size as a function of the parameters . We then leverage results by Balcan et al . [ 8 ] that show how to use structure exhibited by dual functions ( measuring an algorithm ’ s performance , such as its tree size , as a function of its parameters ) to derive sample complexity bounds . Our second main contribution is a sample complexity bound for learning cut-selection policies , which allow B & C to adaptively select cuts as it solves the input IP . These cut-selection policies assign a number of real-valued scores to a set of cutting planes and then apply the cut that has the maximum weighted sum of scores . Tree size is a volatile function of these weights , though we prove that it is piecewise constant , as illustrated in Figure 1 , which allows us to prove our sample complexity bound . Finally , as our third main contribution , we provide guarantees for tuning weighted combinations of scoring rules for other aspects of tree search beyond cut selection , including node and variable selection . We prove that there is a set of hyperplanes splitting the parameter space into regions such that if tree search uses any configuration from a single region , it will take the same sequence of actions . This structure allows us to prove our sample complexity bound . This is the first paper to provide guarantees for tree search configuration that apply simultaneously to multiple different aspects of the algorithm—prior research was specific to variable selection [ 5 ] . Sample complexity bounds are important because if the parameterized class of cuts or cut-selection policies that we optimize over is highly complex and the training set is too small , the learned cut or cut-selection policy might have great average empirical performance over the training set but terrible future performance . In other words , the parameter configuration procedure may overfit to the training set . The sample complexity bounds we provide are uniform-convergence : we prove that given enough samples , uniformly across all parameter settings , the difference between average and empirical performance is small . In other words , these bounds hold for any procedure one might use to optimize over the training set : manual or automated , optimal or suboptimal . No matter what parameter setting the configuration procedure comes up with , the user can be guaranteed that so long as that parameter setting has good average empirical performance over the training set , it will also have strong future performance . 1.2 Related work . Applied research on tree search configuration . Over the past decade , a substantial literature has developed on the use of machine learning for integer programming and tree search [ e.g. , 2 , 7 , 9 , 13 , 19 , 23–25 , 29 , 31–33 , 35 , 36 , 41–43 ] . This has included research that improves specific aspects of B & C such as variable selection [ 2 , 13 , 24 , 29 , 32 , 41 ] , node selection [ 19 , 35 , 44 ] , and heuristic scheduling [ 25 ] . These papers are applied , whereas we focus on providing theoretical guarantees . With respect to cutting plane selection , the focus of this paper , Sandholm [ 36 ] uses machine learning techniques to customize B & C for combinatorial auction winner determination , including cutting plane selection . Tang et al . [ 37 ] and Huang et al . [ 20 ] study machine learning approaches to cutting plane selection . The former work formulates this problem as a reinforcement learning problem and shows that their approach can outperform human-designed heuristics for a variety of tasks . The latter work studies cutting plane selection in the multiple-instance-learning framework and proposes a neural-network architecture for scoring and ranking cutting planes . Meanwhile , the focus of our paper is to provide the first provable guarantees for cutting plane selection via machine learning . Ferber et al . [ 15 ] study a problem where the IP objective vector c is unknown , but an estimate ĉ can be obtained from data . Their goal is to optimize the quality of the solutions obtained by solving the IP defined by ĉ , with respect to the true vector c. They do so by formulating the IP as a differentiable layer in a neural network . The nonconvex nature of the IP does not allow for straightforward gradient computation for the backward pass , so they obtain a continuous surrogate using cutting planes . Provable guarantees for algorithm configuration . Gupta and Roughgarden [ 18 ] initiated the study of sample complexity bounds for algorithm configuration . In research most related to ours , Balcan et al . [ 5 ] provide sample complexity bounds for learning tree search variable selection policies ( VSPs ) . They prove their bounds by showing that for any IP , hyperplanes partition the VSP parameter space into regions where the B & C tree size is a constant function of the parameters . The analysis in this paper requires new techniques because although we prove that the B & C tree size is a piecewiseconstant function of the CG cutting plane parameters , the boundaries between pieces are far more complex than hyperplanes : they are hypersurfaces defined by multivariate polynomials . Kleinberg et al . [ 26 , 27 ] and Weisz et al . [ 38 , 39 ] design configuration procedures for runtime minimization that come with theoretical guarantees . Their algorithms are designed for the case where there are a finitely-many parameter settings to choose from ( although they are still able to provide guarantees for infinite parameter spaces by running their procedure on a finite sample of configurations ; Balcan et al . [ 5 , 6 ] analyze when discretization approaches can and can not be gainfully employed ) . In contrast , our guarantees are designed for infinite parameter spaces . 2 Problem formulation . In this section we give a more detailed technical overview of branch-and-cut , as well as an overview of the tools from learning theory we use to prove sample complexity guarantees . 2.1 Branch-and-cut . We study integer programs ( IPs ) in canonical form given by max { cTx : Ax ≤ b , x ≥ 0 , x ∈ Zn } , ( 1 ) where A ∈ Zm×n , b ∈ Zm , and c ∈ Rn . Branch-and-cut ( B & C ) works by recursively partitioning the input IP ’ s feasible region , searching for the locally optimal solution within each set of the partition until it can verify that it has found the globally optimal solution . It organizes this partition as a search tree , with the input IP stored at the root . It begins by solving the LP relaxation of the input IP ; we denote the solution as x∗LP ∈ Rn . If x∗LP satisfies the IP ’ s integrality constraints ( x∗LP ∈ Zn ) , then the procedure terminates—x∗LP is the globally optimal solution . Otherwise , it uses a variable selection policy to choose a variable x [ i ] . In the left child of the root , it stores the original IP with the additional constraint that x [ i ] ≤ bx∗LP [ i ] c , and in the right child , with the additional constraint that x [ i ] ≥ dx∗LP [ i ] e. It then uses a node selection policy to select a leaf of the tree and repeats this procedure—solving the LP relaxation and branching on a variable . B & C can fathom a node , meaning that it will stop searching along that branch , if 1 ) the LP relaxation satisfies the IP ’ s integrality constraints , 2 ) the LP relaxation is infeasible , or 3 ) the objective value of the LP relaxation ’ s solution is no better than the best integral solution found thus far . We assume there is a bound κ on the size of the tree we allow B & C to build before we terminate , as is common in prior research [ 5 , 21 , 26 , 27 ] . Cutting planes are a means of ensuring that at each iteration of B & C , the solution to the LP relaxation is as close to the optimal integral solution as possible . Formally , let P = { x ∈ Rn : Ax ≤ b , x ≥ 0 } denote the feasible region obtained by taking the LP relaxation of IP ( 1 ) . Let PI = conv ( P ∩ Zn ) denote the integer hull of P . A valid cutting plane is any hyperplane αTx ≤ β such that if x is in the integer hull ( x ∈ PI ) , then x satisfies the inequality αTx ≤ β . In other words , a valid cut does not remove any integral point from the LP relaxation ’ s feasible region . A valid cutting plane separates x ∈ P \ PI if it does not satisfy the inequality , or in other words , αTx > β . At any node of the search tree , B & C can add valid cutting planes that separate the optimal solution to the node ’ s LP relaxation , thus improving the solution estimates used to prune the search tree . However , adding too many cuts will increase the time it takes to solve the LP relaxation at each node . Therefore , solvers such as SCIP [ 16 ] , the leading open-source solver , bound the number of cuts that will be applied . A famous class of cutting planes is the family of Chvátal-Gomory ( CG ) cuts1 [ 10 , 17 ] , which are parameterized by vectors u ∈ Rm . The CG cut defined by u ∈ Rm is the hyperplane buTAcx ≤ buT bc , which is guaranteed to be valid . Throughout this paper we primarily restrict our attention to u ∈ [ 0 , 1 ) m. This is without loss of generality , since the facets of P ∩ { x ∈ Rn : buTAcx ≤ buT bc ∀u ∈ Rm } can be described by the finitely many u ∈ [ 0 , 1 ) m such that uTA ∈ Zn ( Lemma 5.13 of Conforti et al . [ 11 ] ) . Some IP solvers such as SCIP use scoring rules to select among cutting planes , which are meant to measure the quality of a cut . Some commonly-used scoring rules include efficacy [ 4 ] ( score1 ) , objective parallelism [ 1 ] ( score2 ) , directed cutoff distance [ 16 ] ( score3 ) , and integral support [ 40 ] ( score4 ) ( defined in Appendix A ) . Efficacy measures the distance between the cut αTx ≤ β and x∗LP : score1 ( α Tx ≤ β ) = ( αTx∗LP − β ) / ‖α‖2 , as illustrated in Figure 2a . Objective parallelism measures the angle between the objective c and the cut ’ s normal vector α : score2 ( αTx ≤ β ) =∣∣cTα∣∣ / ( ‖α‖2 ‖c‖2 ) , as illustrated in Figures 2b and 2c . Directed cutoff distance measures the distance between the LP optimal solution and the cut in a more relevant direction than the efficacy scoring rule . Specifically , let x be the incumbent solution , which is the best-known feasible solution to the input IP . The directed cutoff distance is the distance between the hyperplane ( α , β ) and the current LP solution x∗LP along the direction of the incumbent x , as illustrated in Figures 2d and 2e : score3 ( αTx ≤ β ) = ‖x− x∗LP‖2 · ( α Tx∗LP − β ) / ∣∣αT ( x− x∗LP ) ∣∣ . SCIP uses the scoring rule 3 5score1 + 1 10score2 + 1 2score3 + 1 10score4 [ 16 ] . 1The set of CG cuts is equivalent to the set of Gomory ( fractional ) cuts [ 12 ] , another commonly studied family of cutting planes with a slightly different parameterization .

Summary: Combinatorial optimization solvers often add cutting planes during solving, which are additional constraints in the problem that accelerate solving. These cutting planes have a major impact on the size of the final branch-and-bound tree, and therefore of the speed of the overall solving, but their impact is badly understood theoretically. At minimum, it is empirically understood that the size of the branch-and-bound tree is a complicated and sensitive function of the parameters of the cutting planes that were added during solving: a slightly different cutting plane might have been much more, or much less, effective.  In this paper the authors look at the problem of trying to learn these functions that relate the size of the branch-and-bound tree (a measure of solving performance) as a function of the instance data (A, b, c) when cuts with given parameters u_1, ..., u_k are added to the problem. They compute an order of magnitude of samples required to learn well this function, both for a single cut, a sequence of cuts, or rounds of sequences of cuts (as are used in practice). They also compute the number of samples required for the generalized problem of predicting tree size as a function of the instance data (A, b, c) for an algorithm that would pick the cut parameters to maximize a convex combination of scores, for given algorithm hyperparameters (the convex weights.) Finally, they do the same for more general algorithms that would pick an action, during solving, by maximizing a convex combination of scores, such as when doing node selection or variable selection.