Learning Large Neighborhood Search Policy for Integer Programming

1 Introduction . Combinatorial optimization problems ( COPs ) have been widely studied in computer science and operations research , which cover numerous real-world tasks in many fields such as communication , transportation and manufacturing [ 1 ] . Most COPs are very difficult to solve efficiently due to their NP-hardness . The performance of classic methods , including exact and heuristic algorithms [ 2 ] , is generally limited by hand-crafted policies that are costly to design , since considerable trial-and-error and domain knowledge are needed . On the other hand , it is common in practice that similar instances with shared structure are frequently solved , and differ only in data that normally follows a distribution [ 3 ] . This provides a chance for machine learning to automatically generate heuristics or policies . In doing so , the learned alternatives are expected to save massive manual work in algorithm design , and raise the performance of the algorithm on a class of problems . Recently , a number of works apply deep ( reinforcement ) learning to automatically design heuristic algorithms , either in constructive or improving fashion . Different from construction heuristics that sequentially extend partial solutions to complete ones [ 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 ] , learning improvement heuristics can often deliver high solution quality by iteratively reoptimizing an initial solution using ∗Wen Song is the corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . local operations [ 12 , 13 , 14 ] . In this line , some methods are developed under the Large Neighborhood Search ( LNS ) framework [ 15 , 16 ] , which is a powerful improving paradigm to find near-optimal solutions for COPs . However , the above methods are restricted to specific problem types , and can not generalize to those from different domains . This motivates the studies of learning to directly solve Integer Programs ( IPs ) , which is very powerful and flexible in modelling a wide range of COPs . The standard approach to solve IPs is branch-and-bound ( B & B ) [ 17 ] , which lies at the core of common solvers such as SCIP , Gurobi , and CPLEX . Thus , most of existing methods improve the performance of a solver on a distribution of instances , by training models for critical search decisions in B & B such as variable and node selection [ 18 , 19 , 20 ] . Nevertheless , these methods are generally limited to small instances and require sufficient interface access to the internal solving process of the solvers . This paper mainly tackle the issue that how to improve a solver from externals such that it can find high-quality solutions more quickly ? In specific , we propose a high-level , learning based LNS method to solve general IP problems . Based on deep reinforcement learning ( RL ) , we train a policy network as the destroy operator in LNS , which decides a subset of variables in the current solution for reoptimization . Then we use a solver as the repair operator , which solves sub-IPs to reoptimize the destroyed variables . Despite being heuristic , our method can effectively handle the large-scale IP by solving a series of smaller sub-IPs . Moreover , complex interface to the solver ’ s internal logic is not required . However , the above RL task is challenging , mainly because the action space , i.e. , number of variable subsets at each LNS step , is exponentially large . To resolve this issue , we represent all the subsets by factorizing them into binary decisions on each variable , i.e. , whether a variable should be destroyed . In doing so , we make it possible to learn a policy to select any subset from large discrete action spaces ( at least 21000 candidates in our experiments ) . To this end , we design a Graph Neural Network ( GNN ) based policy network that enables learning policies for each variable in parallel , and train it by a customized actor-critic algorithm . A recent work [ 21 ] also attempts to learn LNS policy to solve IP problems , and we generalize this framework to enable learning more flexible and powerful LNS algorithms . One limitation of [ 21 ] is that it hypothesizes a constant cardinality of the destroyed variable subset at each LNS step , which is a predefined hyperparameter . However , the number and choice of optimized variables at each step should be adaptive according to instance information and solving status , which is achieved in our method . In doing so , the LNS policies trained by our method significantly outperform those trained by the method in [ 21 ] . We evaluate our method on four NP-hard benchmark problems with SCIP as the repair solver . Extensive results show that our method generally delivers better solutions than SCIP with mere 1/2 or 1/5 of runtime , and significantly outperforms LNS baselines with the same runtime . These advantages notably persist when the trained policies are directly applied to much larger problems . We also apply our LNS framework to Gurobi , which shows superiority over the solver itself and other baselines . 2 Related work . In this section , we briefly review existing works related to ours . We first describe two main streams of learning based methods to solve COPs , and then introduce the literature that study RL with large discrete action space , which is also an essential issue we confront in this paper . Learning to solve specific COPs . Quite a few works attempt to learn heuristics to solve certain types of COPs . Compared to construction ones , methods that learn improvement heuristics can often deliver smaller optimality gap , by training policies to iteratively improve the solution . Chen and Tian [ 12 ] propose to learn how to locally rewrite a solution ; Wu et al . [ 13 ] train policies to pick the next solution in local moves ; Lu et al . [ 14 ] learn to select local operators to reform a solution . These methods are generally limited by simple local operations . A few methods learn more powerful operators under the LNS framework . Hottung and Tierney [ 15 ] train an attention model to repair the solution every time it is broken by a predefined destroy operator . Similarly , Gao et al . [ 16 ] combine GNN and Recurrent Neural Network to learn a reinsertion operator to repair sequence-based solution . However , all the above methods are limited to specific problem types , e.g. , the LNS methods in [ 15 , 16 ] are designed only for routing problems . In contrast , this paper aims to solve general IP problems with a high-level LNS framework and raise its performance by learning better policies . Learning to solve IP problems . Most of learning based methods for IPs aim to improve inner policies of B & B algorithm . For example , He et al . [ 18 ] learn to explore nodes in B & B tree by imitating an oracle . Gasse et al . [ 20 ] train a GNN model to predict the strong branching rule by imitation learning . Khalil et al . [ 19 ] predict the use of primal heuristics by logistic regression . Other components of B & B are also considered to improve its practical performance , such as learning to select cutting planes by Tang et al . [ 22 ] , predicting local branching at root node by Ding et al . [ 23 ] or refining the primal heuristic and branching rule concurrently by Nair et al . [ 24 ] . Different from these works , we employ RL to improve practical performance of IP solvers especially for large-scale problems , without much engineering effort on interfacing with inner process of solvers.2 This is also noted in [ 21 ] , which proposes to combine learning and LNS to solve IPs . However , a major drawback of this method is that the subsets of destroyed variables are assumed to be fixed-sized , which limits its performance . In contrast , our LNS framework allows picking variable subsets in a more adaptive way , and thus empowers learning broader classes of policies . RL with large action spaces . Learning with large-sized , high-dimensional discrete action spaces is still intricate in current RL research . In this direction , Pazis and Parr [ 25 ] use binary format to encode all actions , and learn value functions for each bit . Tavakoli et al . [ 26 ] design neural networks with branches to decide on each component of the action . Besides , Dulac-Arnold et al . [ 27 ] and Chandak et al . [ 28 ] update the action representation by solving continuous control problems so as to alleviate learning complexity via generalization . Other similar works can be found in [ 29 , 30 , 31 ] . However , we note that all these methods are generally designed for tasks with ( discretized ) continuous action spaces , which are relatively low-dimensional and small-sized ( at most 1025 as in [ 26 ] ) . In contrast , action spaces in our scenario are much larger ( at least 21000 ) . In this paper , we propose to factorize the action space into binary actions on each dimension , and train the individual policies in parallel through parameter sharing . 3 Preliminaries . Integer Program ( IP ) is typically defined as argminx { µ⊤x|Ax ≤ b ; x ≥ 0 ; x ∈ Zn } , where x is a vector of n decision variables ; µ ∈ Rn denotes the vector of objective coefficients ; the incidence matrix A ∈ Rm×n and right-hand-side ( RHS ) vector b ∈ Rm together define m linear constraints . With the above formulation , the size of an IP problem is generally reflected by the number of variables ( n ) and constraints ( m ) . Large Neighborhood Search ( LNS ) is a type of improvement heuristics , which iteratively reoptimizes a solution by the destroy and repair operator until certain termination condition is reached [ 32 ] . Specifically , the former one breaks part of the solution xt at step t , then the latter one fixes the broken solution to derive the next solution xt+1 . The destroy and repair operator together define the solution neighborhood N ( xt ) , i.e. , solution candidates that can be accessed at time step t. Compared to typical local search heuristics , LNS is more effective in exploring the solution space , since a larger neighborhood is considered at each step [ 32 ] . Most of existing LNS methods rely on problem specific operators , e.g. , removal and reinsertion for solutions with sequential structure [ 33 , 34 ] . Instead , we propose a RL based LNS for general IP problems , with the learned destroy operator to select variables for reoptimization at each step . 4 Methodology . In this section , we first formulate our LNS framework as a Markov Decision Process ( MDP ) . Then , we present the factorized representation of the large-scale action space , and parametrize the policy by a specialized GNN . Finally , we introduce a customized actor-critic algorithm to train our policy network for deciding the variable subset . 4.1 MDP formulation . Most of existing works learn LNS for specific problems [ 15 , 16 ] , which rely on extra domain knowledge and hinder their applicability to other COPs . In this paper , we apply LNS to general IP 2Nevertheless , our method can also work with solvers enhanced by the above methods as repair operators . problems , in which RL is employed to learn a policy that at each step , selects a variable subset from the solution to be reoptimized . We formulate this sequential decision problem as a discrete-time MDP . Specifically , we regard the destroy operator as the agent and the remainder in LNS as the environment . In one episode of solving an IP instance , the MDP can be described as follows : States . A state st ∈ S at each LNS step needs to not only reflect the instance information but also the dynamic solving status . For the former , we represent it by static features of variables and constraints , along with the incidence matrix in an instance . The latter is represented by dynamic features , including the current solution xt and dynamic statistics of incumbent solutions up to step t. Specifically , both the current and incumbent solution at t = 0 are defined as the initial solution x0 . Actions . At each state st , an action of the agent is to select a variable subset at from all candidate subsets A for reoptimization . Transition . The repair operator ( i.e. , an IP solver ) solves the sub-IP , where only the variables in action at are optimized with the others equaling to their current values in xt . Accordingly , the next state st+1 is deterministically attained by updating st with the new solution : xt+1 = argmin x { µ⊤x|Ax ≤ b ; x ≥ 0 ; x ∈ Zn ; xi = xit , ∀xi /∈ at } . ( 1 ) Rewards . The reward function is defined as rt = r ( st , at ) = µ⊤ ( xt − xt+1 ) , which is the change of the objective value . Given the step limit T of the interaction between the agent and environment , the return ( i.e. , cumulative rewards ) from step t of the episode is Rt = ∑T k=t γ k−trk , with the discount factor γ ∈ [ 0 , 1 ] . The goal of RL is to maximize the expected return E [ R1 ] over all episodes , i.e. , the expected improvement over initial solutions , by learning a policy . Policy . The stochastic policy π represents a conditional probability distribution over all possible variable subsets given a state . Starting from s0 , it iteratively picks an action at based on the sate st at each step t , until reaching the step limit T . An example of the proposed LNS framework solving an IP instance is illustrated in Figure 1 , in which the GNN based policy network has the potential to process instances of any size via sharing parameters across all variables .

Summary: The last few years have seen numerous projects leveraging ML/RL to speedup the optimization of ILP problems. Many of these attempts fall under 2 categories:   * RL driven branch-and-bound. By integrating a policy deeply inside an ILP solver to drive the branch&bound process, these approaches have the ability to extract a rich set of features from the solver and can leverage them to inform the decisions made by the policy. The work of Gasse et al. (Exact combinatorial optimization with graph convolution neural networks)  is an example of such an approach. * Large Neighborhood Search. This technique has been applied with success to speedup the optimization of integer programs, in particular in the work of Song et al. (A general large neighborhood search framework for solving integer linear programs). This approach is easier to implement since it doesn't require a deep integration into an ILP solver (it merely calls one to solve subproblems).  This paper proposes a hybrid of the two approaches. It leverages a neural network architecture similar to the Gasse GCN as well as the featurization introduced by Gasse to build and inform a policy that drives the large neighborhood search pioneered by Song et al. It also offers some novel aspects, including a customized actor critic algorithm to train the policy.