Learning Large Neighborhood Search Policy for Integer Programming

1 Introduction . Combinatorial optimization problems ( COPs ) have been widely studied in computer science and operations research , which cover numerous real-world tasks in many fields such as communication , transportation and manufacturing [ 1 ] . Most COPs are very difficult to solve efficiently due to their NP-hardness . The performance of classic methods , including exact and heuristic algorithms [ 2 ] , is generally limited by hand-crafted policies that are costly to design , since considerable trial-and-error and domain knowledge are needed . On the other hand , it is common in practice that similar instances with shared structure are frequently solved , and differ only in data that normally follows a distribution [ 3 ] . This provides a chance for machine learning to automatically generate heuristics or policies . In doing so , the learned alternatives are expected to save massive manual work in algorithm design , and raise the performance of the algorithm on a class of problems . Recently , a number of works apply deep ( reinforcement ) learning to automatically design heuristic algorithms , either in constructive or improving fashion . Different from construction heuristics that sequentially extend partial solutions to complete ones [ 4 , 5 , 6 , 7 , 8 , 9 , 10 , 11 ] , learning improvement heuristics can often deliver high solution quality by iteratively reoptimizing an initial solution using ∗Wen Song is the corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . local operations [ 12 , 13 , 14 ] . In this line , some methods are developed under the Large Neighborhood Search ( LNS ) framework [ 15 , 16 ] , which is a powerful improving paradigm to find near-optimal solutions for COPs . However , the above methods are restricted to specific problem types , and can not generalize to those from different domains . This motivates the studies of learning to directly solve Integer Programs ( IPs ) , which is very powerful and flexible in modelling a wide range of COPs . The standard approach to solve IPs is branch-and-bound ( B & B ) [ 17 ] , which lies at the core of common solvers such as SCIP , Gurobi , and CPLEX . Thus , most of existing methods improve the performance of a solver on a distribution of instances , by training models for critical search decisions in B & B such as variable and node selection [ 18 , 19 , 20 ] . Nevertheless , these methods are generally limited to small instances and require sufficient interface access to the internal solving process of the solvers . This paper mainly tackle the issue that how to improve a solver from externals such that it can find high-quality solutions more quickly ? In specific , we propose a high-level , learning based LNS method to solve general IP problems . Based on deep reinforcement learning ( RL ) , we train a policy network as the destroy operator in LNS , which decides a subset of variables in the current solution for reoptimization . Then we use a solver as the repair operator , which solves sub-IPs to reoptimize the destroyed variables . Despite being heuristic , our method can effectively handle the large-scale IP by solving a series of smaller sub-IPs . Moreover , complex interface to the solver ’ s internal logic is not required . However , the above RL task is challenging , mainly because the action space , i.e. , number of variable subsets at each LNS step , is exponentially large . To resolve this issue , we represent all the subsets by factorizing them into binary decisions on each variable , i.e. , whether a variable should be destroyed . In doing so , we make it possible to learn a policy to select any subset from large discrete action spaces ( at least 21000 candidates in our experiments ) . To this end , we design a Graph Neural Network ( GNN ) based policy network that enables learning policies for each variable in parallel , and train it by a customized actor-critic algorithm . A recent work [ 21 ] also attempts to learn LNS policy to solve IP problems , and we generalize this framework to enable learning more flexible and powerful LNS algorithms . One limitation of [ 21 ] is that it hypothesizes a constant cardinality of the destroyed variable subset at each LNS step , which is a predefined hyperparameter . However , the number and choice of optimized variables at each step should be adaptive according to instance information and solving status , which is achieved in our method . In doing so , the LNS policies trained by our method significantly outperform those trained by the method in [ 21 ] . We evaluate our method on four NP-hard benchmark problems with SCIP as the repair solver . Extensive results show that our method generally delivers better solutions than SCIP with mere 1/2 or 1/5 of runtime , and significantly outperforms LNS baselines with the same runtime . These advantages notably persist when the trained policies are directly applied to much larger problems . We also apply our LNS framework to Gurobi , which shows superiority over the solver itself and other baselines . 2 Related work . In this section , we briefly review existing works related to ours . We first describe two main streams of learning based methods to solve COPs , and then introduce the literature that study RL with large discrete action space , which is also an essential issue we confront in this paper . Learning to solve specific COPs . Quite a few works attempt to learn heuristics to solve certain types of COPs . Compared to construction ones , methods that learn improvement heuristics can often deliver smaller optimality gap , by training policies to iteratively improve the solution . Chen and Tian [ 12 ] propose to learn how to locally rewrite a solution ; Wu et al . [ 13 ] train policies to pick the next solution in local moves ; Lu et al . [ 14 ] learn to select local operators to reform a solution . These methods are generally limited by simple local operations . A few methods learn more powerful operators under the LNS framework . Hottung and Tierney [ 15 ] train an attention model to repair the solution every time it is broken by a predefined destroy operator . Similarly , Gao et al . [ 16 ] combine GNN and Recurrent Neural Network to learn a reinsertion operator to repair sequence-based solution . However , all the above methods are limited to specific problem types , e.g. , the LNS methods in [ 15 , 16 ] are designed only for routing problems . In contrast , this paper aims to solve general IP problems with a high-level LNS framework and raise its performance by learning better policies . Learning to solve IP problems . Most of learning based methods for IPs aim to improve inner policies of B & B algorithm . For example , He et al . [ 18 ] learn to explore nodes in B & B tree by imitating an oracle . Gasse et al . [ 20 ] train a GNN model to predict the strong branching rule by imitation learning . Khalil et al . [ 19 ] predict the use of primal heuristics by logistic regression . Other components of B & B are also considered to improve its practical performance , such as learning to select cutting planes by Tang et al . [ 22 ] , predicting local branching at root node by Ding et al . [ 23 ] or refining the primal heuristic and branching rule concurrently by Nair et al . [ 24 ] . Different from these works , we employ RL to improve practical performance of IP solvers especially for large-scale problems , without much engineering effort on interfacing with inner process of solvers.2 This is also noted in [ 21 ] , which proposes to combine learning and LNS to solve IPs . However , a major drawback of this method is that the subsets of destroyed variables are assumed to be fixed-sized , which limits its performance . In contrast , our LNS framework allows picking variable subsets in a more adaptive way , and thus empowers learning broader classes of policies . RL with large action spaces . Learning with large-sized , high-dimensional discrete action spaces is still intricate in current RL research . In this direction , Pazis and Parr [ 25 ] use binary format to encode all actions , and learn value functions for each bit . Tavakoli et al . [ 26 ] design neural networks with branches to decide on each component of the action . Besides , Dulac-Arnold et al . [ 27 ] and Chandak et al . [ 28 ] update the action representation by solving continuous control problems so as to alleviate learning complexity via generalization . Other similar works can be found in [ 29 , 30 , 31 ] . However , we note that all these methods are generally designed for tasks with ( discretized ) continuous action spaces , which are relatively low-dimensional and small-sized ( at most 1025 as in [ 26 ] ) . In contrast , action spaces in our scenario are much larger ( at least 21000 ) . In this paper , we propose to factorize the action space into binary actions on each dimension , and train the individual policies in parallel through parameter sharing . 3 Preliminaries . Integer Program ( IP ) is typically defined as argminx { µ⊤x|Ax ≤ b ; x ≥ 0 ; x ∈ Zn } , where x is a vector of n decision variables ; µ ∈ Rn denotes the vector of objective coefficients ; the incidence matrix A ∈ Rm×n and right-hand-side ( RHS ) vector b ∈ Rm together define m linear constraints . With the above formulation , the size of an IP problem is generally reflected by the number of variables ( n ) and constraints ( m ) . Large Neighborhood Search ( LNS ) is a type of improvement heuristics , which iteratively reoptimizes a solution by the destroy and repair operator until certain termination condition is reached [ 32 ] . Specifically , the former one breaks part of the solution xt at step t , then the latter one fixes the broken solution to derive the next solution xt+1 . The destroy and repair operator together define the solution neighborhood N ( xt ) , i.e. , solution candidates that can be accessed at time step t. Compared to typical local search heuristics , LNS is more effective in exploring the solution space , since a larger neighborhood is considered at each step [ 32 ] . Most of existing LNS methods rely on problem specific operators , e.g. , removal and reinsertion for solutions with sequential structure [ 33 , 34 ] . Instead , we propose a RL based LNS for general IP problems , with the learned destroy operator to select variables for reoptimization at each step . 4 Methodology . In this section , we first formulate our LNS framework as a Markov Decision Process ( MDP ) . Then , we present the factorized representation of the large-scale action space , and parametrize the policy by a specialized GNN . Finally , we introduce a customized actor-critic algorithm to train our policy network for deciding the variable subset . 4.1 MDP formulation . Most of existing works learn LNS for specific problems [ 15 , 16 ] , which rely on extra domain knowledge and hinder their applicability to other COPs . In this paper , we apply LNS to general IP 2Nevertheless , our method can also work with solvers enhanced by the above methods as repair operators . problems , in which RL is employed to learn a policy that at each step , selects a variable subset from the solution to be reoptimized . We formulate this sequential decision problem as a discrete-time MDP . Specifically , we regard the destroy operator as the agent and the remainder in LNS as the environment . In one episode of solving an IP instance , the MDP can be described as follows : States . A state st ∈ S at each LNS step needs to not only reflect the instance information but also the dynamic solving status . For the former , we represent it by static features of variables and constraints , along with the incidence matrix in an instance . The latter is represented by dynamic features , including the current solution xt and dynamic statistics of incumbent solutions up to step t. Specifically , both the current and incumbent solution at t = 0 are defined as the initial solution x0 . Actions . At each state st , an action of the agent is to select a variable subset at from all candidate subsets A for reoptimization . Transition . The repair operator ( i.e. , an IP solver ) solves the sub-IP , where only the variables in action at are optimized with the others equaling to their current values in xt . Accordingly , the next state st+1 is deterministically attained by updating st with the new solution : xt+1 = argmin x { µ⊤x|Ax ≤ b ; x ≥ 0 ; x ∈ Zn ; xi = xit , ∀xi /∈ at } . ( 1 ) Rewards . The reward function is defined as rt = r ( st , at ) = µ⊤ ( xt − xt+1 ) , which is the change of the objective value . Given the step limit T of the interaction between the agent and environment , the return ( i.e. , cumulative rewards ) from step t of the episode is Rt = ∑T k=t γ k−trk , with the discount factor γ ∈ [ 0 , 1 ] . The goal of RL is to maximize the expected return E [ R1 ] over all episodes , i.e. , the expected improvement over initial solutions , by learning a policy . Policy . The stochastic policy π represents a conditional probability distribution over all possible variable subsets given a state . Starting from s0 , it iteratively picks an action at based on the sate st at each step t , until reaching the step limit T . An example of the proposed LNS framework solving an IP instance is illustrated in Figure 1 , in which the GNN based policy network has the potential to process instances of any size via sharing parameters across all variables .

Summary: In this paper, the authors propose an actor-critic reinforcement learning algorithm to learn policies for large neighborhood search (LNS). A central innovation is action factorization to tackle the high-dimensional action space. Extensive empirical evaluations are provided, covering different solver integrations (SCIP and Gurobi) as well as generalizations to larger problem instances.