A Learning-based Iterative Method for Solving Vehicle Routing Problems

1 INTRODUCTION . In this paper , we focus on an important class of combinatorial optimization , vehicle routing problems ( VRP ) , which have a wide range of applications in logistics . Capacitated vehicle routing problem ( CVRP ) is a basic variant of VRP , aiming to find a set of routes with minimal cost to fulfill the demands of a set of customers without violating vehicle capacity constraints . The CVRP is NP-hard ( Dantzig & Ramser , 1959 ) , and both exact and heuristic methods have been developed to solve it ( Fukasawa et al. , 2006 ; Golden et al. , 2008 ; Kumar & Panneerselvam , 2012 ; Toth & Vigo , 2014 ) . In recent years , especially after the seminal work of Pointer Networks ( Vinyals et al. , 2015 ) , researchers start to develop new deep learning and reinforcement learning ( RL ) frameworks to solve combinatorial optimization problems ( Bello et al. , 2016 ; Mao et al. , 2016 ; Khalil et al. , 2017 ; Bengio et al. , 2018 ; Kool et al. , 2019 ; Chen & Tian , 2019 ) . For the CVRP itself , a number of RLbased methods have been proposed in the literature ( Nazari et al. , 2018 ; Kool et al. , 2019 ; Chen & Tian , 2019 ) . The learning based methods are trained on a huge number of problem instances , and have been shown to be extremely fast in producing solutions of reasonably good quality . However , when tested with the same benchmark instances , these learning-based methods can not outperform the state-of-the-art method LKH3 ( Helsgaun , 2017 ) , which is a penalty-function-based extension of classical Lin-Kernighan heuristic ( Lin & Kernighan , 1973 ; Helsgaun , 2000 ) . For example , on CVRP with 100 customers , LKH3 is able to produce an average cost of 15.65 . This line of research motivated us to study a framework that combines the strength of Operations Research ( OR ) heuristics with learning capabilities of machine learning ( RL in particular ) . Machine learning can learn ∗Equal contribution . to solve a class of problem instances fast , when test instances are generated from the same distribution as training instances . Classical approaches like search algorithms are effective but may need heavy computation , which is time-consuming . Our research interest is in fusing the strengths of these two worlds . Another related line of research is hyper-heuristics , which is “ a search method or learning mechanism for selecting or generating heuristics to solve computational search problems ” ( Burke et al. , 2013 ) . Instead of developing a high-level methodology without the need of knowing the details of low-level heuristics , we are primarily interested in a closely integrated system that best utilizes the strength of OR operators and learning capability . Our Contributions . Instead of directly constructing a solution from the problem instance ( Graves et al. , 2014 ; Sutskever et al. , 2014 ; Vinyals et al. , 2015 ) , we propose a framework that iteratively searches among solutions , until a certain termination condition is satisfied . Our main contributions are as follows : • We present a learning-based algorithm for solving CVRP , achieving new state-of-the-art results . The recent line of works using RL to solve CVRP shows the potential of machine learning algorithms . They can solve CVRP faster , but can not beat classical OR solvers like LKH3 in term of solution quality . Our algorithm is the first machine learning framework that outperforms LKH3 on CVRP . • We propose a novel hierarchical framework . Instead of putting all operators in one action pool , we separate heuristic operators into two classes , namely improvement operators and perturbation operators . At each state , we choose the class first and then choose operators within the class . Learning from the current solution is made easier by focusing RL on the improvement operators only . • We propose an ensemble method , which trains several RL policies at the same time , but with different state input features . The ensemble method is shown to produce superior results than individual policies with an equivalent amount of computation . Related Work . In recent years , there have been many studies using deep learning and RL to solve combinatorial optimization problems ( Smith , 1999 ; Mao et al. , 2016 ; Lodi & Zarpellon , 2017 ; Veličković et al. , 2017 ; Lombardi & Milano , 2018 ; Bengio et al. , 2018 ) . Routing problems , especially traveling salesman problems ( TSP ) and VRP , have been explored by a sequence of works ( Vinyals et al. , 2015 ; Bello et al. , 2016 ; Khalil et al. , 2017 ; Li et al. , 2018 ; Deudon et al. , 2018 ; Kaempfer & Wolf , 2018 ; Nazari et al. , 2018 ; Kool et al. , 2019 ; Chen & Tian , 2019 ) . Most of these works , with the exception of Chen & Tian ( 2019 ) , follow an end-to-end approach , which is directly constructing a solution from scratch . Vinyals et al . ( 2015 ) first introduce the Pointer Network , inspired by sequence-to-sequence models , to solve TSP . They use an attention model to learn the order of different nodes in a supervised fashion . Later Bello et al . ( 2016 ) develop an RL algorithm to train the Pointer Network . Their framework learns the optimal policy from problem instances and needs no supervised solutions . Nazari et al . ( 2018 ) improve the Pointer Network with a new design , making the model invariant with respect to the input sequence , and extend it to solve VRP . Kool et al . ( 2019 ) propose a model based on attention layers , and an RL algorithm to train this model with a simple but effective baseline . Chen & Tian ( 2019 ) propose a NeuRewriter model for VRP . They define a rewriting rule set and train two policy networks , a region-picking policy and a rule-picking policy , to obtain the next state . Given an initial solution , their goal is to find a sequence of steps towards the solution with minimal cost . 1.1 NOTATIONS . In CVRP , there is a depot and a set ofN customers . Each customer i , i ∈ { 1 , . . . , N } , has a demand di to be satisfied . A vehicle , which always starts at and ends at the depot , can serve a set of customers as long as the total customer demand does not exceed the capacity of the vehicle . The traveling cost ci , j is the cost of a vehicle going from node i to j , with i , j ∈ { 0 , 1 , . . . , N } ( where the depot is denoted by node 0 for convenience ) . The objective is to find a routing plan with minimal cost that serves all customers without violating vehicle capacity constraints . Figure 1 gives an illustration of CVRP , while a mathematical formulation is given in the Appendix . Route . A route is a sequence of nodes visited in order , with the depot ( node 0 ) as the starting and ending node . For example , [ 0 , 1 , 3 , 4 , 0 ] is a traveling plan that starts at the depot , visits node 1 , 3 and 4 sequentially , and returns to the depot . Solution . A solution is a set of routes such that each customer is visited exactly once , and the total demand along each route is less than the vehicle capacity . Operator . An operator is a mapping from one solution to another . In this paper , instead of directly constructing a solution from scratch , we improve or perturb the solution iteratively using operators . 2 LEARN TO IMPROVE . In this section we formally introduce our “ Learn to Improve ” ( L2I ) framework , including main components of the system as well as the design intuitions behind individual components . Figure 2 illustrates the overall framework . As mentioned previously , the framework is iterative in nature , that is , we always start with a feasible solution , continuously improving the solution or perturbing it . Along the way , all constraints remain satisfied . By always maintaining the feasibility of the solution , we are exploring the space of feasible solutions and any of them found by our search process could potentially be a good solution . After T ( a parameter set in advance ) steps the algorithm stops , and we choose the one with the minimum traveling cost as our final solution . Our framework has a few distinct components ( e.g. , how to improve the solution , when and how to perturb it ) , which could be rule-based , learned by machine learning or mixed , and thus allows us to experiment with different ways of integrating machine learning and OR , some of which could lead to superior methods either in terms of better solution quality or in terms of computational efficiency . It is worthwhile to point out that , as contrast to the work of Chen & Tian ( 2019 ) , the solution space we explore is much larger since we are employing a rich set of improvement and perturbation operators . For this research work , we implemented and experimented with a number of design choices , and ended up with a method that is both computationally efficient and able to produce state-of-the-art results . The details of the main components of our method are as follows . Given a history of most recent solutions , our method uses a threshold-based rule to decide whether we should continue to improve the current solution , or should perturb it and restart with the perturbed solution . If it decides that the current solution could still be improved , it will use an RL-based controller to choose one of the improvement operators , and try to improve the solution with the chosen operator . We have a rich set of improvement operators ( the list of improvement operators and their details are given in the Appendix ) , where intra-route ones attempt to reduce the cost of current solution by moving customers to different positions in individual routes , while inter-route ones attempt to reduce the cost by moving customers among different routes . Given that the improvement operators are of distinct characteristics , it is not straightforward to know in advance which operators are most effective for the problem under investigation . It is also difficult to know a pre-defined ordering of the operators that is best for the problem . Thus , an RL-based controller is a good choice to learn the set of improvement operators that is more effective for the problem ( or more specifically , the problem and the data distribution from which the training and test instances are sampled ) . As will be shown in Section 3 , our RL model is able to differentiate more useful improvement operators from less useful ones for CVRP , as well as generate an implicit ordering on how the operators will be applied . On the other hand , upon reaching a local minimum the perturbation controller randomly chooses a perturbation operator that destroys ( completely or partially ) and reconstructs a number of routes to generate a new starting solution ( Table 6 in the Appendix gives the list of perturbation operators and their details ) . Specifically , if no cost reduction has been made for L improvement steps , we perturb the solution and restart the improvement iteration ( where for ease of explanation a maximum consecutive sequence of improvement operators applied before perturbation is called an improvement iteration ) . As a perturbation changes the solution quite dramatically ( by producing a substantially different solution that is usually worse than the current one ) , we found that it is useful to start a new improvement iteration with a reasonably good starting point ( e.g . by filtering out the restarting solutions that are significantly worse than current solution or currently best solution ) . It is clear that we purposely separate improvement operators from perturbation ones , and an alternative design would be to mix them all together and have a single controller deciding which operator to apply next . However , the improvement operators are of different nature from the perturbation ones , and their impacts are different since the perturbation operators have long-lasting effect by affecting an entire improvement iteration . Our experience also suggests that learning is made easier by focusing RL on the improvement operators only . Lastly , it is worthwhile to point out that although the rulebased perturbation controller is shown to be effective , we do not rule out the possibility that it can also be learning-based . The framework described above provides a way of combining the strength of OR operators , which are powerful since they are custom-made for routing problems , with learning capabilities of RL , which is flexible and can be adapted to a given problem and its associated data distribution . Having described our overall framework , we are now ready to present the details of the improvement controller and operators .

Summary: This paper proposes a framework combining RL and OR for solving the capacitated vehicle routing problem (CVRP). The main idea is to train an RL-based controller for choosing the OR operations necessary for improving & reinitializing the solution. Additionally, an ensemble method for diversifying the solution is proposed. The proposed method is shown to empirically outperform the existing OR method LKH3 for solving CVRP.