Rewriting by Generating: Learn Heuristics for Large-scale Vehicle Routing Problems

1 INTRODUCTION . The Large-Scale Vehicle Routing Problems ( VRPs ) is an important combinatorial optimization problem defined upon an enormous distribution of customer nodes , usually more than a thousand . An efficient and high-quality solution to large-scale VRPs is critical to many real-world applications . Meanwhile , most existing works focus on finding near-optimal solutions with only no more than a hundred customers because of the computational complexity ( Laporte , 1992 ; Golden et al. , 2008 ; Braekers et al. , 2016 ) . Originated from the NP-hard nature as a VRPs , the exponential expansion of solution space makes it much more difficult than solving a small-scale one . Therefore , providing effective and efficient solutions for large-scale VRPs is a challenging problem ( Fukasawa et al. , 2006 ) . Current algorithms proposed for routing problems can be divided into traditional non-learning based heuristics and reinforcement learning ( RL ) based models . Many routing solvers involve heuristics as their core algorithms , for instance , ant colony optimization ( Gambardella et al. , 1999 ) and LKH3 ( Helsgaun , 2017 ) , which can find a near optimal solution by greedy exploration . However , they become inefficient when the problem scale extends . Apart from traditional heuristics , RL based VRPs solvers have been widely studied recently to find more efficient and effective solutions ( Dai et al. , 2017 ; Nazari et al. , 2018 ; Bello et al. , 2017 ; Kool et al. , 2019 ; Chen & Tian , 2019 ; Lu et al. , 2020 ) . Thanks to the learning manner that takes every feedback from learning attempts as signals , RL based methods rely on few hand-crafted rules and thus can be widely used in different customer distributions without human intervention and expert knowledge . Besides , these RL methods benefit from a pre-training process allowing them to infer solutions for new instances much faster than traditional heuristics . However , current RL agents are still insufficient to learn a feasible policy and generate solutions directly on large-scale VRPs due to the vast solution space , which is usually N ! for N customers . More specifically , the solution space of a large-scale VRPs with 1000 customers is e2409 much larger than that of a small-scale one with only 100 customers . Consequently , the complexity makes the agent difficult to fully explore and makes the model hard to learn useful knowledge in large-scale VRPs . To avoid the explosion of solution space in large-scale VRPs , we consider leveraging the classic Divide-and-Conquer idea to decompose the enormous scale of the original problem . In particularly , 1Codes and data will be released at https : //github.com/RBG4VRPs/Rewriting-By-Generating dividing the large-scale customer distributions into small-scale ones and then generating individual regional solutions to reduce the problem complexity . However , how to obtain a refined region division where the local VRPs can be handled effectively and how to coordinate iterations between global and local optimization efficiently remain two challenges of our VRPs solvers . To tackle those two challenges above , we propose an RL-based framework , named Rewriting-byGenerating ( RBG ) , to solve large-scale VRPs . The framework adopts a hierarchical RL structure , which consists of a `` Generator '' and a `` Rewriter '' . Firstly , We divide customers into regions and use an elementary RL-based VRPs solver to solve them locally , known as the `` Generation '' process . After that , from a global perspective , a special `` Rewriting '' process is designed based on all regional generations , which rewrites the previous solution with new divisions and the corresponding new regional VRPs results . Within each rewriting step , we select and merge two regions into a hyperregion , and then further divide it into two new sub-regions according to the hyper-regional VRPs solution . By doing this , the problem scale is decomposed into pieces and the problem could be solved efficiently using regional RL-based solvers , and can still preserve the solution quality which is improved by the rewriter continuously . Extensive experiments demonstrate that our RBG framework achieves significant performance in a much more efficient manner . It has a significant advantage on solution quality to other RL-based methods , and outperforms the state-of-the-art LKH3 ( Helsgaun , 2017 ) , by 2.43 % with the problem size of N = 2000 and could infer solutions about 100 times faster . Moreover , it also has a growing superiority to other methods when the problem scale increases . Notations : We introduce some fundamental notations of large-scale VRPs , while the complete formulation is presented in the Appendix . Let G ( V , E ) denote the entire graph of all customers and the depot . Specifically , V = { v0 , v1 , ... , vi , ... , vN } , where v0 denotes the depot , and vi ( 1 ≤ i ≤ N ) denotes the i-th customer with its location ( xi , yi ) and its demand di . The edge ei , j , or E ( vi , vj ) in another manner represents the traveling distance between vi and vj . Within the RBG framework , the generated regional VRPs solution πk = { vk,0 , vk,1 , vk,2 , ... , vk , Nk } of the divided region Gk has a corresponding traveling cost C ( πk ) = ∑Nk i=0E ( vk , i , vk , i+1 ) . The entire solution of all customers is denoted by π . 2 RELATED WORK . We discuss previous works which are related to our research in the following two directions : Traditional Heuristics . Since the exact methods ( Laporte , 1992 ; Laporte & Nobert , 1987 ; Holland , 1992 ; Baldacci et al. , 2010 ) are almost impossible to solve VRPs within a reasonable time due to the high computation complexity , researchers developed heuristics , i.e. , non-exact methods , to find approximation solutions instead . Tabu search is one of the old metaheuristics ( Glover , 1990b ; a ; Gendreau et al. , 1994 ; Battiti & Tecchiolli , 1994 ) , which keeps searching for new solutions in the neighborhood of the current solution . Instead of focusing on improving merely one solution , genetic algorithms operate in a series of solutions ( Goldberg , 1989 ; Holland , 1992 ) . It constructs new structures continuously based on parent structures . Instead of treating objectives to be optimized altogether , ant colony optimizations as another widely accepted solver , utilize several ant colonies to optimize different functions : the number of vehicles , the total distance and others ( Gambardella et al. , 1999 ; Dorigo et al. , 2006 ; Dorigo & Di Caro , 1999 ) . Meanwhile , recreate search methods keep constructing the current solution and ruining the current ones to build better solutions . ( Schrimpf et al. , 2000 ) . This helps to expand the exploration space to prevent the local optimum . Among these categories , LKH3 is a state-of-the-art heuristic solver that empirically finds optimal solutions ( Helsgaun , 2017 ) . Although these heuristics , compared to exact methods , can improve searching efficiency , they are still much too time-consuming when applied to large-scale VRPs with the acceptable performance required , and may fail to respond to any real-time solution requests . RL based VRPs Solutions . Since the learning manner of reinforcement learning allows the agent model to directly infer solutions based on a pre-trained model with much shorter computation time , RL becomes a compelling direction on solving combinatorial optimizations . It has been successfully applied in VRPs particularly ( Bello et al. , 2017 ; Nazari et al. , 2018 ; Kool et al. , 2019 ) . Vinyals et al . ( 2015 ) was the first to adopt deep learning in combinatorial optimizations by a novel Pointer Network model . Inspired by this , Bello et al . ( 2017 ) proposed to use RL to learn model parameters as an optimal strategy instead of relying on ground-truth in a supervised learning way , which demonstrates the effectiveness on TSP and the knapsack problem . Nazari et al . ( 2018 ) further followed the idea to solve VRPs with attention mechanism as augmentation , and Kool et al . ( 2019 ) solved more generalized combinatorial optimization problems . Other than using the idea of PointerNetwork , Dai et al . ( 2017 ) develops their method over graphs via Q-learning ( Sutton & Barto , 2018 ) , so that the solutions could have better generalization ability . Chen & Tian ( 2019 ) proposed a local rewriting rule that keeps rewriting the local components of the current situation via a Q-Actor-Critic training process ( Sutton & Barto , 2018 ) . Lu et al . ( 2020 ) further developed a Learn-to-Iterate structure that not only improves the solution exploration but also generates perturbations to avoid local optimum . This is the first machine learning framework that outperforms LKH3 on CVRPs ( capacitated VRPs ) , both in computation time and solution quality . However , these existing RL based methods only achieve promising results without any hand-craft rules and expertise at small scales with usually no more than a hundred customers . The proposed models can not be trained for thousand-customer-level VRPs because the state space and action space extend exponentially as the number of customers increases , and it will be hard for the model to learn useful route generation policy . In contrast , we propose an RL based framework formed upon the classical idea of Divide-and-Conquer to solve the large-scale challenge . 3 REWRITING-BY-GENERATING . Figure 1 shows the overview structure of our proposed framework , named Rewriting-by-Generating ( RBG ) . Along with the fundamental idea of Divide-and-Conquer to decompose the enormous problem scale as discussed previously , we aim at dividing the total customers into separate regions and generate near-optimal solutions individually . To achieve this , we design a hierarchical RL structure including two agents which take different functions . First , to refine and obtain more reasonable division results , we design the `` Rewriting '' process which keeps up- dating new divisions by rewriting the previous ones and their corresponding regional solutions . The division quality is critical to the final solution since customers from different regions can not be scheduled upon the same route . Within each rewriting step , the agent selects and merges two regions based on their generated solutions . A new solution will be generated upon the merged hyper-region in the following step , and the rewriter will further divide the merge hyper-region back to two new regions . Since the exploration on different customer composition is complicated and it is not trivial to measure the direct influence to the final performance in terms of traveling distance , an RL-based rewriter is a wise choice to learn the selection and merging action . We will show that the model converges and achieves high performance when the rewriter agent learns a stable division result in Section 4 . Second , to reach the global solution from the regional scratches , we employ an elementary VRPs generator that generates solutions to each region , known as the `` Generating '' process . Considering the time efficiency and the ability to learn to solve certain customer distributions when the division updates continuously , we also apply an RL agent to learn to generate solutions on these smaller-scale regions . Overall , we develop a hierarchical RL framework by coordinating the rewriter and the generator in two different scales iteratively . The rewriter updates new division and brings new customer distributions to the generator , while the solutions from the generator formulate a key component of the rewriter . From the technical perspective , it is worthy to note that the merging-repartioning operation that our rewriter conducts is also adopted in previous meta-heuristics ( Baker & Ayechew , 2003 ; Bell & McMullen , 2004 ) , while we replace the handcrafted heuristic with a learning agent . The global RL based rewriter is responsible for managing inter-regional exploration while the generator optimizes local results . The combination of Operation Research ( OR ) heuristics and RL guarantees an effective exploration process as well as achieving high computation efficiency from the prospect of fast solution generation on inference instances . For brevity and clarity , we summary the pipeline as five steps , as shown in Figure 2 . First , we cluster customers into several initialized hyper-regions . Second , we generate an initial regional VRPs solution in individual hyper-regions via our elementary VRPs generator . Third , we utilize the rewriter to partition the merged graph to two sub-regions . Then our rewriter picks up two sub-regions via the attention mechanism and merges them into one hyper-region , and then generates the hyper-regional solution of the merged hyper-region . After that , we go back to the third step to re-partition the hyper-region into sub-regions in a loop . Through this process , the partition becomes more reasonable and the solution gets better and better . Finally after enough steps of rewriting we are able to reach a good solution .

Summary: The paper presents a hierarchical reinforcement learning approach to solve large-scale vehicle routing problems (VRPs). A “rewriting agent” is responsible for dividing the customers into regions while a “generating agent” is responsible for computing the vehicle routes in each region, independently. The rewriting agent learns to score pairs of regions to be merged, using as a reward the reduction of VRP cost gained by the merge. The VRP costs are computed by the generating agent, which is based on the attention model of (Kool et al 2019), that is known to perform well for smaller scale VRPs.