Learning to delegate for large-scale vehicle routing

1 Introduction . Vehicle routing problems ( VRPs ) have enjoyed ample applications in logistics and ride-hailing services [ 23 ] around the world for decades . While determining the optimal solution to a VRP is NP-hard [ 25 ] , there have been numerous attempts to solve VRPs both exactly and approximately : provable algorithms have been designed for specific problem instances up to size 130 [ 28 ] , and powerful heuristic solvers such as LKH-3 [ 13 ] and HGS [ 49 , 48 ] find good solutions in practice for problems of size more than 1000 . However , heuristics methods often suffer from inflexibility due to extensive hand-crafting and the heavy computational burden from lengthy iterative procedures , as in the case of LKH-3 . For example , LKH-3 takes more than an hour to solve a size 2000 CVRP instance ( Table 1 ) , which is impractical for applications such as large-scale courier or municipal services . More recently , machine learning methods inspired by the Pointer Network [ 50 ] provide alternatives to traditional solvers : the learning-based methods greatly reduce computation time while maintaining decent solution quality on small problem instances ( less than 100 cities ) , by training on diverse sets of problem distributions either via supervised [ 50 ] or reinforcement learning [ 29 , 20 ] . However , these methods remain difficult to scale , and few report results on problems of size more than 200 . Our work aims to address scalability by learning to identify smaller subproblems which can be readily solved by existing methods . Our learned subproblem selector guides the problem-solving process to focus local improvement on promising subregions . While there exists a combinatorial number ⇤Equal Contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of subproblems that can be selected at each step , we leverage spatial locality commonly found in combinatorial optimization problems to restrict the selection space size to be linear in the problem size . Intuitively , objects far away from each other generally have very small influence on each other ’ s solution and are likely to be in different routes , so they should not be part of the same subproblem . The greatly reduced search space enables us to feasibly train an attention-based subproblem selector . Our framework combines the advantages of learning and heuristics : our network identifies promising subproblems to improve upon , dramatically speeding up solution times . Using a competitive subsolver on subproblems , we achieve good solution quality without the high computational costs of running the subsolver on large problem instances . In summary , our contributions are : • We propose learning-to-delegate , a learning-based framework for solving large-scale VRPs by iteratively identifying and solving smaller subproblems . • Despite the high dimensionality and NP-hardness of subproblems , we design a Transformer architecture that effectively predicts the subsolver ’ s solution quality for a subproblem . • With extensive validation , we show that learning-to-delegate offers significant speedups and/or objective improvements over both its base subsolver and random ( or heuristic ) subproblem selection , for a variety of VRP variants , distributions , and solvers . 2 Preliminary : Capacitated Vehicle Routing Problems ( CVRP ) . In CVRP , there is a depot node 0 and city nodes { 1 , ... , N } . Each city node i has demand di to fulfill . A vehicle with capacity C starts and ends at the depot and visits a route of city nodes such that the sum of city demands along the route does not exceed C , after which the vehicle starts a new route again . The objective is to find a valid solution minimizing the solution cost . We define the following : • Route : a sequence of nodes , where the first and last node are the depot 0 , and the rest are city nodes . In a valid route , the sum of demands of the city nodes does not exceed C. • Route cost : the sum of edge costs for the sequence of nodes . For an edge from node i to node j , the edge cost is the Euclidean distance between node i and j . • Solution : a feasible solution consists of a set of valid routes visiting each city exactly once . • Solution cost : the sum of route costs for all routes in the solution . An optimal solution is a feasible solution with the minimum solution cost . • Subproblem : a CVRP consisting of the depot , a subset of cities , and corresponding demands . 3 Related Work . Classical methods . Heuristics for solving combinatorial optimization problems have been studied for decades . The most powerful methods , such as local search [ 1 ] , genetic algorithms [ 36 ] , and ant colony methods [ 10 ] , involve iteratively improving the solution in a hand-designed neighborhood . For example , move , swap [ 52 ] , and 2-opt [ 9 ] are well-known heuristics for the traveling salesman and vehicle routing problems . The competitive VRP solver LKH-3 [ 13 ] uses the Lin–Kernighan heuristic [ 26 ] as a backbone , which involves swapping pairs of sub-routes to create new routes , whereas the CVRP solver HGS [ 49 , 48 ] uses a hybrid genetic and local search procedure to achieve state-of-the-art solution qualities on problems up to size 1000 . While LKH-32 tackles a variety of VRP variants , the publicly available implementation of HGS 3 only solves CVRP . For large problems , low-level heuristics are combined with meta-heuristics , including Tabu Search with Adaptive Memory [ 40 ] , guided local search [ 51 ] , and Large Neighborhood Search [ 35 ] . The inspiration for our work derives from Partial OPtimization Metaheuristic Under Special Intensification Conditions ( POPMUSIC ) , which iteratively optimizes problem subparts and has been used to solve problems as diverse as map labeling [ 24 ] , berth allocation [ 22 ] , and p-median clustering [ 39 ] . Despite the promise of the POPMUSIC framework for large-scale combinatorial optimization , the impact of certain design choices is not well understood , such as the subproblem selection ordering 2We refer to the LKH-3 code at http : //webhotel4.ruc.dk/~keld/research/LKH-3/ . 3We refer to the HGS code at https : //github.com/vidalt/HGS-CVRP . and size [ 43 ] . For example , it is not clear how to meaningfully order the subproblems . One early work in this direction demonstrated that a last-in-first-out stack order performs better than random [ 3 ] . Our work provides a natural approach to order the subproblems based on predicted improvement . Deep learning methods . Recently , there has been a surge of interest in training deep neural networks ( DNN ) to solve combinatorial problems . As observed by Kwon et al . [ 21 ] , most methods fall into one of the following two categories : 1 ) construction methods [ 29 , 20 ] , where an autoregressive model such as the Pointer Network [ 50 ] directly outputs a solution , and 2 ) improvement methods [ 7 , 15 ] , where the DNN iteratively performs local updates to the solution , resembling local search . These methods are approaching the solution quality of LKH-3 [ 13 ] on small problem instances . For example , Kwon et al . [ 21 ] extends a construction approach [ 20 ] to encourage more diverse output samples on N 100 VRPs . Among improvement methods , Lu et al . [ 27 ] learn a meta-controller to select among a set of local search heuristics , marking the first learning-based approach to outperform LKH-3 on N 100 VRPs . Despite these successes , learning-based approaches for large-scale VRPs are poorly understood . In addition , all of the aforementioned methods are trained using deep reinforcement learning ; for large problems , trajectory collection becomes prohibitively expensive . Scaling up learning methods . A few recent works have begun to investigate scaling of learned networks for NP-hard graph problems . For example , Ahn et al . [ 2 ] propose an iterative scheme called learning-what-to-defer for maximum independent set . At each stage , for each node in the graph , the network either outputs the solution for the node or defers the determination to later stages . Song et al . [ 38 ] proposes an imitation-learning-based pipeline for Integer Linear Programs ( ILP ) , where at each stage they partition all variables into disjoint subsets , and use the Gurobi [ 12 ] ILP solver to solve the partitions . Due to differences in graph structure , our work presents a more natural scheme to handle VRP constraints and structures . A few works attempt to incorporate learning to decompose large-scale VRPs [ 6 , 47 , 31 ] . However , the decomposition approaches proposed appear to be experimentally less effective than ours . 4 An Iterative Framework for VRPs . While CVRPs are NP-hard and thus require worst-case exponential time in the problem size to solve optimally , typical CVRPs exhibit structure that practical solvers may exploit . We hypothesize that in such situations , the larger problem can be efficiently approximately solved as a sequence of smaller subproblems , which can be delegated to efficient subsolvers . To test this hypothesis , we propose a learning-based , iterative framework with two components : 1 ) a subsolver capable of solving a small problem instance ( exactly or approximately ) , and 2 ) a learned model for identifying suitable subproblems within a larger problem to delegate to the subsolver . Algorithm 1 : Learning to Delegate Input : Problem instance P , initialized solution X , subproblem selector f✓ , subsolver Subsolver , number of steps T , parameter k denoting the size of subproblems 1 for Step t = 1 : T do 2 Sk , local ConstructSubproblems ( P , X , k ) 3 S f✓ ( Sk , local ) 4 X 0S Subsolver ( S ) 5 X X 0S [ XP\S 6 end for We illustrate our iterative framework in Figure 1 , which takes in a problem instance P with a feasible initial solution X0 . At each step , given the current solution X , we select a smaller subproblem S ⇢ P with our learned model f✓ then apply the subsolver to solve S ; we then update X with the new solution for S. To maintain feasibility after an update , we restrict S to be the set S of visited cities of a subset of routes from X . Since routes with cities in P \ S remain valid routes , we obtain a new feasible solution X 0 = X 0S [ XP\S , where X 0S is the subsolution for S and XP\S consists of unselected routes from X . Intuitively , a strong f✓ should identify a subproblem S such that the subsolver solution X 0S results in a large improvement in objective from X to X 0 . 4.1 The Restricted Subproblem Selection Space . As the number of routes in P is R = O ( N ) , the cardinality of the selection space S is O ( 2R ) , which is exponential in the problem size N and difficult for learned models to consider . If we restrict each subproblem to cities from exactly k routes , there are still R k = O ( Rk ) subproblems to consider . Therefore we further restrict selection to subproblems with spatial locality . As shown in Figure 1 ( d ) , we only consider subproblems from Sk , local , where each subproblem is centered around a particular route r and contains the k routes whose centroids have the smallest Euclidean distance to the centroid of r. In this way , we reduce the selection space to |Sk , local| = R = O ( N ) from |S| = O ( 2N ) . In Algorithm 1 , we refer to this restriction as ConstructSubproblems . Our restriction to a local selection space is motivated by the fact that many combinatorial optimizations problems have inherent spatial locality , i.e . problem entities are more strongly affected by nearby entities than faraway entities . Earlier heuristical methods such as POPMUSIC [ 42 ] leverage similar spatial locality .

Summary: This paper proposes an iterative framework to solve large-scale Vehicle Routing Problems. At each iteration, given the current feasible solution, a learned component selects subroutes that form a VRP subproblem. Then the subproblem is solved using a standard VRP solver (LKH-3) and the subsolution is used to update the current solution. The subproblem selector uses a Transformer encoder and predicts the subsolution cost; it is trained in a supervised way. Experimental results show that the method is able to reach the performance of the VRP solver LKH with significantly shorter computation times.