Deep Policy Dynamic Programming for Vehicle Routing Problems

Routing problems are a class of combinatorial problems with many practical1 applications . Recently , end-to-end deep learning methods have been proposed2 to learn approximate solution heuristics for such problems . In contrast , classical3 dynamic programming ( DP ) algorithms guarantee optimal solutions , but scale4 badly with the problem size . We propose Deep Policy Dynamic Programming5 ( DPDP ) , which aims to combine the strengths of learned neural heuristics with6 those of DP algorithms . DPDP prioritizes and restricts the DP state space using7 a policy derived from a deep neural network , which is trained to predict edges8 from example solutions . We evaluate our framework on the travelling salesman9 problem ( TSP ) , the vehicle routing problem ( VRP ) and TSP with time windows10 ( TSPTW ) and show that the neural policy improves the performance of ( restricted ) 11 DP algorithms , making them competitive to strong alternatives such as LKH , while12 also outperforming most other ‘ neural approaches ’ for solving TSPs , VRPs and13 TSPTWs with 100 nodes.14 1 Introduction15 Dynamic programming ( DP ) is a powerful framework for solving optimization problems by solving16 smaller subproblems through the principle of optimality [ 3 ] . Famous examples are Dijkstra ’ s17 algorithm [ 14 ] for the shortest route between two locations , and the classic Held-Karp algorithm for18 the travelling salesman problem ( TSP ) [ 23 , 4 ] . Despite their long history , dynamic programming19 algorithms for vehicle routing problems ( VRPs ) have seen limited use in practice , primarily due to20 their bad scaling performance . More recently , a line of research has attempted the use of machine21 learning ( especially deep learning ) to automatically learn heuristics for solving routing problems22 [ 57 , 5 , 41 , 29 , 7 ] . While the results are promising , most learned heuristics are not ( yet ) competitive23 to ‘ traditional ’ algorithms such as LKH [ 24 ] and lack ( asymptotic ) guarantees on their performance.24 In this paper , we propose Deep Policy Dynamic Programming ( DPDP ) as a framework for solving25 vehicle routing problems . The key of DPDP is to combine the strengths of deep learning and DP,26 by restricting the DP state space ( the search space ) using a policy derived from a neural network.27 In Figure 1 it can be seen how the neural network indicates promising parts of the search space28 ( through a heatmap over the edges of the graph ) , which is then used by the DP algorithm to find a29 good solution . DPDP is more powerful than some related ideas [ 62 , 52 , 61 , 6 , 34 ] as it combines30 supervised training of a large neural network with just a single model evaluation at test time , to enable31 running a large scale guided search using DP . The DP framework is flexible as it can model a variety32 of realistic routing problems with difficult practical constraints [ 20 ] . We illustrate this by testing33 DPDP on the TSP , the capacitated VRP and the TSP with ( hard ) time window constraints ( TSPTW ) .34 In more detail , the starting point of our proposed approach is a restricted dynamic programming35 algorithm [ 20 ] . Such an algorithm heuristically reduces the search space by retaining only the B most36 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . promising solutions per iteration . The selection process is very important as it defines the part of the37 DP state space considered and , thus , the quality of the solution found ( see Fig . 2 ) . Instead of manually38 defining a selection criterion , DPDP defines it using a ( sparse ) heatmap of promising route segments39 obtained by pre-processing the problem instance using a ( deep ) graph neural network ( GNN ) [ 26 ] .40 This approach is reminiscent of neural branching policies for branch-and-bound algorithms [ 19 , 40 ] .41 In this work , we thus aim for a ‘ neural boost ’ of DP algorithms , by using a graph neural network42 for scoring partial solutions . Prior work on ‘ neural ’ vehicle routing has focused on auto-regressive43 models [ 57 , 5 , 13 , 29 ] , but they have high computational cost when combined with ( any form of ) 44 search , as the model needs to be evaluated for each partial solution considered . Instead , we use ( for45 TSP ) and adapt ( for VRP and TSPTW ) a model to predict a heatmap indicating promising edges [ 26 ] ,46 and define the score of a partial solution as the ‘ heat ’ of the edges it contains ( plus an estimate of the47 ‘ heat-to-go ’ or potential of the solution ) . As the neural network only needs to be evaluated once for48 each instance , this enables a much larger search ( defined by B ) , making a good trade-off between49 quality and computational cost . Additionally , we can apply a threshold to the heatmap to define a50 sparse graph on which to run the DP algorithm , reducing the runtime by eliminating many solutions.51 Figure 2 illustrates the overall DPDP algorithm . In Section 4 , we show that DPDP significantly52 improves over ‘ classic ’ restricted DP algorithms ( with the same B ) . Additionally , we show that53 DPDP outperformes most other ‘ neural ’ approaches for TSP , VRP and TSPTW and is competitive54 with the highly-optimized LKH solver [ 24 ] for VRP , while achieving similar results much faster for55 TSP and TSPTW . For TSPTW , DPDP also outperforms the best open-source solver we could find56 [ 10 ] , illustrating the power of DPDP to handle difficult hard constraints ( time windows ) .57 2 Related work58 DP has a long history as an exact solution method for routing problems [ 31 , 50 ] , e.g . for the TSP59 with time windows [ 15 ] and precedence constraints [ 39 ] , but typically limited to small problems only,60 due to the curse of dimensionality . Restricted DP ( with heuristic policies ) has been used to address,61 e.g. , the time dependent TSP [ 37 ] , and has been generalized into a flexible framework for VRPs with62 different types of practical constraints [ 20 ] . DP approaches have also been shown to be useful in63 settings with difficult practical issues such as time-dependent travel times and driving regulations [ 28 ] 64 or stochastic demands [ 42 ] . For a thorough investigation of modelling choices of DP for routing ( and65 scheduling ) , see [ 53 ] . For sparse graphs , alternative , but less flexible , formulations can be used [ 8 ] .66 Despite the flexibility , constructive DP methods have not gained much popularity compared to67 heuristic search approaches such as Ruin and Recreate [ 47 ] , Adaptive Large Neighborhood Search68 [ 46 ] , LKH [ 24 ] or FILO [ 1 ] . While highly effective , these methods are limited in their flexibility as69 special operators need to be engineered for different types of problems . While restricted DP was70 shown to have superior performance on realistic VRPs with many constraints [ 20 ] , the performance71 gap of around 10 % for standard ( benchmark ) VRPs ( with time windows ) is too large to popularize72 the restricted dynamic programming approach . We argue that the missing ingredient for restricted73 dynamic programming is the availability of a strong but computationally cheap policy for selecting74 which solutions should be considered , which is the motivation behind DPDP.75 In the machine learning community , recent advances have significantly improved deep neural networks76 ( DNNs ) to perform tasks such as image classification and machine translation [ 32 ] . After the first77 deep learning model was trained ( using example solutions ) to construct TSP tours [ 57 ] , many78 improvements have been proposed , e.g . different training strategies such as reinforcement learning79 ( RL ) [ 5 , 27 , 12 , 30 ] and model architectures , which enabled the same idea to be used for other80 routing problems [ 41 , 29 , 13 , 45 , 16 , 60 ] . Most constructive neural methods are auto-regressive,81 evaluating the model many times to predict one node at the time , but other works have considered82 predicting a ‘ heatmap ’ of promising edges at once [ 43 , 26 , 17 ] , which allows a tour to be constructed83 ( using sampling or beam search ) without further evaluating the model . An alternative direction is84 ‘ learning to search ’ , where a neural network is used to guide a search procedure such as local search85 [ 7 , 35 , 18 , 59 , 25 ] . Some works have attempted scaling to larger instances beyond 100 nodes , which86 remains challenging [ 36 , 17 ] . The combination of machine learning with DP has been proposed in87 limited settings [ 62 , 52 , 61 ] . Most related to our approach , a DP algorithm for TSPTW , guided by an88 RL agent , was implemented using an existing solver [ 6 ] and a neural network predicting edges has89 been combined with tree search [ 34 ] and local search for maximum independent set ( MIS ) . For a90 wider view on machine learning for routing problems and combinatorial optimization , see [ 38 , 54 ] .91 3 Deep Policy Dynamic Programming92 DPDP uses an existing graph neural network [ 26 ] ( which we modify for VRP and TSPTW ) to predict93 a heatmap of promising edges , which is used to derive the policy for scoring partial solutions in94 the DP algorithm . The DP algorithm starts with a beam of a single initial ( empty ) solution . It then95 proceeds by iterating the following steps : ( 1 ) all solutions on the beam are expanded , ( 2 ) dominated96 solutions are removed for each DP state , ( 3 ) the B best solutions according to the scoring policy97 define the beam for the next iteration . These steps are illustrated in Fig . 2 . The objective function is98 used to select the best solution from the final beam . The resulting algorithm is a beam search over the99 DP state space ( which is not a ‘ standard beam search ’ over the solution space ! ) and we call B the100 beam size . DPDP is asymptotically optimal as using B = n · 2n for a TSP with n nodes guarantees101 optimal results , but choosing smaller B allows to trade performance for computational cost.102 DPDP is a generic framework that can be applied to different problems , by defining the following103 ingredients : ( 1 ) the state variables to track while constructing solutions , ( 2 ) the initial solution,104 ( 3 ) feasible actions to expand solutions , ( 4 ) rules to define dominated solutions and ( 5 ) a scoring105 policy for selecting the B solutions to keep . A solution is always ( uniquely ) defined as a sequence of106 actions , which allows the DP algorithm to construct the final solution by backtracking . In the next107 sections , we define these ingredients for the TSP , VRP and TSPTW.108 3.1 Travelling Salesman Problem109 We implement DPDP for Euclidean TSPs with n nodes on a ( sparse ) graph , where the cost for edge110 ( i , j ) is given by cij , the Euclidean distance between the coordinates of nodes i and j.111 For each partial solution , defined by a sequence of actions a , the state variables are cost ( a ) , the112 total cost ( distance ) , current ( a ) , the current node , and visited ( a ) , the set of visited nodes ( including113 the start node ) . Without loss of generality , we let 0 be the start node , so we initialize the beam at step114 t = 0 with the empty initial solution with cost ( a ) = 0 , current ( a ) = 0 and visited ( a ) = { 0 } . At115 step t , the action at ∈ { 0 , ... , n − 1 } indicates the next node to visit , and is a feasible action for a116 partial solution a = ( a0 , ... , at−1 ) if ( at−1 , at ) is an edge in the graph and at 6∈ visited ( a ) , or , when117 all are visited , if at = 0 to return to the start node . When expanding the solution to a′ = ( a0 , ... , at ) ,118 we can compute the state variables incrementally as:119 cost ( a′ ) = cost ( a ) + ccurrent ( a ) , at , current ( a ′ ) = at , visited ( a ′ ) = visited ( a ) ∪ { at } . ( 1 ) A ( partial ) solution a is a dominated solution if there exists a ( dominating ) solution a∗ such120 that visited ( a∗ ) = visited ( a ) , current ( a∗ ) = current ( a ) and cost ( a∗ ) < cost ( a ) . The tuple121 ( visited ( a ) , current ( a ) ) we refer to as the DP state , so removing all dominated partial solutions,122 we keep exactly one minimum-cost solution for each unique DP state1 . Since a solution can only123 dominate other solutions with the same set of visited nodes , we only need to remove dominated124 solutions from sets of solutions with the same number of actions . Therefore , we can efficiently125 execute the DP algorithm in iterations , where at step t all solutions have ( after t actions ) t+ 1 visited126 nodes ( including the start node ) , keeping the memory need at O ( B ) states ( with B the beam size ) .127 We define the scoring policy using a pretrained model [ 26 ] , which takes as input node coordinates128 and edge distances to predict a raw ‘ heatmap ’ value ĥij ∈ ( 0 , 1 ) for each edge ( i , j ) . The model was129 trained to predict optimal solutions , so ĥij can be seen as the probability that edge ( i , j ) is in the130 optimal tour . We force the heatmap to be symmetric thus we define hij = max { ĥij , ĥji } . The policy131 is defined using the heatmap values , in such a way to select the ( partial ) solutions with the largest132 total heat , while also taking into account the ( heat ) potential for the unvisited nodes . The policy thus133 selects the B solutions which have the highest score , defined as score ( a ) = heat ( a ) +potential ( a ) ,134 with heat ( a ) = ∑t−1 i=1 hai−1 , ai , i.e . the sum of the heat of the edges , which can be computed135 incrementally when expanding a solution . The potential is added as an estimate of the ‘ heat-to-136 go ’ ( similar to the heuristic in A∗ search ) for the remaining nodes , and avoids the ‘ greedy pitfall ’ 137 of selecting the best edges while skipping over nearby nodes , which would prevent good edges138 from being used later . It is defined as potential ( a ) = potential0 ( a ) + ∑ i6∈visited ( a ) potentiali ( a ) 139 with potentiali ( a ) = wi ∑ j 6∈visited ( a ) hji∑n−1 k=0 hki , where wi is the node potential weight given by140 wi = ( maxj hji ) · ( 1 − 0.1 ( ci0maxj cj0 − 0.5 ) ) . By normalizing the heatmap values for incoming141 edges , the ( remaining ) potential for node i is initially equal to wi but decreases as good edges142 become infeasible due to neighbours being visited . The node potential weight wi is equal to the143 maximum incoming edge heatmap value ( an upper bound to the heat contributed by node i ) , which144 gets multiplied by a factor 0.95 to 1.05 to give a higher weight to nodes closer to the start node , which145 we found helps to encourage the algorithm to keep edges that enable to return to the start node . The146 overall heat + potential function identifies promising partial solutions and is computationally cheap.147 3.2 Vehicle Routing Problem148 For the VRP , we add a special depot node to the graph , indicated by DEP . Each node i has a demand149 di , and the goal is to find multiple routes , which have a limited capacity denoted by CAPACITY.150 Additionally to the TSP state variables cost ( a ) , current ( a ) and visited ( a ) , we keep track of151 capacity ( a ) , which is the remaining capacity in the current route/vehicle . A solution starts at the152 depot , so we initialize the beam at step t = 0 with the empty initial solution with cost ( a ) = 0,153 current ( a ) = DEP , visited ( a ) = ∅ and capacity ( a ) = CAPACITY . For the VRP , we do not consider154 visiting the depot as a separate action . Instead , we define 2n actions , where at ∈ { 0 , ... , 2n − 1 } .155 The actions 0 , ... , n− 1 indicate a direct move from the current node to node at , whereas the actions156 1If we have multiple partial solutions with the same state and cost , we can arbitrarily choose one to dominate the other ( s ) , for example the one with the lowest index of the current node . n , ... , 2n − 1 indicate a move to node at − n via the depot . Feasible actions are those that move157 to unvisited nodes via edges in the graph and obey the following constraints . For the first action158 a0 there is no choice and we constrain ( for convenience of implementation ) a0 ∈ { n , ... , 2n − 1 } .159 A direct move ( at < n ) is only feasible if dat ≤ capacity ( a ) and updates the state similar to TSP160 but reduces remaining capacity by dat . A move via the depot is always feasible ( respecting the161 graph edges and assuming di ≤ CAPACITY ∀i ) as it resets the vehicle CAPACITY before subtracting162 demand , but incurs the ‘ via-depot cost ’ cDEPij = ci , DEP + cDEP , j . When all nodes are visited , we allow a163 special action to return to the depot . This somewhat unusual way of representing a CVRP solution164 has desirable properties similar to the TSP formulation : at step t we have exactly t nodes visited , and165 we can run the DP in iterations , removing dominated solutions at each step t.166 For VRP , a partial solution a is a dominated solution dominated by a∗ if visited ( a∗ ) = visited ( a ) 167 and current ( a∗ ) = current ( a ) ( i.e . a∗ corresponds to the same DP state ) and cost ( a∗ ) ≤ cost ( a ) 168 and capacity ( a∗ ) ≥ capacity ( a ) , with at least one of the two inequalities being strict . This means169 that for each DP state , given by the set of visited nodes and the current node , we do not only keep170 the ( single ) solution with lowest cost ( as in the TSP algorithm ) , but keep the complete set of pareto-171 efficient solutions in terms of cost and remaining vehicle capacity . This is because a higher cost172 partial solution may still be preferred if it has more remaining vehicle capacity , and vice versa.173 For the VRP scoring policy , we modify the model [ 26 ] to include the depot node and demands . The174 special depot node gets a separate learned initial embedding parameter , and we add additional edge175 types for connections to the depot , to mark the depot as being special . Additionally , each node gets176 an extra input ( next to its coordinates ) corresponding to di/CAPACITY ( where we set dDEP = 0 ) .177 Apart from this , the model remains exactly the same2 . The model is trained on example solutions178 from LKH [ 24 ] ( see Section 4.2 ) , which are not optimal , but still provide a useful training signal.179 Compared to TSP , the definition of the heat is slightly changed to accommodate for the ‘ via-depot180 actions ’ and is best defined incrementally using the ‘ via-depot heat ’ hDEPij = hi , DEP · hDEP , j · 0.1,181 where multiplication is used to keep heat values interpretable as probabilities and in the range ( 0 , 1 ) .182 The additional penalty factor of 0.1 for visiting the depot encourages the algorithm to minimize the183 number of vehicles/routes . The initial heat is 0 and when expanding a solution a to a′ using action184 at , the heat is incremented with either hcurrent ( a ) , at ( if at < n ) or h DEP current ( a ) , at−n ( if at ≥ n ) . The185 potential is defined similarly to TSP , replacing the start node 0 by DEP.186 3.3 Travelling Salesman Problem with Time Windows187 For the TSPTW , we also have a special depot / start node 0 , and each node i has a time window188 defined by ( li , ui ) in which the node should be visited , assuming travel time is equal to cost/distance.189 It is allowed to wait if arrival at a node is before li , but arrival can not be after ui ( i.e . the constraint is190 hard ) . We consider the objective to be minimizing total cost , but minimizing total time ( or makespan ) 191 only requires training on different example solutions . Due to the hard constraints , TSPTW is typically192 considered more challenging to solve than plain TSP , for which every solution is feasible.193 The state variables and initial solution are equal to TSP except that we add time ( a ) which is194 initially 0 ( = l0 ) . Feasible actions at ∈ { 0 , ... , n − 1 } are those that move to unvisited nodes via195 edges in the graph such that the arrival time is no later than uat and do not directly eliminate the196 possibility to visit other nodes in time3 . Expanding a solution a to a′ using action at updates the197 time as time ( a′ ) = max { time ( a ) + ccurrent ( a ) , at , lat } .198 For each DP state , we keep all efficient solutions in terms of cost and time , so a partial solution a is a199 dominated solution dominated by a∗ if a∗ has the same DP state ( visited ( a∗ ) = visited ( a ) and200 current ( a∗ ) = current ( a ) ) and is strictly better in terms of cost and time , i.e . cost ( a∗ ) ≤ cost ( a ) 201 and time ( a∗ ) ≤ time ( a ) , with at least one of the two inequalities being strict.202 The model [ 26 ] for the scoring policy is adapted to include the time windows ( li , ui ) as node features203 ( in the same unit as coordinates and costs ) , and we use a special embedding for the depot similar to204 VRP . Due to the time dimension , a TSPTW solution is directed , and edge ( i , j ) may be good whereas205 ( j , i ) may be not , so we adapt the model to enable predictions hij 6= hji ( see details in Appendix206 B ) . We generated example training solutions using ( heuristic ) DP with a large beam size , which was207 faster than using LKH . Given the heat predictions , the score ( heat + potential ) is exactly as for TSP.208 2Except that we do not use the K-nearest neighbour feature [ 26 ] as it contains no additional information . 3E.g. , arriving at a node i at t = 10 ( including waiting ) is not feasible if node j has uj = 12 and cij = 3 . 3.4 Graph sparsity209 As described , the DP algorithm can take into account a sparse graph to define feasible expansions . As210 our problems are defined on sets of nodes rather than graphs , the use of a sparse graph is an artificial211 design choice , which allows to significantly reduce the runtime but may sacrifice the possibility to212 find good or optimal tours . We propose two different strategies for defining the sparse graph on213 which to run the DP : thresholding the heatmap values hij and using the K-nearest neighbour ( KNN ) 214 graph . By default , we use a ( low ) heatmap threshold of 10−5 , which rules out most of the edges as215 the model confidently predicts ( close to ) 0 for most edges . This is a secondary way to leverage the216 neural network ( independent of the scoring policy ) , which can be seen as a form of learned problem217 reduction [ 49 ] . For symmetric problems ( TSP and VRP ) , we add KNN edges in both directions . For218 the VRP , we additionally connect each node to the depot ( and vice versa ) to ensure feasibility.219 3.5 Implementation & hyperparameters220 We implement DPDP using PyTorch [ 44 ] to leverage batched computation on the GPU . For details,221 see Appendix A . Our code is publicly available.4 DPDP has very few hyperparameters , but the222 heatmap threshold of 10−5 and details like the functional form of e.g . the scoring policy are ‘ educated223 guesses ’ or manually tuned on a few validation instances and can likely be improved . The runtime is224 influenced by implementation choices which were manually selected using a few validation instances.225 4 Experiments226 4.1 Travelling Salesman Problem227 In Table 1 we report our main results for DPDP with beam sizes of 10K ( 10 thousand ) and 100K,228 for the TSP with 100 nodes on a commonly used test set [ 29 ] . We report results using Concorde [ 2 ] ,229 LKH [ 24 ] and Gurobi [ 22 ] , as well as recent results of the strongest methods using neural networks230 ( ‘ neural approaches ’ ) from literature . Running times for solving 10000 instances after training should231 be taken as rough indications as some are on different machines , typically with 1 GPU or a many-core232 CPU ( 8 - 32 ) . The costs indicated with * are not directly comparable due to slight dataset differences233 [ 17 ] . Times for generating heatmaps ( if applicable ) is reported separately ( as the first term ) from the234 running time for MCTS [ 17 ] or DP . DPDP achieves close to optimal results , strictly outperforming235 the neural baselines achieving better results in less time ( except POMO [ 30 ] , see Section 4.2 ) .236 4https : //github.com/ ? ? ? ? ? , to be disclosed after review 4.2 Vehicle Routing Problem237 For the VRP , we train the model using 1 million instances of 100 nodes , generated according to the238 distribution described by [ 41 ] and solved using one run of LKH [ 24 ] . We train using a batch size of239 48 and a learning rate of 10−3 ( selected as the result of manual trials to best use our GPUs ) , for ( at240 most ) 1500 epochs of 500 training steps ( following [ 26 ] ) from which we select the saved checkpoint241 with the lowest validation loss . We use the validation and test sets by [ 29 ] .242 Table 1 shows the results compared to a recent implementation of Hybrid Genetic Search ( HGS ) 5,243 a SOTA heuristic VRP solver [ 56 , 55 ] . HGS is faster and improves around 0.5 % over LKH , which244 is typically considered the baseline in related work . We present the results for LKH , as well as the245 strongest neural approaches and DPDP with beam sizes up to 1 million . Some results used 2000246 ( different ) instances [ 35 ] and can not be directly compared6 . DPDP outperforms all other neural247 baselines , except POMO [ 30 ] , which delivers good results very quickly by exploiting symmetries in248 the problem . However , as it can not ( easily ) improve further with additional runtime , we consider this249 contribution orthogonal to DPDP . DPDP is competitive to LKH ( see also Section 4.4 ) .250 More realistic instances We also train the model and run experiments with instances with 100251 nodes from a more realistic and challenging data distribution [ 51 ] . This distribution , commonly used252 in the routing community , has greater variability , in terms of node clustering and demand distributions.253 LKH failed to solve two of the test instances , which we found out is because LKH by default uses254 a fixed number of routes equal to a lower bound , given by ⌈∑n−1 i=0 di CAPACITY ⌉ , which may be infeasible7.255 Therefore we solve these instances by rerunning LKH with an unlimited number of allowed routes256 ( which in general gives worse results , see Section 4.4 ) .257 DPDP was run on a machine with 4 GPUs , but we also report ( estimated ) runtimes for 1 GPU258 ( 1080Ti ) , and we compare against 16 or 32 CPUs for HGS and LKH . In Table 2 it can be seen that259 the difference with LKH is , as expected , slightly larger than for the simpler dataset , but still below260 1 % for beam sizes of 100K-1M . We also observed a higher validation loss , so it may be possible to261 improve results using more training data . Nevertheless , finding solutions within 1 % of the specialized262 SOTA HGS algorithm , and even closer to LKH , is impressive for these challenging instances , and we263 consider the runtime ( for solving 10K instances ) acceptable , especially when using multiple GPUs.264 4.3 TSP with Time Windows265 For the TSP with hard time window constraints , we use the data distribution by [ 6 ] and use their set266 of 100 test instances with 100 nodes . These were generated with small time windows , resulting in267 a small feasible search space , such that even with very small beam sizes , our DP implementation268 solves these instances optimally , eliminating the need for a policy . Therefore , we also consider a269 more difficult distribution similar to [ 10 ] , which has larger time windows which are more difficult270 as the feasible search space is larger8 [ 15 ] . For details , see Appendix B . For both distributions , we271 generate training data and train the model exactly as we did for the VRP.272 5https : //github.com/vidalt/HGS-CVRP 6The running time of 4000 hours ( 167 days ) for 10K instances is estimated from 24min/instance [ 35 ] . 7For example , three nodes with a demand of two can not be assigned to two routes with a capacity of three . 8Up to a limit , as making the time windows infinite size reduces the problem to plain TSP .

Summary: The paper investigates dynamic programming techniques for routing that relies on auxiliary neural network models. The idea is to predict a "heatmap" of edges indicating whether they are promising or not for good-quality solutions, which needs to be generated only once per instance. The problem is then solved by a (forward) dynamic program that expands states based on scores extracted by the heatmap. The methodology is applied to the vehicle routing problem (VRP), the traveling salesperson problem (TSP), and the TSP with time windows (TSPTW).