Learning a Transferable Scheduling Policy for Various Vehicle Routing Problems based on Graph-centric Representation Learning

1 INTRODUCTION . The Vehicle Routing Problem ( VRP ) , a well-known NP-hard problem , has been enormously studied since it appeared by Dantzig & Ramser ( 1959 ) . There have been numerous attempts to compute the exact ( optimal ) or approximate solutions for various types of vehicle routing problems by using mixed integer linear programming ( MILP ) , which uses mostly a branch-and-price algorithm appeared in Desrochers et al . ( 1992 ) or a column generation method ( Chabrier , 2006 ) , or heuristics ( ( Cordeau et al. , 2002 ; Clarke & Wright , 1964 ; Gillett & Miller , 1974 ; Gendreau et al. , 1994 ) ) . However , these approaches typically require huge computational time to find the near optimum solution . For more information for VRP , see good survey papers ( Cordeau et al. , 2002 ; Toth & Vigo , 2002 ) . There have been attempts to solve such vehicle routing problems using learning based approaches . These approaches can be categorized into supervised-learning based approaches and reinforcementlearning based approaches ( Bengio et al. , 2020 ) ; supervised learning approaches try to map a target VRP with a solution or try to solve sub-problems appears during optimization procedure , while reinforcement learning ( RL ) approaches seek to learn to solve routing problems without supervision ( i.e , solution ) but using only repeated trials and the associated reward signal . Furthermore , the RL approaches can be further categorized into improvement heuristics and construction heuristics ( Mazyavkina et al. , 2020 ) ; improvement heuristics learn to modify the current solution for a better solution , while construction heuristics learn to construct a solution in a sequential decision making framework . The current study focuses on the RL-based construction heuristic for solving various routing problems . Various RL-based solution construction approaches have been employed to solve the traveling salesman problem ( TSP ) ( Bello et al. , 2016 ; Khalil et al. , 2017 ; Nazari et al. , 2018 ; Kool et al. , 2018 ) or the capacitated vehicle routing problem ( CVRP ) ( Nazari et al. , 2018 ; Kool et al. , 2018 ) . ( Bello et al. , 2016 ; Nazari et al. , 2018 ; Kool et al. , 2018 ) has used the encoder-decoder structure to sequentially generate routing schedules , and ( Khalil et al. , 2017 ) uses graph based embedding to determine the next assignment action . Although these approaches have shown the potential that the RL based approaches can learn to solve some types of routing problems , these approaches have the major two limitations : ( 1 ) only focus on routing a single vehicle over cities for minimizing the total traveling distance ( i.e. , min-sum problem ) and ( 2 ) the trained policy for a specific routing problem can not be used for solving other routing problems with different objective and constraints ( they show that trained policy can be used to solve the same type of the routing problems with different problem sizes ) . In this study , We proposed the Graph-centric RL-based Transferable Scheduler ( GRLTS ) for various vehicle routing problems . GRLTS is composed of graph-centric representation learning and RLbased scheduling policy learning . GRLTS is mainly designed to solve min-max capacititated multi vehicle routing problems ( mCVRP ) ; the problem seeks to minimize the total completion time for multiple vehicles whose one-time traveling distance is constrained by their fuel levels to serve the geographically distributed customer nodes . The method represents the relationships among vehicles , customers , and fuel stations using relationship-specific graphs to consider their topological relationships and employ graph neural network ( GNN ) to extract the graph ’ s embedding to be used to make a routing action . To effectively train the policy for minimizing the total completion time while satisfying the fuel constraints , we use the specially designed reward signal in RL framework . The representation learning for graph and the decision making policy are trained in an end-to-end fashion in an MARL framework . In addition , to effectively explore the joint combinatorial action space , we employ curriculum learning while controlling the difficulty ( complexity ) of a target problem . The proposed GRLTS resolves the two issues raised in other RL-based routing algorithms : • GRLTS learns to coordinate multiple vehicles to minimize the total completion time ( makespan ) . It can resolve the first issue of other RL-based routing algorithms and can be used to solve practical routing problems of scheduling multiple vehicles simultaneously . ( Kang et al. , 2019 ) also employed the graph based embedding ( random graph embedding ) to solve identical parallel machine scheduling problem , the problem seeking to minimize the makespan by scheduling multiple machines . However , our approach is more general in that it can consider capacity constraint and more fast and scalable node embedding strategies . • GRLTS transfers the trained scheduling policy with random mCVRP instances to be used for solving not only new mCVRP problems with different complexity but also different routing problems ( CVRP , mTSP , TSP ) with different objectives and constraints . 2 FORMULATION . 2.1 MIN-MAX SOLUTION FOR MCVRP . We define the set of vehicles VV = 1 , ... , NV , the set of customers VC = 1 , ... , NC , and the set of refueling stations VR = 1 , ... , NR , where NA , NC , and NR are the numbers of vehicles , customers , and refueling stations , respectively . The objective of min-max mCVRP is minimizing the makespan that is the longest distance among all vehicle ’ s traveling distance , i.e. , min maxi∈VV Li with Li being the traveling distance of vehicle i , while each vehicle ’ s one-time traveling distance is constrained by its remaining fuel . The detailed mathematical formulation using mixed integer linear programming ( MILP ) is provided in Appendix . Figure ( 1 ) ( left ) shows a snapshot of a mCVRP state and Figure ( 1 ) ( right ) represents a feasible solution of the mCVRP . 2.2 DEC-MDP FORMULATION FOR MCVRP . We seek to sequentially construct an optimum solution . Thus , we frame the solution construction procedure as a decentralized Markov decision problem ( Dec-MDP ) as follows . 2.2.1 STATE . We define the vehicle state svt , ∀v ∈ VV , the customer state sct , ∀c ∈ VC , and the refueling station state srt , ∀r ∈ VR as follows : • State of vehicle v , svt = ( xvt , fvt , qvt ) . xvt is the allocated node that vehicle v to visit ; fvt is the current fuel level ; and qvt is the number of customers served by the vehicle v so far . • State of a customer c , sct = ( xc , vc ) . xc is the location of customer node c ( static ) . Visit indicator v c ∈ { 0 , 1 } becomes 1 if the customer c is visited and 0 , otherwise . • State of a refueling station r , srt = xr . xr is the location of the refueling station r ( static ) . The global state st then becomes st = ( { svt } Nv v=1 , { sct } NC c=1 , { srt } NR r=1 ) . 2.2.2 ACTIONS & STATE TRANSITION . Action avt for vehicle v at time t is indicating a node to be visited by vehicle v at time t+ 1 , that is , avt = x v t+1 ∈ { VC ∪ VR } . Therefore , the next state of vehicle v becomes svt+1 = ( xvt+1 , fvt+1 , qvt+1 ) where fvt+1 and q v t+1 are determined deterministically by an action a v t as follows : • Fuel capacity update : fvt+1 = { F v , if avt ∈ VR ftv − d ( xvt , avt ) , otherwise . • Customer visit number update : qvt+1 = { qvt , if a v t ∈ VR qvt + 1 , otherwise . 2.2.3 REWARDS . The goal of mCVRP is to force all agents to coordinate to finish the distributed tasks quickly while satisfying the fuel constraints . To achieve this global goal in a distributed manner , we use the specially designed independent reward for each agent as : • visiting reward : To encourage vehicles to visit the customer nodes faster , in turn , minimizing makespan , we define customer visit reward rvvisit = q v t . This reward is provided when an agent visits a customer ; the more customer nodes a vehicle agent n visits , the greater reward it can earn . • Refueling reward : To induce a strategic refueling , we introduce refuel reward rvrefuel = qvt × ( ( F v−fvt ) / ( Fv−1 ) ) α . We define the refuel reward as an opportunity cost . That is , vehicles with sufficient fuel are not necessary to refuel ( small reward ) . In contrast , refueling vehicles with a lack of fuel is worth as much as visiting customers . In this study , we set F v = 10 ( which is the equivalent to the total traveling distance that vehicle v can travel with the fuel tank fully loaded ) and α = 2 . 2.3 RELATIONSHIPS WITH OTHER CLASS OF VRPS . mCVRP , the target problem of this study , has three key properties : 1 ) the problem seeks to minimize the total completion time of vehicles by forcing all vehicles to coordinate ( in a distributed manner ) , 2 ) the problem employs fuel capacity constraints requiring the vehicles to visit the refueling stations strategically , 3 ) the problem considers multiple refueling depots ( revisit allowed ) . If some of these requirements are relaxed , min-max mCVRP can be degenerated into simpler conventional routing problems : • TSP is the problem where a single vehicle is operated to serve every customer while minimizing the total traveling distance . The agent needs or needs not come back to the depot . This problem does not have capacity constraints . • CVRP ( capacity-constrained TSP ) is the problem where a single vehicle is operated to serve every customer while minimizing the total traveling distance and satisfying the fuel constraint . The vehicles need to comeback depot to charge . • mTSP ( multi-agent TSP ) is the problem where multiple vehicles should serve all the customers as quickly as possible . This problem does not have capacity constraints . • mCVRP ( multi-agent , capacity-constrained TSP ) is our target problem having the properties of both mTSP and CVRP . Additionally , we add more than one refueling depot . The mathematical formulations for these problems are provided in Appendix . We train the policy using random mCVRP instances with varying numbers of agents and customers and employ the trained policy without parameter changes to solve TSP , CVRP and mTSP to test its domain transferability . 3 METHOD . This section explains how the proposed model , given a state ( a partial solution ) , assigns an idle vehicle to next node to visit under the sequential decision-making framework ( see Figure 2 ) . 3.1 STATE REPRESENTATION USING RELATIONSHIP-SPECIFIC MULTIPLE GRAPHS . The proposed model represents the global state st using as a weighted graph Gt = G ( V , E , w ) where V = { VV , VC , VR } , and E is the set of edges between node i , j ∈ V and w is weight for edge ( i , j ) ( here , distance dij ) . Each node corresponding to vehicle , customer , and refueling station will be initialized with its associated states defined earlier . Although we can assume that all nodes are connected with each other regardless of types and distance , we restrict the edge connection to its neighboring nodes to reduce the computational cost . Specifically , each type of node can define its connectivity range and connect an edge if any node is located within its range as follows ( see Figure 2 ) : evj = 1 ∀v ∈ Vv , d ( v , j ) ≤ RV = fvt ( 1 ) ecj = 1 ∀c ∈ Vc , d ( c , j ) ≤ RC ( 2 ) erj = 1 ∀r ∈ VR , d ( r , j ) ≤ RR = max v∈Vv F v ( 3 ) That is , the vehicle node v ∈ Vv connects the edges with nodes that are located within its traveling distance ( i.e. , the current fuel level fvt ) . In addition , the customer nodes c ∈ Vc connects the edges with nodes that are located within the constant range RC . We set RC = 5 while following a typical hyperparameter selection procedure . Finally , the refueling node r ∈ VR connects the edges with nodes within maximum distance that the vehicle with the largest fuel capacity can travel with the full loaded fuel ( in this study F v = 10 for all vehicles ) . Note that the target node j that can be connected to each node can be any types of nodes .

Summary: The paper presents a reinforcement learning approach to learn a routing policy for a family of Vehicle Routing Problems (VRPs). More precisely, the authors train a model for the min-max capacitated multi vehicle routing problem (mCVRP), then use it to solve variants of the problem that correspond to various VRP problems (with a single vehicle, no capacity constraints, no fueling stations, etc). They use a GNN to represent the states and the PPO algorithm to learn the policy. They validate their approach on both random instances and literature benchmarks.