Learning a Transferable Scheduling Policy for Various Vehicle Routing Problems based on Graph-centric Representation Learning

1 INTRODUCTION . The Vehicle Routing Problem ( VRP ) , a well-known NP-hard problem , has been enormously studied since it appeared by Dantzig & Ramser ( 1959 ) . There have been numerous attempts to compute the exact ( optimal ) or approximate solutions for various types of vehicle routing problems by using mixed integer linear programming ( MILP ) , which uses mostly a branch-and-price algorithm appeared in Desrochers et al . ( 1992 ) or a column generation method ( Chabrier , 2006 ) , or heuristics ( ( Cordeau et al. , 2002 ; Clarke & Wright , 1964 ; Gillett & Miller , 1974 ; Gendreau et al. , 1994 ) ) . However , these approaches typically require huge computational time to find the near optimum solution . For more information for VRP , see good survey papers ( Cordeau et al. , 2002 ; Toth & Vigo , 2002 ) . There have been attempts to solve such vehicle routing problems using learning based approaches . These approaches can be categorized into supervised-learning based approaches and reinforcementlearning based approaches ( Bengio et al. , 2020 ) ; supervised learning approaches try to map a target VRP with a solution or try to solve sub-problems appears during optimization procedure , while reinforcement learning ( RL ) approaches seek to learn to solve routing problems without supervision ( i.e , solution ) but using only repeated trials and the associated reward signal . Furthermore , the RL approaches can be further categorized into improvement heuristics and construction heuristics ( Mazyavkina et al. , 2020 ) ; improvement heuristics learn to modify the current solution for a better solution , while construction heuristics learn to construct a solution in a sequential decision making framework . The current study focuses on the RL-based construction heuristic for solving various routing problems . Various RL-based solution construction approaches have been employed to solve the traveling salesman problem ( TSP ) ( Bello et al. , 2016 ; Khalil et al. , 2017 ; Nazari et al. , 2018 ; Kool et al. , 2018 ) or the capacitated vehicle routing problem ( CVRP ) ( Nazari et al. , 2018 ; Kool et al. , 2018 ) . ( Bello et al. , 2016 ; Nazari et al. , 2018 ; Kool et al. , 2018 ) has used the encoder-decoder structure to sequentially generate routing schedules , and ( Khalil et al. , 2017 ) uses graph based embedding to determine the next assignment action . Although these approaches have shown the potential that the RL based approaches can learn to solve some types of routing problems , these approaches have the major two limitations : ( 1 ) only focus on routing a single vehicle over cities for minimizing the total traveling distance ( i.e. , min-sum problem ) and ( 2 ) the trained policy for a specific routing problem can not be used for solving other routing problems with different objective and constraints ( they show that trained policy can be used to solve the same type of the routing problems with different problem sizes ) . In this study , We proposed the Graph-centric RL-based Transferable Scheduler ( GRLTS ) for various vehicle routing problems . GRLTS is composed of graph-centric representation learning and RLbased scheduling policy learning . GRLTS is mainly designed to solve min-max capacititated multi vehicle routing problems ( mCVRP ) ; the problem seeks to minimize the total completion time for multiple vehicles whose one-time traveling distance is constrained by their fuel levels to serve the geographically distributed customer nodes . The method represents the relationships among vehicles , customers , and fuel stations using relationship-specific graphs to consider their topological relationships and employ graph neural network ( GNN ) to extract the graph ’ s embedding to be used to make a routing action . To effectively train the policy for minimizing the total completion time while satisfying the fuel constraints , we use the specially designed reward signal in RL framework . The representation learning for graph and the decision making policy are trained in an end-to-end fashion in an MARL framework . In addition , to effectively explore the joint combinatorial action space , we employ curriculum learning while controlling the difficulty ( complexity ) of a target problem . The proposed GRLTS resolves the two issues raised in other RL-based routing algorithms : • GRLTS learns to coordinate multiple vehicles to minimize the total completion time ( makespan ) . It can resolve the first issue of other RL-based routing algorithms and can be used to solve practical routing problems of scheduling multiple vehicles simultaneously . ( Kang et al. , 2019 ) also employed the graph based embedding ( random graph embedding ) to solve identical parallel machine scheduling problem , the problem seeking to minimize the makespan by scheduling multiple machines . However , our approach is more general in that it can consider capacity constraint and more fast and scalable node embedding strategies . • GRLTS transfers the trained scheduling policy with random mCVRP instances to be used for solving not only new mCVRP problems with different complexity but also different routing problems ( CVRP , mTSP , TSP ) with different objectives and constraints . 2 FORMULATION . 2.1 MIN-MAX SOLUTION FOR MCVRP . We define the set of vehicles VV = 1 , ... , NV , the set of customers VC = 1 , ... , NC , and the set of refueling stations VR = 1 , ... , NR , where NA , NC , and NR are the numbers of vehicles , customers , and refueling stations , respectively . The objective of min-max mCVRP is minimizing the makespan that is the longest distance among all vehicle ’ s traveling distance , i.e. , min maxi∈VV Li with Li being the traveling distance of vehicle i , while each vehicle ’ s one-time traveling distance is constrained by its remaining fuel . The detailed mathematical formulation using mixed integer linear programming ( MILP ) is provided in Appendix . Figure ( 1 ) ( left ) shows a snapshot of a mCVRP state and Figure ( 1 ) ( right ) represents a feasible solution of the mCVRP . 2.2 DEC-MDP FORMULATION FOR MCVRP . We seek to sequentially construct an optimum solution . Thus , we frame the solution construction procedure as a decentralized Markov decision problem ( Dec-MDP ) as follows . 2.2.1 STATE . We define the vehicle state svt , ∀v ∈ VV , the customer state sct , ∀c ∈ VC , and the refueling station state srt , ∀r ∈ VR as follows : • State of vehicle v , svt = ( xvt , fvt , qvt ) . xvt is the allocated node that vehicle v to visit ; fvt is the current fuel level ; and qvt is the number of customers served by the vehicle v so far . • State of a customer c , sct = ( xc , vc ) . xc is the location of customer node c ( static ) . Visit indicator v c ∈ { 0 , 1 } becomes 1 if the customer c is visited and 0 , otherwise . • State of a refueling station r , srt = xr . xr is the location of the refueling station r ( static ) . The global state st then becomes st = ( { svt } Nv v=1 , { sct } NC c=1 , { srt } NR r=1 ) . 2.2.2 ACTIONS & STATE TRANSITION . Action avt for vehicle v at time t is indicating a node to be visited by vehicle v at time t+ 1 , that is , avt = x v t+1 ∈ { VC ∪ VR } . Therefore , the next state of vehicle v becomes svt+1 = ( xvt+1 , fvt+1 , qvt+1 ) where fvt+1 and q v t+1 are determined deterministically by an action a v t as follows : • Fuel capacity update : fvt+1 = { F v , if avt ∈ VR ftv − d ( xvt , avt ) , otherwise . • Customer visit number update : qvt+1 = { qvt , if a v t ∈ VR qvt + 1 , otherwise . 2.2.3 REWARDS . The goal of mCVRP is to force all agents to coordinate to finish the distributed tasks quickly while satisfying the fuel constraints . To achieve this global goal in a distributed manner , we use the specially designed independent reward for each agent as : • visiting reward : To encourage vehicles to visit the customer nodes faster , in turn , minimizing makespan , we define customer visit reward rvvisit = q v t . This reward is provided when an agent visits a customer ; the more customer nodes a vehicle agent n visits , the greater reward it can earn . • Refueling reward : To induce a strategic refueling , we introduce refuel reward rvrefuel = qvt × ( ( F v−fvt ) / ( Fv−1 ) ) α . We define the refuel reward as an opportunity cost . That is , vehicles with sufficient fuel are not necessary to refuel ( small reward ) . In contrast , refueling vehicles with a lack of fuel is worth as much as visiting customers . In this study , we set F v = 10 ( which is the equivalent to the total traveling distance that vehicle v can travel with the fuel tank fully loaded ) and α = 2 . 2.3 RELATIONSHIPS WITH OTHER CLASS OF VRPS . mCVRP , the target problem of this study , has three key properties : 1 ) the problem seeks to minimize the total completion time of vehicles by forcing all vehicles to coordinate ( in a distributed manner ) , 2 ) the problem employs fuel capacity constraints requiring the vehicles to visit the refueling stations strategically , 3 ) the problem considers multiple refueling depots ( revisit allowed ) . If some of these requirements are relaxed , min-max mCVRP can be degenerated into simpler conventional routing problems : • TSP is the problem where a single vehicle is operated to serve every customer while minimizing the total traveling distance . The agent needs or needs not come back to the depot . This problem does not have capacity constraints . • CVRP ( capacity-constrained TSP ) is the problem where a single vehicle is operated to serve every customer while minimizing the total traveling distance and satisfying the fuel constraint . The vehicles need to comeback depot to charge . • mTSP ( multi-agent TSP ) is the problem where multiple vehicles should serve all the customers as quickly as possible . This problem does not have capacity constraints . • mCVRP ( multi-agent , capacity-constrained TSP ) is our target problem having the properties of both mTSP and CVRP . Additionally , we add more than one refueling depot . The mathematical formulations for these problems are provided in Appendix . We train the policy using random mCVRP instances with varying numbers of agents and customers and employ the trained policy without parameter changes to solve TSP , CVRP and mTSP to test its domain transferability . 3 METHOD . This section explains how the proposed model , given a state ( a partial solution ) , assigns an idle vehicle to next node to visit under the sequential decision-making framework ( see Figure 2 ) . 3.1 STATE REPRESENTATION USING RELATIONSHIP-SPECIFIC MULTIPLE GRAPHS . The proposed model represents the global state st using as a weighted graph Gt = G ( V , E , w ) where V = { VV , VC , VR } , and E is the set of edges between node i , j ∈ V and w is weight for edge ( i , j ) ( here , distance dij ) . Each node corresponding to vehicle , customer , and refueling station will be initialized with its associated states defined earlier . Although we can assume that all nodes are connected with each other regardless of types and distance , we restrict the edge connection to its neighboring nodes to reduce the computational cost . Specifically , each type of node can define its connectivity range and connect an edge if any node is located within its range as follows ( see Figure 2 ) : evj = 1 ∀v ∈ Vv , d ( v , j ) ≤ RV = fvt ( 1 ) ecj = 1 ∀c ∈ Vc , d ( c , j ) ≤ RC ( 2 ) erj = 1 ∀r ∈ VR , d ( r , j ) ≤ RR = max v∈Vv F v ( 3 ) That is , the vehicle node v ∈ Vv connects the edges with nodes that are located within its traveling distance ( i.e. , the current fuel level fvt ) . In addition , the customer nodes c ∈ Vc connects the edges with nodes that are located within the constant range RC . We set RC = 5 while following a typical hyperparameter selection procedure . Finally , the refueling node r ∈ VR connects the edges with nodes within maximum distance that the vehicle with the largest fuel capacity can travel with the full loaded fuel ( in this study F v = 10 for all vehicles ) . Note that the target node j that can be connected to each node can be any types of nodes .

Summary: This paper considers the problem of capacitated vehicle routing which is a famous combinatorial optimization problem that is known to be NP-hard. This paper takes the approach of solving instances of this problem using RL. The goal is this problem is to minimize the maximum time (or makespan objective) for multiple vehicles to complete various tasks subject to fuel constraint. The paper trains a graph embedding from random instances and then show that it solves new instances of this problem with reasonable accuracy. Moreover, they also show that the embedding can be transferred to other related objectives.