A Hierarchical Reinforcement Learning Based Optimization Framework for Large-scale Dynamic Pickup and Delivery Problems

1 Introduction . The Dynamic Pickup and Delivery Problem ( DPDP ) constitutes an important family of routing problems , which generally contains three key elements : orders , goods and vehicles as shown in Figure 1 . Orders are generated in real-time . Different orders contain different types and quantities of goods . A number of vehicles are scheduled to serve the orders by transporting the desired goods from different origins to different destinations . The objective of DPDP is to dynamically assign each order to the most appropriate vehicle so that the overall transportation cost ( e.g. , overall distances ) could be minimized . DPDPs are widespread in order dispatching systems of the supply chain , express mail delivery services and elsewhere . ∗Equal contribution . † Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . DPDP is a complex variant of the Travelling Salesman Problem ( TSP ) and Vehicle Routing Problem ( VRP ) , which are both NP-Hard combinatorial optimization problems [ 25 ] . The main difficulty of DPDP comes from the dynamically generated orders in real-time , thus the order dispatching decisions can not be made beforehand in an offline style . Besides , compared with TSP and VRP , there exist various additional complex constraints in DPDP such as pickup and delivery constraint , Last-In-First-Out ( LIFO ) constraint , time window constraint , split demand constraint , etc . Traditional methods for DPDP . Existing solutions for DPDP fall into two categories . The first category maintains a fixed buffer to cache the most recent generated orders and periodically dispatches all cached orders in a delayed mode . By this way , the overall dynamic problem is partitioned into a series of static sub-problems with subsets of known orders , i.e. , static Pickup and Delivery Problems ( PDPs ) . Then , operational research-based ( OR ) methods [ 22 , 20 ] , heuristic and meta-heuristic methods [ 7 , 16 , 25 , 4 , 5 , 3 , 11 , 26 , 23 , 9 ] are designed to solve each sub-problem . However , myopically optimizing each static sub-problem can not guarantee the overall dynamic problem could be optimized from a long-term perspective since the split sub-problems are not independent of each other . The main reasons are previous orders assignment results will influence ( 1 ) the number of remaining orders to be dispatched , ( 2 ) the vehicle ’ s remaining capacity and ( 3 ) the relative positions to the following orders . To acquire better solutions , the second category methods [ 24 , 8 , 12 ] try to predict the distribution of future orders and take the predicted orders into consideration when computing the solution for each sub-problem . However , predicting future orders is not realistic due to the high uncertainty in the real world . Inaccurate predictions will mislead the order dispatcher and route planner , and result in poor solution quality . Learning-based methods for VRP . Additionally , a common flaw of traditional OR and metaheuristic methods is that they are computationally expensive and normally unable to obtain a desired solution within the allowable time . Besides , the design of them heavily relies on complex domain knowledge . To improve the solution computing efficiency and ease the difficulty of the algorithm design , recently , several learning-based methods are proposed [ 27 , 1 , 18 , 6 , 13 ] . These methods have demonstrated that the solution computing efficiency can be significantly improved by leveraging the generalization ability of the trained models . Besides , they could obtain solutions with competitive qualities compared with the state-of-the-art traditional methods . Although these methods mainly focus on TSPs or VRPs , of which all orders ’ information is known in advance and much fewer constraints are considered comparing with DPDP , learning-based methods have shown great potential to help solve large-scale DPDPs and reach superior performance . In this paper , we propose a novel hierarchical reinforcement learning ( RL ) based optimization framework to solve the real-world large-scale DPDPs . Considering that order dispatching has a longterm impact on the overall optimization objective , the upper-level RL agent dynamically determines whether to wait longer at each moment for caching more future orders . In this way , the orders can be more flexibly assigned to vehicles ( since each vehicle will have more candidate orders to choose ) and the routes of vehicles could be optimized towards globally better solutions . The lower-level RL agent is responsible for assigning the cached orders to the most appropriate vehicles by sequentially manipulating heuristic operators to improve the solution quality iteratively . To verify the effectiveness of the framework , we collected real historical data from the order dispatching system of Huawei Supply Chain and built a simulator to simulate the order dispatching and vehicle transportation process . Further , we deployed our method on the company ’ s Supply Chain Business Unit . Extensive offline simulation and online testing showed the superior performance of our algorithm . Our main contributions are as follows : • We are the first to propose a practical hierarchical RL framework to efficiently and farsightedly compute superior solutions for the real-world large-scale DPDPs with complex constraints . • We design a simulator using real industrial data to be the experimental benchmark to verify the proposed method , which is available here for interested researchers . • We show that our approach considerably improves the optimization objectives compared with existing algorithms both in the offline evaluation and online testing . The ablation study indicates our approach can obtain high-quality solutions with fast running speed and has strong generalization ability . 2 Problem Formulation . We now give the formulation of DPDP in our logistics scenario . For the orders dynamically generated in real-time at different nodes ( i.e. , factories and warehouses ) within a day , vehicles should be scheduled to transport the goods from pickup nodes to delivery nodes to fulfil the orders with minimal transportation cost . In our case , the objective is to minimize K vehicles average travelling distances D ( K ) of the entire DPDP : minD ( K ) ( 1 ) while meeting several constraints : Pickup and Delivery Constraint , Capacity Constraint , LIFO Constraint , Time Window Constraint , etc . Detailed constraints are shown in Appendix A . In practice , however , some orders are destined to violate time window constraints2 . Thus , we add it to the objective function as an associated penalty to convert the hard time window constraint to a soft one . The penalty function is defined as the overtime beyond the specified completion time of each order . The optimization objective is then reformulated as minimizing the weighted sum 3 of average vehicles travelling distances ( kilometers ) D ( K ) and total overtime ( seconds ) OT of all orders C : minD ( K ) + λ ∗OT ( C ) ( 2 ) Apart from the various complex constraints mentioned above , the additional difficulties of this problem mainly come from two aspects : ( 1 ) The problem scale is very large . In practical logistic scenarios of the company , millions of products and intermediate materials are manufactured every day . As these products and materials might be used in the subsequent phases ( e.g. , assembling or selling ) , they have to be scheduled and transported between hundreds of factories and warehouses by dozens of vehicles within stringent timeline constraints , which constitutes a very large-scale and complex DPDP . ( 2 ) Besides , as the orders are generated online in real-time , the schedule planning can not be made aforehand in an offline style . From the oracle ’ s point of view , i.e. , when all orders of a day are known in advance , the uncertainty is eliminated and this DPDP can be formulated as a complex Mixed Integer Programming ( MIP ) Problem , of which the optimal solution could be obtained utilizing exact algorithms ( e.g. , cutting plane algorithms , branch-and-bound algorithms or modern solvers such as Gurobi [ 20 ] ) [ 22 ] . However , in reality , it ’ s impossible to know all the orders in advance , thus these approaches are not applicable . To eliminate the uncertainties brought by the unknown orders , a practical way is to utilize a buffer to cache the most recent generated orders and periodically dispatches all cached orders in a little delayed mode . With the known orders in the cache , the static PDP can be formulated as an MIP as shown in Appendix A . We could resort to modern solvers to solve this MIP . However , even for the static PDP with very few orders , it still costs several hours to compute a feasible solution , which is beyond the acceptable limits ( details are shown in Table 2 and 3 ) . Besides , even if we could obtain the optimal solution for each fixed split static PDP , we still can not guarantee the global DPDP can be optimized as these static sub-problems are not independent of each other . 2For example , goods of an order need to be transported to a far destination in a short time . Even if a vehicle is used to serve the order as soon as the order is generated , violation of time window constraints can still occur . 3If the overtime happens , the subsequent assembling and selling will be delayed or cancelled , resulting in great economic loss . Therefore , we set λ to a very large number , e.g. , 10000 , to severely punish the overtime . 3 Method . 3.1 Overall Framework In this paper , considering the challenges mentioned above , we propose a novel hierarchical reinforcement learning based optimization framework , which contains two levels of agents . As shown in Figure 2 , we maintain a buffer to cache the newly generated orders and periodically dispatch all cached orders at once . But instead of dispatching the cached orders of fixed numbers or predicting future orders , we design an upper-level agent to dynamically determine whether to wait longer for caching more future orders at each moment . Though waiting longer will postpone the dispatching and transportation of the earlier cached orders , additional future orders can be taken into account for the vehicle-order matching . In this way , each vehicle will have more candidate orders to choose , thus the overall travelling distances will be more potentially to be optimized for shorter4 . This process could be regarded as sacrificing a little time in exchange for a precise estimation of future orders . However , waiting for too long will also increase the risk of overtime of the earlier cached orders . Thus , whether to wait longer to cache more orders at each moment will have a long-term impact on the overall dispatching results , and can be naturally modeled as a sequential decision-making problem . We model this procedure as a Markov Decision Process ( MDP ) . Depending on whether to wait longer at each moment , the overall DPDP can be dynamically partitioned into a series of static sub-problems , each of which includes different numbers of orders . As shown in Figure 2 , the generated orders are accumulated in the buffer until the upper-level agent decides to stop caching at time ti+k . Then , the agent releases the cached orders to the lower-level agent and clears the buffer . Given the released orders ( which form a static sub-problem , i.e. , a PDP ) , the lower-level agent is appointed to assign the orders to the most appropriate vehicles and arrange the transportation route of each vehicle , such that the transportation cost of these orders could be minimized . First , a set of basic operators are maintained , whose roles are converting one feasible solution to another . For instance , given an initial solution ’ s route { A- > B- > C } with three nodes A , B and C , a typical operator is swapping two nodes [ 13 ] , e.g. , swapping A and B . After applying this operator , { A- > B- > C } is converted to { B- > A- > C } . If the travelling cost of { B- > A- > C } is less than { A- > B- > C } , the initial solution is improved . On this basis , we design the lower-level agent similar to the traditional metaheuristic algorithms which sequentially manipulates these operators to improve the solution of each PDP . The difference is that we model the process of sequentially manipulating these operators as an MDP and incorporate RL methods to optimize the policy instead of manually designing complex rules . Finally , the best found solution is adopted by the order dispatching system to assign the orders to the vehicles and arrange their transportation routes . In the following two subsections , we will go into more details of the designed two agents . 4In Table 7 of Appendix I , we show that splitting the overall DPDP into different sub-problems with different time spans will lead to different qualities of solutions

Summary: The current paper proposes a hierarchical approach to solve the dynamic pickup and delivery problem (DPDP). The method uses two RL modules. The upper-level RL policy is designed to segment the time window thus transforming DPDP into a static PDP. For the upper-level RL, DQN is used. The lower-level RL policy is trained to select one solution improving heuristics. The lover level policy is parameterized with the GNN network and trained under the framework of Reinforce.