A Hierarchical Reinforcement Learning Based Optimization Framework for Large-scale Dynamic Pickup and Delivery Problems

1 Introduction . The Dynamic Pickup and Delivery Problem ( DPDP ) constitutes an important family of routing problems , which generally contains three key elements : orders , goods and vehicles as shown in Figure 1 . Orders are generated in real-time . Different orders contain different types and quantities of goods . A number of vehicles are scheduled to serve the orders by transporting the desired goods from different origins to different destinations . The objective of DPDP is to dynamically assign each order to the most appropriate vehicle so that the overall transportation cost ( e.g. , overall distances ) could be minimized . DPDPs are widespread in order dispatching systems of the supply chain , express mail delivery services and elsewhere . ∗Equal contribution . † Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . DPDP is a complex variant of the Travelling Salesman Problem ( TSP ) and Vehicle Routing Problem ( VRP ) , which are both NP-Hard combinatorial optimization problems [ 25 ] . The main difficulty of DPDP comes from the dynamically generated orders in real-time , thus the order dispatching decisions can not be made beforehand in an offline style . Besides , compared with TSP and VRP , there exist various additional complex constraints in DPDP such as pickup and delivery constraint , Last-In-First-Out ( LIFO ) constraint , time window constraint , split demand constraint , etc . Traditional methods for DPDP . Existing solutions for DPDP fall into two categories . The first category maintains a fixed buffer to cache the most recent generated orders and periodically dispatches all cached orders in a delayed mode . By this way , the overall dynamic problem is partitioned into a series of static sub-problems with subsets of known orders , i.e. , static Pickup and Delivery Problems ( PDPs ) . Then , operational research-based ( OR ) methods [ 22 , 20 ] , heuristic and meta-heuristic methods [ 7 , 16 , 25 , 4 , 5 , 3 , 11 , 26 , 23 , 9 ] are designed to solve each sub-problem . However , myopically optimizing each static sub-problem can not guarantee the overall dynamic problem could be optimized from a long-term perspective since the split sub-problems are not independent of each other . The main reasons are previous orders assignment results will influence ( 1 ) the number of remaining orders to be dispatched , ( 2 ) the vehicle ’ s remaining capacity and ( 3 ) the relative positions to the following orders . To acquire better solutions , the second category methods [ 24 , 8 , 12 ] try to predict the distribution of future orders and take the predicted orders into consideration when computing the solution for each sub-problem . However , predicting future orders is not realistic due to the high uncertainty in the real world . Inaccurate predictions will mislead the order dispatcher and route planner , and result in poor solution quality . Learning-based methods for VRP . Additionally , a common flaw of traditional OR and metaheuristic methods is that they are computationally expensive and normally unable to obtain a desired solution within the allowable time . Besides , the design of them heavily relies on complex domain knowledge . To improve the solution computing efficiency and ease the difficulty of the algorithm design , recently , several learning-based methods are proposed [ 27 , 1 , 18 , 6 , 13 ] . These methods have demonstrated that the solution computing efficiency can be significantly improved by leveraging the generalization ability of the trained models . Besides , they could obtain solutions with competitive qualities compared with the state-of-the-art traditional methods . Although these methods mainly focus on TSPs or VRPs , of which all orders ’ information is known in advance and much fewer constraints are considered comparing with DPDP , learning-based methods have shown great potential to help solve large-scale DPDPs and reach superior performance . In this paper , we propose a novel hierarchical reinforcement learning ( RL ) based optimization framework to solve the real-world large-scale DPDPs . Considering that order dispatching has a longterm impact on the overall optimization objective , the upper-level RL agent dynamically determines whether to wait longer at each moment for caching more future orders . In this way , the orders can be more flexibly assigned to vehicles ( since each vehicle will have more candidate orders to choose ) and the routes of vehicles could be optimized towards globally better solutions . The lower-level RL agent is responsible for assigning the cached orders to the most appropriate vehicles by sequentially manipulating heuristic operators to improve the solution quality iteratively . To verify the effectiveness of the framework , we collected real historical data from the order dispatching system of Huawei Supply Chain and built a simulator to simulate the order dispatching and vehicle transportation process . Further , we deployed our method on the company ’ s Supply Chain Business Unit . Extensive offline simulation and online testing showed the superior performance of our algorithm . Our main contributions are as follows : • We are the first to propose a practical hierarchical RL framework to efficiently and farsightedly compute superior solutions for the real-world large-scale DPDPs with complex constraints . • We design a simulator using real industrial data to be the experimental benchmark to verify the proposed method , which is available here for interested researchers . • We show that our approach considerably improves the optimization objectives compared with existing algorithms both in the offline evaluation and online testing . The ablation study indicates our approach can obtain high-quality solutions with fast running speed and has strong generalization ability . 2 Problem Formulation . We now give the formulation of DPDP in our logistics scenario . For the orders dynamically generated in real-time at different nodes ( i.e. , factories and warehouses ) within a day , vehicles should be scheduled to transport the goods from pickup nodes to delivery nodes to fulfil the orders with minimal transportation cost . In our case , the objective is to minimize K vehicles average travelling distances D ( K ) of the entire DPDP : minD ( K ) ( 1 ) while meeting several constraints : Pickup and Delivery Constraint , Capacity Constraint , LIFO Constraint , Time Window Constraint , etc . Detailed constraints are shown in Appendix A . In practice , however , some orders are destined to violate time window constraints2 . Thus , we add it to the objective function as an associated penalty to convert the hard time window constraint to a soft one . The penalty function is defined as the overtime beyond the specified completion time of each order . The optimization objective is then reformulated as minimizing the weighted sum 3 of average vehicles travelling distances ( kilometers ) D ( K ) and total overtime ( seconds ) OT of all orders C : minD ( K ) + λ ∗OT ( C ) ( 2 ) Apart from the various complex constraints mentioned above , the additional difficulties of this problem mainly come from two aspects : ( 1 ) The problem scale is very large . In practical logistic scenarios of the company , millions of products and intermediate materials are manufactured every day . As these products and materials might be used in the subsequent phases ( e.g. , assembling or selling ) , they have to be scheduled and transported between hundreds of factories and warehouses by dozens of vehicles within stringent timeline constraints , which constitutes a very large-scale and complex DPDP . ( 2 ) Besides , as the orders are generated online in real-time , the schedule planning can not be made aforehand in an offline style . From the oracle ’ s point of view , i.e. , when all orders of a day are known in advance , the uncertainty is eliminated and this DPDP can be formulated as a complex Mixed Integer Programming ( MIP ) Problem , of which the optimal solution could be obtained utilizing exact algorithms ( e.g. , cutting plane algorithms , branch-and-bound algorithms or modern solvers such as Gurobi [ 20 ] ) [ 22 ] . However , in reality , it ’ s impossible to know all the orders in advance , thus these approaches are not applicable . To eliminate the uncertainties brought by the unknown orders , a practical way is to utilize a buffer to cache the most recent generated orders and periodically dispatches all cached orders in a little delayed mode . With the known orders in the cache , the static PDP can be formulated as an MIP as shown in Appendix A . We could resort to modern solvers to solve this MIP . However , even for the static PDP with very few orders , it still costs several hours to compute a feasible solution , which is beyond the acceptable limits ( details are shown in Table 2 and 3 ) . Besides , even if we could obtain the optimal solution for each fixed split static PDP , we still can not guarantee the global DPDP can be optimized as these static sub-problems are not independent of each other . 2For example , goods of an order need to be transported to a far destination in a short time . Even if a vehicle is used to serve the order as soon as the order is generated , violation of time window constraints can still occur . 3If the overtime happens , the subsequent assembling and selling will be delayed or cancelled , resulting in great economic loss . Therefore , we set λ to a very large number , e.g. , 10000 , to severely punish the overtime . 3 Method . 3.1 Overall Framework In this paper , considering the challenges mentioned above , we propose a novel hierarchical reinforcement learning based optimization framework , which contains two levels of agents . As shown in Figure 2 , we maintain a buffer to cache the newly generated orders and periodically dispatch all cached orders at once . But instead of dispatching the cached orders of fixed numbers or predicting future orders , we design an upper-level agent to dynamically determine whether to wait longer for caching more future orders at each moment . Though waiting longer will postpone the dispatching and transportation of the earlier cached orders , additional future orders can be taken into account for the vehicle-order matching . In this way , each vehicle will have more candidate orders to choose , thus the overall travelling distances will be more potentially to be optimized for shorter4 . This process could be regarded as sacrificing a little time in exchange for a precise estimation of future orders . However , waiting for too long will also increase the risk of overtime of the earlier cached orders . Thus , whether to wait longer to cache more orders at each moment will have a long-term impact on the overall dispatching results , and can be naturally modeled as a sequential decision-making problem . We model this procedure as a Markov Decision Process ( MDP ) . Depending on whether to wait longer at each moment , the overall DPDP can be dynamically partitioned into a series of static sub-problems , each of which includes different numbers of orders . As shown in Figure 2 , the generated orders are accumulated in the buffer until the upper-level agent decides to stop caching at time ti+k . Then , the agent releases the cached orders to the lower-level agent and clears the buffer . Given the released orders ( which form a static sub-problem , i.e. , a PDP ) , the lower-level agent is appointed to assign the orders to the most appropriate vehicles and arrange the transportation route of each vehicle , such that the transportation cost of these orders could be minimized . First , a set of basic operators are maintained , whose roles are converting one feasible solution to another . For instance , given an initial solution ’ s route { A- > B- > C } with three nodes A , B and C , a typical operator is swapping two nodes [ 13 ] , e.g. , swapping A and B . After applying this operator , { A- > B- > C } is converted to { B- > A- > C } . If the travelling cost of { B- > A- > C } is less than { A- > B- > C } , the initial solution is improved . On this basis , we design the lower-level agent similar to the traditional metaheuristic algorithms which sequentially manipulates these operators to improve the solution of each PDP . The difference is that we model the process of sequentially manipulating these operators as an MDP and incorporate RL methods to optimize the policy instead of manually designing complex rules . Finally , the best found solution is adopted by the order dispatching system to assign the orders to the vehicles and arrange their transportation routes . In the following two subsections , we will go into more details of the designed two agents . 4In Table 7 of Appendix I , we show that splitting the overall DPDP into different sub-problems with different time spans will lead to different qualities of solutions

Summary: The paper considers dynamic pickup and delivery problems in which a limited number of vehicles need to be scheduled to handle all demands with the goal of minimizing the average traveling distance. The problem involves capacity constraints of the vehicles, LIFO constraint, and soft time window constraint, where violating the time windows result in some penalty.   A new algorithm is proposed which is based on the existing idea: Make a buffer to cache the newly arrived orders, then periodically dispatch all the orders in the buffer. The current approaches add the orders to the buffer up to a given size or based on the prediction of the future, but the new proposed approach uses an upper-level agent which determines whether to wait for more items (and get some delay penalty) or dispatch the current orders. For this purpose, a day is partitioned into 144 time-slots of each 10 minutes and the goal is to decide to continue or wait at each of those steps. Then, the received ordered in the buffer with all the orders that are not picked up will be passed to the lower-level agent to solve the static pick-up and delivery sub-problem. The upper-level agent uses DQN to train the agent in which the same goal as of the problem is used as the reward. The state includes "the number of orders accumulated in the buffer, the number of available vehicles, the amount of time left before exceeding the time limit of each order".  The lower level agent learns the assignment of the orders to the vehicles and the sequence to pass over the orders. The state includes the position information of the customers and vehicles. Action is choosing one of four introduced heuristic improvements for routes. They can be applied to the routes and if there are any improvements, the new route will be accepted. A graph neural network with REINFORCE algorithm is used to train the policy.