Neural Combinatorial Optimization with Reinforcement Learning : Solving theVehicle Routing Problem with Time Windows

1 INTRODUCTION . Vehicle routing problem with time windows ( VRPTW ) can be defined as an extension of the wellknown vehicle routing problem ( VRP ) in which the objective is to design a network of routes to satisfy customers demands with minimal total costsGan et al . ( 2012 ) . Each route starts from and ends at the depot , and for which the total demand is strictly under the vehicle capacity . Except for the depot , all clients are visited once within the corresponding time window . Moreover , when this previous constraint is violated , a penalty cost will be applied . Plenty of research have focused on studying and solving this problem commonly referred to as NP-hard Lenstra & Kan ( 1981 ) . Introducing time windows increases its computational complexity , therefore VRPTW requires more advanced techniques to get reliable solutions . The literature is full of hand-engineered heuristics that provide near-optimal solutions within practical runtime Bräysy & Gendreau ( 2005 ) . In general , these classical approaches fulfill the trade-off between optimality and complexity . However , the challenge becomes greater when new problem instances are defined or new features are inserted . In this case , a manual adaptation and business knowledge are required to maintain the heuristic ’ s efficiency . Considering this tedious maintenance process and the huge advancement in learning methods , some works have been focusing on using RNN and RL to learn independent heuristics for solving the VRP . In this paper , we are extending those works by adding the time windows constraint . Precisely , we develop an end-to-end model able to provide near-optimal solutions for every problem instance . In other words , as long as the trained model receives data coming roughly from the same generating process , it yields a reliable solution without the need to build another model for this specific data . The framework we propose is suitable for introducing more flexibility regarding inputs variance because of the used reinforcement learning approach . The learning process can be formulated as a Markov decision process in a well-defined environment . Therefore , the optimal solution is designed following a dynamic perspective , where the policy is architected through an attention model and trained to maximize the reward which corresponds to the negative tour length . Additionally , the state can be defined as the data relating to each customer ( cartesian coordinates , demand , service time , allowed time windows ) . This state clearly combines static and dynamic parameters . Its dynamic dimension reflects directly the demand change over the learning steps in a manner that once a client is visited its demand turns into zero . Finally , the environment actions could be seen as the set of customers to include in the solution at each stage . Our approach is an extension of some existing works namely : Bello et al . ( 2016 ) , Nazari et al . ( 2018 ) , and Kool et al . ( 2018 ) . Our added value lies in generalizing those works to solve the VRPTW , one of the most common combinatorial optimization problems . This paper aims also to strengthen the use of machine learning for solving hard combinatorial problems Bengio et al . ( 2021 ) . Including time windows constraint increases the VRP complexity and changes the learning and optimization strategy . Consequently , it requires customization in the data generating process , and a new architecture of the reinforcement learning space , especially the environment policy and the transition function . Concretely , the complete model is made up of a neural network which receives the embedding of dynamic and static inputs . The outputs of this sub-model are processed through an attention mechanism within the reinforcement learning space to deliver at the end the near-optimal sequence . 2 APPROACH BACKGROUND . Before we deep dive in the model architecture and present its main components , we should briefly highlight some problem-related concepts . In addition to the commonly known ideas about VRP , the treated problem presents some specific assumptions . The halting condition is attained when all nodes demand are satisfied . Furthermore , the vehicle of capacity D returns to the depot to refill when its load runs out without resetting the time . Each customer has a service time si strictly lower than the corresponding time window range [ T ( i ) min , T ( i ) max ] to unload its demand . We assume in our case that the time spent to go from customer i to customer j is proportional to the distance dij . In addition , we define T to be the needed time to serve all clients . In short , VRPTW can be formulated as a graph G = ( V , E ) , such that X = { x0 , x1 , ... , xn } is the set of vertices where x0 stands for the depot , and E = { eij = ( xi , xj ) | ( xi , xj ) ∈ V 2 } is the set of edges . Besides , each xi is associated with a tuple of features ( ci , di , si , TWi ) and each eij is associated with a cost dij , where : ci : is the two dimensional coordinates of node i. di : is the demand of node i. si : is the service time of node i. TWi : is the time window to serve node i. dij : is the distance between node i and node j . The ultimate goal of solving VRP TW is to find the path π = ( π1 , ... , πN ) that minimizes the cost within the following space : Π = { ( π1 , ... , πN ) , πi ∈ { x0 , x1 , ... , xn } } . Remark : As a part of the problem setting , it is important to mention that the split deliveries are not allowed , and only one vehicle drive to serve all clients . Many research in the literature tackled the above-described problem relying on hand-crafted approaches Fisher ( 1994 ) , Cordeau & Groupe d ’ études et de recherche en analyse des décisions ( Montréal ( 2000 ) . However , in the recent few years , a new dedicated branch called Neural Combinatorial Optimization has dawned . Precisely , Vinyals et al . ( 2015 ) was the first research that presented its main foundations through a sequence to sequence model called pointer network . This latter consists of two coupled RNN , the first one is used to encode inputs to a specific representation , and the other one to decode the processed output and render it as a sequence . In addition , this work follows a supervised learning approach to train this model using labeled data for the traveling salesman problem ( TSP ) . Even though , this study delivered promising insights about using neural networks for solving combinatorial problems , its results are strongly linked to the quality of the used labels . Furthermore it is a hard task to find enough TSP labeled data for training . To overcome these limitations a reinforcement learning environment to train the RNN models was proposed by Bello et al . ( 2016 ) . Besides , their approach is fully supported by the fact that roughly all combinatorial problems are evaluated through a specific reward policy . They provide significant results for both TSP and Knapsack problems in terms of the solution quality and the computational time . Others attempts in the literature considered the same perspective for solving the VRP namely Nazari et al . ( 2018 ) and Peng et al . ( 2019 ) . Unlike TSP , VRP includes some dynamic parameters , especially regarding node features as the case of the demand which changes after visiting a particular customer . As a consequence , the major changes they proposed deal with the attention mechanism , the transition function in the decoding steps , and the embedding of both dynamic and static inputs . Taking into account the above-described evolution of neural combinatorial optimization , we will present the model architecture to solve the VRPTW . 3 THE MODEL ARCHITECTURE . The configuration encoder-decoder has proved an important efficiency to deal with many problems including VRP Sutskever et al . ( 2014 ) . As shown in Figure 1 , the encoder receives raw data X as described in section 2 and converts it to a convenient representation through many layers . These useful features constructed by the encoder are grasped by the decoder to build progressively the near-optimal sequence . Concretely , the encoder gradually picks out one node to include in the sequence depending on a calculated distribution for each node . Thus , one can give the joint probability of a solution π using the chain rule as follows : pθ ( π | X ) = N∏ i=1 pθ ( πi | X , π1 : i−1 ) ( 1 ) Such that , θ are the estimated distribution parameters , and π1 : i−1 = ( π1 , π2 , ... , πi−1 ) . 3.1 ENCODER . More precisely , the encoder incrementally gets the input sequence and transforms it into a set of embeddings . For each node xi with features of dimension d ( d = 5 in the case of VRPTW ) , the starting embedding h ( 0 ) i with dimension d ′ is calculated via a linear transformation as follows : { h ( 0 ) i = Wxi + b if i 6= 0 h ( 0 ) i = W0xi + b0 if i = 0 } ( 2 ) Where W ∈ IRd ′ ×d and b ∈ IRd ′ are the learnable parameters , while W0 and b0 are the ones used for the case of the depot . As shown in the Figure 1 , this set of embeddings crosses a network of 3 layers for further updates . Each layer comprises two sublayers : a self-attention sublayer and then a feed-forward sublayer . • Self attention sublayer : is called also multi-head attention , it brings specific updates to the primary embeddings Vaswani et al . ( 2017 ) . Let h ( l ) i stands for the embedding of the node i in the layer l ∈ { 1 , 2 , 3 } . Taking into account the recursive relation between layers , one can compute the multi-head attention vector MHA ( l ) i following these equations Peng et al . ( 2019 ) : q ( l ) im = W Q mh ( l−1 ) i k ( l ) im = W K m h ( l−1 ) i v ( l ) im = W V mh ( l−1 ) i ( 3 ) u ( l ) ijm = q ( l ) imk ( l ) jm ( 4 ) a ( l ) ijm = eu ( l ) ijm∑n y e u ( l ) iym ( 5 ) h ′ ( l ) im = n∑ j=0 a ( l ) ijmv ( l ) jm ( 6 ) MHA ( l ) i ( h ( l−1 ) 0 , h ( l−1 ) 1 , ... , h ( l−1 ) n ) = M∑ m=1 WOmh ′ ( l ) im ( 7 ) Where , WQm ∈ IRd×d ′ , WKm ∈ IRd×d ′ , WVm ∈ IRd×d ′ are learnable parameters , M is the number of heads , q ( l ) im ∈ IRd is the query vector , k ( l ) im ∈ IRd is the key vector , and v ( l ) im ∈ IRd is the value vector . • Feed forward sublayer : Using the multi-head attention vector , the update made up through this sublayer for each node i is computed as follows Peng et al . ( 2019 ) : ĥ ( l ) i = tanh ( h ( l−1 ) i +MHA ( l ) i ( h ( l−1 ) 0 , h ( l−1 ) 1 , ... , h ( l−1 ) n ) ) ( 8 ) FF ( ĥ ( l ) i ) = W F 1 ReLu ( W F 0 + b F 0 ĥ ( l ) i ) ( 9 ) h ( l ) i = tanh ( ĥ ( l ) i + FF ( ĥ ( l ) i ) ) ( 10 ) Such that , h ( l−1 ) i is the embedding of node i at the layer l − 1 , WF0 ∈ IRd×d ′ , and WF0 ∈ IRd ′ ×d . This computation process is replicated at each layer , consequently the final embedding vector ( h ( N ) 0 , h ( N ) 1 , ... , h ( N ) n ) is the one obtained while ending the layer N . This output vector is considered as the main structural features for the decoder component .

Summary: This paper proposes a Neural Combinatorial Optimization approach to solving the Vehicle Routing Problem with Time Windows. It uses a policy gradient method to optimize an attention model, paired with a masking scheme that prevents unwanted actions during the policy rollout. Performance is compared with OR-Tools and LKH-3 solvers.