ScheduleNet: Learn to Solve MinMax mTSP Using Reinforcement Learning with Delayed Reward

1 INTRODUCTION . There have been numerous approaches to solve combinatorial optimization ( CO ) problems using machine learning . Bengio et al . ( 2020 ) have categorized these approaches into demonstration and experience . In demonstration setting , supervised learning has been employed to mimic the behavior of the existing expert ( e.g. , exact solvers or heuristics ) . On the other hand , in the experience setting , typically , reinforcement learning ( RL ) has been employed to learn a parameterized policy that can solve newly given target problems without direct supervision . While the demonstration policy can not outperform its guiding expert , RL-based policy can outperform the expert because it improves its policy using a reward signal . Concurrently , Mazyavkina et al . ( 2020 ) have further categorized the RL approaches into improvement and construction heuristics . An improvement heuristics start from the arbitrary ( complete ) solution of the CO problem and iteratively improve it with the learned policy until the improvement stops ( Chen & Tian , 2019 ; Ahn et al. , 2019 ) . On the other hand , the construction heuristics start from the empty solution and incrementally extend the partial solution using a learned sequential decision-making policy until it becomes complete . There has been continuous effort to learn to solve famous CO problems such as Traveling Salesman Problem ( TSP ) and Vehicle Routing Problem ( VRP ) using RL-based construction heuristics ( Bello et al. , 2016 ; Kool et al. , 2018 ; Khalil et al. , 2017 ; Nazari et al. , 2018 ) . Although they have shown good optimality and computational efficiency performance , these approaches have been limited to only scheduling a single-vehicle . The multi-extensions of these routing problems , such as multiple TSP and multiple VRP , are underrepresented in the deep learning research community , even though they capture a broader set of the real-world problems and pose a more significant scientific challenge . The multiple traveling salesmen problem ( mTSP ) aims to determine a set of subroutes for each salesman , given m salesmen and N cities that need to be visited by one of the salesmen , and a depot where salesmen are initially located and to which they return . The objective of a mTSP is either minimizing the sum of subroute lengths ( MinSum ) or minimizing the length of the longest subroute ( MinMax ) . In general , the MinMax objective is more practical , as one seeks to visit all cities as soon as possible ( i.e. , total completion time minimization ) . In contrast , the MinSum formulation , in general , leads to highly imbalanced solutions where one of the salesmen visits most of the cities , which results in longer total completion time ( Lupoaie et al. , 2019 ) . In this study , we propose a learning-based decentralized and sequential decision-making algorithm for solving Minmax mTSP problem ; the trained policy , which is a construction heuristic , can be employed to solve mTSP instances with any numbers of salesman and cities . Learning a transferable mTSP solver in a construction heuristic framework is significantly challenging comparing to its single-agent variants ( TSP and CVRP ) because ( 1 ) we need to use the state representation that is flexible enough to represent any arbitrary number of salesman and cities ( 2 ) we need to introduce the coordination among multiple agents to complete the geographically distributed tasks as quickly as possible using a sequential and decentralized decision making strategy and ( 3 ) we need to learn such decentralized cooperative policy using only a delayed and sparse reward signal , makespan , that is revealed only at the end of the episode . To tackle such a challenging task , we formulate mTSP as a semi-MDP and derive a decentralized decision making policy in a multi-agent reinforcement learning framework using only a sparse and delayed episodic reward signal . The major components of the proposed method and their importance are summarized as follows : • Decentralized cooperative decision-making strategy : Decentralization of scheduling policy is essential to ensure the learned policy can be employed to schedule any size of mTSP problems in a scalable manner ; decentralized policy maps local observation of each idle salesman one of feasible individual action while joint policy maps the global state to the joint scheduling actions . • State representation using type-award graph attention ( TGA ) : the proposed method represents a state ( partial solution to mTSP ) as a set of graphs , each of which captures specific relationships among works , cities , and a depot . The proposed method then employs TGA to compute the node embeddings for all nodes ( salesman and cities ) , which are used to assign idle salesman to an unvisited city sequentially . • Training decentralized policy using a single delayed shared reward signal : Training decentralized cooperative strategy using a single sparse and delayed reward is extremely difficult in that we need to distribute credits of a single scalar reward ( makespan ) over the time and agents . To resolve this , we propose a stable MARL training scheme which significantly stabilizes the training and improves the generalization performance . We have empirically shown that the proposed method achieves the performance comparable to Google OR-Tools , a highly optimized meta-heuristic baseline . The proposed approach outperforms OR-Tools in many cases on in-training , out-of-training problem distributions , and real-world problem instances . We also verified that scheduleNet can provide an efficient routing service to customers . 2 RELATED WORK . Construction RL approaches A seminal body of work focused on the construction approach in the RL setting for solving CO problems ( Bello et al. , 2016 ; Nazari et al. , 2018 ; Kool et al. , 2018 ; Khalil et al. , 2017 ) . These approaches utilize encoder-decoder architecture , that encodes the problem structure into a hidden embedding first , and then autoregressively decodes the complete solution . Bello et al . ( 2016 ) utilized LSTM ( Hochreiter & Schmidhuber , 1997 ) based encoder and decode the complete solution ( tour ) using Pointer Network ( Vinyals et al. , 2015 ) scheme . Since the routing tasks are often represented as graphs , Nazari et al . ( 2018 ) proposed an attention based encoder , while using LSTM decoder . Recently , Kool et al . ( 2018 ) proposed to use Transformer-like architecture ( Vaswani et al. , 2017 ) to solve several variants of TSP and single-vehicle CVRP . On the contrary , Khalil et al . ( 2017 ) do not use encoder-decoder architecture , but a single graph embedding model , structure2vec ( Dai et al. , 2016 ) , that embeds a partial solution of the TSP and outputs the next city in the ( sub ) tour . ( Kang et al. , 2019 ) has extended structure2vec to random graph and employed this random graph embedding to solve identical parallel machine scheduling problems , the problem seeking to minimize the makespan by scheduling multiple machines . Learned mTSP solvers The machine learning approaches for solving mTSP date back to Hopfield & Tank ( 1985 ) . However , these approaches require per problem instance training . ( Hopfield & Tank , 1985 ; Wacholder et al. , 1989 ; Somhom et al. , 1999 ) . Among the recent learning methods , Kaempfer & Wolf ( 2018 ) encodes MinSum mTSP with a set-specialized variant of Transformer architecture that uses permutation invariant pooling layers . To obtain the feasible solution , they use a combination of the softassign method Gold & Rangarajan ( 1996 ) and a beam search . Their model is trained in a supervised setting using mTSP solutions obtained by Integer Linear Programming ( ILP ) solver . Hu et al . ( 2020 ) utilizes a GNN encoder and self-attention Vaswani et al . ( 2017 ) policy outputs a probability of assignment to each salesman per city . Once cities are assigned to specific salesmen , they use existing TSP solver , OR-Tools ( Perron & Furnon ) , to obtain each worker ’ s subroutes . Their method shows impressive scalability in terms of the number of cities , as they present results for mTSP instances with 1000 cities and ten workers . However , the trained model is not scalable in terms of the number of workers and can only solve mTSP problems with a pre-specified , fixed number of workers . 3 PROBLEM FORMULATION . We define the set of m salesmen indexed by VT = { 1 , 2 , ... , m } , and the set of N cities indexed by VC = { m + 1 , 2 , ... , m + N } . Following mTSP conventions , we define the first city as the depot . We also define the 2D-coordinates of entities ( salesmen , cities , and the depot ) as pi . The objective of MinMax mTSP is to minimize the length of the longest subtour of salesmen , while subtours covers all cities and all subtours of salesmen end at the depot . For the clarity of explanation , we will refer to salesman as a workers , and cities as a tasks . 3.1 MDP FORMULATION FOR MINMAX MTSP . In this paper , the objective is to construct an optimal solution with a construction RL approach . Thus , we cast the solution construction process of MinMax mTSP as a Markov decision process ( MDP ) . The components of the proposed MDP are as follows . Transition The proposed MDP transits based on events . We define an event as the the case where any worker reaches its assigned city . We enumerate the event with the index τ for avoiding confusion from the elapsed time of the mTSP problem . t ( τ ) is a function that returns the time of event τ . In the proposed event-based transition setup , the state transitions coincide with the sequential expansion of the partial scheduling solution . State Each entity i has its own state siτ = ( piτ ,1 active τ ,1 assigned τ ) at the τ -th event . the coordinates piτ is time-dependent for workers and static for tasks and the depot . Indicator 1activeτ describes whether the entity is active or inactive In case of tasks , inactive indicates that the task is already visited ; in case of worker , inactive means that worker returned to the depot . Similarly , 1assignedτ indicates whether worker is assigned to a task or not . We also define the environment state senvτ that contains the current time of the environment , and the sequence of tasks visited by each worker , i.e. , partial solution of the mTSP . The state sτ of the MDP at the τ -th event becomes sτ = ( { siτ } m+Ni=1 , senvτ ) . The first state s0 corresponds to the empty solution of the given problem instance , i.e. , no cities have been visited , and all salesmen are in the depot . The terminal state sT corresponds to a complete solution of the given mTSP instance , i.e. , when every task has been visited , and every worker returned to the depot ( See Figure 1 ) . Action A scheduling action aτ is defined as the worker-to-task assignment , i.e . salesman has to visit the assigned city . Reward . We formulate the problem in a delayed reward setting . Specifically , the sparse reward function is defined as r ( sτ ) = 0 for all non-terminal events , and r ( sT ) = t ( T ) , where T is the index of the terminal state . In other words , a single reward signal , which is obtained only for the terminal state , is equals to the makespan of the problem instance .

Summary: The mTSP with the goal of minimizing the longest route is considered, which minimizes the maximum route. This objective results in a balanced-length set of routes, which compared to the sum of routes obtains a more practical result. A graph representation based on the worker and assigned tasks is defined, then a type-aware graph attention (TGA) embedding procedure is proposed in which it obtains an embedding for node and edge representation. The state for each entity involves the 2D coordinates of the entity and the boolean indicator of idleness and assigned task of the worker. The action is worker-to-task assignment, and the reward is the makespan of finishing the problem, which is a sparse function. In type-aware graph attention (TGA) embedding, the embedding of node and edge are obtained and using the attention mechanism, an important weight for the embedding of each edge is obtained. The embedding functions are type-dependent which means the embedding considers the source node-type. The final message value for each node is obtained by multiplying the weight and edge embedding value of its neighbors. Then, an MLP is used to obtain a value for each source and possible target node, which takes the final value of the source, target, and edge between those as input. Then, the output of the MLP is used to get the final probability of choosing the next node.