ScheduleNet: Learn to solve multi-agent scheduling problems with reinforcement learning

1 INTRODUCTION . Optimal assignments of multiple autonomous agents for sequential completion of distributed tasks are necessary to solve various types of scheduling problems in the logistics , transportation , and manufacturing industries . Examples of such scheduling problem include finding the optimal delivery plans for vaccines , customer pickup orders for ride-sharing services , and machine operation sequence in modern manufacturing facilities . As the size of the problems increases , solving largescale scheduling problems using mathematical programming becomes infeasible or ineffective due to the expensive computational cost . Furthermore , such optimization approaches can not solve realtime scheduling problems where the new tasks appear dynamically . Target Problem & Challenges . In this paper , we propose ScheduleNet , an RL-based decentralized constructive scheduler for coordinating multiple agents to finish tasks with minimum completion time . The objectives of the target problems and their associated technical challenges are as follows : • Min-Max vs Min-Sum : ScheduleNet seeks to minimize the total completion time ( i.e. , makespan ) for various time-critical distributed tasks ( e.g. , vaccine delivery ) . Most multi-agent scheduling problems are designed to minimize the total traveling distance of all agents , which often result in unbalanced task assignments among the agents ( Bakach et al. , 2021 ) . Although the makespan is the most direct and intuitive reward for inducing coordination , it is notoriously difficult to train the decentralized policy with this sparse and delayed reward due to the temporal and spatial credit assignment issues . • Construction Heuristic vs Improvement heuristic : ScheduleNet builds a solution sequentially by assigning an idle agent to one of the remaining tasks while considering the relationships among the remaining tasks and agents . This construction heuristic ensures that the learned policy can reschedule whenever a new event occurs ( i.e. , an agent finishes the assigned task or a new task appears ) . However , it is more challenging to find a better plan due to its sequential solution construction ; a wrong choice at the early stage can cause irreversible poor results at the end . • Decentralized vs Centralized : ScheduleNet allows each agent to choose its destination independently while using its local observations and incorporating other agents ’ assignments . This decentralization ensures that the learned policy can solve a large-scale problem without having to search over the joint action space for all agents . However , to make the independently-chosen scheduling decision produce an excellent global performance , a sophisticated coordination mechanism should be incorporated into decentralized policy implicitly . Proposed Decision-Making Scheme . We formulate the multi-agent scheduling problems ( mSPs ) as an event-based Markov decision process ( MDP ) with an episodic reward , and derive a decentralized decision-making policy using reinforcement learning . At every step , ScheduleNet accepts the MDP state as an input and assigns an idle agent to one of the feasible tasks . The decision-making procedure of ScheduleNet is as follows : • ScheduleNet first represents the MDP state as an agent-task graph , which captures the complex relationships among the entities effectively and is general enough to be applied to various mSPs . • ScheduleNet then employs the type-aware graph attention ( TGA ) to extract important relational features among the agents and tasks for making the best cooperative task assignment . • Lastly , ScheduleNet computes the agent-task assignment probability by utilizing the computed node embeddings . Training Method . Although the makespan ( shared team reward ) is the most direct and general reward design for solving mSPs , training a decentralized scheduling policy using this reward is extremely difficult due to the credit assignment issues ( Riedmiller et al. , 2018 ; Hare , 2019 ) . Additionally , makespan is highly volatile due to the combinatorial aspect of mSPs ’ solution space ; a slight change in the solution can drastically alter the outcome . To overcome these issues , we employ the Clip-REINFORCE algorithm with normalized reward to train the decentralized cooperative policy effectively . Novelties . The proposed method that derives the decentralized constructive schedulers to coordinate multiple agents has the following novelties and advantages : • Balance between Representability & Scalability : ScheduleNet can extract crucial features effectively using TGA and make the best cooperative task assignment ( representability ) in a computationally efficient manner . The computationally-efficient representation scheme and the constructive decision-making scheme allows ScheduleNet to solve large-scale scheduling problems . • Transferability & Generalizability : The type-aware graph representation allows the trained policy to solve problems with different numbers of agents and tasks ( size transferability ) . Furthermore , this general state representation with the universal reward signal ( i.e. , makespan ) allows ScheduleNet to be used to solve various multi-agent scheduling problems ( generalizability ) . We validate this by showing that ScheduleNet can learn to solve multiple traveling salesmen problem ( mTSP ) , as well as jop-shop scheduling problem ( JSP ) whose constraints are more complex than that of mTSP . 2 RELATED WORKS . RL approaches that solve vehicle routing problems . According to Mazyavkina et al . ( 2020 ) , the RL approaches that solve vehicle routing problems can be categorized into : ( 1 ) the improvement heuristics that rewrite the complete solution iteratively to obtain a better routing plan ( Wu et al. , 2020 ; da Costa et al. , 2020 ; Chen & Tian , 2019 ; Lu et al. , 2020 ) ; ( 2 ) the construction heuristics that construct the solution sequentially by assigning idle vehicles to unvisited cities until the complete routing plan ( sequence ) is constructed ( Bello et al. , 2016 ; Nazari et al. , 2018 ; Kool et al. , 2018 ; Khalil et al. , 2017 ) , and ( 3 ) the hybrid approaches that blend both approaches ( Joshi et al. , 2020 ; Fu et al. , 2021 ; Kool et al. , 2021 ; Ahn et al. , 2020 ) . Typically , the improvement heuristics show better performances than construction heuristics as they revise the complete plan iteratively . However , the construction heuristics are more effective for online vehicle routing problems , where the routes should be updated whenever a new customer appears . These RL approaches have exclusively focused on static planning in a single-agent perspective , which is far from the settings of real applications . RL approaches that solve min-max mTSP . There are only few RL approaches that solve min-max mTSP , which involves minimizing the makespan for multiple salesmen to visit all cities . Hu et al . ( 2020 ) applies RL to train the clustering algorithm that groups cities , and strong TSP heuristics ( e.g. , OR-Tool ) to optimize the sub-tours of the city clusters . This is fundamentally different from ScheduleNet , which derives a complete end-to-end learned heuristic that constructs a feasible solution from “ scratch ” without relying on any existing solvers . Cao et al . ( 2021 ) , which proposes a transformer-based construction policy to solve min-max mTSP , is the most similar approach to ScheduleNet . RL approaches that solve Job-shop scheduling problems . There are only few RL approaches that solve the JSPs , which involves minimizing the makespan for multiple machines to finish the sequence of operations that is required to finish their jobs . For example , Gabel & Riedmiller ( 2012 ) ; Lin et al . ( 2019 ) propose to learn a scheduling policy for each machine ; hence , it requires additional training to solve JSPs with a different number of machines . Recently , Park et al . ( 2021 ) ; Zhang et al . ( 2020 ) have proposed to learn a shared scheduling policy that can be used for all machines to minimize the makespan . Similar to ScheduleNet , these studies utilize the disjunctive graph representation of JSP . However , these methods utilize human-engineered dense reward , while ScheduleNet uses sparse and delayed reward ( i.e. , makespan ) to train a policy . 3 PROBLEM FORMULATION . We formulate mSP as a MDP with sparse reward , and aim to derive a decentralized scheduling decision-making policy that can be shared by all agents . The MDP is defined as : State . We define state sτ as the τ -th partial solution of mSP ( i.e. , the completed/uncompleted tasks , the status of agents , and the sequence of the past assignments ) . The initial s0 and terminal state sT are defined as an empty and a complete solution , respectively . Action . We define action aτ as the act of assigning an idle agent to one of the feasible tasks ( unassigned tasks ) . We refer to aτ as the agent-to-task assignment . When the multiple agents are idle at the same time t , we randomly choose one agent and assign an action to the agent . This is repeated until no agent is idle . Note that such randomness does not alter the resulting solutions , since the agents are considered to be homogeneous and the scheduling policy is shared . Transition . The proposed MDP is formulated with an event-based transition . An event is defined as the case where any agent finishes the assigned task ( e.g. , a salesman reaches the assigned city in mTSP ) . Whenever an event occurs , the idle agent is assigned to a new task , and the status of the agent and the target task are updated accordingly . We enumerate the event with τ to avoid confusion from the elapsed time of the problem ; t ( τ ) is a function that returns the time of event τ . Reward . The proposed MDP uses the negative makespan ( i.e . total completion time of tasks ) as the reward ( i.e. , r ( sT ) = −t ( T ) ) that is realized only at sT . 3.1 EXAMPLE : MDP FORMULATION OF MTSP . Let us consider the single-depot mTSP with two types of entities : m salesmen ( i.e. , m agents ) and N cities ( i.e. , N tasks ) . All salesmen start their journey from the depot , and come back to the depot after visiting all cities ( each city can be visited by only one salesman ) . The solution to mTSP is considered to be complete when all the cities have been visited , and all salesmen have returned to the depot . The MDP formulation for mTSP is similar to that of the general mSP . The specific definition of the state for mTSP is as follows : State . We define sτ = ( { siτ } N+mi=1 , senvτ ) , which is composed of two types of states : entity state siτ and environment state senvτ . • siτ = ( p i τ ,1 active τ ,1 assigned τ ) is the state of the i-th entity . piτ is the 2D Cartesian coordinate of the i-th entity at the τ -th event . 1activeτ indicates whether the i-th agent/task is active ( agent is working/ task is not visited ) . Similarly , 1assignedτ indicates whether agent/task is assigned . • senvτ contains the current time of the environment , and the sequences of cities visited by the salesmen . 4 SCHEDULENET . In this section , we explain how ScheduleNet recommends a scheduling action aτ of an idle agent from input sτ ( partial solution ) . This is done by ( 1 ) constructing the agent-task graph Gτ , ( 2 ) embedding Gτ using TGA , and ( 3 ) computing the assignment probabilities . Figure 1 illustrates the decision-making process of ScheduleNet .

Summary: This paper tries to solve traditional scheduling problems by using the deep reinforcement learning (RL) framework. To make the RL framework applicable to such problems, the authors model the problem states as an agent-task graph and encode the nodes in the graph utilizing a type-aware graph attention technique they proposed. The obtained node embeddings are next used to compute the final assignment probabilities. The authors use implementation tricks such as reward normalization and clip-loss to accelerate training or achieve better performance in the training process. The proposed method performs well in the traveling salesmen problem and the job-shop scheduling problem.