Learning to Solve Multi-Robot Task Allocation with a Covariant-Attention based Neural Architecture

1 INTRODUCTION . In multi-robot task allocation ( MRTA ) problems , we study how to coordinate tasks among a team of cooperative robotic systems such that the decisions are free of conflict and optimize a quantity of interest ( Gerkey & Matarić , 2004 ) . The potential real-world applications of MRTA are immense , considering that multi-robotics is one of the most important emerging directions of robotics research and development , and task allocation is fundamental to most multi-robotic or swarm-robotic operations . Example applications include disaster response ( Ghassemi & Chowdhury , 2018 ) , last-mile delivery ( Aurambout et al. , 2019 ) , environment monitoring ( Espina et al. , 2011 ) , reconnaissance ( Olson et al. , 2012 ) and combat Behjat et al . ( 2021 ) . Although various approaches ( e.g. , graph-based methods ( Ghassemi et al. , 2019 ) , integer-linear programming ( ILP ) approaches ( Nallusamy et al. , 2009 ; Toth & Vigo , 2014 ) , and auction-based methods ( Dias et al. , 2006 ; Schneider et al. , 2015 ) ) have been proposed to solve the combinatorial optimization problem underlying MRTA operations , they usually do not scale well with number of robots and/or tasks , and do not readily adapt to complex problem characteristics without tedious hand-crafting of the underlying heuristics . In the recent years , a rich body of work has emerged on using learning-based techniques to model solutions or intelligent heuristics for combinatorial optimization ( CO ) problems over graphs . The existing methods are mostly limited to classical CO problems , such as multi-traveling salesman ( mTSP ) , vehicle routing ( VRP ) , and max-cut type of problems . We specifically focus on a class of MRTA problems that falls into the Single-task Robots , and Single-robot Tasks ( SR-ST ) class defined in ( Gerkey & Matarić , 2004 ; Nunes et al. , 2017 ) . Based on iTax taxonomoy as defined in Gerkey & Matarić ( 2004 ) , these problems fall into the In-schedule Dependencies ( ID ) category . Here , a feasible and conflict-free task allocation is defined as assigning any task to only one robot ( Ghassemi et al. , 2019 ) . For solving these problems , we propose a new covariant attention-based model ( aka CAM ) , a neural architecture for learning over graphs to construct the MRTA policies . This architecture builds upon the attention mechanism concept and innovatively integrates an equivariant embedding of the graph to capture graph structure while remaining agnostic to node ordering . We implement CAM on an original MRTA problem , a suite of benchmark MRTA problems and benchmark VRP problems to perform generalizability , scalability and comparative analyses of the new method . We also perform an analysis on the impact of the neighborhood size on the performance on the benchmark MRTA problem . 1.1 MULTI-ROBOT TASK ALLOCATION . In recent years , learning approaches based on Graph Neural Networks or GNN are being increasingly used to solve planning problems with a CO formulation , e.g. , TSP , VRP , Max-Cut , Min-Vertex , and MRTA Kool et al . ( 2019 ) ; Barrett et al . ( 2019 ) ; Khalil et al . ( 2017 ) ; Kaempfer & Wolf ( 2018 ) ; Mittal et al . ( 2019 ) ; Li et al . ( 2018 ) ; Nowak et al . ( 2017 ) ; Wang & Gombolay ( 2020 ) ; Tolstaya et al . ( 2020 ) ; Sykora et al . ( 2020 ) ; Dai et al . ( 2017 ) . Further details on these related learning-based studies can be found in Appendix A . Some of the conventional ILP , MILP , and INLP based methods for MRTA have been discussed in Appendix D.1 . GNNs provide the advantage of being able to capture both Euclidean features ( e.g. , task location ) , as well as non-Euclidean features such as task capacity , task deadline and the local structure of task neighborhoods . The latter serves as higher-level meaningful features that assist in generalized decision-making These existing studies are however limited in three key aspects : 1 ) They address simplified problems that often exclude common real-world factors such as resource and capacity constraints Kool et al . ( 2019 ) ; Kaempfer & Wolf ( 2018 ) ; Khalil et al . ( 2017 ) ; Tolstaya et al . ( 2020 ) ) . 2 ) They are mostly focused on smaller sized problems ( ≤ 100 tasks and 10 robots ) Paleja et al . ( 2020 ) ; Strens & Windelinckx ( 2005 ) ; Wang & Gombolay ( 2020 ) ; Sykora et al . ( 2020 ) , with their scalability remaining unclear . 3 ) They rarely provide evidence of generalizing to problem scenarios that are larger in size than those used for training . This capability would be particularly critical since real-world MRTA problems often involve simulating episodes whose costs scale with the number of tasks and robots , making re-training efforts burdensome . To address these gaps , here we propose a new learning framework that can solve large-sized MRTA problems ( SR-ST ) with commonly considered constraints – involving up to 1000+ tasks and 200+ robots – and generalize across even larger problem scenarios without the need to re-train . For most practical scenarios with larger number of locations , a highly optimal solution is not always desired , while a good feasible solution is the priority as pointed out by Cappart et al . ( 2021 ) . Therefore , to enable scalable policies , we design a novel encoder based on the concept of Covariant Compositional Networks ( CCN ) Hy et al . ( 2018 ) , which is hypothesized to effectively combine local structural information with permutation invariance . The encoder is followed by a decoder based on a Multi-head Attention mechanism Kool et al . ( 2019 ) ; Vaswani et al . ( 2017 ) which fuses the encoded information and problem/mission-specific information ( Context ) using simple matrix multiplication , in order to enable decentralized sequential decision-making . 1.2 CONTRIBUTIONS OF THIS PAPER . The primary contributions of this paper can thus be stated as follows : 1 ) We formulate the general SR-ST class of MRTA problems as a Markov Decision Process or MDP over graphs with the multirobot state information embedded as the context portion of the policy model , such that the ( task allocation ) policy can be learned using an RL approach . 2 ) We design the GNN that acts as the policy network as an encoder-decoder architecture , where the encoder is innovatively based on covariant compositional networks ( CCN ) , whose embedding capabilities significantly improve generalizability and scalability to larger task graphs and multi-robot teams . 3 ) We implement an attention based decoder ( inspired by Kool et al . ( 2019 ) ) to enable sequential decision-making , and specifically extend it to a multi-agent combinatorial optimization setting . The proposed CAM architecture is evaluated on a representative MRTA problem that involves coordinating a team of unmanned aerial vehicles ( UAVs ) to time-efficiently deliver flood relief . The results of this case study demonstrate how CAM clearly outperforms the state-of-the-art attention based method AM ( Kool et al. , 2019 ) , in terms of scalabilty and convergence , thereby emphasizing the effectiveness of the new encoder . Further case studies show that CAM continues to compare favorably to AM over benchmark MRTA problems ( with time and capacity constraints ) and CVRP problems . Comparisons to non-learning baselines for these benchmark problems demonstrate the significant online computation advantages of learnt policies , with the latter being 10-100 times faster . The remainder of the paper is organized as follows : Section 2 defines the MRTA problem and its formulation as an MDP over graphs . Section 3 presents our proposed new GNN architecture . Section 4 describes simulation settings and different case studies . Results are discussed in Section 5 . 2 MRTA : PROBLEM DEFINITION AND FORMULATIONS . The multi-robot task allocation ( MRTA ) problem is defined as the allocation of tasks and resources among several robots that act together without conflict in the same environment to accomplish a common mission . The optimum solution ( decision ) of the MRTA problem is a sequence of tasks for each robot ( conflict-free allocation ) that maximizes the mission outcome ( e.g. , fraction of tasks completed ) or minimize the mission cost ( e.g. , total distance travelled ) subject to the robots ’ range constraints . Here , the following assumptions are made : 1 ) All robots are identical and start/end at the same depot ; 2 ) There are no environmental uncertainties ; 3 ) The location ( xi , yi ) of task-i and its time deadline τi are known to all robots ; 4 ) Each robot can share its state and its world view with other robots ; and 5 ) There is a depot ( Task-0 ) , where each robot starts from and visits if no other tasks are feasible to be undertaken due to the lack of available range . Each tour is defined as departing from the depot , undertaking at least one task , and returning to the depot . 6 ) Motivated by the multi-UAV relief delivery problem , tasks are considered to be instantaneous , which means that reaching the waypoint associated with a task completes that task . This MRTA problem is a class of combinatorial optimization problems , which can be modeled in graph space . In order to learn policies that yield solutions to this CO problem , we express the MRTA problem as a Markov Decision Process ( MDP ) over a graph , described next . The optimization formulation of MRTA is then given in Section 2.2 . 2.1 MDP OVER A GRAPH . The MRTA problem involves a set of nodes/vertices ( V ) and a set of edges ( E ) that connect the vertices to each other , which can be represented as a complete graph G = ( V , E ) . Each node represents a task , and each edge connects a pair of nodes . Let Ω be a weight matrix where the weight of the edge ( ωij ∈ Ω ) represents the cost ( e.g. , distance ) incurred by a robot to take task-j after achieving task-i . For MRTA with N tasks , the number of vertices and the number of edges are N and N ( N − 1 ) /2 , respectively . Node i is assigned a 3-dimensional feature vector denoting the task location and time deadline , i.e. , di = [ xi , yi , τi ] where i ∈ [ 1 , N ] . Here , ωij can be computed as ωij = √ ( xi − xj ) 2 + ( yi − yj ) 2 , where i , j ∈ [ 1 , N ] . The MDP defined in a decentralized manner for each individual robot ( to capture its task selection process ) can be expressed as a tuple < S , A , Pa , R > . The components of the MDP can be defined as : State Space ( S ) : A robot at its decision-making instance uses a state s ∈ S , which contains the following information : 1 ) the current mission time , 2 ) its current location , 3 ) its remaining ferry-range ( battery state ) , 4 ) the planned ( allocated ) tasks of its peers , 5 ) the remaining ferry-range of its peers , and 6 ) the states of tasks . The states of tasks contain the location , the time deadline , and the task status – active , completed , and missed ( i.e. , deadline is passed ) . Here we assume that each robot can broadcast its information to its peers without the need for a centralized system for communication , as aligned with modern communication capabilities Sykora et al . ( 2020 ) . Action Space ( A ) : The set of actions is represented as A , where each action a is defined as the index of the selected task , { 0 , . . . , N } with the index of the depot as 0 . The task 0 ( the depot ) can be selected by multiple robots , but the other tasks are allowed to be chosen once if they are active ( not completed or missed tasks ) . Pa ( s′|s , a ) : A robot by taking action a at state s reaches the next state s′ in a deterministic manner ( i.e. , deterministic transition model is defined ) . Reward ( R ) : The reward function is defined as −fcost , and is calculated when there is no more active tasks ( all tasks has been visited once irrespective of it being completed or missed ) . Transition : The transition is an event-based trigger . An event is defined as the condition that a robot reaches its selected task or visits the depot location .

Summary: This paper proposes a new method, including neural network architecture, for solving time-constrained multi-robot task allocation (MRTA) problems. The proposed approach models the target problem as a Markov Decision Process (MDP) over graphs and use Reinforcement Learning (RL) methods to solve the problem. The proposed learning architecture is called Covariant Attention-based Mechanism (CAM).  The architecture is shown to have better performance than an existing state-of-the-art encoder-decoder method regarding task completion, cost function, and scalability. Though the performance is still lower than non-learning-based baseline methods, i.e., BiG-MRTA, the computational cost is significantly smaller than the baselines.