Learning to Solve Multi-Robot Task Allocation with a Covariant-Attention based Neural Architecture

1 INTRODUCTION . In multi-robot task allocation ( MRTA ) problems , we study how to coordinate tasks among a team of cooperative robotic systems such that the decisions are free of conflict and optimize a quantity of interest ( Gerkey & Matarić , 2004 ) . The potential real-world applications of MRTA are immense , considering that multi-robotics is one of the most important emerging directions of robotics research and development , and task allocation is fundamental to most multi-robotic or swarm-robotic operations . Example applications include disaster response ( Ghassemi & Chowdhury , 2018 ) , last-mile delivery ( Aurambout et al. , 2019 ) , environment monitoring ( Espina et al. , 2011 ) , reconnaissance ( Olson et al. , 2012 ) and combat Behjat et al . ( 2021 ) . Although various approaches ( e.g. , graph-based methods ( Ghassemi et al. , 2019 ) , integer-linear programming ( ILP ) approaches ( Nallusamy et al. , 2009 ; Toth & Vigo , 2014 ) , and auction-based methods ( Dias et al. , 2006 ; Schneider et al. , 2015 ) ) have been proposed to solve the combinatorial optimization problem underlying MRTA operations , they usually do not scale well with number of robots and/or tasks , and do not readily adapt to complex problem characteristics without tedious hand-crafting of the underlying heuristics . In the recent years , a rich body of work has emerged on using learning-based techniques to model solutions or intelligent heuristics for combinatorial optimization ( CO ) problems over graphs . The existing methods are mostly limited to classical CO problems , such as multi-traveling salesman ( mTSP ) , vehicle routing ( VRP ) , and max-cut type of problems . We specifically focus on a class of MRTA problems that falls into the Single-task Robots , and Single-robot Tasks ( SR-ST ) class defined in ( Gerkey & Matarić , 2004 ; Nunes et al. , 2017 ) . Based on iTax taxonomoy as defined in Gerkey & Matarić ( 2004 ) , these problems fall into the In-schedule Dependencies ( ID ) category . Here , a feasible and conflict-free task allocation is defined as assigning any task to only one robot ( Ghassemi et al. , 2019 ) . For solving these problems , we propose a new covariant attention-based model ( aka CAM ) , a neural architecture for learning over graphs to construct the MRTA policies . This architecture builds upon the attention mechanism concept and innovatively integrates an equivariant embedding of the graph to capture graph structure while remaining agnostic to node ordering . We implement CAM on an original MRTA problem , a suite of benchmark MRTA problems and benchmark VRP problems to perform generalizability , scalability and comparative analyses of the new method . We also perform an analysis on the impact of the neighborhood size on the performance on the benchmark MRTA problem . 1.1 MULTI-ROBOT TASK ALLOCATION . In recent years , learning approaches based on Graph Neural Networks or GNN are being increasingly used to solve planning problems with a CO formulation , e.g. , TSP , VRP , Max-Cut , Min-Vertex , and MRTA Kool et al . ( 2019 ) ; Barrett et al . ( 2019 ) ; Khalil et al . ( 2017 ) ; Kaempfer & Wolf ( 2018 ) ; Mittal et al . ( 2019 ) ; Li et al . ( 2018 ) ; Nowak et al . ( 2017 ) ; Wang & Gombolay ( 2020 ) ; Tolstaya et al . ( 2020 ) ; Sykora et al . ( 2020 ) ; Dai et al . ( 2017 ) . Further details on these related learning-based studies can be found in Appendix A . Some of the conventional ILP , MILP , and INLP based methods for MRTA have been discussed in Appendix D.1 . GNNs provide the advantage of being able to capture both Euclidean features ( e.g. , task location ) , as well as non-Euclidean features such as task capacity , task deadline and the local structure of task neighborhoods . The latter serves as higher-level meaningful features that assist in generalized decision-making These existing studies are however limited in three key aspects : 1 ) They address simplified problems that often exclude common real-world factors such as resource and capacity constraints Kool et al . ( 2019 ) ; Kaempfer & Wolf ( 2018 ) ; Khalil et al . ( 2017 ) ; Tolstaya et al . ( 2020 ) ) . 2 ) They are mostly focused on smaller sized problems ( ≤ 100 tasks and 10 robots ) Paleja et al . ( 2020 ) ; Strens & Windelinckx ( 2005 ) ; Wang & Gombolay ( 2020 ) ; Sykora et al . ( 2020 ) , with their scalability remaining unclear . 3 ) They rarely provide evidence of generalizing to problem scenarios that are larger in size than those used for training . This capability would be particularly critical since real-world MRTA problems often involve simulating episodes whose costs scale with the number of tasks and robots , making re-training efforts burdensome . To address these gaps , here we propose a new learning framework that can solve large-sized MRTA problems ( SR-ST ) with commonly considered constraints – involving up to 1000+ tasks and 200+ robots – and generalize across even larger problem scenarios without the need to re-train . For most practical scenarios with larger number of locations , a highly optimal solution is not always desired , while a good feasible solution is the priority as pointed out by Cappart et al . ( 2021 ) . Therefore , to enable scalable policies , we design a novel encoder based on the concept of Covariant Compositional Networks ( CCN ) Hy et al . ( 2018 ) , which is hypothesized to effectively combine local structural information with permutation invariance . The encoder is followed by a decoder based on a Multi-head Attention mechanism Kool et al . ( 2019 ) ; Vaswani et al . ( 2017 ) which fuses the encoded information and problem/mission-specific information ( Context ) using simple matrix multiplication , in order to enable decentralized sequential decision-making . 1.2 CONTRIBUTIONS OF THIS PAPER . The primary contributions of this paper can thus be stated as follows : 1 ) We formulate the general SR-ST class of MRTA problems as a Markov Decision Process or MDP over graphs with the multirobot state information embedded as the context portion of the policy model , such that the ( task allocation ) policy can be learned using an RL approach . 2 ) We design the GNN that acts as the policy network as an encoder-decoder architecture , where the encoder is innovatively based on covariant compositional networks ( CCN ) , whose embedding capabilities significantly improve generalizability and scalability to larger task graphs and multi-robot teams . 3 ) We implement an attention based decoder ( inspired by Kool et al . ( 2019 ) ) to enable sequential decision-making , and specifically extend it to a multi-agent combinatorial optimization setting . The proposed CAM architecture is evaluated on a representative MRTA problem that involves coordinating a team of unmanned aerial vehicles ( UAVs ) to time-efficiently deliver flood relief . The results of this case study demonstrate how CAM clearly outperforms the state-of-the-art attention based method AM ( Kool et al. , 2019 ) , in terms of scalabilty and convergence , thereby emphasizing the effectiveness of the new encoder . Further case studies show that CAM continues to compare favorably to AM over benchmark MRTA problems ( with time and capacity constraints ) and CVRP problems . Comparisons to non-learning baselines for these benchmark problems demonstrate the significant online computation advantages of learnt policies , with the latter being 10-100 times faster . The remainder of the paper is organized as follows : Section 2 defines the MRTA problem and its formulation as an MDP over graphs . Section 3 presents our proposed new GNN architecture . Section 4 describes simulation settings and different case studies . Results are discussed in Section 5 . 2 MRTA : PROBLEM DEFINITION AND FORMULATIONS . The multi-robot task allocation ( MRTA ) problem is defined as the allocation of tasks and resources among several robots that act together without conflict in the same environment to accomplish a common mission . The optimum solution ( decision ) of the MRTA problem is a sequence of tasks for each robot ( conflict-free allocation ) that maximizes the mission outcome ( e.g. , fraction of tasks completed ) or minimize the mission cost ( e.g. , total distance travelled ) subject to the robots ’ range constraints . Here , the following assumptions are made : 1 ) All robots are identical and start/end at the same depot ; 2 ) There are no environmental uncertainties ; 3 ) The location ( xi , yi ) of task-i and its time deadline τi are known to all robots ; 4 ) Each robot can share its state and its world view with other robots ; and 5 ) There is a depot ( Task-0 ) , where each robot starts from and visits if no other tasks are feasible to be undertaken due to the lack of available range . Each tour is defined as departing from the depot , undertaking at least one task , and returning to the depot . 6 ) Motivated by the multi-UAV relief delivery problem , tasks are considered to be instantaneous , which means that reaching the waypoint associated with a task completes that task . This MRTA problem is a class of combinatorial optimization problems , which can be modeled in graph space . In order to learn policies that yield solutions to this CO problem , we express the MRTA problem as a Markov Decision Process ( MDP ) over a graph , described next . The optimization formulation of MRTA is then given in Section 2.2 . 2.1 MDP OVER A GRAPH . The MRTA problem involves a set of nodes/vertices ( V ) and a set of edges ( E ) that connect the vertices to each other , which can be represented as a complete graph G = ( V , E ) . Each node represents a task , and each edge connects a pair of nodes . Let Ω be a weight matrix where the weight of the edge ( ωij ∈ Ω ) represents the cost ( e.g. , distance ) incurred by a robot to take task-j after achieving task-i . For MRTA with N tasks , the number of vertices and the number of edges are N and N ( N − 1 ) /2 , respectively . Node i is assigned a 3-dimensional feature vector denoting the task location and time deadline , i.e. , di = [ xi , yi , τi ] where i ∈ [ 1 , N ] . Here , ωij can be computed as ωij = √ ( xi − xj ) 2 + ( yi − yj ) 2 , where i , j ∈ [ 1 , N ] . The MDP defined in a decentralized manner for each individual robot ( to capture its task selection process ) can be expressed as a tuple < S , A , Pa , R > . The components of the MDP can be defined as : State Space ( S ) : A robot at its decision-making instance uses a state s ∈ S , which contains the following information : 1 ) the current mission time , 2 ) its current location , 3 ) its remaining ferry-range ( battery state ) , 4 ) the planned ( allocated ) tasks of its peers , 5 ) the remaining ferry-range of its peers , and 6 ) the states of tasks . The states of tasks contain the location , the time deadline , and the task status – active , completed , and missed ( i.e. , deadline is passed ) . Here we assume that each robot can broadcast its information to its peers without the need for a centralized system for communication , as aligned with modern communication capabilities Sykora et al . ( 2020 ) . Action Space ( A ) : The set of actions is represented as A , where each action a is defined as the index of the selected task , { 0 , . . . , N } with the index of the depot as 0 . The task 0 ( the depot ) can be selected by multiple robots , but the other tasks are allowed to be chosen once if they are active ( not completed or missed tasks ) . Pa ( s′|s , a ) : A robot by taking action a at state s reaches the next state s′ in a deterministic manner ( i.e. , deterministic transition model is defined ) . Reward ( R ) : The reward function is defined as −fcost , and is calculated when there is no more active tasks ( all tasks has been visited once irrespective of it being completed or missed ) . Transition : The transition is an event-based trigger . An event is defined as the condition that a robot reaches its selected task or visits the depot location .

Summary: This paper presents an interesting idea of using neural-network-based RL to solve a type of vehicle routing problems, where the vehicles are tasked with visiting spatial locations to deliver items, and are subject to load capacity and delivery time constraints. In order to solve this problem, the authors propose an encoder-decoder architecture to decide for each robot, where to move next. The encoder is inspired from the Covariance Compositional Networks and the decoder utilizes an attention module, and the network is trained via REINFORCE. The results show that the proposed method outperforms the baseline in unseen test cases, in terms of task completion rate and the specified cost function.