TarMAC: Targeted Multi-Agent Communication

We explore a collaborative multi-agent reinforcement learning setting where a team of agents attempts to solve cooperative tasks in partially-observable environments . In this scenario , learning an effective communication protocol is key . We propose a communication architecture that allows for targeted communication , where agents learn both what messages to send and who to send them to , solely from downstream task-specific reward without any communication supervision . Additionally , we introduce a multi-stage communication approach where the agents co-ordinate via multiple rounds of communication before taking actions in the environment . We evaluate our approach on a diverse set of cooperative multi-agent tasks , of varying difficulties , with varying number of agents , in a variety of environments ranging from 2D grid layouts of shapes and simulated traffic junctions to complex 3D indoor environments . We demonstrate the benefits of targeted as well as multi-stage communication . Moreover , we show that the targeted communication strategies learned by agents are both interpretable and intuitive . 1 INTRODUCTION . Effective communication is a key ability for collaborative multi-agents systems . Indeed , intelligent agents ( humans or artificial ) in real-world scenarios can significantly benefit from exchanging information that enables them to coordinate , strategize , and utilize their combined sensory experiences to act in the physical world . The ability to communicate has wide-ranging applications for artificial agents – from multi-player gameplay in simulated games ( e.g . DoTA , Quake , StarCraft ) or physical worlds ( e.g . robot soccer ) , to networks of self-driving cars communicating with each other to achieve safe and swift transport , to teams of robots on search-and-rescue missions deployed in hostile and fast-evolving environments . A salient property of human communication is the ability to hold targeted interactions . Rather than the ‘ one-size-fits-all ’ approach of broadcasting messages to all participating agents , as has been previously explored ( Sukhbaatar et al. , 2016 ; Foerster et al. , 2016 ) , it can be useful to direct certain messages to specific recipients . This enables a more flexible collaboration strategy in complex environments . For example , within a team of search-and-rescue robots with a diverse set of roles and goals , a message for a fire-fighter ( “ smoke is coming from the kitchen ” ) is largely meaningless for a bomb-defuser . In this work we develop a collaborative multi-agent deep reinforcement learning approach that supports targeted communication . Crucially , each individual agent actively selects which other agents to send messages to . This targeted communication behavior is operationalized via a simple signaturebased soft attention mechanism : along with the message , the sender broadcasts a key which encodes properties of agents the message is intended for , and is used by receivers to gauge the relevance of the message . This communication mechanism is learned implicitly , without any attention supervision , as a result of end-to-end training using a downstream task-specific team reward . The inductive bias provided by soft attention in the communication architecture is sufficient to enable agents to 1 ) communicate agent-goal-specific messages ( e.g . guide fire-fighter towards fire , bomb-defuser towards bomb , etc . ) , 2 ) be adaptive to variable team sizes ( e.g . the size of the local neighborhood a self-driving car can communicate with changes as it moves ) , and 3 ) be interpretable through predicted attention probabilities that allow for inspection of which agent is communicating what message and to whom . Our results however show that just using targeted communication is not enough . Complex realworld tasks might require large populations of agents to go through multiple stages of collaborative communication and reasoning , involving large amounts of information to be persistent in memory and exchanged via high-bandwidth communication channels . To this end , our actor-critic framework combines centralized training with decentralized execution ( Lowe et al. , 2017 ) , thus enabling scaling to a large number of agents . In this context , our inter-agent communication architecture supports multiple stages of targeted interactions at every time-step , and the agents ’ recurrent policies support persistent relevant information in internal states . While natural language , i.e . a finite set of discrete tokens with pre-specified human-conventionalized meanings , may seem like an intuitive protocol for inter-agent communication – one that enables human-interpretability of interactions – forcing machines to communicate among themselves in discrete tokens presents additional training challenges . Since our work focuses on machine-only multi-agent teams , we allow agents to communicate via continuous vectors ( rather than discrete symbols ) , and via the learning process , agents have the flexibility to discover and optimize their communication protocol as per task requirements . We provide extensive empirical demonstration of the efficacy of our approach across a range of tasks , environments , and team sizes . We begin by benchmarking multi-agent communication with and without attention on a cooperative navigation task derived from the SHAPES environment ( Andreas et al. , 2016 ) . We show that agents learn intuitive attention behavior across a spectrum of task difficulties . Next , we evaluate the same targeted multi-agent communication architecture on the traffic junction environment ( Sukhbaatar et al. , 2016 ) , and show that agents are able to adaptively focus on ‘ active ’ agents in the case of varying team sizes . Finally , we demonstrate effective multi-agent communication in 3D environments on a cooperative first-person point-goal navigation task in the rich House3D environment ( Wu et al. , 2018 ) . 2 RELATED WORK . Multi-agent systems fall at the intersection of game theory , distributed systems , and Artificial Intelligence in general ( Shoham & Leyton-Brown , 2008 ) , and thus have a rich and diverse literature . Our work builds on and is related to prior work in deep multi-agent reinforcement learning , the centralized training and decentralized execution paradigm , and emergent communication protocols . Multi-Agent Reinforcement Learning ( MARL ) . Within MARL ( see Busoniu et al . ( 2008 ) for a survey ) , our work is related to recent efforts on using recurrent neural networks to approximate agent policies ( Hausknecht & Stone , 2015 ) , algorithms stabilizing multi-agent training ( Lowe et al. , 2017 ; Foerster et al. , 2018 ) , and tasks in novel application domains such as coordination and navigation in 3D simulated environments ( Peng et al. , 2017 ; OpenAI , 2018 ; Jaderberg et al. , 2018 ) . Centralized Training & Decentralized Execution . Both Sukhbaatar et al . ( 2016 ) and Hoshen ( 2017 ) adopt a fully centralized framework at both training and test time – a central controller processes local observations from all agents and outputs a probability distribution over joint actions . In this setting , any controller ( e.g . a fully-connected network ) can be viewed as implicitly encoding communication . Sukhbaatar et al . ( 2016 ) present an efficient architecture to learn a centralized controller invariant to agent permutations – by sharing weights and averaging as in Zaheer et al . ( 2017 ) . Meanwhile Hoshen ( 2017 ) proposes to replace averaging by an attentional mechanism to allow targeted interactions between agents . While closely related to our communication architecture , his work only considers fully supervised one-next-step prediction tasks , while we tackle the full reinforcement learning problem with tasks requiring planning over long time horizons . Moreover , a centralized controller quickly becomes intractable in real-world tasks with many agents and high-dimensional observation spaces ( e.g . navigation in House3D ( Wu et al. , 2018 ) ) . To address these weaknesses , we adopt the framework of centralized learning but decentralized execution ( following Foerster et al . ( 2016 ) ; Lowe et al . ( 2017 ) ) and further relax it by allowing agents to communicate . While agents can use extra information during training , at test time , they pick actions solely based on local observations and communication messages received from other agents . Finally , we note that fully decentralized execution at test time without communication is very restrictive . It means 1 ) each agent must act myopically based solely on its local observation and 2 ) agents can not coordinate their actions . In our setting , communication between agents offers a rea- sonable trade-off between allowing agents to globally coordinate while retaining tractability ( since the communicated messages are much lower-dimensional than the observation space ) . Emergent Communication Protocols . Our work is also related to recent work on learning communication protocols in a completely end-to-end manner with reinforcement learning – from perceptual input ( e.g . pixels ) to communication symbols ( discrete or continuous ) to actions ( e.g . navigating in an environment ) . While ( Foerster et al. , 2016 ; Jorge et al. , 2016 ; Das et al. , 2017 ; Kottur et al. , 2017 ; Mordatch & Abbeel , 2017 ; Lazaridou et al. , 2017 ) constrain agents to communicate with discrete symbols with the explicit goal to study emergence of language , our work operates in the paradigm of learning a continuous communication protocol in order to solve a downstream task ( Sukhbaatar et al. , 2016 ; Hoshen , 2017 ; Jiang & Lu , 2018 ) . While ( Jiang & Lu , 2018 ) also operate in a decentralized execution setting and use an attentional communication mechanism , their setup is significantly different from ours as they use attention to decide when to communicate , not who to communicate with ( ‘ who ’ depends on a hand-tuned neighborhood parameter in their work ) . Table 1 summarizes the main axes of comparison between our work and previous efforts in this exciting space . 3 TECHNICAL BACKGROUND . Decentralized Partially Observable Markov Decision Processes ( Dec-POMDPs ) . A DecPOMDP is a cooperative multi-agent extension of a partially observable Markov decision process ( Oliehoek ( 2012 ) ) . For N agents , it is defined by a set of states S describing possible configurations of all agents , a global reward function R , a transition probability function T , and for each agent i P 1 , ... , N a set of allowed actions Ai , a set of possible observations Ωi and an observation functionOi . Operationally , at each time step every agent picks an action ai based on its local observation ωi following its own stochastic policy πθipai|ωiq . The system randomly transitions to the next state s1 given the current state and joint action T ps1|s , a1 , ... , aN q . The agent team receives a global reward r “ Rps , a1 , ... , aN q while each agent receives a local observation of the new state Oipωi|s1q . Agents aim to maximize the total expected return J “ řT t “ 0 γ trt where γ is a discount factor and T is the episode time horizon . Actor-Critic Algorithms . Policy gradient methods directly adjust the parameters θ of the policy in order to maximize the objective Jpθq “ Es „ pπ , a „ πθpsq rRps , aqs by taking steps in the direction of ∇Jpθq . We can write the gradient with respect to the policy parameters as ∇θJpθq “ Es „ pπ , a „ πθpsq r∇θ log πθpa|sqQπps , aqs , where Qπps , aq is called the action-value , it is the expected remaining discounted reward if we take action a in state s and follow policy π thereafter . Actor-Critic algorithms learn an approximation of the unknown true action-value function Q̂ps , aq by e.g . temporal-difference learning ( Sutton & Barto , 1998 ) . This Q̂ps , aq is called the Critic while the policy πθ is called the Actor . Multi-Agent Actor-Critic . Lowe et al . ( 2017 ) propose a multi-agent Actor-Critic algorithm adapted to centralized learning and decentralized execution . Each agent learns its own individual policy πθipai|ωiq conditioned on local observation ωi , using a centralized Critic which estimates the joint action-value Q̂ps , a1 , ... , aN q .

Summary: The authors propose a new architecture for learning communication protocols. In this architecture each message consists of a key and a value. When receiving the message the listener produces an attention key that is used to selectively attend to some messages more than other using soft attention. This differs from the typical 'broadcasting' protocols learned in literature.