Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks

1 INTRODUCTION . Communication is an essential element of intelligence as it helps in learning from others experience , work better in teams and pass down knowledge . In multi-agent settings , communication allows agents to cooperate towards common goals . Particularly in partially observable environments , when the agents are observing different parts of the environment , they can share information and learnings from their observation through communication . Recently , there have been a lot of success in the field of reinforcement learning ( RL ) in playing Atari Games ( Mnih et al. , 2015 ) to playing Go ( Silver et al. , 2016 ) , most of which have been limited to the single agent domain . However , the number of systems and applications having multi-agents have been growing ( Lazaridou et al. , 2017 ; Mordatch & Abbeel , 2018 ) ; where size can be from a team of robots working in manufacturing plants to a network of self-driving cars . Thus , it is crucial to successfully scale RL to multi-agent environments in order to build intelligent systems capable of higher productivity . Furthermore , scenarios other than cooperative , namely semi-cooperative ( or mixed ) and competitive scenarios have not even been studied as extensively for multi-agent systems . The mixed scenarios can be compared to most of the real life scenarios as humans are cooperative but not fully-cooperative in nature . Humans work towards their individual goals while cooperating with each other . In competitive scenarios , agents are essentially competing with each other for better rewards . In real life , humans always have an option to communicate but can choose when to actually communicate . For example , in a sports match two teams which can communicate , can choose to not communicate at all ( to prevent sharing strategies ) or use dishonest signaling ( to misdirect opponents ) ( Lehman et al. , 2018 ) in order to optimize their own reward and handicap opponents ; making it important to learn when to communicate . ∗Equal contribution . †Current affiliation . This work was completed when authors were at New York University . Teaching agents how to communicate makes it is unnecessary to hand code the communication protocol with expert knowledge ( Sukhbaatar et al. , 2016 ) ( Kottur et al. , 2017 ) . While the content of communication is important , it is also important to know when to communicate either to increase scalability and performance or to increase competitive edge . For example , a prey needs to learn when to communicate to avoid communicating its location with predators . Sukhbaatar et al . ( 2016 ) showed that agents communicating through a continuous vector are easier to train and have a higher information throughput than communication based on discrete symbols . Their continuous communication is differentiable , so it can be trained efficiently with back-propagation . However , their model assumes full-cooperation between agents and uses average global rewards . This restricts the model from being used in mixed or competitive scenarios as full-cooperation involves sharing hidden states to everyone ; exposing everything and leading to poor performance by all agents as shown by our results . Furthermore , the average global reward for all agents makes the credit assignment problem even harder and difficult to scale as agents don ’ t know their individual contributions in mixed or competitive scenarios where they want themselves to succeed before others . To solve above mentioned issues , we make the following contributions : 1 . We propose Individualized Controlled Continuous Communication Model ( IC3Net ) , in which each agent is trained with its individualized reward and can be applied to any scenario whether cooperative or not . 2 . We empirically show that based on the given scenario–using the gating mechanism–our model can learn when to communicate . The gating mechanism allows agents to block their communication ; which is useful in competitive scenarios . 3 . We conduct experiments on different scales in three chosen environments including StarCraft and show that IC3Net outperforms the baselines with performance gaps that increase with scale . The results show that individual rewards converge faster and better than global rewards . 2 RELATED WORK . The simplest approach in multi-agent reinforcement learning ( MARL ) settings is to use an independent controller for each agent . This was attempted with Q-learning in Tan ( 1993 ) . However , in practice it performs poorly ( Matignon et al. , 2012 ) , which we also show in comparison with our model . The major issue with this approach is that due to multiple agents , the stationarity of the environment is lost and naïve application of experience replay doesn ’ t work well . The nature of interaction between agents can either be cooperative , competitive , or a mix of both . Most algorithms are designed only for a particular nature of interaction , mainly cooperative settings ( Omidshafiei et al. , 2017 ; Lauer & Riedmiller , 2000 ; Matignon et al. , 2007 ) , with strategies which indirectly arrive at cooperation via sharing policy parameters ( Gupta et al. , 2017 ) . These algorithms are generally not applicable in competitive or mixed settings . See Busoniu et al . ( 2008 ) for survey of MARL in general and Panait & Luke ( 2005 ) for survey of cooperative multi-agent learning . Our work can be considered as an all-scenario extension of Sukhbaatar et al . ( 2016 ) ’ s CommNet for collaboration among multiple agents using continuous communication ; usable only in cooperative settings as stated in their work and shown by our experiments . Due to continuous communication , the controller can be learned via backpropagation . However , this model is restricted to fully cooperative tasks as hidden states are fully communicated to others which exposes everything about agent . On the other hand , due to global reward for all agents , CommNet also suffers from credit assignment issue . The Multi-Agent Deep Deterministic Policy Gradient ( MADDPG ) model presented by Lowe et al . ( 2017 ) also tries to achieve similar goals . However , they differ in the way of providing the coordination signal . In their case , there is no direct communication among agents ( actors with different policy per agent ) , instead a different centralized critic per agent – which can access the actions of all the agents – provides the signal . Concurrently , a similar model using centralized critic and decentralized actors with additional counterfactual reward , COMA by Foerster et al . ( 2018 ) was proposed to tackle the challenge of multiagent credit assignment by letting agents know their individual contributions . Vertex Attention Interaction Networks ( VAIN ) ( Hoshen , 2017 ) also models multi-agent communication through the use of Interaction Networks ( Battaglia et al. , 2016 ) with attention mechanism ( Bahdanau et al. , 2015 ) for predictive modelling using supervised settings . The work by Foerster et al . ( 2016b ) also learns a communication protocol where agents communicate in a discrete manner through their actions . This contrasts with our model where multiple continuous communication cycles can be used at each time step to decide the actions of all agents . Furthermore , our approach is amenable to dynamic number of agents . Peng et al . ( 2017 ) also attempts to solve micromanagement tasks in StarCraft using communication . However , they have non-symmetric addition of agents in communication channel and are restricted to only cooperative scenarios . In contrast , a lot of work has focused on understanding agents ’ communication content ; mostly in discrete settings with two agents ( Wang et al. , 2016 ; Havrylov & Titov , 2017 ; Kottur et al. , 2017 ; Lazaridou et al. , 2017 ; Lee et al. , 2018 ) . Lazaridou et al . ( 2017 ) showed that given two neural network agents and a referential game , the agents learn to coordinate . Havrylov & Titov ( 2017 ) extended this by grounding communication protocol to a symbols ’ s sequence while Kottur et al . ( 2017 ) showed that this language can be made more human-like by placing certain restrictions . Lee et al . ( 2018 ) demonstrated that agents speaking different languages can learn to translate in referential games . 3 MODEL . In this section , we introduce our model Individualized Controlled Continuous Communication Model ( IC3Net ) as shown in Figure 1 to work in multi-agent cooperative , competitive and mixed settings where agents learn what to communicate as well as when to communicate . First , let us describe an independent controller model where each agent is controlled by an individual LSTM . For the j-th agent , its policy takes the form of : ht+1j , s t+1 j = LSTM ( e ( o t j ) , h t j , s t j ) atj = π ( h t j ) , where otj is the observation of the j-th agent at time t , e ( · ) is an encoder function parameterized by a fully-connected neural network and π is an agent ’ s action policy . Also , htj and s t j are the hidden and cell states of the LSTM . We use the same LSTM model for all agents , sharing their parameters . This way , the model is invariant to permutations of the agents . IC3Net extends this independent controller model by allowing agents to communicate their internal state , gated by a discrete action . The policy of the j-th agent in a IC3Net is given by gt+1j = f g ( htj ) ht+1j , s t+1 j = LSTM ( e ( o t j ) + c t j , h t j , s t j ) ct+1j = 1 J − 1 C ∑ j′ 6=j ht+1j′ g t+1 j′ atj = π ( h t j ) , where ctj is the communication vector for the j-th agent , C is a linear transformation matrix for transforming gated average hidden state to a communication tensor , J is the number of alive agents currently present in the system and fg ( . ) is a simple network containing a soft-max layer for 2 actions ( communicate or not ) on top of a linear layer with non-linearity . The binary action gtj specifies whether agent j wants to communicate with others , and act as a gating function when calculating the communication vector . Note that the gating action for next time-step is calculated at current time-step . We train both the action policy π and the gating function fg with REINFORCE ( Williams , 1992 ) . In Sukhbaatar et al . ( 2016 ) , individual networks controlling agents were interconnected , and they as a whole were considered as a single big neural network . This single big network controller approach required a definition of an unified loss function during training , thus making it impossible to train agents with different rewards . In this work , however , we move away from the single big network controller approach . Instead , we consider multiple big networks with shared parameters each controlling a single agent separately . Each big network consists of multiple LSTM networks , each processing an observation of a single agent . However , only one of the LSTMs need to output an action because the big network is only controlling a single agent . Although this view has a little effect on the implementation ( we can still use a single big network in practice ) , it allows us to train each agent to maximize its individual reward instead of a single global reward . This has two benefits : ( i ) it allows the model to be applied to both cooperative and competitive scenarios , ( ii ) it also helps resolve the credit assignment issue faced by many multi-agent ( Sukhbaatar et al. , 2016 ; Foerster et al. , 2016a ) algorithms while improving performance with scalability and is coherent with the findings in Chang et al . ( 2003 ) .

Summary: From a methodological perspective, this paper describes a simple bu clever learning architecture with individual agents able to decide when to communicate through a learned gating mechanism. Each agent is an LSTM able to decide at each time point which aspects of its internal state should be exposed to other agents through this gating mechanism. The presentation of this method is clear to a level that should allows the reader to implement this him/herself. It would be great if the code associated to this could be released but the presentation allows for reproducibility.