Learning when to Communicate at Scale in Multiagent Cooperative and Competitive Tasks

1 INTRODUCTION . Communication is an essential element of intelligence as it helps in learning from others experience , work better in teams and pass down knowledge . In multi-agent settings , communication allows agents to cooperate towards common goals . Particularly in partially observable environments , when the agents are observing different parts of the environment , they can share information and learnings from their observation through communication . Recently , there have been a lot of success in the field of reinforcement learning ( RL ) in playing Atari Games ( Mnih et al. , 2015 ) to playing Go ( Silver et al. , 2016 ) , most of which have been limited to the single agent domain . However , the number of systems and applications having multi-agents have been growing ( Lazaridou et al. , 2017 ; Mordatch & Abbeel , 2018 ) ; where size can be from a team of robots working in manufacturing plants to a network of self-driving cars . Thus , it is crucial to successfully scale RL to multi-agent environments in order to build intelligent systems capable of higher productivity . Furthermore , scenarios other than cooperative , namely semi-cooperative ( or mixed ) and competitive scenarios have not even been studied as extensively for multi-agent systems . The mixed scenarios can be compared to most of the real life scenarios as humans are cooperative but not fully-cooperative in nature . Humans work towards their individual goals while cooperating with each other . In competitive scenarios , agents are essentially competing with each other for better rewards . In real life , humans always have an option to communicate but can choose when to actually communicate . For example , in a sports match two teams which can communicate , can choose to not communicate at all ( to prevent sharing strategies ) or use dishonest signaling ( to misdirect opponents ) ( Lehman et al. , 2018 ) in order to optimize their own reward and handicap opponents ; making it important to learn when to communicate . ∗Equal contribution . †Current affiliation . This work was completed when authors were at New York University . Teaching agents how to communicate makes it is unnecessary to hand code the communication protocol with expert knowledge ( Sukhbaatar et al. , 2016 ) ( Kottur et al. , 2017 ) . While the content of communication is important , it is also important to know when to communicate either to increase scalability and performance or to increase competitive edge . For example , a prey needs to learn when to communicate to avoid communicating its location with predators . Sukhbaatar et al . ( 2016 ) showed that agents communicating through a continuous vector are easier to train and have a higher information throughput than communication based on discrete symbols . Their continuous communication is differentiable , so it can be trained efficiently with back-propagation . However , their model assumes full-cooperation between agents and uses average global rewards . This restricts the model from being used in mixed or competitive scenarios as full-cooperation involves sharing hidden states to everyone ; exposing everything and leading to poor performance by all agents as shown by our results . Furthermore , the average global reward for all agents makes the credit assignment problem even harder and difficult to scale as agents don ’ t know their individual contributions in mixed or competitive scenarios where they want themselves to succeed before others . To solve above mentioned issues , we make the following contributions : 1 . We propose Individualized Controlled Continuous Communication Model ( IC3Net ) , in which each agent is trained with its individualized reward and can be applied to any scenario whether cooperative or not . 2 . We empirically show that based on the given scenario–using the gating mechanism–our model can learn when to communicate . The gating mechanism allows agents to block their communication ; which is useful in competitive scenarios . 3 . We conduct experiments on different scales in three chosen environments including StarCraft and show that IC3Net outperforms the baselines with performance gaps that increase with scale . The results show that individual rewards converge faster and better than global rewards . 2 RELATED WORK . The simplest approach in multi-agent reinforcement learning ( MARL ) settings is to use an independent controller for each agent . This was attempted with Q-learning in Tan ( 1993 ) . However , in practice it performs poorly ( Matignon et al. , 2012 ) , which we also show in comparison with our model . The major issue with this approach is that due to multiple agents , the stationarity of the environment is lost and naïve application of experience replay doesn ’ t work well . The nature of interaction between agents can either be cooperative , competitive , or a mix of both . Most algorithms are designed only for a particular nature of interaction , mainly cooperative settings ( Omidshafiei et al. , 2017 ; Lauer & Riedmiller , 2000 ; Matignon et al. , 2007 ) , with strategies which indirectly arrive at cooperation via sharing policy parameters ( Gupta et al. , 2017 ) . These algorithms are generally not applicable in competitive or mixed settings . See Busoniu et al . ( 2008 ) for survey of MARL in general and Panait & Luke ( 2005 ) for survey of cooperative multi-agent learning . Our work can be considered as an all-scenario extension of Sukhbaatar et al . ( 2016 ) ’ s CommNet for collaboration among multiple agents using continuous communication ; usable only in cooperative settings as stated in their work and shown by our experiments . Due to continuous communication , the controller can be learned via backpropagation . However , this model is restricted to fully cooperative tasks as hidden states are fully communicated to others which exposes everything about agent . On the other hand , due to global reward for all agents , CommNet also suffers from credit assignment issue . The Multi-Agent Deep Deterministic Policy Gradient ( MADDPG ) model presented by Lowe et al . ( 2017 ) also tries to achieve similar goals . However , they differ in the way of providing the coordination signal . In their case , there is no direct communication among agents ( actors with different policy per agent ) , instead a different centralized critic per agent – which can access the actions of all the agents – provides the signal . Concurrently , a similar model using centralized critic and decentralized actors with additional counterfactual reward , COMA by Foerster et al . ( 2018 ) was proposed to tackle the challenge of multiagent credit assignment by letting agents know their individual contributions . Vertex Attention Interaction Networks ( VAIN ) ( Hoshen , 2017 ) also models multi-agent communication through the use of Interaction Networks ( Battaglia et al. , 2016 ) with attention mechanism ( Bahdanau et al. , 2015 ) for predictive modelling using supervised settings . The work by Foerster et al . ( 2016b ) also learns a communication protocol where agents communicate in a discrete manner through their actions . This contrasts with our model where multiple continuous communication cycles can be used at each time step to decide the actions of all agents . Furthermore , our approach is amenable to dynamic number of agents . Peng et al . ( 2017 ) also attempts to solve micromanagement tasks in StarCraft using communication . However , they have non-symmetric addition of agents in communication channel and are restricted to only cooperative scenarios . In contrast , a lot of work has focused on understanding agents ’ communication content ; mostly in discrete settings with two agents ( Wang et al. , 2016 ; Havrylov & Titov , 2017 ; Kottur et al. , 2017 ; Lazaridou et al. , 2017 ; Lee et al. , 2018 ) . Lazaridou et al . ( 2017 ) showed that given two neural network agents and a referential game , the agents learn to coordinate . Havrylov & Titov ( 2017 ) extended this by grounding communication protocol to a symbols ’ s sequence while Kottur et al . ( 2017 ) showed that this language can be made more human-like by placing certain restrictions . Lee et al . ( 2018 ) demonstrated that agents speaking different languages can learn to translate in referential games . 3 MODEL . In this section , we introduce our model Individualized Controlled Continuous Communication Model ( IC3Net ) as shown in Figure 1 to work in multi-agent cooperative , competitive and mixed settings where agents learn what to communicate as well as when to communicate . First , let us describe an independent controller model where each agent is controlled by an individual LSTM . For the j-th agent , its policy takes the form of : ht+1j , s t+1 j = LSTM ( e ( o t j ) , h t j , s t j ) atj = π ( h t j ) , where otj is the observation of the j-th agent at time t , e ( · ) is an encoder function parameterized by a fully-connected neural network and π is an agent ’ s action policy . Also , htj and s t j are the hidden and cell states of the LSTM . We use the same LSTM model for all agents , sharing their parameters . This way , the model is invariant to permutations of the agents . IC3Net extends this independent controller model by allowing agents to communicate their internal state , gated by a discrete action . The policy of the j-th agent in a IC3Net is given by gt+1j = f g ( htj ) ht+1j , s t+1 j = LSTM ( e ( o t j ) + c t j , h t j , s t j ) ct+1j = 1 J − 1 C ∑ j′ 6=j ht+1j′ g t+1 j′ atj = π ( h t j ) , where ctj is the communication vector for the j-th agent , C is a linear transformation matrix for transforming gated average hidden state to a communication tensor , J is the number of alive agents currently present in the system and fg ( . ) is a simple network containing a soft-max layer for 2 actions ( communicate or not ) on top of a linear layer with non-linearity . The binary action gtj specifies whether agent j wants to communicate with others , and act as a gating function when calculating the communication vector . Note that the gating action for next time-step is calculated at current time-step . We train both the action policy π and the gating function fg with REINFORCE ( Williams , 1992 ) . In Sukhbaatar et al . ( 2016 ) , individual networks controlling agents were interconnected , and they as a whole were considered as a single big neural network . This single big network controller approach required a definition of an unified loss function during training , thus making it impossible to train agents with different rewards . In this work , however , we move away from the single big network controller approach . Instead , we consider multiple big networks with shared parameters each controlling a single agent separately . Each big network consists of multiple LSTM networks , each processing an observation of a single agent . However , only one of the LSTMs need to output an action because the big network is only controlling a single agent . Although this view has a little effect on the implementation ( we can still use a single big network in practice ) , it allows us to train each agent to maximize its individual reward instead of a single global reward . This has two benefits : ( i ) it allows the model to be applied to both cooperative and competitive scenarios , ( ii ) it also helps resolve the credit assignment issue faced by many multi-agent ( Sukhbaatar et al. , 2016 ; Foerster et al. , 2016a ) algorithms while improving performance with scalability and is coherent with the findings in Chang et al . ( 2003 ) .

Summary: The authors propose a new network architecture for multi-agent reinforcement learning. The new architecture addresses three issues: (1) the applicability of existing algorithms to semi-cooperative or competitive settings; (2) the ability to use local rewards during agent training; (3) the credit assignment problem with global multi-agent rewards. The authors address these issues with a new architecture that is comprised of several LSTM controllers with tied weights that transmit a continuous vector to each other, and that are augment with a gating mechanism that allows them to abstain from communicating.