Learning to Ground Multi-Agent Communication with Autoencoders

1 Introduction . An essential aspect of communication is that each pair of speaker and listener must share a common understanding of the symbols being used [ 8 ] . For artificial agents interacting in an environment , with a communication channel but without an agreed-upon communication protocol , this raises the question : how can meaningful communication emerge before a common language has been established ? To address this challenge , prior works have used supervised learning [ 18 ] , centralized learning [ 14 , 17 , 28 ] , or differentiable communication [ 6 , 17 , 28 , 32 , 41 ] . Yet , none of these mechanisms is representative of how communication emerges in nature , where animals and humans have evolved communication protocols without supervision and without a centralized coordinator [ 35 ] . The communication model that most closely resembles language learning in nature is a fully decentralized model , where agents ’ policies are independently optimized . However , decentralized models perform poorly even in simple communication tasks [ 17 ] or with additional inductive biases [ 10 ] . We tackle this challenge by first making the following observations on why emergent communication is difficult in a decentralized multi-agent reinforcement learning setting . A key problem that prevents agents from learning meaningful communication is the lack of a common grounding in communication symbols [ 3 , 10 , 19 ] . In nature , the emergence of a common language is thought to be aided by physical biases and embodiment [ 29 , 42 ] – we can only produce certain vocalizations , these sounds only can be heard a certain distance away , these sounds bear similarity to natural sounds in the environment , etc . – yet artificial communication protocols are not a priori grounded in aspects of the environment Project page , code , and videos can be found at https : //toruowo.github.io/marl-ae-comm/ . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . dynamics . This poses a severe exploration problem as the chances of a consistent protocol being found and rewarded is extremely small [ 17 ] . Moreover , before a communication protocol is found , the random utterances transmitted between agents add to the already high variance of multi-agent reinforcement learning , making the learning problem even more challenging [ 10 , 28 ] . To overcome the grounding problem , an important question to ask is : do agents really need to learn language grounding from scratch through random exploration in an environment where success is determined by chance ? Perhaps nature has a different answer ; previous studies in cognitive science and evolutionary linguistics [ 20 , 36 , 39 , 40 ] have provided evidence for the hypothesis that communication first started from sounds whose meaning are grounded in the physical environment , then creatures adapted to make sense of those sounds and make use of them . Inspired by language learning in natural species , we propose a novel framework for grounding multi-agent communication : first ground speaking through learned representations of the world , then learn listening to interpret these grounded utterances . Surprisingly , even with the simple representation learning task of autoencoding , our approach eases the learning of communication in fully decentralized multi-agent settings and greatly improves agents ’ performance in multi-agent coordination tasks that are nearly unsolvable without communication . The contribution of our work can be summarized as follows : • We formulate communication grounding as a representation learning problem and propose to use observation autoencoding to learn a common grounding across all agents . • We experimentally validate that this is an effective approach for learning decentralized communication in MARL settings : a communication model trained with a simple autoencoder can consistently outperform baselines across various MARL environments . • In turn , our work highlights the need to rethink the problem of emergent communication , where we demonstrate the essential need for visual grounding . 2 Related Work . In multi-agent reinforcement learning ( MARL ) , achieving successful emergent communication with decentralized training and non-differentiable communication channel is an important yet challenging task that has not been satisfactorily addressed by existing works . Due to the non-stationary and nonMarkovian transition dynamics in multi-agent settings , straightforward implementation of standard reinforcement learning methods such as Actor-Critic [ 22 ] and DQN [ 31 ] perform poorly [ 17 , 28 ] . Centralized learning is often used to alleviate the problem of high variance in MARL , for example learning a centralized value function that has access to the joint observation of all agents [ 9 , 14 , 28 ] . However , it turns out that MARL models are unable to solve tasks that rely on emergent communication , even with centralized learning and shared policy parameters across the agents [ 17 ] . Eccles et al . [ 10 ] provides an analysis that illustrates how MARL with communication poses a more difficult exploration problem than standard MARL , which is confirmed by empirical results in [ 17 , 28 ] : communication exacerbates the sparse and high variance reward signal in MARL . Many works therefore resort to differentiable communication [ 6 , 17 , 28 , 32 , 41 ] , where agents are allowed to directly optimize each other ’ s communication policies through gradients . Among them , Choi el al . [ 6 ] explore a high-level idea that is similar to ours : generating messages that the model itself can interpret . However , these approaches impose a strong constraint on the nature of communication , which limits their applicability to many real-world multi-agent coordination tasks . Jaques et al . [ 21 ] proposes a method that allows independently trained RL agents to communicate and coordinate . However , the proposed method requires that an agent either has access to policies of other agents or stays in close proximity to other agents . These constraints make it difficult for the same method to be applied to a wider range of tasks , such as those in which agents are not embodied or do not observe others directly . Eccles et al . [ 10 ] attempts to solve the same issue by introducing inductive biases for positive signaling and positive listening , but implementation requires numerous task-specific hyperparameter tuning , and the effectiveness is limited . It is also worth noting that , while a large number of existing works on multi-agent communication take structured state information as input [ 4 , 15 , 16 , 17 , 32 , 34 ] , we train agents to learn a communication protocol directly from raw pixel observations . This presents additional challenges due to the unstructured and ungrounded nature of pixel data , as shown in [ 3 , 6 , 24 ] . To our knowledge , this work is the first to effectively use representation learning to aid communication learning from pixel inputs in a wide range of MARL task settings . 3 Preliminaries . We model multi-agent reinforcement learning ( MARL ) with communication as a partiallyobservable general-sum Markov game [ 26 , 38 ] , where each agent can broadcast information to a shared communication channel . Each agent receives a partial observation of the underlying world state at every time step , including all information communicated in the shared channel . This observation is used to learn an appropriate policy that maximizes the agent ’ s environment reward . In this work , we parameterize the policy function using a deep neural network . Formally , a decentralized MARL can be expressed as a partially observable Markov decision process as M = ⟨S , A , C , O , T , R , γ⟩ , where N is the number of agents , S is a set of states spaces , A = { A1 , ... , AN } , C = { C1 , ... , CN } , and O = { O1 , ... , ON } are a set of action , of communication , and of observation spaces respectively . At time step t , an agent k observes a partial view o ( k ) t of the underlying true state st , and a set of communicated messages from the previous time step ct−1 = { c ( 1 ) t−1 , ... , c ( N ) t−1 } . The agent then chooses an action a ( k ) t ∈ Ak and a subsequent message to broadcast c ( k ) t ∈ Ck . Given the joint actions of all N agents at = { a ( 1 ) t , ... , a ( N ) t } ∈ ( A1 , ... , AN ) , the transition function T : S×A1× ... ×AN → ∆ ( S ) maps the current state st and set of agent actions at to a distribution over the next state st+1 . Since the transition function T is non-determinstic , we denote the probability distributions over S as ∆ ( S ) . Finally , each agent receives an individual reward r ( k ) t ∈ R ( st , at ) where R : S×A1× ... ×AN → R. In our work , we consider a fully cooperative setting in which the objective of each agent is to maximize the total expected return of all agents : maximize π : S→A×C E [ ∑ t∈T ∑ k∈N γtR ( st , at ) ∣∣∣ ( at , ct ) ∼ π ( k ) , st ∼ T ( st−1 ) ] ( 1 ) for some finite time horizon T and discount factor γ . In MARL , the aforementioned objective function is optimized using policy gradient . Specifically , we use asynchronous advantage actor-critic ( A3C ) [ 30 ] with Generalized Advantage Estimation [ 37 ] to optimize our policy network . The policy network in A3C outputs a distribution over the actions and the discounted future returns . Any other policy optimization algorithms could have been used in lieu of A3C . We do not use centralized training or self-play , and only consider decentralized training where each agent is parameterized by an independent policy . 4 Grounding Representation for Communication with Autoencoders . The main challenge of learning to communicate in fully decentralized MARL settings is that there is no grounded information to which agents can associate their symbolic utterances . This lack of grounding creates a dissonance across agents and poses a difficult exploration problem . Ultimately , the gradient signals received by agents are therefore largely inconsistent . As the time horizon , communication space , and the number of agents grow , this grounding problem becomes even more pronounced . This difficulty is highlighted in numerous prior works , with empirical results showing that agents often fail to use the communication channel at all during decentralized learning [ 3 , 10 , 19 , 28 ] . We propose a simple and surprisingly effective approach to mitigate this issue : using a self-supervised representation learning task to learn a common grounding [ 43 ] across all agents . Having such a grounding enables speakers to communicate messages that are understandable to the listeners and convey meaningful information about entities in the environment , though agents need not use the same symbols to mean the same things . Specifically , we train each agent to independently learn to auto-encode its own observation and use the learned representation for communication . This approach offers the benefits of allowing fully decentralized training without needing additional architectural image encoder message encoder ( st ) Policy Gradient Communication Grounding agent 1 a ( 2 ) tc ( 2 ) t c ( n ) t a ( n ) ta ( 1 ) tc ( 1 ) t agent 2 agent n c ( n ) t 1c ( 1 ) t 1 c ( 2 ) t 1 Communicate s ( 1 ) t s ( 2 ) t s ( n ) t s ( n ) t c ( 1 ) t 1 . . . c ( n ) t 1 ( ct 1 ) a ( n ) tc ( n ) t ( n ) ( 2 ) ( 1 ) Figure 1 : Overview : The overall schematic of our multi-agent system . All agents share the same individual model architecture , but each agent is independently trained to learn to auto-encode its own observation and use the learned representation for communication . At each time step , each agent observes an image representation of the environment as well as messages broadcasted by other agents during the last time step . The image pixels are processed through an Image Encoder ; the broadcasted messages are processed through a Message Encoder ; the image features and the message features are concatenated and passed through a Policy Network to predict the next action . The image features are also used to generate the next communication messages using the Communication Autoencoder . bias or supervision signal . In Section 5 , we show the effectiveness of our approach on a variety of MARL communication tasks .

Summary: The goal of the paper is to show that learning a representation to ground language can improve communication in multi-agent settings.  Their method: each agent encodes an image and the other agents messages (through a small conn-net and mlp, respectively). A GRU-based policy outputs an action. A communication autoencoder predicts a message from the image feature and is trained with an auxiliary construction objective (i.e., to reconstruct the image).  They evaluate their model on a referential game with CIFAR images and two grid environments: one where the agents must reach a goal collaboratively and another where one agent has to open a door for another to pass.  They show that their method with an auto encoder strongly outperforms the non-autoencoder variants on the CIFAR task and door task.