Learning to Ground Multi-Agent Communication with Autoencoders

1 Introduction . An essential aspect of communication is that each pair of speaker and listener must share a common understanding of the symbols being used [ 8 ] . For artificial agents interacting in an environment , with a communication channel but without an agreed-upon communication protocol , this raises the question : how can meaningful communication emerge before a common language has been established ? To address this challenge , prior works have used supervised learning [ 18 ] , centralized learning [ 14 , 17 , 28 ] , or differentiable communication [ 6 , 17 , 28 , 32 , 41 ] . Yet , none of these mechanisms is representative of how communication emerges in nature , where animals and humans have evolved communication protocols without supervision and without a centralized coordinator [ 35 ] . The communication model that most closely resembles language learning in nature is a fully decentralized model , where agents ’ policies are independently optimized . However , decentralized models perform poorly even in simple communication tasks [ 17 ] or with additional inductive biases [ 10 ] . We tackle this challenge by first making the following observations on why emergent communication is difficult in a decentralized multi-agent reinforcement learning setting . A key problem that prevents agents from learning meaningful communication is the lack of a common grounding in communication symbols [ 3 , 10 , 19 ] . In nature , the emergence of a common language is thought to be aided by physical biases and embodiment [ 29 , 42 ] – we can only produce certain vocalizations , these sounds only can be heard a certain distance away , these sounds bear similarity to natural sounds in the environment , etc . – yet artificial communication protocols are not a priori grounded in aspects of the environment Project page , code , and videos can be found at https : //toruowo.github.io/marl-ae-comm/ . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . dynamics . This poses a severe exploration problem as the chances of a consistent protocol being found and rewarded is extremely small [ 17 ] . Moreover , before a communication protocol is found , the random utterances transmitted between agents add to the already high variance of multi-agent reinforcement learning , making the learning problem even more challenging [ 10 , 28 ] . To overcome the grounding problem , an important question to ask is : do agents really need to learn language grounding from scratch through random exploration in an environment where success is determined by chance ? Perhaps nature has a different answer ; previous studies in cognitive science and evolutionary linguistics [ 20 , 36 , 39 , 40 ] have provided evidence for the hypothesis that communication first started from sounds whose meaning are grounded in the physical environment , then creatures adapted to make sense of those sounds and make use of them . Inspired by language learning in natural species , we propose a novel framework for grounding multi-agent communication : first ground speaking through learned representations of the world , then learn listening to interpret these grounded utterances . Surprisingly , even with the simple representation learning task of autoencoding , our approach eases the learning of communication in fully decentralized multi-agent settings and greatly improves agents ’ performance in multi-agent coordination tasks that are nearly unsolvable without communication . The contribution of our work can be summarized as follows : • We formulate communication grounding as a representation learning problem and propose to use observation autoencoding to learn a common grounding across all agents . • We experimentally validate that this is an effective approach for learning decentralized communication in MARL settings : a communication model trained with a simple autoencoder can consistently outperform baselines across various MARL environments . • In turn , our work highlights the need to rethink the problem of emergent communication , where we demonstrate the essential need for visual grounding . 2 Related Work . In multi-agent reinforcement learning ( MARL ) , achieving successful emergent communication with decentralized training and non-differentiable communication channel is an important yet challenging task that has not been satisfactorily addressed by existing works . Due to the non-stationary and nonMarkovian transition dynamics in multi-agent settings , straightforward implementation of standard reinforcement learning methods such as Actor-Critic [ 22 ] and DQN [ 31 ] perform poorly [ 17 , 28 ] . Centralized learning is often used to alleviate the problem of high variance in MARL , for example learning a centralized value function that has access to the joint observation of all agents [ 9 , 14 , 28 ] . However , it turns out that MARL models are unable to solve tasks that rely on emergent communication , even with centralized learning and shared policy parameters across the agents [ 17 ] . Eccles et al . [ 10 ] provides an analysis that illustrates how MARL with communication poses a more difficult exploration problem than standard MARL , which is confirmed by empirical results in [ 17 , 28 ] : communication exacerbates the sparse and high variance reward signal in MARL . Many works therefore resort to differentiable communication [ 6 , 17 , 28 , 32 , 41 ] , where agents are allowed to directly optimize each other ’ s communication policies through gradients . Among them , Choi el al . [ 6 ] explore a high-level idea that is similar to ours : generating messages that the model itself can interpret . However , these approaches impose a strong constraint on the nature of communication , which limits their applicability to many real-world multi-agent coordination tasks . Jaques et al . [ 21 ] proposes a method that allows independently trained RL agents to communicate and coordinate . However , the proposed method requires that an agent either has access to policies of other agents or stays in close proximity to other agents . These constraints make it difficult for the same method to be applied to a wider range of tasks , such as those in which agents are not embodied or do not observe others directly . Eccles et al . [ 10 ] attempts to solve the same issue by introducing inductive biases for positive signaling and positive listening , but implementation requires numerous task-specific hyperparameter tuning , and the effectiveness is limited . It is also worth noting that , while a large number of existing works on multi-agent communication take structured state information as input [ 4 , 15 , 16 , 17 , 32 , 34 ] , we train agents to learn a communication protocol directly from raw pixel observations . This presents additional challenges due to the unstructured and ungrounded nature of pixel data , as shown in [ 3 , 6 , 24 ] . To our knowledge , this work is the first to effectively use representation learning to aid communication learning from pixel inputs in a wide range of MARL task settings . 3 Preliminaries . We model multi-agent reinforcement learning ( MARL ) with communication as a partiallyobservable general-sum Markov game [ 26 , 38 ] , where each agent can broadcast information to a shared communication channel . Each agent receives a partial observation of the underlying world state at every time step , including all information communicated in the shared channel . This observation is used to learn an appropriate policy that maximizes the agent ’ s environment reward . In this work , we parameterize the policy function using a deep neural network . Formally , a decentralized MARL can be expressed as a partially observable Markov decision process as M = ⟨S , A , C , O , T , R , γ⟩ , where N is the number of agents , S is a set of states spaces , A = { A1 , ... , AN } , C = { C1 , ... , CN } , and O = { O1 , ... , ON } are a set of action , of communication , and of observation spaces respectively . At time step t , an agent k observes a partial view o ( k ) t of the underlying true state st , and a set of communicated messages from the previous time step ct−1 = { c ( 1 ) t−1 , ... , c ( N ) t−1 } . The agent then chooses an action a ( k ) t ∈ Ak and a subsequent message to broadcast c ( k ) t ∈ Ck . Given the joint actions of all N agents at = { a ( 1 ) t , ... , a ( N ) t } ∈ ( A1 , ... , AN ) , the transition function T : S×A1× ... ×AN → ∆ ( S ) maps the current state st and set of agent actions at to a distribution over the next state st+1 . Since the transition function T is non-determinstic , we denote the probability distributions over S as ∆ ( S ) . Finally , each agent receives an individual reward r ( k ) t ∈ R ( st , at ) where R : S×A1× ... ×AN → R. In our work , we consider a fully cooperative setting in which the objective of each agent is to maximize the total expected return of all agents : maximize π : S→A×C E [ ∑ t∈T ∑ k∈N γtR ( st , at ) ∣∣∣ ( at , ct ) ∼ π ( k ) , st ∼ T ( st−1 ) ] ( 1 ) for some finite time horizon T and discount factor γ . In MARL , the aforementioned objective function is optimized using policy gradient . Specifically , we use asynchronous advantage actor-critic ( A3C ) [ 30 ] with Generalized Advantage Estimation [ 37 ] to optimize our policy network . The policy network in A3C outputs a distribution over the actions and the discounted future returns . Any other policy optimization algorithms could have been used in lieu of A3C . We do not use centralized training or self-play , and only consider decentralized training where each agent is parameterized by an independent policy . 4 Grounding Representation for Communication with Autoencoders . The main challenge of learning to communicate in fully decentralized MARL settings is that there is no grounded information to which agents can associate their symbolic utterances . This lack of grounding creates a dissonance across agents and poses a difficult exploration problem . Ultimately , the gradient signals received by agents are therefore largely inconsistent . As the time horizon , communication space , and the number of agents grow , this grounding problem becomes even more pronounced . This difficulty is highlighted in numerous prior works , with empirical results showing that agents often fail to use the communication channel at all during decentralized learning [ 3 , 10 , 19 , 28 ] . We propose a simple and surprisingly effective approach to mitigate this issue : using a self-supervised representation learning task to learn a common grounding [ 43 ] across all agents . Having such a grounding enables speakers to communicate messages that are understandable to the listeners and convey meaningful information about entities in the environment , though agents need not use the same symbols to mean the same things . Specifically , we train each agent to independently learn to auto-encode its own observation and use the learned representation for communication . This approach offers the benefits of allowing fully decentralized training without needing additional architectural image encoder message encoder ( st ) Policy Gradient Communication Grounding agent 1 a ( 2 ) tc ( 2 ) t c ( n ) t a ( n ) ta ( 1 ) tc ( 1 ) t agent 2 agent n c ( n ) t 1c ( 1 ) t 1 c ( 2 ) t 1 Communicate s ( 1 ) t s ( 2 ) t s ( n ) t s ( n ) t c ( 1 ) t 1 . . . c ( n ) t 1 ( ct 1 ) a ( n ) tc ( n ) t ( n ) ( 2 ) ( 1 ) Figure 1 : Overview : The overall schematic of our multi-agent system . All agents share the same individual model architecture , but each agent is independently trained to learn to auto-encode its own observation and use the learned representation for communication . At each time step , each agent observes an image representation of the environment as well as messages broadcasted by other agents during the last time step . The image pixels are processed through an Image Encoder ; the broadcasted messages are processed through a Message Encoder ; the image features and the message features are concatenated and passed through a Policy Network to predict the next action . The image features are also used to generate the next communication messages using the Communication Autoencoder . bias or supervision signal . In Section 5 , we show the effectiveness of our approach on a variety of MARL communication tasks .

Summary: This paper proposes an approach towards encouraging the development of communication protocols in multi-agent reinforcement learning systems. The authors rightly observe that naive training of a communication policy according to task reward often fails, due to high variability in the initial messages, the difficulty of connecting this communication to reward, and lack of common grounding among the agents.  The authors' idea is to imbue agents with a message-generating mechanism cast as an autoencoder, where the agents aim to generate discrete messages that can reconstruct the observations of the agents. Thus, the message generated for a given timestep is essentially a discretized summary of an agents' observation at that time. This encourages the messages to be maximally informative about an agents' current state—something that is shown by qualitatively.  The authors evaluate this communication idea on a variety of environments, from a reference game variant to a few multi-agent minigrid environments, and show that their autoencoded communication outperforms (1) non-communicative baselines and (2) communication policies trained with standard RL objectives. They then have a few analyses that show that the agents are making use of the generated messages, in that (1) messages are informative about agents' current states (by design) and (2) messages affect other agents' entropy over decisions once they indicate a change in state.