Learning to Ground Multi-Agent Communication with Autoencoders
Authors: Toru Lin
Minyoung Huh
Chris Stauffer
Ser-Nam Lim
Phillip Isola
Venue: NeurIPS 2021 Poster (2021)
Keywords: multi-agent reinforcement learning
multi-agent communication

Abstract: Communication requires having a common language, a lingua franca, between agents. This language could emerge via a consensus process, but it may require many generations of trial and error. Alternatively, the lingua franca can be given by the environment, where agents ground their language in representations of the observed world. We demonstrate a simple way to ground language in learned representations, which facilitates decentralized multi-agent communication and coordination. We find that a standard representation learning algorithm -- autoencoding -- is sufficient for arriving at a grounded common language. When agents broadcast these representations, they learn to understand and respond to each other's utterances and achieve surprisingly strong task performance across a variety of multi-agent communication environments.

Learning to Ground Multi-Agent Communication with Autoencoders
Toru Lin MIT CSAIL torulk@mit edu
Minyoung Huh MIT CSAIL
Chris Stauffer Facebook AI cstauffer com
Ser-Nam Lim Facebook AI
minhuh@mit.. edu
sernamlim@fb.. com
Phillip Isola
MIT CSAIL phillipi@mit.
Abstract
Communication requires having common language, a lingua franca, between agents. This language could emerge via a consensus process, but it may require many generations of trial and error. Alternatively, the lingua franca can be given by the environment, where agents ground their language in representations of the observed world. We demonstrate simple way to ground language in learned representations, which facilitates decentralized multi-agent communication and coordination. We find that standard representation learning algorithm autoen- coding is sufficient for arriving at grounded common language. When agents broadcast these representations, they learn understand and respond each other's 's utterances and achieve surprisingly strong task performance across variety multi-agent communication environments.
1 Introduction
An essential aspect of communication is that each pair of speaker and listener must share common understanding of the symbols being used [8]. For artificial agents interacting in an environment, with a communication channel but without an agreed-upon communication protocol, this raises the question: how can meaningful communication emerge before a common language has been established?
To address this challenge, prior works have used supervised learning [18], centralized learning [14, 17, 28), or differentiable communication [6, 17, 28, 32, 41J. Yet, none of these mechanisms is representative of how communication emerges in nature, where animals and humans have evolved communication protocols without supervision and without a centralized coordinator [35]. The communication model that most closely resembles language learning in nature is fully decentralized model, where agents' policies are independently optimized. However, decentralized models perform poorly even simple communication tasks [17] with additional inductive biases [10].
We tackle this challenge by first making the following observations on why emergent communication difficult in a decentralized multi-age aeelt- reinforcement learning setting. A key problem that prevents agents from learning meaningful communication is the lack a common grounding in communication symbols [3, 10, 19]. In nature, the emergence of common language is thought to be aided by physical biases and embodiment [29, 42] - we can only produce certain vocalizations, these sounds only can be heard certain distance away, these sounds bear similarity to natural sounds in the environment, etc. yet artificial communication protocols are not priori grounded in aspects of the environment
Project page, code, videos can found at httpp::///oruuee github. io marrllrl--a ae- comm/.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

dynamics. This poses a severe exploration problem as the chances of consistent protocol being found and rewarded is extremely small [17]. Moreover, before a communication protocol found, the random utterances transmitted between agents add to the already high variance of multi-agent reinforcement learning, making the learning problem even more challenging [10, 28].
To overcome the grounding problem, an important question to ask is: do agents really need to learn language grounding from scratch through random exploration an environment where success is determined by chance? Perhaps nature has a different answer, previous studies in cognitive science and evolutionary linguistics [20, 36, 39, 40] have provided evidence for the hypothesis that commu- nication first started from sounds whose meaning are grounded in the physical environment, then creatures adapted to make sense of those sounds and make use of them. Inspired by language learning in natural species, we propose novel framework for grounding multi-agent communication: first ground speaking through learned representations the world, then learn listening to interpret these grounded utterances. Surprisingly, even with the simple representation learning task of autoencoding, our approach eases the learning of communication in fully decentralized aulti-agent settings and greatly improves agents' performance in multi-agent coordination tasks that are nearly unsolvable without communication.
The contribution of our work can be summarized as follows:
We formulate communication grounding as representation learning problem and propose to use observation autoencoding to learn a common grounding across all agents.
We experimentally validate that this is an effective approach for learning decentralized communica- tion in MARL settings: a communication model trained with a simple autoencoder can consistently outperform baselines across various MARL environments.
In turn, our work highlights the need to rethink the problem of emergent communication, where we demonstrate the essential need for visual grounding.
2
Related Work
In multi-agent reinforcement learning (MARL), achieving successful emergent communication with decentralized training and non-differemtiable communication channel is an important yet challenging task that has not been satisfactorily addressed by existing works. Due the non-stationary and non- Markovian transition dynamics in multi-agent settings, straightforward implementation of standard reinforcement learning methods such as Actor-Critic [22] and DQN [31] perform poorly [17, 28).
Centralized learning is often used to alleviate the problem of high variance in MARL, for example learning a centralized value function that has access to the joint observation of all agents [9, 14, 28). However, it turns out that MARL models are unable to solve tasks that rely on emergent communication, even with centralized learning and shared policy parameters across the agents [17]. Eccles et al. [10] provides an analysis that illustrates how MARL with communication poses a more difficult exploration problem than standard MARL, which is confirmed by empirical results in [17, 28]: communication exacerbates the sparse and high variance reward signal in MARL.
Many works therefore resort differentiable communication [6, 17, 28, 32, 41], where agents are allowed to directly optimize each other's 's communication policies through gradients. Among them, Choi el al. [6] explore a high-level idea that is similar to ours: generating messages that the model itself can interpret. However, these approaches impose a strong constraint on the nature of communication, which limits their applicability to many real-world multi-agent coordination tasks.
Jaques et al. [21] proposes a method that allows independently trained RL agents to communicate and coordinate. However, the proposed method requires that an agent either has access to policies of other agents or stays in close proximity to other agents. These constraints make it difficult for the same method to be applied or do not observe others directly. Eccles et al. [10] attempts to solve the same issue by introducing inductive biases for positive signaling and positive listening, but implementation requires numerous tpec-spe hyperparameter tuning, and the effectiveness is limited.
wider range of tasks, such as those in which agents are not embodied
It is also worth noting that, while a large number of existing works on multi-agent communication take structured state information as input [4, 15, 16, 17, 32, 34], we train agents to learn commu- nication protocol directly from raw pixel observations. This presents additional challenges due to

---

the unstructured and ungrounded nature of pixel data, as shown in [3, 6, 24]. To our knowledge, this work is the first effectively use representation learning to aid communication learning from pixel inputs in a wide range of MARL task settings.
3 Preliminaries
We model multi-agent reinforcement learning (MARL) with communication as a partially- observable general-sum Markov game [26, 38], where each agent can broadcast information to shared communication channel. Each agent receives a partial observation of the underlying world state at every time step, including all information communicated in the shared channel. This observation is used learn an appropriate policy that maximizes the agent's environment reward. In this work, we parameterize the policy function using deep neural network.
Formally, a decentralized MARL can be expressed as a partially observable Markov decision process as M = (S, O,T.C ?), where N is the number of agents, S is a set of states spaces, A = {AA ..... A..,, J.C {Ct, ,.., C*J. and O {0L ,..., ON are a set of action, of communication, and of observation spaces respectively.
(k) of the underlying true state St and set of At time step t, agent & observes partial view of communicated messages from the previous time step 4-1 = (6-11,... (.)). The agent then chooses an action aha)) A and subsequent message to broadcast (( C Given the joint actions of all N
agents = {a, ..... a; ( (4' A"), the transition function T S x A' x x AA + (S) maps the current state 8t and set of agent actions a, to a distribution over the next state 814 Since
the transition function T is non-determinstic, we denote the probability distributions over as ASS).
Finally, each agent receives an individual reward r (k) R[st, where R: : A x... A R.
In our work, we consider a fully cooperative setting in which the objective of each agent to maximize the total expected return of all agents:
maximize T AXC
E
ΣV Σ
R'R(s,, at) (()) ~ 180 T(s
(I)
1ET KEN
for some finite time horizon T and discount factor Y-
In MARL, the aforementioned objective function optimized using policy gradient. Specifically, we use asynchronous advantage actor-critic (A3C) [30] with Generalized Advantage Estimation [37] to optimize our policy network. The policy network in A3C outputs distribution over the actions and the discounted future returns. Any other policy optimization algorithms could have been used in lieu of A3C. We do not use centralized training or self-play, and only consider decentralized training where each agent is parameterized by an independent policy.
4 Grounding Representation for Communication with Autoencoders
The main challenge of learning to communicate fully decentralized MARL settings that there no grounded information to which agents can associate their symbolic utterances. This lack of grounding creates a dissonance across agents and poses difficult exploration problem. Ultimately, the gradient signals received by agents are therefore largely inconsistent. As the time horizon, communication space, and the number of agents grow, this grounding problem becomes even more pronounced. This difficulty is highlighted in numerous prior works, with empirical results showing that agents often fail to use the communication channel at all during decentralized learning [3, 10, 19, 28].
We propose simple and surprisingly effective approach to mitigate this issue: using a self superviedd representation learning task learn a common grounding [43] across all agents. Having such a grounding enables speakers to communicate messages that are understandable to the listeners and convey meaningful information about entities the environment, though agents need not use the same symbols to mean the same things. Specifically, we train each agent to independently learn to auto-encode its own observation and use the learned representation for communication. This approach offers the benefits of allowing fully decentralized training without needing additional architectural
3

---

af
c
c
m
Pin (o(
Palcy Gradient
Communication Autoencodar
22
(n)
(9)
(m)
Communicate
60m (8)
deccom (Ce-1)
image encoder
message emooder
68)
agent
agent
agent II
The overall schematic of our multi-agent system. All agents share the same individual model architecture, but each agent is independently trained learn to auto-encode its own observation and use the learned representation for communication. At each time step, each agent observes an image representation the environment as well as messages broadcasted by other agents during the last time step. The image pixels are processed through an Image Encoder; the broadcasted messages are processed through Message Encoder; the image features and the message features are concatenated and passed through Policy Network to predict the next action. The image features also used to generate the next communication messages using the Communication Autoencoder.
Figure 1: Overview:
bias or supervision signal. In Section 5, we show the effectiveness of our approach on a variety of MARL communication tasks.
overview of our method is shown Figure 1, which illustrates the communication flow among agents at some arbitrary time step t. All agents share the same individual model architecture, and each agent consists of two modules; a speaker module and a listener module. We describe the architecture details of single agent below.
4.1 Speaker Module
At each time step the speaker module takes in the agent's observation of (k) and outputs the agent's next communication message c
Image Encoder Given the raw pixel observation, the module first uses a image encoder to embed the pixels into a now-dimensional feature (a) + @m (o (*) E 1228 The image encoder convolutional neural network with 4 convolutional layers, and the output of this network is spatially pooled. We use
same image encoder the listener module.
Communication Autoencoder The goal of the communication autoencoder is to take the current state observation and generate the next subsequent message. We use an autoencoder to learn
mapping from the feature space of image encoder to communication symbols, i.e. dim (o, (A)) + c The autoencoder consists of encoder and decoder, both parameterized by 3-layer MLP. The
decoder tries to reconstruct the input state from the communication message (1) + (o communication messages are quantized before being passed through the decoder. We use straight- through estimator to differentiate through the quantization [1]. The auxiliary objective function of our model is to minimize the reconstruction loss | (o, (ox*)) (k) ) (o; (ox) I|3. This loss is optimized
(k)). The
jointly with the policy gradient loss from the listener module.
4.2 Listener Module
While the goal of the speaker module is to output grounded communication based on the agent's private observation of the goal of the listener module is to learn an optimal action policy based on both the observation 0f4 and communicated messages c--1- At each time step t, the listener module outputs the agent's 's next action d;

---

FindGoal (non-ordinal)
RedBlueDoors (ordinal)
Bue apent walls
Blue doc opanid
Fied olor opanio fist
Good miafI Praggerr book
Vaubdlyy
Fist slcoe speenee
Figure 2: MarlGrid Environment: We introduce two new grid environments: FindGoal (left) and RedBlueDoors (right). These environments are adapted from the GridWorld enviromment [5, 33]. Environment states are randomized at every episode and are partially observable to the agents. In FindGoal, the task is to reach the green goal location. Each agent receives reward of when they reach the goal, and an additional reward of when all 3 agents reach the goal within the time frame. In RedBlueDoors the task is ordinal, where the ordering of actions matter. A reward of 1 is given to both agents if and only if the red door is opened first and then the blue door.
Message Encoder The message encoder linearly projects all messages communicated from the previous time step C--1 using shared embedding layer. The information across all agent message embeddings combined through concatenation and passed through 3-layer MLP. The resulting message feature has fixed dimension of 128, i.c. @comm (ca) E R118
Policy Network Each agent uses an independent policy head, which is standard GRU [7] policy with linear layer. The GRU policy concatenates the encoded image features and the message features 0 = Óim(( (om (k)) (room(ct)) where is the concatenation operator across the feature dimension. the actions
The GRU policy predicts distribution
(Ø) and the corresponding expected returns. The predicted action distribution and expected returns are used for computing the policy gradient loss. This loss is jointly optimized with the autoencoder reconstruction loss from the speaker module.
The same setup is used for all experiments. The exact details of the network architecture and the corresponding training details are in the Appendix.
Experiments
In this section, we demonstrate that autoencoding is simple and effective representation learning algorithm to ground communication in MARL. We evaluate our method on various multi- environments and qualitatively show that our method outperforms baseline methods. We then provide further ablations and analyses on the learned communication.
5.1 Environments
We introduce three multi-agent communication environments: CIFAR Game, FindGoal, and RedBlueDoors. Our work focuses on fully cooperative scenarios, but can also be extended to competitive or mixed scenarios.
Our environments cover a wide range of communication task settings, including (1) referential or non-referential, (2) ordinal non-ordinal, and (3) two-agent versus generalized multi-agent. A referential game, often credited to Lewis signaling game [25], refers to a setup in which agents communicate through series of message exchanges solve a task. In contrast to non-referential games, constructing communication protocol is critical to solving the task where one can only arrive solution through communication. Referential games are referred as grounded learning environment, and therefore, communication in MARL has been studied mainly through the lens of referential games [12, 24]. Lastly, ordinal tasks refer family of problems where the ordering of the actions is critical for solving the task. The difference between ordinal and non-ordinal settings illustrated in Figure 2, where the blue agent must wait for the red agent to open the door to successfully receive reward. In contrast, non-ordinal tasks could benefit from shared information, but is not necessary complete the task. We now describe the environments used in our work in more detail:
5

---

CIFAR Game We design CIFAR Game following the setup of Multi-Step MNIST Game in [17], but with CIFAR-10 dataset [23] instead. This is non-ordinal, two-agent, referential game. In CIFAR game, each agent independently observes a randomly drawn image from the CIFAR-10 dataset, and the goal is to communicate the observed image to the other agent within environment time steps. At each time step, each agent broadcasts set of communication symbols of length 1. At the final time step, each agent must choose a class label from the 10 possible choices, At the end of the episode, an agent receives a reward of 0.5 for each correctly guessed class label, and both agents receive reward of 1 only when both images are classified correctly.
MarlGrid Environments The second and third environments we consider are: FindGoal (Figure left) and RedBlueDoors (Figure 2 right). Both environments are adapted from the GridWorld environment [5, 33] and environment states are randomized at every episode.
FindGoal is non-ordinal, multi-agent, non-referential game. We use N = 3 agents, and at each time step, each agent observes partial view of the environment centered at its current position. The task of agents is to reach the green goal location as fast as possible. Each agent receives an individual reward of for completing the task and an additional reward of 1 when all agents have reached the goal. Hence, the optimal strategy of an agent is to communicate the goal location once it observes the goal. If all agents learn sufficiently optimized search algorithm, they can maximize their reward without communication.
RedBl ueoooss is an ordinal, two-agent, non-referential game. The environment consists of red door and a blue door, both initially closed. The task of agents is to open both doors, but unlike in the previous two games, the ordering of actions executed by agents matters. A reward of is given to both agents if and only if the red door is opened first and then the blue door. This means that any time the blue door is opened first, both agents receive reward of O, and the episode ends immediately. Hence, the optimal strategy for agents is to convey the information that the red door was opened. Since is possible to solve the task through visual observation or by a single agent that opens both doors, communication is not necessary.
Compared to CIFAR Game, the MarIGrid environments have a higher- nigeer-dimensional observation space and a more complex action space. The fact that these environments are non-referential exacerbates the visual-language grounding problem since communication can only exist in the form of cheap talk (i.e., costless, nonbinding, nonverifiable communication [13] that has no direct effect on the game state and agent payoffs). We hope to show from this set of environments that autoencoders can be used as a surprisingly simple and adaptable representation learning task to ground communication. It requires little effort to implement and almost no change across environments. Most importantly. as we will see in Section 5.4, autoencoded representation shows an impressive improvement over communication trained with reinforcement learning.
5.2 Baselines
To evaluate the effectiveness of grounded communication, we compare our method (ae- comm) against the following baselines: (1) a no-comm baseline, where agents are trained without a communication channel; (2) arl-comm baseline'. where we do not make a distinction between the listener module and the speaker module, and the communication policy is learned in a way similar to the environment policy; (3) a rl- comm-wo wi th-biiasss baseline, where inductive biases for positive signaling and positive listening are added to rl- comm training as proposed in [10]: (4) ae-rl-come baseline, where the communication policy is learned by an additional policy network trained on top of the autoencoded representation in speaker module.
5.3
The Effectiveness of Grounded Communication
In Table and Figure 3, we compare task performance of ae-comn agents with performance of baseline agents. We report all results with 95% confidence intervals, evaluated 100 episodes per seed over 10 random seeds.
In CIFAR Game environment, all three baselines (no comm, r1 r1-comm, r1-comm- teimm-withhte with-biasess biases) could only obtain an average reward close to that of random guesses throughout the training process.
"The rl-comm baseline be seen an A3C version of RIAL [17] without parameter sharing.

---

RooBlueDoors 1
CIFAR Dialogue
FindGoal
0.5
Environment reward
Environment remnandteewrd reward
400
0.8
0.4
Episode length
0.3
0.4
0.
200
0.1
100
0.
100000
200000
300000
100000
200000
300000
100000
200000
30000
Train iterations
Train iterations
Train iterations
r comm
r comm w/ bias
comm
Figure 3: Comparison with baselines: Comparison between our method that uses an autoencoded communication (ae-comm), baseline that is trained without communication (no-comm), base- line where communication policy trained using reinforcement learning (rl-comm) and another baseline where inductive biases for positive signaling and positive listening added to rl- comm training (r1- (rl-comm-i with with-inddus te--nnductiin weriiieebisseee biases). For FindGoal, we visualize the amount of time takes for all agents reach the goal, all methods reach the goal within the time frame. For each set of results, we report the mean and 95% confidence intervals evaluated on 100 episodes per seed over 10 random seeds.
In comparison, ae- ae-comm agents achieve a much higher reward on average. Since this environment a referential game, our results directly indicate that ae- ae-comm agents learn to communicate more effectively than the baseline agents.
RedBlueDoors environment poses challenging multi-agent coordination problem since the reward is extremely sparse. As shown in Figure 3(b), neither of agents
methods
CIFAR
RedBlueDoors
FindGoal
avg. r
avg. 1
avg. 1.).
nо-comm
0.082 = 0.009
0.123 0.096
169.0 + 26.8
the no- comm and was able learn successful coordination strategy. Although rl-comm-wi ta-tth-bbee biass agents outperform the other two baseline agents, they not learn an optimal strategy that guarantees an average reward close to 1. In contrast, ae-comm agents converge to an optimal strategy after 150k of training.
rl-comu
0.099 1: 0.013
0.174 1 0.009
184.8 1: 25.2
ri-comm w/ bias [10]
0.142 + 0.019
0.729 0.072
158.0 +: 12.3
ac-comm (ours)
0.348 + 0.041
0.984 0.002
103.5 + 20,2
Table 1: Comparison with baselines: We compute the average reward for CIFAR Game and RedBlueDoors environments, and average episode length for FindGoal environment. For each set of results, report the mean and 95% confidence intervals evaluated on 100 episodes per seed over 10 random seeds.
In FindGoal environment, agents are able solve the task without communication, but their perfor- mance can be improved with communication. Therefore, we use episode length instead of reward as performance metric for this environment. numerical results in table below. While all agents are able to obtain full rewards, Figure 3 shows that ae- ce-comm agents are able to complete the episode much faster than other agents. We further verify that this improvement is indeed result of the successful communication by providing further analysis in Section 5.5.
resolve the ambiguity in Figure 3(c), we also include
Our results indicate that communication model trained with autoencoding tasks consistently outperforms the baselines across all environments. The observation that communication does not work well with reinforcement learning is consistent with observations made in prior works [10, 17, 28]. Furthermore, our results with autoencoders a task that often considered trivial - highlight that we community may have overlooked a critical representation learning component in MARL communication. In Section 5.4, we provide more detailed discussion on the difficulty of training emergent communication via policy optimization.
7

---

ReoBlueDoors f
CIFAR Dialogue
FindGoal
450
Environment reward
Environment reward
400
0.8
0.8
Episode length
350
250
0.4
0.4
200
150
0.2
100
0.
100000
200000
300000
0.0
100000
200000
300000
0
100000
200000
30000
Train iterations
Train iterations
Train iterations
comm
comm
Comparison between a speaker module trained with only an autoencoding task (ae- comm) and another one trained with both autoen- coding task and reinforcement learning (ae-rl-on comm). We observed that further training a policy on top of the autoencoder representation degrades performance across all environments (except for in FindGoal, where performance of ae-rl- comm and ae comm stayed about the same).
Figure 4: Representation learning with reinforcement learning:
5.4 The Role of Autoencoding
Given the success of agents trained with autoencoders, it is natural to ask whether a better communica- tion protocol can emerge from the speaker module by jointly training it with reinforcement learning policy. To this end, we train a GRU policy on top of the autoencoded representation (ae-rl-comm) and compare it against our previous model that was trained just with autoencoder (ae-comm). The communication policy head is independent of the environment action policy head.
Surprisingly, we observed in Figure 4 that the model trained jointly with reinforcement learning consistently performed worse (except for FindGoal, where performance of ae-rl-comm and comm stayed about the same). We hypothesize that the lack of correlation between visual observation and the communication rewards hurts the agents' performance. This lack of visual- reward grounding could introduce conflicting gradient updates to the action policy, and thereby exacerbate the high-variance problem that already exists reinforcement learning. other words, optimization is harder whenever the explooraton problem for learning speaker and listener policies introduced. Our observation suggests that specialized reward design and training at the level of [2] might be required for decentralized MARL communication. This prompts to rethink how to address the lack of visual grounding in communication policy, where this work serves as a first step in this direction.
5.5 Analyzing the Effects of Communication Signals Agent Behavior
Communication Embedding To analyze whether the agents have learned a meaningful visual grounding for communication, we first visualize the communication embedding. In Figure 5, we visualize the communication symbols transmitted by the agents trained on RedBlueDoors. The communication symbols are discrete with length of 10 (1024 possible embedding choices), and we use upproximately 4096 communication samples across 10 episodic runs. We embed the commu- nication symbols using t-SNE [44] and further cluster them using DBSCAN [11]. In the figure, visualize clusters by observing the correspondence between image states and communication symbols produced by agents in those states. For example, corresponded to an environment state when the red door was opened; this suggests a communication action where one agent signals the other agent to open the blue door and complete the task.
observed that a specific communication cluster
Entropy of Action Distribution To measure whether communicated information directly influences other agents' actions, we visualize the entropy of action distribution during episode rollouts. Suppose agent shares information that is vital to solving the task. In that case, decrease in entropy should be observed in the action distributions of other agents, as they act more deterministically towards solving the task.
As shown Figure 6, we visualize the entropy of action distribution across 256 random episodic runs using policy parameters from fully trained ae comm model. The entropies are aligned using

---

Communication Embedding
No doors visible
Red door was opened
Figure 5: Communication clusters: 4096 communication messages are embedded into low- dimensional representation using t-SNE [44] and is clustered using DBSCAN [11]. We visualize the images corresponding to the communication messages. We observed that the message clusters correspond to various meaningful phases throughout the task. The communication symbol of the purple cluster corresponds to when doors are visible by either agent, and the light green cluster corresponds to when the red door is opened.
environment milestone events: for FindGoal, this is when the first agent reaches the goal; for RedBlueDoors this when the red door opened. Since the identity of the agents that solve the task first does not matter, entropy plots are computed with respect to the listener agents (i.e., agents that receive vital information from others). In FindGoal, this corresponds to the last agent to reach the goal; in RedBlueDoors, this corresponds to the agent opening the blue door. For both environments, we see a sharp fall-off in entropy as soon as the first agents finish the task. In contrast, agents trained without autoencoding act randomly regardless of whether other agents have completed the task. This reaffirms that the agents trained with autoencoder can effectively transmit information to other agents.
Positive Signaling and Positive Listening We additionally investigate the two metrics suggested by [27] for measuring effectiveness of communication, positive signaling and pos- itive listening. Since ae ae-comm agents have communicate their learned representation, the presence of representation learning task loss means that ae- comm agents are intrinsically optimized for positive signaling (i.e., sending messages that are related to their observation or action). In Table 2, we report the increase in average reward when adding a communi- cation channel, comparing ae- comm with its baselines; this metric is suggested by [27] to be a sufficient metric for positive listening (i.e., communication messages influence the behavior of agents). We observe that agent task performance improves most substantially in ce-com with the addition of communication channel.
CIFAR
RedBlueDoors
methods
gain Ar
gain Ar n
rl-comm
0.017 + 0.022
-0.030 + 0.155
rl-comm w/ bias [10]
0.060 1: 0.029
0.552 1: 0.155
ae-comm (ours)
0.266 1 0.058
0.807 0.185
Table 2: Performance gain with communication: Positive listening measured by the increase in reward after adding a communication channel. Per- formance of co-comm agents improves more than the baselines.
6 Discussion and Societal Impacts
We present a framework for grounding multi-agent communication through autoencoding, a simple self-supervised representation learning task. Our method allows agents to learn non-differentiable communication in fully decentralized settings and does not impose constraints on input structures (e.g., state inputs pixel inputs) or task nature (e.g., referential or non-referential) Our results demonstrate that agents trained with the proposed method achieve much better performance on suite of coordination tasks compared to baselines.

---

FindGoal
RedBlueDoor
1.6
1.8
Action entropy
1.6
Action entropy
1.5
1.4
1.4-
1.2
1.3
1.0
1.2
0.8
1.1
Red door opened
Both doors opened
First agent reached
Second agent resched
without grounding
with grounding
Figure 6: Policy entropy with communication: We visualize the entropy of the action policy throughout the task (lower is better). The graph is generated with 256 random episodes. For F FindGoal, the entropy is measured on the last agent to enter the goal, and for RedBlueDoors the entropy is measured on the agent that opens the blue door (the second door). All 256 runs are aligned to the dotted red lines which corresponds to the time in which the first and second agent enters the goal for (FindGoal), and the time which the red and the blue doors are opened for )eedBiueDoors) The model trained with auto-encoder transmits messages that are effectively used by other agents.
We believe this work on multi-agent communication of importance to our society for two reasons. First, it extends computational framework under which scientific inquiries concerning language acquisition, language evolution, and social learning can be made. Second, unlike works in which agents can only learn latent representations of other agents through passive observations [45, 46), it opens up new ways for artificial learning agents to improve their coordination and cooperative skills, increasing their reliability and usability when deployed to real-world tasks and interacting with humans.
However, we highlight two constraints of our work. First, method assumes that all agents have the same model architecture and the same autoencoding loss, while real-world applications may involve heterogeneous agents. Second, our method may fail in scenarios where agents need to be more selective of the information communicated since they are designed to communicate their observed information indiscriminately. We believe that testing the limits of these constraints, and relaxing them, will be important steps for future work.
We would also like to clarify that communication between the agents in our work is only part of what true "communication" should eventually entail. We position our work as one partial step toward true communication, in that our method provides strong bias toward positive signaling and allows decentralized agents to coordinate their behavior toward common goals by passing messages amongst each other. It does not address other aspects of communication, such as joint optimization between a speaker and a listener. In the appendix section, we highlight the difficulty of training emergent communication via policy optimization alongside empirical results.
Another limitation of this work, which is also one concern we have regarding its potential negative societal impact, is that the environments we consider are fully cooperative. If the communication method we present in this work is to be deployed to the real world, we need to either make sure the environment is rid of adversaries, or conduct additional research to come up with robust counter- strategies in the face of adversaries, which could use better communication policies as a way to lie, spread misinformation, maliciously manipulate other agents.
Acknowledgements
We sincerely thank all the anonymous reviewers for their extensive discussions on and valuable contributions to this paper. We thank Lucy Chai and Xiang Fu for helpful comments on the manuscript. We thank Jakob Foerster for providing inspiring advice multi-agent training. Additionally, TL would like to thank Sophie and Sofia; MH would like to thank Sally, Leo and Mila; PI would like to thank Moxie and Momo. This work was supported by grant from Facebook.
10

---

References
[1] Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons
for conditional computation. arXiv preprint 138.32230...432 2013. 4
[2] C. Berner, G. Brockman, B. Chan, V. Cheung. P. Dębiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme,
C. Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv: 1912.0660.
2019. 8
[3] B. Bogin, M. Geva, and J. Berant. Emergence of communication in an interactive world with consistent speakers. arXiv preprint N.KI::88090005444 2018. 1.3
[4] K. Cao, A. Lazaridou, M. Lanctot, J. Z. Leibo, K. Tuyls, and Clark. Emergent communication through
negotiation. arXiv preprint arXiv: 03980. 2018. 2
[5] M. Chevalier- Boisserr, L. Willems, and S. Pal. Minimalistic gridworld environment for openai gym. https: // tigb. com/ / htmmmmaxmmmccc gegggggmmmnn minigrid, 2018. 6,
[6] E. Choi, A. Lazaridou, and de Freitas. Compositional obverter communication learning from raw visual input. arXiv preprint arXiv: 1100.800444444 2018. 1, 2,
[7] J. Chung. C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint 1412.355, 2014.
[8] A. Dafoc, E. Hughes, Y. Bachrach, T. Collins, K. R. McKee, J. Z. Leibo, K. Larson, and T. Graepel. Open problems in cooperative ai. arXiv preprint arXiv:2012.0880 2006300, 2020.
[9] A. Das, T. Gervet, J. Romoff, D. Batra, D. Parikh, M. Rabbat, and J. Pincau. Tarmac: Targeted multi-agen communication. In International Conference on Machine Learning, pages 1538-1546, PMLR, 2019. 2 [10] T. Eccles. Bachrach, G. Lever, A. Lazaridou, and T. Graepel. Biases for emergent communication in multi-agent reinforcement learning. arXiv preprint a912: 25676, 2019. 1, 6, 7.9
[11] M. Ester, H.-P. Kriegel, J. Sander, X. Xu, et al. A density- based algorithm for discovering clusters in large spatial databases with noise. In Kdd, volume 96, pages 221,,11, 1996. 9
[12] K. Evtimova, A. Drozdov, D. Kicla, and K. Cho. Emergent communication in a multi-modal, multi-step referential game. arXiv preprint arXiv: 100.:1705.1000 10369, 2017. 5
[13] J. Farrell. Cheap talk, coordination, and entry. The RAND Journal of Economics, pages 34-39, 1987, 6 [14] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. 1,2
[15] J. Foerster, F. Song. E. Hughes, N. Burch, I. Dunning, S. Whiteson, M. Botvinick, and M. Bowling. Bayesian action decoder for deep multi-agent reinforeement learning. In International Conference on Machine Learning, pages 1942-1951. PMLR, 2019. 2
[16] J. N. Foerster, Y. M. Assacl, N. de Freitas, and S. Whiteson. Learning to communicate to solve riddles with deep distributed recurrent q-networks. arXiv preprint arXiv: 02072, 2016. 2
[17] J. N. Foerster. M. Assael, N. De Freitas, and S. Whiteson. Learning to communicate with deep multi-agent reinforcement learning. arXiv preprini 1oXXiv::6050666 2016. 1, 2. 6,7
[18] L. Graesser, K. Cho, and D. Kiela. Emergent linguistic phenomena in multi-agent communication games. arXiv preprint arXiv: 087066 2019.
[19] S. Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3):335-34 346,-34,, 1990. 1,
[20] J. R. Hurford. Biological evolution of the saussurean sign as a component of the language acquisition device. Lingua, 187-22,, 1989. 2
[21] N. Jaques, A. Lazaridou. E. Hughes, C. Gulcehre, P. Ortega, D. Strouse, Z. Leibo, and N. De Freitas. Social influence as intrinsic motivation for multi-agent deep reinforcement learning. In International
Conference on Machine Learning, pages 3040-3049. PMLR, 2019. 2
[22] V. R. Konda and J. N. Tsitsiklis. Actor-critic algorithms. In Advances in neural information processing systems, pages 1014-1014, Citeseer, 2000. 2
[23] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009. 6
[24] A. Lazaridou, K. M. Hermann, K. Tuyls, and S. Clark. Emergence of linguistic communication from 2018. 3.5
referential games with symbolic and pixel input. arXiv preprini arXiv:1804
[25] D. Lewis. Convention: A philosophical study. John Wiley & Sons, 2008.
[26] M. L. Littman. Markov games as framework for multi-agent reinforcement learning. In Machine learning
proceedings 1994, pages 157-163 Elsevier, 1994. 3
[27] R. Lowe, J. Foerster, Y.-L. Boureau, J. Pineau, and Y. Dauphin. On the pitfalls of measuring emergent communication. arXiv preprint arXiv: 1...:::993305//1/ 05168, 2019. 9
[28] R. Lowe, Y. Wu, A. Tamar, Harb, P. Abbeel, and Mordatch. Multi-agent actor-critic for mixed cooperative-compettive environments. arXiv preprint arXiv:1700.02275. 2017. 1. 3.
[29] B. MacWhinney. The emergence of language from embodiment. Psychology Press. 2013.
[30] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning. pages 1928 1937, PMLR, 2016.

---

[31] V. Mnih, K. Kavukcuogiu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing
atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. 2
[32] I. Mordatch and P. Abbeel. Emergence of grounded compositional language in multi-agent populations. In
P Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018. 1.2
[33]
Ndousse. marlgrid. bttps://githiie com/ waanduss/marrlrrd,, 2020.
[34] M. Noukhowitch, T. LaCroix, A. Lazaridou, and A, Courville. Emergent communication under competition.
arXiv preprint arXiv:2 10276, 2021. 2
[35] M. A. Nowak and D. C. Krakauer. The evolution of language. Proceedings of the National Academy of 1999.
Sciences, 96( (0(14):80888808)
[36] D. K. Roy and Pentland, Learning words from sights and sounds: computational model. Cognitive science, 26(1): (6(1))113-33 13-16,, 2002.
[37] Schulman, Moritz, S. Levine, M. Jordan, and P. Abheel. High-dimensional continuous control using
generalized advantage estimation. arXiv preprint arXiv: 1.....00..222 506.00238, 2015.
[38] L. S. Shapley. Stochastic games. Proceedings of the national academy of sciences, 39(10)
1953. 3
[39]
L.. Steels. The synthetic modeling of language origins. Evolution of communication, 1(1):1-34, 1997.
[40]
L. Steels and F. Kaplan. Aibo's first words: The social learning language and meaning. Evolution of communication, 4(1):3 32, 2000.
[41] S. Sukhbaatar, A. Szlam, and R. Fergus, Learning multiagent communication with backpropagation. arXiv
preprint ar0iv: 0,7766 2016. 1.2
[42] M. Tomasello. Origins of human communication. MIT press, 2010.
[43] T. Udagawa and A. Aizawa. natural language corpus common grounding under continuous and partially-observabe context. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1120--127, 2019, 3
[44] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research,
9111). 2008. 8,
[45] A. Xic, D. Losey, R. Tolsma, C. Finn, and D. Sadigh. Learning latent representations influence multi-agent agent interaction. arXiv preprint arXiv:2011.066 2020. 10
[46] L. Zintgraf, S. Devlin, K. Ciosek, S. Whiteson, and K. Hofmann. Deep interactive bayesian reinforcement learning via meta- meta-learning arXiv preprint arXiv:2101.03886 2021. 10
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proof' of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you'ne using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "sJU4n6Chlv", "reviewer": "Reviewer_rwT3", "summary": "This paper proposes an approach towards encouraging the development of communication protocols in multi-agent reinforcement learning systems. The authors rightly observe that naive training of a communication policy according to task reward often fails, due to high variability in the initial messages, the difficulty of connecting this communication to reward, and lack of common grounding among the agents.\n\nThe authors' idea is to imbue agents with a message-generating mechanism cast as an autoencoder, where the agents aim to generate discrete messages that can reconstruct the observations of the agents. Thus, the message generated for a given timestep is essentially a discretized summary of an agents' observation at that time. This encourages the messages to be maximally informative about an agents' current state—something that is shown by qualitatively.\n\nThe authors evaluate this communication idea on a variety of environments, from a reference game variant to a few multi-agent minigrid environments, and show that their autoencoded communication outperforms (1) non-communicative baselines and (2) communication policies trained with standard RL objectives. They then have a few analyses that show that the agents are making use of the generated messages, in that (1) messages are informative about agents' current states (by design) and (2) messages affect other agents' entropy over decisions once they indicate a change in state.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this is an interesting and clear paper with a nice idea at its core, but I see an obvious missing comparison to related work and some more philosophical issues with the papers' framing that make me hesitant to recommend the paper outright. I'm looking forward to the other reviews and the author response.\n\n## Strengths\n\n- The paper is well written and clear.\n- It has several clean experiments across a variety of environments that clearly show the benefits of autoencoded communication over non-communication and naive baselines.\n- The idea of messages as autoencoding agent observations is an interesting one—it has appeared here and there in the literature (see missing references) but is clearly presented here and will be of interest to those working in MARL. It's likely that such an objective can be combined with more directly communicative objectives, to both encourage grounding and optimize for communication.\n- While some more analyses could be done, I buy the authors' analysis in Section 5.5 that the agents learn to use and understand the messages, both due to the inherent *positive signaling* bias of the autoencoding objective and Figure 7 (which neatly shows that agents become more confident when their partner reaches a subgoal of the environment, and that this confidence can only be a result of the availability of the partners' message).\n\n## Weaknesses\n\nI have two major concerns with this work, and some other issues/questions.\n\n### No comparison to Eccles et al.\n\nEccles et al. [7] is perhaps the most relevant paper for this work. It too suggests that naively optimizing communication via the typical MARL reward signal is brittle and often fails, and propose several auxiliary objectives. Yet the paper is quickly dismissed in L81-83 as \"requiring task-specific hyperparameter tuning\" and \"limited effectiveness.\" Task-specific hyperparameter tuning is a fair criticism, and one claimed advantage of the current method could be its relative ease-of-implementation. But \"limited effectiveness\" is an unsubstantiated claim, and taken together these reasons are not sufficient to completely discard comparison to their proposed approaches. Yet no such comparisons are done in this paper.\n\n### Is this \"communication\"?\n\nMy other concern is more conceptual: it makes me uncomfortable to imagine the experiments in this paper as true \"communication\" among agents, since the learned messages are not trained via any inherent communicative objective (i.e.: generate some message to maximize some reward given some dialog context). Rather, the messages are generated solely from an autoencoding objective: essentially learning a discrete representation of an agents' current observation space (without any regard to what influence this has on the ultimate success of the group).\n\nOne question I have for the authors is, is this a significantly different setup than simply doing centralized MARL, where each agent has full access to the other agents' observation space and internal state? Especially if the agents policies have some discrete latent variable representation which we could then label as \"messages\"?\n\nIn some ways this paper is an extension of the Eccles et al. paper: one part of it proposes a bias for positive signaling (that we generate messages correlated with our current state), and this paper takes it to an extreme where we actually don't consider *anything* besides positive signaling (i.e. the *only* objective is to maximize MI between messages and observations).\n\nThis means that many of the purported benefits of the \"communication\" learned by the agents in this paper seem given \"for free\". For example, the authors admit in L282 that the autoencoder agents are intrinsically optimized for \"positive signaling\" - but indeed, the messages are generated with no other objective besides positive signaling. Similarly, the visualization in Figure 5 suggests that \"messages\" cluster according to episode state. But this is unsurprising: we would expect an optimal policy's internal representations to also cluster according to episode state.\n\nI can see this somewhat philosophical observation resulting in more concrete limitations of the proposed method: if the communication channel is very limited compared to the agents observation state—either because of limited bandwidth (i.e. vocabulary) or a cost to sending messages—then such an autoencoding objective may not work. (Indeed we need more details about the communication channel size, as I'll mention again below). In general, there might be many tasks when agents must learn to be more selective with the information communicated at each timestep, and a purely \"descriptive\" objective like the one presented here might fail, and we need agents that learn *what* to communicate. (This is the same reason why we humans do not constantly vocalize our environmental state every passing second). Admittedly, these constraints are not often considered in current MARL research, and more communicative objectives could be combined with the proposed autoencoding objective (i.e. a \"first step towards grounding\" as in L58-59, though again, this idea is similar to Eccles et al.).\n\n### Other issues/questions\n\n- Insufficient description of baselines in L223-224, specifically with regards to learning the communication via a separate policy network. Why is the communication module in the listener module instead of the speaker module (L224)? isn't the autoencoded representation of the speaker model precisely some sort of discrete message - here you are learning a policy network on top of this to go from this message to another message?\n- L240: I don't understand how figure 3c shows that ae-comm agents are able to complete the episode much faster than other agents. All 3 lines seem to be within the shaded error regions? What does the error region represent? In cases that are visually ambiguous like this one, it would be good to have some sort of significance test.\n- I'm very suspicious of the RL-communication policy not working above chance (no communication) in something as simple as the CIFAR referential game. This is an odd conclusion given the large literature on emergent communication in referential games that finds that we can learn successful (albeit uninterpretable) communication protocols with policy gradient, differentiable relaxations, mixtures of both, etc. Could the authors explain this result more?\n- Would be nice to explicit calculate the proposed CIC metric (i.e. mutual information between messages and other agents' actions) in L274-276, which is precisely what Lowe et al. [22] suggests.\n- It's not clear to me how Figure 6 illustrates positive listening. Autoencoder loss and task performance are correlated, but that's because we're jointly optimizing the two - it doesn't mean that the increase in task performance is *caused* by the improved autoencoder reconstruction. (I do buy the Figure 7 analysis more).\n\n## Missing references\n\nThe idea generating messages that the model itself can interpret (in this case: reconstruct its own state) is similar to the obverter communication method proposed by Choi et al., Compositional Obverter Communication Learning From Raw Visual Input, ICLR 2018\n\n## Minor\n\n- L146 Missing Citation\n- The communication spaces C could be more precisely defined across both environments.\n- Figure 6 would be more clear if Loss was labeled \"Autoencoder Loss\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "oNmvLzq2XDT", "reviewer": "Reviewer_HBSZ", "summary": "The goal of the paper is to show that learning a representation to ground language can improve communication in multi-agent settings.\n\nTheir method: each agent encodes an image and the other agents messages (through a small conn-net and mlp, respectively). A GRU-based policy outputs an action. A communication autoencoder predicts a message from the image feature and is trained with an auxiliary construction objective (i.e., to reconstruct the image).\n\nThey evaluate their model on a referential game with CIFAR images and two grid environments: one where the agents must reach a goal collaboratively and another where one agent has to open a door for another to pass.\n\nThey show that their method with an auto encoder strongly outperforms the non-autoencoder variants on the CIFAR task and door task.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: their main contribution is the design of a multi-agent communication architecture that uses an auto encoder as an auxiliary loss.\n\nQuality: The work is evaluated on a very limited set of tasks and there are no baselines they compare against. The main result (Figure 3) is an ablation. The experiments were also conducted without any hyper parameter tuning.\n\nClarity: The introduction is very high level, which makes it difficult to understandl. I think it would help to ground the introduction in more specific technical contributions in related work to make it more accessible.\n\nSignificance: It's difficult to contextualize the significance of the work because of it's lack of comparisons to other multi-agent work. \n\nMinor comments:\n- The visualization on the right side of figure 1 is confusing. Specifically, why are there bidirectional arrows between the communication encoder and message (and policy network and action) if it's generating the message? It may help to visualize these as modules instead, like the encoders below.\n\nHere are some other works on multi-agent interaction and communication that may be relevant:\nLearning Latent Representations to Influence Multi-Agent Interaction. Xie et. al., 2020.\nTarMAC: Targeted Multi-Agent Communication. Das et. al., 2019.\nEmergent Linguistic Phenomena in Multi-Agent Communication Games. Graesser et. al., 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss societal impacts.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "iBB6BxdU_FJ", "reviewer": "Reviewer_ux54", "summary": "In this paper, the authors present a novel method for learning decentralized communication policies in fully cooperative multi-agent RL. The method is based on autoencoding agent observations and using these as messages, thus sidestepping the usual \"chicken and egg\" problem for simultaneously learning speaker and listener policies, which is a hard joint-exploration problem. The authors present results in which this method outperforms both non-communication and RIAL baselines, and include some analyses and ablations that shed light on how and why this method is successful.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive aspects\n\n- The paper establishes a convincing argument and method for grounding decentralized communication via autoencoding. \n- The introduction and abstract are well-written, and most of the relevant literature is cited clearly. \n- The method is elegant and well-described, and in my opinion the results would therefore be reproducible. \n- The proposed method outperforms RIAL and no communication baselines on several different fully cooperative tasks. \n- The t-SNE analysis in Figure 5 and the entropy analysis in Figure 7 are convincing methods that uncover the mechanisms underlying improved performance of agents under this method.\n\nAspects requiring improvement\n\n- The paper seems to rely on a strong assumption that all agents have the same model architecture, and the same autoencoding loss. This reduces the impact of the work. It would be very interesting to have further experiments that assess to what extent this constraint is required. See also comments on \"limitations\" below.\n- An important baseline is missing. Since the \"speaker\" is transmitting exactly an autoencoded latent state, a good baseline would be one where the speaker transmits a latent state from the policy network, without autoencoding. This would help to establish that it is the autoencoding itself that is necessary, rather than merely the simplification in the multi-agent learning dynamics that comes from fixing the speaker policy to be a latent state.\n- The authors invent their own environments rather than using environments from the literature. This means that their work cannot be quantitatively compared to previous papers like (in their citations) [7,13,16]. Could the authors add one additional environment from the existing literature to make this comparison easier? In particular, does their method outperform some of the methods with inductive biases? To what extent does their method regain the \"topline\" performance afforded by fully differentiable centralized models like DIAL?\n- The results in Figure 3(c) are inconclusive. The authors claim that AE comm is better, but the confidence intervals on their plot suggest that no such conclusion can be drawn. Could the authors comment on the statistically validity of their claim? If the task is simply too easy to see a meaningful difference between the 3 methods, perhaps it would be better to try a harder task here?\n- Parts of the Experiments section are unconvincing. In particular, I did not find the claims about positive listening and positive signaling sufficiently rigorous: it feels like the authors are observing a correlation rather than demonstrating a causal relationship. This section would benefit from reworking or removing. Moreover Section 5.4 misses important hypotheses about why adding RL to the AE loss doesn't help. This might be because the optimization problem is harder when you reintroduce the \"chicken and egg\" joint-exploration problem for learning speaker and listener policies at the same time. Alternatively, or additionally, it might be because the multi-objective optimization problem is harder than the single-objective optimization problem, so training requires more iterations. It would be useful to have further experiments here to disambiguate these possibilities with the representation-learning hypothesis which the authors present. \n- Some relevant citations are missing. For example there has been much recent work on learning latent representations to influence behavior in multi-agent systems, e.g. Xie, Losey, Tolsma, Finn and Sadigh 2020; Zintgraf, Devlin, Ciosek, Whiteson, Hofmann 2021. It would be useful to contextualise this contribution with reference to those papers.\n\nSmall comments\n\n- In several places the authors refer to \"cooperative\" games where they could more precisely say \"fully cooperative\" or \"common payoff\" to distinguish from mixed-motive games, which are sometimes also referred to as cooperative.\n- In lines 233 and 237 the names of the environments are the wrong way round, I believe. \n- In line 214, the authors could consider using the more precise definition of \"cheap talk\" from Farrer 1987.\n- The authors might want to consider citing Dafoe et. al, \"Cooperative AI\", 2020 in their introduction; this review contains a section on the importance of grounding. \n- In their introduction, the authors may wish to also discuss the importance of the gestural modality in the origins of human communication, as in Tomasello, \"Origins of Human Communication\", 2008. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the work should be discussed more clearly. For example, is it essential that agents have identical model architectures, as stated at the bottom of page 3? If so, this seems like quite a strong limitation, since most real-world applications may involve heterogeneous agents. On the other hand, if this limitation can be reduced or mitigated, the authors should provide experiments to this effect, or strong arguments for future work. \n\nThe societal impact is well addressed on page 9.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "J_r9Lgnem-F", "reviewer": "Reviewer_tRBg", "summary": "The paper proposes to learn the communication protocol in an emergent communication environment via an autoencoder; i.e. they train an autoencoder to learn a discrete representation of the image part of the observation, and use that as the message that gets broadcasted to all other agents at each timestep. The paper shows that this achieves significant gains over learning the communication protocol from scratch in a decentralized way.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is very well written, with several beautiful plots and figures, many auxiliary experiments that help understand what's going on. It -looks- like a NeurIPS paper.\n\nAt the core of the paper is a simple experiment --- rather than learning a communication protocol from the environmental reward, have agents share a discrete representation of their state at each time step. It turns out that this works very well in the environments that the paper considers (a CIFAR-based referential game, and some GridWorld environments requiring communication). The reason for this is pretty intuitive: communicating a discrete representation of your state is really all of the information you need to solve these tasks (i.e. it's all the information that the other agents need). Decentralized communication protocols are hard to learn, so it makes sense that they perform way worse. \n\nTo me, this is an interesting experiment! I'm a bit surprised that nobody has thought of it before (to my knowledge), but autoencoding is definitely a strong communication prior, so it's a very reasonable thing to try.\n\nThat being said, I have a couple of concerns about the paper:\n- I don't think the paper quite moves beyond \"this is an interesting experiment\" into \"this could actually be useful / fundamental\" territory. To do this, I would like to have seen either: (a) this approach compared to other methods, e.g. Jaques et al. or Eccles et al., showing that it outperforms them (I don't really like when a paper claims 'the effectiveness of previous approach X is limited, but don't compare to that approach); or (b) testing how the method works in environments where communicating only the state is not sufficient for coordination (e.g. communicating about past actions, or future intentions). I think the fact that doing RL on top of the autoencoder prior does *worse* is indicative that communicating about the state is indeed all you need for these tasks (or maybe that the RL component was poorly tuned).\n- I'm not a big fan of the framing of the approach as 'grounding' the communication, implying that learning the communication protocol from scratch is 'ungrounded'. To me, if the agent learns to communicate with other agents in a grounded environment (such that it improves reward / is actually used), then that communication is grounded. The overall effect is that the paper seems to be a bit 'over-hyped'. This is reinforced by the allusion to cognitive science + evolutionary linguistics as motivation (which isn't wrong, per se, but feels a bit stretched).\n\nThat being said, compared to other papers in emergent communication I do think this paper comes close to meeting the bar (most other emergent communication papers don't compare to other papers in the field). It's still a nice paper. But I find myself wanting either more substance to the paper, or a modification in the framing.\n\nAll things considered, I'd rate this paper as borderline. \n\nSmall points:\n- Reference missing on the straight-through estimator", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the limitations I mentioned above / framing could be improved, but otherwise no concerns.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers found the proposed method a simple yet intriguing contribution to the emergent communication literature. The authors addressed important initial concerns about baselines and ablations in the rebuttal period, and reviewers were satisfied with the new results. There was concern about whether sending autoencoded latents "really" constitutes communication, however I believe that makes the work all the more thought-provoking for the community. I expect this paper will be a useful contribution for other emergent communication research to use as a baseline or build upon.
Decision: Accept (Poster)