Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors present a novel method for learning decentralized communication policies in fully cooperative multi-agent RL. The method is based on autoencoding agent observations and using these as messages, thus sidestepping the usual "chicken and egg" problem for simultaneously learning speaker and listener policies, which is a hard joint-exploration problem. The authors present results in which this method outperforms both non-communication and RIAL baselines, and include some analyses and ablations that shed light on how and why this method is successful.
main_review: Positive aspects

- The paper establishes a convincing argument and method for grounding decentralized communication via autoencoding. 
- The introduction and abstract are well-written, and most of the relevant literature is cited clearly. 
- The method is elegant and well-described, and in my opinion the results would therefore be reproducible. 
- The proposed method outperforms RIAL and no communication baselines on several different fully cooperative tasks. 
- The t-SNE analysis in Figure 5 and the entropy analysis in Figure 7 are convincing methods that uncover the mechanisms underlying improved performance of agents under this method.

Aspects requiring improvement

- The paper seems to rely on a strong assumption that all agents have the same model architecture, and the same autoencoding loss. This reduces the impact of the work. It would be very interesting to have further experiments that assess to what extent this constraint is required. See also comments on "limitations" below.
- An important baseline is missing. Since the "speaker" is transmitting exactly an autoencoded latent state, a good baseline would be one where the speaker transmits a latent state from the policy network, without autoencoding. This would help to establish that it is the autoencoding itself that is necessary, rather than merely the simplification in the multi-agent learning dynamics that comes from fixing the speaker policy to be a latent state.
- The authors invent their own environments rather than using environments from the literature. This means that their work cannot be quantitatively compared to previous papers like (in their citations) [7,13,16]. Could the authors add one additional environment from the existing literature to make this comparison easier? In particular, does their method outperform some of the methods with inductive biases? To what extent does their method regain the "topline" performance afforded by fully differentiable centralized models like DIAL?
- The results in Figure 3(c) are inconclusive. The authors claim that AE comm is better, but the confidence intervals on their plot suggest that no such conclusion can be drawn. Could the authors comment on the statistically validity of their claim? If the task is simply too easy to see a meaningful difference between the 3 methods, perhaps it would be better to try a harder task here?
- Parts of the Experiments section are unconvincing. In particular, I did not find the claims about positive listening and positive signaling sufficiently rigorous: it feels like the authors are observing a correlation rather than demonstrating a causal relationship. This section would benefit from reworking or removing. Moreover Section 5.4 misses important hypotheses about why adding RL to the AE loss doesn't help. This might be because the optimization problem is harder when you reintroduce the "chicken and egg" joint-exploration problem for learning speaker and listener policies at the same time. Alternatively, or additionally, it might be because the multi-objective optimization problem is harder than the single-objective optimization problem, so training requires more iterations. It would be useful to have further experiments here to disambiguate these possibilities with the representation-learning hypothesis which the authors present. 
- Some relevant citations are missing. For example there has been much recent work on learning latent representations to influence behavior in multi-agent systems, e.g. Xie, Losey, Tolsma, Finn and Sadigh 2020; Zintgraf, Devlin, Ciosek, Whiteson, Hofmann 2021. It would be useful to contextualise this contribution with reference to those papers.

Small comments

- In several places the authors refer to "cooperative" games where they could more precisely say "fully cooperative" or "common payoff" to distinguish from mixed-motive games, which are sometimes also referred to as cooperative.
- In lines 233 and 237 the names of the environments are the wrong way round, I believe. 
- In line 214, the authors could consider using the more precise definition of "cheap talk" from Farrer 1987.
- The authors might want to consider citing Dafoe et. al, "Cooperative AI", 2020 in their introduction; this review contains a section on the importance of grounding. 
- In their introduction, the authors may wish to also discuss the importance of the gestural modality in the origins of human communication, as in Tomasello, "Origins of Human Communication", 2008. 
limitations_and_societal_impact: The limitations of the work should be discussed more clearly. For example, is it essential that agents have identical model architectures, as stated at the bottom of page 3? If so, this seems like quite a strong limitation, since most real-world applications may involve heterogeneous agents. On the other hand, if this limitation can be reduced or mitigated, the authors should provide experiments to this effect, or strong arguments for future work. 

The societal impact is well addressed on page 9.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The goal of the paper is to show that learning a representation to ground language can improve communication in multi-agent settings.

Their method: each agent encodes an image and the other agents messages (through a small conn-net and mlp, respectively). A GRU-based policy outputs an action. A communication autoencoder predicts a message from the image feature and is trained with an auxiliary construction objective (i.e., to reconstruct the image).

They evaluate their model on a referential game with CIFAR images and two grid environments: one where the agents must reach a goal collaboratively and another where one agent has to open a door for another to pass.

They show that their method with an auto encoder strongly outperforms the non-autoencoder variants on the CIFAR task and door task.
main_review: Originality: their main contribution is the design of a multi-agent communication architecture that uses an auto encoder as an auxiliary loss.

Quality: The work is evaluated on a very limited set of tasks and there are no baselines they compare against. The main result (Figure 3) is an ablation. The experiments were also conducted without any hyper parameter tuning.

Clarity: The introduction is very high level, which makes it difficult to understandl. I think it would help to ground the introduction in more specific technical contributions in related work to make it more accessible.

Significance: It's difficult to contextualize the significance of the work because of it's lack of comparisons to other multi-agent work. 

Minor comments:
- The visualization on the right side of figure 1 is confusing. Specifically, why are there bidirectional arrows between the communication encoder and message (and policy network and action) if it's generating the message? It may help to visualize these as modules instead, like the encoders below.

Here are some other works on multi-agent interaction and communication that may be relevant:
Learning Latent Representations to Influence Multi-Agent Interaction. Xie et. al., 2020.
TarMAC: Targeted Multi-Agent Communication. Das et. al., 2019.
Emergent Linguistic Phenomena in Multi-Agent Communication Games. Graesser et. al., 2019.
limitations_and_societal_impact: The authors discuss societal impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes an approach towards encouraging the development of communication protocols in multi-agent reinforcement learning systems. The authors rightly observe that naive training of a communication policy according to task reward often fails, due to high variability in the initial messages, the difficulty of connecting this communication to reward, and lack of common grounding among the agents.

The authors' idea is to imbue agents with a message-generating mechanism cast as an autoencoder, where the agents aim to generate discrete messages that can reconstruct the observations of the agents. Thus, the message generated for a given timestep is essentially a discretized summary of an agents' observation at that time. This encourages the messages to be maximally informative about an agents' current state—something that is shown by qualitatively.

The authors evaluate this communication idea on a variety of environments, from a reference game variant to a few multi-agent minigrid environments, and show that their autoencoded communication outperforms (1) non-communicative baselines and (2) communication policies trained with standard RL objectives. They then have a few analyses that show that the agents are making use of the generated messages, in that (1) messages are informative about agents' current states (by design) and (2) messages affect other agents' entropy over decisions once they indicate a change in state.
main_review: Overall, this is an interesting and clear paper with a nice idea at its core, but I see an obvious missing comparison to related work and some more philosophical issues with the papers' framing that make me hesitant to recommend the paper outright. I'm looking forward to the other reviews and the author response.

## Strengths

- The paper is well written and clear.
- It has several clean experiments across a variety of environments that clearly show the benefits of autoencoded communication over non-communication and naive baselines.
- The idea of messages as autoencoding agent observations is an interesting one—it has appeared here and there in the literature (see missing references) but is clearly presented here and will be of interest to those working in MARL. It's likely that such an objective can be combined with more directly communicative objectives, to both encourage grounding and optimize for communication.
- While some more analyses could be done, I buy the authors' analysis in Section 5.5 that the agents learn to use and understand the messages, both due to the inherent *positive signaling* bias of the autoencoding objective and Figure 7 (which neatly shows that agents become more confident when their partner reaches a subgoal of the environment, and that this confidence can only be a result of the availability of the partners' message).

## Weaknesses

I have two major concerns with this work, and some other issues/questions.

### No comparison to Eccles et al.

Eccles et al. [7] is perhaps the most relevant paper for this work. It too suggests that naively optimizing communication via the typical MARL reward signal is brittle and often fails, and propose several auxiliary objectives. Yet the paper is quickly dismissed in L81-83 as "requiring task-specific hyperparameter tuning" and "limited effectiveness." Task-specific hyperparameter tuning is a fair criticism, and one claimed advantage of the current method could be its relative ease-of-implementation. But "limited effectiveness" is an unsubstantiated claim, and taken together these reasons are not sufficient to completely discard comparison to their proposed approaches. Yet no such comparisons are done in this paper.

### Is this "communication"?

My other concern is more conceptual: it makes me uncomfortable to imagine the experiments in this paper as true "communication" among agents, since the learned messages are not trained via any inherent communicative objective (i.e.: generate some message to maximize some reward given some dialog context). Rather, the messages are generated solely from an autoencoding objective: essentially learning a discrete representation of an agents' current observation space (without any regard to what influence this has on the ultimate success of the group).

One question I have for the authors is, is this a significantly different setup than simply doing centralized MARL, where each agent has full access to the other agents' observation space and internal state? Especially if the agents policies have some discrete latent variable representation which we could then label as "messages"?

In some ways this paper is an extension of the Eccles et al. paper: one part of it proposes a bias for positive signaling (that we generate messages correlated with our current state), and this paper takes it to an extreme where we actually don't consider *anything* besides positive signaling (i.e. the *only* objective is to maximize MI between messages and observations).

This means that many of the purported benefits of the "communication" learned by the agents in this paper seem given "for free". For example, the authors admit in L282 that the autoencoder agents are intrinsically optimized for "positive signaling" - but indeed, the messages are generated with no other objective besides positive signaling. Similarly, the visualization in Figure 5 suggests that "messages" cluster according to episode state. But this is unsurprising: we would expect an optimal policy's internal representations to also cluster according to episode state.

I can see this somewhat philosophical observation resulting in more concrete limitations of the proposed method: if the communication channel is very limited compared to the agents observation state—either because of limited bandwidth (i.e. vocabulary) or a cost to sending messages—then such an autoencoding objective may not work. (Indeed we need more details about the communication channel size, as I'll mention again below). In general, there might be many tasks when agents must learn to be more selective with the information communicated at each timestep, and a purely "descriptive" objective like the one presented here might fail, and we need agents that learn *what* to communicate. (This is the same reason why we humans do not constantly vocalize our environmental state every passing second). Admittedly, these constraints are not often considered in current MARL research, and more communicative objectives could be combined with the proposed autoencoding objective (i.e. a "first step towards grounding" as in L58-59, though again, this idea is similar to Eccles et al.).

### Other issues/questions

- Insufficient description of baselines in L223-224, specifically with regards to learning the communication via a separate policy network. Why is the communication module in the listener module instead of the speaker module (L224)? isn't the autoencoded representation of the speaker model precisely some sort of discrete message - here you are learning a policy network on top of this to go from this message to another message?
- L240: I don't understand how figure 3c shows that ae-comm agents are able to complete the episode much faster than other agents. All 3 lines seem to be within the shaded error regions? What does the error region represent? In cases that are visually ambiguous like this one, it would be good to have some sort of significance test.
- I'm very suspicious of the RL-communication policy not working above chance (no communication) in something as simple as the CIFAR referential game. This is an odd conclusion given the large literature on emergent communication in referential games that finds that we can learn successful (albeit uninterpretable) communication protocols with policy gradient, differentiable relaxations, mixtures of both, etc. Could the authors explain this result more?
- Would be nice to explicit calculate the proposed CIC metric (i.e. mutual information between messages and other agents' actions) in L274-276, which is precisely what Lowe et al. [22] suggests.
- It's not clear to me how Figure 6 illustrates positive listening. Autoencoder loss and task performance are correlated, but that's because we're jointly optimizing the two - it doesn't mean that the increase in task performance is *caused* by the improved autoencoder reconstruction. (I do buy the Figure 7 analysis more).

## Missing references

The idea generating messages that the model itself can interpret (in this case: reconstruct its own state) is similar to the obverter communication method proposed by Choi et al., Compositional Obverter Communication Learning From Raw Visual Input, ICLR 2018

## Minor

- L146 Missing Citation
- The communication spaces C could be more precisely defined across both environments.
- Figure 6 would be more clear if Loss was labeled "Autoencoder Loss"
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes to learn the communication protocol in an emergent communication environment via an autoencoder; i.e. they train an autoencoder to learn a discrete representation of the image part of the observation, and use that as the message that gets broadcasted to all other agents at each timestep. The paper shows that this achieves significant gains over learning the communication protocol from scratch in a decentralized way.
main_review: This paper is very well written, with several beautiful plots and figures, many auxiliary experiments that help understand what's going on. It -looks- like a NeurIPS paper.

At the core of the paper is a simple experiment --- rather than learning a communication protocol from the environmental reward, have agents share a discrete representation of their state at each time step. It turns out that this works very well in the environments that the paper considers (a CIFAR-based referential game, and some GridWorld environments requiring communication). The reason for this is pretty intuitive: communicating a discrete representation of your state is really all of the information you need to solve these tasks (i.e. it's all the information that the other agents need). Decentralized communication protocols are hard to learn, so it makes sense that they perform way worse. 

To me, this is an interesting experiment! I'm a bit surprised that nobody has thought of it before (to my knowledge), but autoencoding is definitely a strong communication prior, so it's a very reasonable thing to try.

That being said, I have a couple of concerns about the paper:
- I don't think the paper quite moves beyond "this is an interesting experiment" into "this could actually be useful / fundamental" territory. To do this, I would like to have seen either: (a) this approach compared to other methods, e.g. Jaques et al. or Eccles et al., showing that it outperforms them (I don't really like when a paper claims 'the effectiveness of previous approach X is limited, but don't compare to that approach); or (b) testing how the method works in environments where communicating only the state is not sufficient for coordination (e.g. communicating about past actions, or future intentions). I think the fact that doing RL on top of the autoencoder prior does *worse* is indicative that communicating about the state is indeed all you need for these tasks (or maybe that the RL component was poorly tuned).
- I'm not a big fan of the framing of the approach as 'grounding' the communication, implying that learning the communication protocol from scratch is 'ungrounded'. To me, if the agent learns to communicate with other agents in a grounded environment (such that it improves reward / is actually used), then that communication is grounded. The overall effect is that the paper seems to be a bit 'over-hyped'. This is reinforced by the allusion to cognitive science + evolutionary linguistics as motivation (which isn't wrong, per se, but feels a bit stretched).

That being said, compared to other papers in emergent communication I do think this paper comes close to meeting the bar (most other emergent communication papers don't compare to other papers in the field). It's still a nice paper. But I find myself wanting either more substance to the paper, or a modification in the framing.

All things considered, I'd rate this paper as borderline. 

Small points:
- Reference missing on the straight-through estimator
limitations_and_societal_impact: I think the limitations I mentioned above / framing could be improved, but otherwise no concerns.
review_title: null

Meta-review: Reviewers found the proposed method a simple yet intriguing contribution to the emergent communication literature. The authors addressed important initial concerns about baselines and ablations in the rebuttal period, and reviewers were satisfied with the new results. There was concern about whether sending autoencoded latents "really" constitutes communication, however I believe that makes the work all the more thought-provoking for the community. I expect this paper will be a useful contribution for other emergent communication research to use as a baseline or build upon.
Decision: Accept (Poster)