Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper argues that using discrete tokens in emergent communication fundamentally limits what agents can learn.

Method: Instead, they propose to learn an embedding matrix in addition to a token selection policy for use in emergent communication. The embeddings are trained in 3 different ways: supervised, learned from scratch, and manual specification. This approach is corroborated by analyzing a noisy channel reference game in an information-theoretic manner.

Their self-play tasks are in a particle-world environment:
- In _triangle_ a speaker must communicate which point of the triangle the listener should go to.
- In _9 points_, a speaker must communicate which point of the triangle the listener should go to. But, the 9 points are grouped into 3 groups of 3. The agent is rewarded based on distance, so going to a point in the same group as the target is less costly.
- In _uniform_, the positions of two landmarks are sampled randomly. A seeker observes the id of the target and an observer observes their locations. They must communicate to guide the seeker to the target.

They find that their proposed discrete prototype model outperforms a one-hot representation. In the _9 point_ task, the embedding space also reflects the point grouping.

In their zero-shot human communication experiments, they train speakers and listeners on a referential game over pairs of CIFAR images. They compare learned embeddings performance against class-label embeddings, and embeddings trained on sentences generated by humans for the task. The aim of these experiments was to show that agents were able to generalize to human communication and to generalize to novel communication settings (ie to images unseen before).
main_review: Originality: Adding embedding tokens to an emergent communication task is a sensible contribution. The creativity of this paper shines in the way the experiments were designed. E.g., the 3 evaluation tasks were well designed to understand how the agents change communication strategies as imprecision becomes less costly (triangle -> points) and how agents use discrete tokens to reason about continuous phenomenon. Changing out the different kinds of embeddings in the CIFAR experiments was also a clever way of understanding how the communication protocol changes across different tokens.

Quality: the experiments were thorough and well executed (e.g., median and standard error reported, experiment design and rationale was highly detailed.)

Clarity: The paper was very straightforward to follow. From the experimental section, I felt I could reproduce the results. Specifically, there visualizations communicated the key concepts and environments very effectively.

Significance: Adding embeddings as tokens in emergent communication could open up many avenues for analysis going forward. E.g., the paper already demonstrated how embeddings can better demonstrate how humans understand emergent communication protocols and how communication can be transferred to new settings (such as differentiating between new images).

limitations_and_societal_impact: Yes, they address limitations under the contributions section.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper extends research in multi-agent emergent communication using discrete tokens. The agents learn to communicate across a noisy channel. The main idea is to have the agents learn to embed discrete tokens into a continuous space, similar to NLP techniques such as word2vec. The authors show that the learned tokens are interpretable to other agents in self-play. They also show that these messages can be adapted to understand embeddings produced by BERT or human labels, and that humans can in turn interpret the embeddings.
main_review: Overall, I enjoyed reading this paper. The interpretability the actions and communications of an RL agent is an important problem which the paper clearly and thoroughly studies. The main idea is simple to understand and well motivated. The results are in simple settings.

Pros: 
1) Equations are explained clearly and the paper is well written.
2) Experimental results show clear improvement over the baselines in the first two environments.
3) Experiments and ablations are thorough. I particularly liked the results on human-interpretability of the agent messages. 

Cons:
1) It would be nice to see some agent-only environments beyond particle worlds, and tasks beyond navigating to the correct -landmark. Similarly, it would be interesting to see performance on a slightly larger set of image classes. 
2) The prototype agent doesn't seem significantly different from the cont agent on the zero-shot reference game. 
3) 100 supervised examples in the training set is not quite the same as zero-shot learning. 

Questions:
1) Do the agents communicate once per episode, or continuously? If the latter, do the messages change? 
2) What are the one hot agent messages when c < z? 
3) In Section 5 Figure 3, why are the final locations only roughly clustered? Given one of five messages, the seeker agent should reach one location and stop to maximize expected reward. Figure 4 seems to support this. 
4) In Appendix D, it would be interesting to see the non-identity reward matrix. 
5) For the zero-shot human understanding experiments, are the held out classes always 1 animal and 1 vehicle? Is there any interesting pattern in human errors e.g. classes that are conflated consistently? Was the same experiment done with the cont model? 

Details:
- (Section 5 Table 1) - Some additional noise ablations would be interesting.
- (Section 5 Table 2) - Some additional experiments varying z would be interesting. 
- (Section 6.2 Line 331) - is essence -> in essence.
- (Section 6.2) - Add reference to the results in Appendix F/Table 8. A reference to Figure 8 is also helpful. 

limitations_and_societal_impact: The authors address the limitations and social impact of this work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors present theoretical and experimental results that support the use of employing discrete tokens derived from continuous word embeddings in place of the standard practice of using one-hot vectors as symbols ('words') in emergent communication. The new approach is evaluated using particle-world environments (Lowe et al 2017), in terms of zero-shot reward in a standard referential game on CIFAR10, and in human evaluation. The main contributions are 1) a practical implementation of the technically sound idea of using discretised embeddings to reflect semantic similarities in language, instead of orthogonal one-hot vectors 2) new theoretical results on the optimality of one-hot tokens as vocabulary, and a theoretical analysis of the cases when one-hot tokens might be suboptimal (when the cost of errors or prior over tokens are not uniform, or when the number of tokens and communication dimensions are not equal) 3) experimental results of using those embedding-based tokens, in terms of robustness in noisy environments, interpretability and zero-shot generalization in agent-to-agent and human-to-agent communication.  The authors examine two behaviour shaping constraints that can be used to learn those tokens: 'lexicon-setting' (manually specified and non-trainable tokens) and token grounding using a small supervised set.
main_review: ## Strengths:

The authors expand on the existing work on bridging emergent communication and NLP (Lazaridou et al 2020) and propose a sound implementation of word embeddings in multi-agent communication with a discrete channel.

The results are promising in terms of practical human-agent communication.

The tokens are shown to be meaningfully related (e.g, a mean of the tokens representing locations $\alpha$ and $\beta$ represents the actual mean of the two locations on a grid), which opens new possibilities in terms of interpretable communication protocols and zero-shot generalization in multi-agent tasks.

The authors include anonymised code in the submission. 

In terms of writing and structure, the submission is significantly above average: for example, the authors clearly state the hypotheses which motivate their experiments. The appendix is detailed and the figures are of (above average) quality. Well done!

The authors motivated their answers in the Checklist.

The writing is quite dense in terms of the amount of technical content, but it is clear and coherent, with no unjustified claims.

## Originality:

The contributions are original and the previous work in multi-agent communication is appropriately cited.

## Quality:

As mentioned in the Strengths section, both the content and the presentation are of high quality.

## Questions to the authors:

In Table 3, how many runs did you use to compute the mean reward and standard errors?

Have you observed structural, semantic or pragmatic language drift (following Lazaridou et al 2020) in your experiments?

## Significance:

The paper proposes a way of deriving a semantically interconnected yet discrete vocabulary in multi-agent communication, which might influence the research on discrete, interpretable communication channels. Zero-shot generalization, interpetability and various means of incorporating a priori knowledge in the agents' protocol based on the task are of interest to the research communities (emergent communication, deep RL) working on multi-agent communication.

## Notation:

Line 108: ‘ in a matrix of $z$ tokens’ suggests that $z$ is a number whereas in Equations (1)-(4) it is used as a set ($i \in z$). Please change the limits of the sums and the maxima to something like $1 \leq i \leq z$.

Line 124: The usual notation for variance is $\sigma^2$, $\sigma$ suggests standard deviation. 

## Typos: 

Line 7: ‘architectures that enables’ -> ‘architectures that enable’

Line 12: ‘allowing them communicate’ -> ‘allowing them to communicate’

Line 138: ‘$T_{z \times c}$’ -> ‘$T_{c \times z}$’

Line 244 and line 245: a hyphen should be replaced with a minus -> $-100$, $-96$

## Latex:

Line 93 $\forall i \in [1,N]$, $A_i$ is a set of actions

'...' in equations should be replaced with '$\ldots$'

## Linguistic comments:

Line 117 ‘We seek to find’ -> ‘We seek’

Line 138 ‘which then is multiplied’ -> ‘which is then multiplied’



limitations_and_societal_impact: The paper lacks the Broader Impact section. Suggestions on how the authors can comment on the broader and societal impact to strengthen their paper:

Discuss the examples of real-life applications of agent-to-agent communication and human-to-agent communication, and the benefits of human-interpretable protocols (refer to the Section 6.2 Zero-Shot Human Understanding of Tokens: the token-based approach achieves higher results than a BERT-based model, which is widely used).

Regarding social impact, the authors could discuss the possible dangers of using the work of human annotators on Amazon Mechanical Turk (AMT) (Section 6.1.).

The authors briefly commented on the biases found in the existing word embeddings, in the future work paragraph. This point could be developed in the Broader Impact section to strengthen the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors extend previous work on communication in multi-agent reinforcement learning in 2-agent games involving a speaker and a listener, where the speaker generates a token and the listener performs an action informed by the token it receives. The contribution of the present paper is to replace the usual one-hot encoding of tokens with learned tokens in a richer, emergent encoding space. A mathematical justification for this step is given, and its advantages are evaluated in a variety of settings, including settings where humans take the part of either listeners or speakers. These experiments, which show interpolation and zero-shot generalisation, vindicate three hypotheses the authors set out concerning the advantages of the proposed learned encodings over one-hot encodings.
main_review: The paper offers a well motivated contribution to the sub-field of multi-agent RL agent communication, and presents a solid evaluation of the resulting model that demonstrates its advantages. The paper is mostly well written and clear. Overall the paper looks good to me, and deserves to be accepted.

For me, section 3.2 (the mathematical motivation) didn’t add very much to the paper, and the main findings stand for themselves. But other readers might take something useful away from the maths here.

I have a few questions and minor issues.

The authors describe their method as using "prototypes". But what exactly is a "prototype"? Is this the same as a learned token (so there are z prototypes)? I assume this is the case. But then “prototype” seems like a strange word to choose, and I'm not sure what the intuition is. Either way, the term isn’t defined anywhere, as far as I can see. Maybe the authors could remedy this.

In Tables 1 and 2, why do you report the median rather than the mean?

Line 200: “Agents were trained and tested in fixed teams” - Presumably a “team” is a pair of agents: a listener and a speaker? Maybe the authors should make this explicit.

The nature of the listener agent in the particle environment could be described briefly in the main text (if there's space). I gather from the Appendix that it moves according to (somewhat) realistic physics (as opposed to making discrete steps in a grid world). Do its actions comprise a continuously-valued force-direction pair?

Words in human language don’t typically cluster according to semantic in the way advocated here (the “arbitrariness of the sign”). Perhaps, if there's space in the final paper, the authors could discuss this issue. Maybe the richer encodings here are analogous to multiple words rather than single words? (Eg: "fluffy animal" and "dangerous animal" are both semantically close and lexically similar.)

TYPOS

Line 139: upper-case theta shoud be lower-case

Equation 5: Should T_{c x z} be T_{z x c} as in line 138 of the text? Or is it the text that has them the wrong way around? (I don’t think you need those subscripts in the equation anyway)

Line 331: “is essence” -> “in essence”
limitations_and_societal_impact: In several of the experiments, the "continual" encoding method out-performs the "prototype" method. Perhaps the authors could discuss this.
review_title: null

Meta-review: This work studies referential games in between a speaker and a listener and proposes to replace the commonly used one-hot encoding of tokens with learned tokens in an encoding space. Moreover, the authors designed a good experimental setup to help them test and understand the effect of their approach and a thorough evaluation, including evaluation with humans.

All reviewers agree that this is a solid contribution to the field of emergent communication, positively commenting not only about the general structure of the paper on putting forward a hypotheses and then thoroughly testing it with targeted experiments, bit also commenting on the creative experimental setup but also the interpretability aspects of their proposed method.

This is a good contribution to the field of emergent communication and hence I suggest acceptance of this work.
Decision: Accept (Poster)