Emergent Discrete Communication in Semantic Spaces

Abstract: Neural agents trained in reinforcement learning settings can learn to communicate among themselves via discrete tokens, accomplishing as a team what agents would be unable to do alone. However, the current standard of using one-hot vectors as discrete communication tokens prevents agents from acquiring more desirable aspects of communication such as zero-shot understanding. Inspired by word embedding techniques from natural language processing, we propose neural agent architectures that enables them to communicate via discrete tokens derived from a learned, continuous space. We show in a decision theoretic framework that our technique optimizes communication over a wide range of scenarios, whereas one-hot tokens are only optimal under restrictive assumptions. In self-play experiments, we validate that our trained agents learn to cluster tokens in semantically-meaningful ways, allowing them communicate in noisy environments where other techniques fail. Lastly, we demonstrate both that agents using our method can effectively respond to novel human communication and that humans can understand unlabeled emergent agent communication, outperforming the use of one-hot communication.


Reviews and author responses:

 Thank you for your response - I appreciate your points, and have to say I find the discussion enjoyable.

That being said, I still feel the paper is a little short of the mark in terms of experimental coverage and grounding in existing methods, and I haven’t been convinced to change my score.

You are proposing a method that can apply quite generally to emergent communication settings, so it would be natural to include results on a few different tasks. The particle world experiments are interesting, but the paper doesn’t have any agent-agent referential games. It does not seem out of scope for this paper to show whether your method works for those too, especially since you do use those games for the human-agent experiments. If your method works there too, your proposal becomes significantly stronger; if it doesn’t, that is interesting and worth investigating.

 We thank the reviewer for their thorough response. We are glad to have clarified the principal concern.

Regarding the specific points raised by the reviewer, they are all well-taken.

We appreciate the necessity of motivating noise in emergent communication environments, as it does not appear to be standard. Literature in psycholinguistics suggests that humans reason about uncertainty over language ("Eye movement evidence that readers maintain and act on uncertainty about past linguistic input" 2009 Levy et al., "Rational integration of noisy evidence and prior semantic expectations in sentence interpretation" 2013 Gibson et al., "Noisy-context surprisal as a human sentence processing cost model" 2017 Levy et al.) so perhaps we should push for a broader adoption of noisy environments in artificial communication. Robustness via compositions of tokens also appears promising, almost like how error correcting bits are added to messages in standard error-correcting codes, and the provided reference appears extremely relevant. We further found another related work, published in 2021: "A Joint Learning and Communication Framework for Multi-Agent Reinforcement Learning over Noisy Channel" (https://www.imperial.ac.uk/media/imperial-college/research-centres-and-groups/ipc-lab/A-Joint-Learning-and-Communication.pdf)

The further notes on zero-shot understanding are appreciated and will be used in future work.

 Thanks to authors for their detailed response to my review.

The authors response has indeed helped me clarify their approach to this paper and alleviated my worry that I'm missing something completely obvious here re: the use of one-hot tokens in emergent communication. The two purported benefits of the prototype based approach are (1) robustness to noise and (2) zero-shot understandnig.

**Re: (1) robustness to noise**

I agree with this potential benefit and mentioned this in my original review (*some of the ideas have merit if we assume a noisy channel...*). However, this is actually a highly non-standard assumption in the emergent communication literature, which was the cause of my confusion; most work assumes perfect transmissibility. (Perhaps it is a mistake for the community to treat noisy communication as non-standard; regardless, it needs to be taken into account when contextualizing the current work with related work). By the way, this makes the following paper more relevant: https://openreview.net/forum?id=ZbXlSL_xwtA.

Actually, besides that paper, I know of no contemporary emergent communication work which investigates communication strategies under a noisy channel. If authors can point to enough literature where this assumption is taken for granted, I would revise my comments about the framing being incorrect.

In context with the existing literature, it thus seems imprecise and confusing to frame the paper as: "compared to the existing work on emergent communication, prototype-based communication does better." The framing should instead focus much more on introducing the noisy channel assumption: "existing work on emergent communication assumes perfect transmission, but realistically, we would expect noisy communication. under the assumption of a noisy channel, prototype-based approach does better than the typical one-hot approach."

That being said, even under this framing of investigating what the optimal communication strategies are under noisy channel assumptions, some more rigorous baselines could be explored, such as those proposed by Reviewer bPM2: it is possible to have discrete communication without the extreme assumption that you only can transmit a single atomic one-hot token, and there are no relations between the different tokens. This is precisely why, instead of human language consisting of infinite unique tokens each representing the meaning of a completely different sentence ("Sentence1", "Sentence2", ...), sentences are composed of reusable words/units with consistent meanings. This is a property of human language which likely makes it more robust to noise, and is also partially why work in emergent communication studies sequences of symbols. Case in point:

>  In another society, people communicate via composable words (e.g., “read,” “able,” and “readable”). 

This baseline is not actually explored in the paper.

**Re: (2) zero-shot understanding**

This is also a fair point which may be easier with prototype-based communication, though this seems to rely on some source of external knowledge to embed novel utterances into the right semantic space (see my discussion on Sentence-BERT below). As in point (1) above, if we are restricted to transmission of a single discrete token with no similarities among tokens, then I buy that there is no zero-shot generalization ability. But there *is* possibility of communicating novel utterances with discrete tokens by composing together the reusable words to form new meanings - i.e. language compositionality! Generalization to OOD inputs is a common test for emergent communication work in the literature and is usually done assuming communication of sequences of tokens for precisely this reason.

**Re: Zero-shot sentence-BERT**

> No agents actually use a language model in their architecture. Rather, Sentence-BERT is used to generate a small amount of supervised data, whereupon Sentence-BERT is discarded and the agents are trained with the data. By excluding some classes from training of the agents, we therefore create zero-shot reference games. When transforming human language to communication, we use Sentence-BERT, but then that communication vector is given to the listener. Thus, Sentence-BERT is not part of the listener agent, as suggested by the reviewer.

It is true that Sentence-BERT is not literally part of the listener agent, but it seems like there is significant external knowledge being injected into the agents insofar as the semantic space is constrained to look like that of Sentence-BERT space, which has been trained with a vast amount of data such to enable meaningful semantic interpolation among open-ended English. Thus the lexicon fitting objective feels somewhat like distillation of BERT's knowledge into the semantic space of the agents. And finally to accept novel communication, human utterances are embedded (via Sentence-BERT) to form a meaningful prototype. I know this is not literally part of the listener, but in some sense the listener "depends" on Sentence-BERT knowing that, e.g., "apple" is a similar class to "orange".

Taking the authors response into account, I will raise my score to 4 (borderline 5). If the authors propose to revise the framing more to more clearly contrast with the existing work (i.e. better putting the noisy channel assumption more front-and-center), I would raise my score to a confident 5. And if there are less restrictive one-hot baselines explored (as also proposed by Reviewer bPM2), I would consider raising my score to 6 or higher.

This paper presents a study of emergent communication where the communication channel is, according to the authors, composed of a finite set of continuous embeddings, rather than one-hot tokens representing unrelated words, as is typically used in the literature to simulate linguistic transmission.

The authors first show that, in a noisy channel communication task where the speaker selects from a finite set of embeddings and the listener's goal is to identify the index of the sent embedding, one-hot vectors are only optimal under certain restrictive assumptions, one of them being when the number of embeddings (vocab size) equals their dimensionality. They use this finding to motivate a communication game where the speaker selects from a set of dense embeddings, and show both in a suite of multi-agent RL tasks an an image-based reference game that semantic embeddings outperform traditional one-hot communication channels in various ways, including:

1. They achieve higher reward and communication success than one-hot tokens
2. Embeddings are interpretable and cluster according to the semantics of their corresponding messages, unlike one-hot vectors that are orthogonal and have no predefined relationships between each otheer
3. Communication channels with semantic embeddings allow a human to play the role of both speaker and listener in this communication task (by generating novel utterances which are then embedded and passed to the listener, or by asking humans to interpret messages sent by a speaker assuming some 2D visualization of the semantic space)  I was very confused by this paper. Unless I am missing something obvious, I feel like this paper has a fundamental understanding of how communication via one-hot tokens is treated in the emergent communication literature. Like most NLP applications, in speaker/listener setups there is indeed a component in the model where distinct words are treated as one-hot tokens with no inherent semantics. However, these one-hot tokens are invariably *embedded* by the listener, where the one-hot tokens are used to lookup continuous semantic word embeddings, which are then provided for further computation in the listener (e.g. via an LSTM or Transformer architecture). Often the speaker has its own embedding matrix, which helps it understand the semantics of its generated message during autoregressive language generation. In other words, these are *precisely* the word embedding mechanisms that the authors claim in Section 2.2 constitute motivation for their present idea.

In other words, I see no difference between what the authors propose and what the approach that standard emergent communication setups already use. Speakers already produce "discrete communication in semantic spaces" insofar as the generated one-hot tokens are used to index into a listener's learned embedding matrix. And this embedding matrix presumably has all of the benefits that the authors propose in this paper: semantic clustering (Figure 2), interpolation (Figure 4), interpretability (6.2), potential for zero-shot novel communication (6.1), etc.

This made me very confused as to what the "one-hot" baselines are - it is really hard to imagine an architecture that does not have an implicit embedding matrix (e.g. even if the vectors are fed as binary features into an MLP, the first dense layer serves the role of a word embedding matrix and presumably learns some semantics). I was thus very interested in how these one-hot baselines are formulated, but unfortunately the paper offers almost no architectural details on the agents in both the RL and reference game experiments.

I think some of the ideas in this paper have merit - if we assume a noisy channel then I can see why communicating only distinct tokens might be too corruptible of a communication channel, which motivates communication via more "continuous"-y measures (presumably this is why we have words composed of reusable/partially redundant parts like characters and phonemes). But noisy channel communication is also an assumption that is not often explored in emergent communication, and so the derivation of the suboptimality of one-hot tokens in Equations 1-4 feels a bit out of place: here, the communication task is presented as a Shannon-style encoder-decoder noisy-communication task, where the goal of the listener is to recover the message. But the literature tends to formulate the objective in terms of listener performance on some downstream task, rather than trying to reconstruct the message, because reconstructing the message is assumed to be trivial. The framing of the paper should be updated to take this into account.

Finally, some of the conditions that must be violated for one-hot tokens to be suboptimal don't make much sense to me. For example, in L130, why would the number of tokens and communication dimensions not be equal in discrete communication? The way to specify a vocabulary of size N for the agents is literally to define exactly N unique one-hot vectors corresponding to each word? If there are more token dimensions, then one-hot tokens don't uniquely identify each token. Similarly, if the number of tokens is less than the token dimensions, the extra token dimensions are completely superfluous. Again this doesn't make much sense to me given my own priors on how discrete communication is treated in the literature.

To summarize, this paper left me very confused, as it seems to fundamentally misunderstand how discrete communication is treated in emergent communication. I don't understand how the presented prototype models differ at all from standard emergent communication models and am not sure what kinds of existing models the one-hot baselines in the paper instantiate, due to lack of architectural details. This means I can't recommend acceptance, as the experiments in the paper do not seem to support some novel contribution, but rather the status quo of emergent communication setups. Indeed, similar experiments into the interpretability of learned embeddings (Lazaridou et al. ICLR 2017), generalizability to human language (Lowe et al., ICLR 2020), etc. have already been explored in depth. I am open to revising my review, and look forward to the authors response, which may be able to clear up my confusion.

## Minor

- Can figure 1 have more labels (what is student, teacher, etc, can you give an example of what a goal looks like?) I assume t_i is the target but I didn't see it in the text.
- "Uniform" as a term feels overloaded in Section 5. E.g. Table 1 Dist = Uniform, and the Uniform environment.
- Can we have z in table 1? also this is minor but I had trouble remembering what z and c correspond to. Could instead e.g. use |C| to denote vocab size, and h_c or c_dim for dimensionality.
- L305 why "BERT-like"? Can you specifically name the model?
- My understanding is that the Appendix should be submitted as supplementary material, otherwise this is possibly a page limit violation.
- Meaning of "In" and "Out" in table 3 could be made more clear. Maybe mention parenthetically in L311-312
- The framing of the 6.1 experiment is a bit odd. I don't think it's "novel communication" if the listener model includes a pretrained BERT model which is trained on a massive amount of external data and can embed a variety of expressions into a semantically meaningful space (the agents are precisely communicating in BERT embedding space due to the lexicon setting technique). BERT must be treated as a component of the listener, and if human generated referring expressions aren't novel to BERT, they aren't really novel to the listener. Yes

 We thank the reviewer for their time and address their specific concerns below:

1) The number of prototypes is a hyperparameter. You are correct in noting that the number of prototypes is a hyperparameter that must be chosen a priori. Admittedly, many discrete communication methods suffer this same drawback: one-hot based communication relies upon the communication dimension hyperparameter. Having too few prototypes effectively underparametrizes agents. The bottom row of Table 1 effectively studies such a scenario: although the agent has 9 prototypes, the high level of environment noise reduces the prototypes to form 3 large clusters. Conversely, when agents are overparametrized, additional tokens are often ignored (i.e., never emitted by the speaker) or redundant (i.e, very near other tokens and eliciting similar responses).

2) Results from Table 1:

    2.1) Uniform vs. Skewed. It makes sense that all agents performed better in the skewed scenario than the uniform one. As discussed throughout the paper, the prototype agent can exploit distributional asymmetry for increased robustness. Even the one-hot agent can benefit to some extent in this limited scenario: in the skewed scenario the one-hot agents were still unable to distinguish any messages because of the high amount of environment noise, but they could go to the mean of the two most likely targets instead of the mean of the 3 targets. 

    2.2) Z vs. C. We changed the relative values of Z and C in Table 1 intentionally, in order to study the effects of over- or under-parametrizing prototype agents. In the Triangle environment in which z < c, it is indeed unfair, but it favors the one-hot agent! Prototype- and one-hot-based agents both communicated in 9 dimensions, but the prototype agents only had 3 tokens available, whereas the one-hot agents had 9. Thus, one-hot agents were given an unfair advantage.

    2.3) Triangle and 9-points environments. Neither of these environments had “absorbing states.” All episodes lasted 50 timesteps, as specified in the first paragraph of Appendix B. The optimal listener policy was therefore to drive straight to the target and stay nearby. 

    2.4) 9-points reward: it is unsurprising that the 9-points environments resulted in higher reward than the triangle environment, given the environment noise values specified in Table 1. Specifically, the 9-points environments used less environment noise, so agents were better able to drive straight to the target. Furthermore, by comparing rows 3 and 5 from Table 1, we are able to identify that prototype-based agents in noisy environments (row 5) achieve similar reward to agents with only 3 tokens with no noise (row 3). We make this point in lines 244 to 246.

3) Human-agent experiments

    3.1) Supervised and RL training: Prototype and cont agents are trained with a small amount of data (100 points) to align with sentence embeddings. The RL training enabled agents to better generalize, supporting conclusions from prior art in combining supervised and RL training (as noted on line 304).

    3.2) One-hot agents. As the reviewer notes, one-hot agents do not support continuous embeddings. That is why, as noted on line 306, one-hot agents used the one-hot representation of classification labels. We used these one-hot labels for supervised data and for “Label” experiments. Thus, one-hot agents were treated quite similarly to cont or prototype agents, just using one-hot embeddings instead of the BERT-based embeddings. One-hot agents do indeed perform well in self out; this corresponds to the speaker agent producing a one-hot vector based on a novel image and the listener agent interpreting that vector. Agents can perform well in self-play experiments like this by overloading previously used tokens (e.g., using the one-hot for “dog” to communicate about a “cat”). Thus, this corresponds to zero-shot reference to novel image classes, but not interpretation of novel communication. Lastly, we note that in all reference game experiments, one-hot agents were given a slight advantage compared to all other agents by using a 10 dimensional communication dimension (instead of 3), which reduced the relative effect of environment noise. This could explain the slightly better performance of one-hot agents that the reviewer observed.

    3.3) Zero-shot understanding. We maintain that we truly have assessed aspects of zero-shot understanding of agents’ tokens. Admittedly, this understanding is relatively crude by only separating animals and vehicles, but this nevertheless represents the fact that humans can understand an unlabeled token based only on its value. That is, simply, zero-shot understanding.

    3.4) Reward matrix dependency. We conducted some ablation studies with the reward matrix in Table 5 (Appendix D). Given that our paper already filled the 9 pages of the main paper limit, and added 10 pages of appendices, further experiments for a full range of reward matrices were not included.


 We understand the reviewer’s interpretation of our work, and seek to clarify the contribution of our method below.

Fundamentally, we differ from prior art in that the communicated messages are selected from a discrete set of vectors in a continuous space, unlike prior art that communicates via one-hot vectors. We acknowledge that, as the reviewer notes, the first thing standard listener agents do when receiving a one-hot vector is to effectively embed it into a semantic space. We argue in this paper, however, that there are substantial benefits conferred by communicating via the “embedding” instead of one-hot tokens.

Intuitively, imagine that humans are developing a communication protocol. In one society, people communicate by saying “Word 1” or “Word 2,005.” In another society, people communicate via composable words (e.g., “read,” “able,” and “readable”). Both societies may have equally expressive languages and induce similar beliefs in listeners. Fundamentally, however, the first society, with only indexed words, will be unable to support a new speaker who has limited knowledge about this language and wishes to communicate a combination of two words. Furthermore, in the first society, if a listener mishears a word and hears “Word 14” instead of “Word 15”, the potential confusion would be significant if the two words had very different meanings. Conversely, in the second society, by communicating via messages in which the form itself of a message denotes some meaning, such costly errors would be less likely, and zero-shot understanding of composition of words may be supported.

In this paper, we argue that our hypothetical society one is to society two as one-hot agents are to prototype agents. Both are theoretically equally capable in the absence of noise or zero-shot understanding, but in more challenging domains, prototype-based agents are better.

Reformulating our work and the one-hot baseline:

In our work, speaker agents produce one of a discrete set of continuous vectors as communication.

In our one-hot baseline, speaker agents produce a one-hot vector as communication.


This distinction matters in several scenarios:

1)  When communicating over a noisy channel, in which noise is added to the communication vector, there are important differences between using one-hot vectors and continuous tokens. The agents can learn to move the continuous tokens to optimally be robust to environment noise, unlike agents that only communicate via one-hot tokens.

2) When communicating novel semantics, learnable tokens enable a continuous interpretation of communication, whereas one-hot tokens do not. For example, consider the human-agent experiments in which we used word embeddings to label images belonging to a class that had never been seen before, e.g. a horse. Using our agents, humans could directly encode text to a continuous vector, which the listener interpreted. Conversely, if agents were trained with only one-hot communication, it is not obvious what the speaker should produce to elicit the correct listener response. Dissecting the listener agent by removing the first layer and trying to interpolate between known classes is theoretically possible, but it is not obvious what the correct interpolation would be, nor does it feel correct for the speaker agent to use part of the listener agent’s weights to formulate communication. 


We appreciate the minor comments and address some of them below:

1) BERT-like model.\
We left out the specific name of the model for space, although the reference (33) includes the name: Sentence-BERT.

2) Zero-shot understanding.\
No agents actually use a language model in their architecture. Rather, Sentence-BERT is used to generate a small amount of supervised data, whereupon Sentence-BERT is discarded and the agents are trained with the data. By excluding some classes from training of the agents, we therefore create zero-shot reference games. When transforming human language to communication, we use Sentence-BERT, but then that communication vector is given to the listener. Thus, Sentence-BERT is not part of the listener agent, as suggested by the reviewer.


Ultimately, the single goal of this rebuttal is to convey to the reviewer the importance of communicating in the continuous space instead of via one-hot embeddings that listeners are then responsible to embed. We hope that the intuitive example provided in this rebuttal provides motivation that bolsters the results already presented in the main paper.



 We appreciate the RIAL vs. DIAL perspective, as our approach seeks to bring some of the expressivity of continuous communication to discrete methods. As noted in the paper, we adopt a decentralized framework in which agents do not share gradients or weights directly; the listener observes noisy speaker communication, which is ultimately a function of the speakers’ weights but is more properly characterized as a standard action.

Similarly, the reviewers comments on projection and embedding are well-taken. Indeed, one may think of agents communicating via one-hot vectors as effectively re-embedding tokens in their first layers. Likewise, we were inspired by word embedding techniques from NLP. Although subtle, the contribution of our work is to say that embeddings are the right communication space, rather than one-hot vectors. This decision is fundamentally important in noisy environments, in which the cost of confusing tokens can vary among pairs of tokens. 

The reviewer noted that our “technique encourages similarity of tokens, rather than similarity of responses…” This is slightly inaccurate. By training in noisy environments, agents that use our technique learn to group tokens that elicit similar responses. Here, similarity of response is dictated by the reward if one confuses tokens (e.g., moves to one landmark instead of another). Thus, our technique, which merely allows tokens to be learnable, allows for desirable semantic properties of tokens to emerge.

The reviewer’s comments on VQ-VAE are insightful and appreciated; we plan on extending this work to examine other discrete representation methods. Indeed, our prototype agents were dubbed “prototype” agents because of initial inspiration from prototype-based classification methods, in which representations must cluster around a set of learnable prototypes. This is clearly connected to VQ-VAE work. We even performed initial experiments in this direction, in which continuous messages were sent and encouraged to cluster around discrete tokens. In early experiments, however, we found that clusters often failed to become tight enough to fully rely upon just the prototypes at test time. Fundamentally, this work is certainly interesting and worth pursuing, but with 9 pages for the main paper and an additional 10 pages dedicated to appendices, this additional work seemed out of scope for the current paper.

 We thank the reviewer for their analysis of the paper, but we seek to clarify the framing of our work. Our technique is part of the emergent discrete communication literature. We therefore assume prior motivation for discrete emergent communication is valid, and we primarily compare our method to one-hot based communication. A continuous baseline, as noted by the reviewer, is a relaxation of discrete methods, but we include it merely as an optimal baseline. If we accept the premise that we wish for agents to learn to communicate via a discrete vocabulary, our method does not need to outperform continuous agents. Studying the emergence of discrete communication among continuous agents is outside the scope of this work.

Our environments were chosen precisely to identify when one-hot and prototype-based communication differed. Each environment setup is motivated in the main text (e.g., for the triangle environment, we used different priors over targets, and for the 9-points environment, we produced a non-identity reward matrix). In the uniform environment, the one-hot agent actually had a greater communication dimension than the prototype agent (5 vs. 3), so it is unsurprising that one-hot did at least as well as prototypes. This setup and the one for row 4 of Table 1 correspond to scenarios specifically designed for one-hot optimality, as requested by the reviewer. Because prototypes are a relaxation of one-hot tokens, this optimality is demonstrated via nearly identical performance for the two techniques.

We appreciate the desire to explain scenarios in which one-hot tokens are suboptimal. For space reasons, we included these examples in Appendix A (page 15). 

We address the reviewer's detailed comments below:

1) Unclear 3.2 notation.\
We will seek to improve the notation of these equations by noting that the tokens are emitted by the speaker, and the listener must identify a token given a noisy observed version of the emitted token.
2) Proof before games.\
All of our experiments were conducted in environments that correspond to the scenarios studied in this formalization. Section 3.2 highlights the importance of non-identity reward matrices and non-uniform priors. The triangle environment is used to study non-uniform priors and the 9-points environment is used to study non-identity reward matrices. The reference games similarly use non-identity reward matrices.
3) Self-play environment details.\
Full details of the self-play environments are listed in Appendix B; we were only able to retain high-level descriptions in the main text due to space constraints.
4) Score boundaries.\
We will provide random and optimal baselines in the tables.
5) Motivating choice of noise values.\
The analysis is dependent on the noise values, as it must be high enough to “matter” and low enough to not drown out all signal. We discuss the choice of noise values for particle environments in Appendix B.
6) Human-agent experiment supervision.\
It is necessary to use supervised data to “ground” the agent communication to the right concepts. Such supervision is widespread in the literature, as indicated by the references on line 304. In essence, without supervision, agents could learn the right “vocabulary” but not the associated meanings.

The paper explores how discrete latent space can be used in emergent communication instead of one-hot vectors. Indeed, the authors claim that one-hot vectors may be too restrictive, while continuous messages provide an interesting venue for research.

More precisely, they propose to replace the use of one-hot vectors by a set of tokens derived from a continuous space (embeddings), that are either learned or inherited from another model. They show that using embeddings leads to:
 - a communication optimisation
 - robustness in noisy environments
 - better understanding in machine-human settings

The main contribution of the paper is thus the introduction of a new model for the communication channel that leads to better communication performances compared to one-hot vectors.   Authors accurately point out the limits of the standard use of one-hot vectors in language emergence. They provide a method that fairly improves communication performances with several scenarios. However, the paper lacks finer analysis to strongly complete language emergence literature. In particular, it could have been fruitful to add a descriptive approach, e.g, under which environmental constraints agents spontaneously develop discreteness within a continuous communication channel ? Finding such constraints would be a strong argument to use embeddings in language emergence as a standard lexicon. Here, the discrete communication of continuous vectors does not emerge per se. 


*Originality/Significance/Quality*

Choice of model: The originality of the paper is to transfer tools inherited from NLP (word embeddings) in order to improve communication performances. Authors are right to notice that researchers  in emergent communication mostly promote constrained communication channels (eg. [1,2]). It is often motivated by:
 -  the sake of simplicity (eg. [3])
 - language emergence studies aim at studying core properties of languages without the need of reaching optimal performances (eg. [4,5])

The paper's goal is here to:
- introduce the notion of continuous semantic space
- show how it improves performances compared to one-hot vectors

The paper mostly considers 3 baselines (one-hot vectors (a), discrete set of continuous vectors (b), continuous vectors (c). Based on the different scenarios, authors show theoretically or empirically that either (a), (b) and (c) are optimal/more performant. Since (a) is a constrained version of (b), and (b) a constrained version of (c), a nice result would have been to show that, without directly constraining the communication channel, either case (a), (b) or (c) emerge only based on environmental constraints. In other words, does discreteness emerge by itself when the environment is structured in a way such that discreteness is optimal ? (eg. paper following this approach [6]) 

Authors provide evidence that introducing a continuous semantic space improves the performances. However, it would be worthwhile adding a descriptive part on the spontaneous emergence of discreteness. 

As baselines, authors chose two extreme cases:
- messages as single one-hot vectors
 - messages as continuous vectors

The paper locates the method (message as continuous vectors taken from a discrete set) as an intersection of the other algorithms. For instance, it could have been interesting to add as a baseline sequences of one-hot vectors or sequences of digits (eg. ${0,1}^{d}$). Indeed, this communication channel is frequently used in emergent communication settings (eg. [5], [7], [8]). With the choice of baseline, it is difficult to analyze the causes explaining the gain of performance (are one-hot vectors limited because of the lack of relationship between tokens ? are continuous tokens more performant because of the continuity or just the dimensionality ?).

Consequently, it could have been fruitful to complete the work with additional baselines.

*Choice of environments*

Authors mainly introduce 2 types of environments:
triangle + 9 points: finite number of objects at fixed (continuous) locations
uniform: infinite number of (continuous) locations

It is a strength of the paper to introduce several games. However, as a reader, it is difficult to understand why those environments have been chosen. First, environment presentations lack some short motivations. Second, the results are not aligned over all environments (in triangle + 9 points, proto shows an improvement compared to One Hot, but it is not the case for uniform). Therefore, adding uniform does not comfort the results. The main contribution of uniform is to introduce the alignment between messages and behaviour. As it is an initial observation for the next section, it could have been moved at the beginning of section 6.

In section 3, authors state assumptions under which one-hot vectors are optimal. Introducing an environment under which one-hots are optimal would have been appreciated to strengthen the theoretical results / showing a “baseline” experiment.

Eventually, it seems that there is a correspondence between the structure of the triangle+9 points and prototype-based models ; and between the structure of uniform and cont. model. It could therefore have been interesting to discuss or show that each method is optimal (or better than the others) when the environment is structurally close to it. 

*Clarity*

The introduction, related section and the experiment preliminaries are well organized. The rest of the paper is harder to read. To me, it is only due to the organization of the paper and small changes could ease the reading. 

The proof (Part 3.2) comes too early in the paper. Since no game is presented at this time, it is difficult to understand the assumptions of the proof and it seems that the proof merely applies for a denoising problem. 
In addition, since the core of the paper is to present the advantage of the presented method compared to one-hots, this paper would benefit from additional details on the suboptimality of one-hots under assumption relaxation. Sub-paragraph (l128-133) could be extended to provide a better intuition on the suboptimality of the one-hot tokens.

Section 5 may be bit too monolithic. Authors alternate environment presentations, results and analysis within the same 2 pages section. To me, it would have been clearer to use a more standard organization (environment presentation / results / analysis). In particular, it would have naturally impose to:
- motivate the choice of environments
- leave more space for technical details such as reward, losses (personally, equations would have been appreciated for the sake of clarity).
 - link the results of the different experiments (especially those of triangle+9 points and uniform)
- provide an overall analysis that summarizes the results of all experiments.

*Details*

Technical approach
 - 3.1 / 3.2. Formalization is quite clear.
 - The proof in subsection 3.2 is not clear to me. Notations could be improved (for example by using a different notation to refer to speaker and listener policy).
 - The proof is provided before presenting any game or environment. It is hard to understand if the proof applies for any game/environment. Can you give additional insights?
- Agent Self-Play Environment: Even if mentioned in the text, it would be nice to have 1 or 2 formal equations defining the inputs/output/messages/rewards in order to have a clear idea of the setting (and the potential biases that could derive from it…)
- Table could be completed with score boundaries (randomness/optimality) in order to have a better sense of the range of the quantitative results
- It could be advantageous to motivate the choice of noise values (even one line telling that the overall analysis are independent on the value of the noise)
- Human-Agent Experiments: l302-309: protocole would benefit additional explanation. It is not very clear the interest of doing the supervision 

[1] Jakob Foester, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiiteson. Learning to communication with multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, 2016

[2] Angeliki Lazaridou, Alexander Peysakhovic and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In 5th International Conference on Learning Representations. ICLR 2017

[3] Miss Tools and Mr Fruit: Emergent communication in agents learning about object affordances. Diane Bouchacourt and Marco Baroni (2019)

[4] Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, Stephen Clark Emergence of linguistic communication from referential games with symbolic and pixel input. In 6th International Conference on Learning Representations. ICLR 2018

[5] Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux and Marco Baroni. Anti-efficient encoding in emergent communication. Advances in Neural Information Processing Systems, 2019

[6] On the Spontaneous Emergence of Discrete and Compositional Signals. Nur Geffen Lan, Emmanuel Chemla, Shane Steinert-Threlked. In Proceedings of the 58th Annual Meeting of the Association for Computation Linguistics (ACL) 2020

[7] Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and Generalization in Emergent Languages. In Proceedings of the 58th Annual Meeting of the Association for Computation Linguistics (ACL) 2020

[8] Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. Emergent Communication in a multi-modal, multi-step referential game. In 6th International Conference on Learning Representations. ICLR 2018
 cf above.

This paper studies a communication game where a speaker has to provide instructions to a listener via a noisy communication channel; the listener then has to execute a navigation task in a particle world. The communication setup differs from the commonly used one by providing the speaker and listener with a shared token embedding table. The authors find that training with a shared embedding table leads to more accurate communication, or at least better task performance by the listener.

There is a second contribution in the form of a set of experiments where human-agent pairs play a referential game on the CIFAR-10 dataset. By training on a subset of classes and evaluating on the held-out ones, the authors evaluate generalization performance under distribution shift, and find that both agents and humans successfully interpret communication about unseen data.
  The technique proposed in this paper interpolates in an interesting way between straightforward differentiable communication (DIAL-style) and non-differentiable symbolic communication (RIAL-style). Because the agents use the same embedding table, which is subject to weight updates from the RL loss, more information flows between the agents than in a pure RIAL-like case, but less than in the case of a continuously valued vector being sent, like with DIAL.

A comment: The tokens being transmitted in natural language are less similar at the level of their physical realization than at the level of their semantic meaning. Your setup encourages similarity of the tokens, rather than similarity of the elicited responses (semantics). One-hot tokens, when processed by a language model, are typically multiplied by an input processing matrix. A one-hot token effectively selects a column from that matrix, which acts like a token embedding for the purposes of the LM, i.e. the purposes of the listening agent. Similarly, the generating agent generates a vector which is projected+softmaxed into a one-hot form. The projection matrix used there contains effective embeddings in a similar way. The difference with respect to your solution here is that the embeddings are not shared between the speaker and listener. This is not a criticism of the proposed method, but more to point out that a setup with one-hot tokens often effectively also involves embedding tables, and it might be good to make that clear in the text.

Another comment: the set-up with a shared codebook, with a speaker / encoder and a listener / decoder bears a lot of similarities to a VQ-VAE. It is obviously not exactly the same, e.g. because the listener doesn’t reconstruct the original input in this paper, but executes some other downstream task. Still, a discussion of the similarities and differences wouldn’t be out of place.

The paper has a number of strong points in its execution: the agent-agent experiments aren’t simply about communication but about doing a downstream task; there are human-agent experiments involving communication about held-out stimuli, and the analysis of the agents’ behavior and codebook is well done.

There are some points for improvement though. One thing that would make the paper stronger is more agent-agent experiments, ideally with similar codebook analysis, but at least with more complexity in the task, since the particle world tasks don't ask a lot from the communication system. An additional baseline method to consider might be VQ-VAE-based, given the similarities - e.g. let the speaker figure out the codebook on its own, in a VQ-VAE, and then see how well the listener can learn to work with that. Potential negative societal impact is not really discussed, but is also not very directly relevant. The limitations are not very different from the exisiting emergent communication literature; if anything, by showing successful human communication, the authors indicate their method is less limiting.

This paper proposes to learn word embedding along with a communication protocol in multiple communication games. More specifically, a word embedding is a bounded vector, acting as a token for communication. To find a point in the middle of discrete and complete continuous communication channels, the author introduced a designated architecture for the communication policy: prototype-based communication. Namely, the agents can only communicate with z tokens (prototypes) from a c-dim hypercube. In their experiments, the authors show that the proposed prototype-based communication protocols are more robust to noise than one-hot communication protocols. They also illustrate better generalizability and semantic alignment of prototype-based protocols.   1. The present work is novel in that it goes beyond communication with discrete tokens but does not lose tractability in a continuous space. This is achieved by a designated communication policy that employs a fixed number of learnable and continuous prototypes as bases. Such a special architecture echoes our intuition that in communication we convey discrete concepts which can be represented with continuous prototypes. It also hints a direction to emerge word embeddings that reflects the semantic space, an unexplored question in the emerging literature of emergent communication. 

2. This work is in general of good quality in both analysis and experimental design. For analysis, the authors formulate a sufficient decision-making condition for one-hot communication being optimal, which supports the claim that one-hot communication may not optimal be if the reward matrix is not Identity or the prior is not uniform. All experiments are designed in accord with this analysis. However, it is unclear how this analysis connects with the main focus of semantic spaces. The experiments are interesting, though with some flaws and confusion to be detailed in Limitations. The authors try to study couples of interesting problems including (1) if the agents can exploit additional dimensions to handle noises, (2) if agents can cluster concepts that are proximal in the return landscape with the prototype-based communication, (3) if the emerged tokens can zero-shot generalize to unseen classes with the guidance from outsourced word embeddings, (4) if humans can understand emerged tokens in zero-shot. 

3. There is considerable room for improvement in clarity. For example, by "clustering tokens with similar meanings" (P5 L195), it is confusing if the authors want to cluster tokens together or cluster similar communication intents around one token. Section 6 and Table 3 are also quite confusing, see Limitations. 

4. The proposed question to study, to emerge semantic word embedding, could be significant if investigated properly. However, this paper mostly works with synthetic tasks, limiting the significance with considerable particularity. I think the significance can be improved if the authors can (1) provide a clear definition of how semantic space should connect with the reward function, (2) discuss the impact of the number of prototypes, (3) find a more natural way to incorporate pretrained word embedding.  I would consider increasing my rating if the authors could resolve my following concerns:

1. The number of prototypes is a hyperparameter and seems to need handtuning with domain knowledge. In the experiments, this number is always set to be the number of clusters underlying the communication intents (goals or classes). In real communication scenario, it is not reasonable to let agents know the number of clusters a priori. I wonder how the presented results would be affected by changing this number to an arbitrary value. 

2. The results reported in Table 1 need further consideration. One may be curious why in the triangle experiments agents perform better in a skewed distribution of goals than in uniform distribution. Moreover, in the triangle experiment, the authors compare (the prototype-based communication with z tokens in c-dim continuous space) to (c tokens in one-hot communication), which occurs unfair to me. What would happen if the one-hot agents are endowed with a z-length one-hot vector and a c-length one-hot vector? Are they still prone to noise? For the 9 Points environment, it is not clear if the authors set all goal states as absorbing states or just the target one. It looks suspicious since the final returns are higher than in the Triangle experiment. The authors claim that this experiment demonstrates the clustering behavior of prototype-based communication. But we cannot figure from the reported final returns if the listener successfully distinguishes the target from its neighbors. Does the design of reward that encourages reaching proximal distractors also induce ambiguity? 

3. The human-agent experiments can be interesting but are largely limited by the current confusing presentation. First, to achieve "zero-shot" communication in unseen classes, the authors introduce an auxiliary training loss to align the emerged word embedding with embeddings from other sources. But it becomes unclear how much the communication game affects the learned tokens. Apparently, the receiver may choose correctly by fitting to the outsourced word embeddings and generalize with the inherent regularity of such embeddings. Second, Table 3 and the explanation in L314 in P8 are quite confusing to me. The authors stated that Self in Table 3 is the trained communication vector, but why one-hot setup has trained communication vectors? Shouldn't their communication vectors be one-hot? Why agents even achieve the best performance in unseen classes with one-hot communication? Third, there is a considerable degree of overclaim in "zero-shot human understanding of tokens". The experiment reported in 6.2 basically test if humans can tell an emerged 2-d vector is close to vector representations of prototypes of "animals" classes or "vehicles" classes. Human subjects can exploit the particular design that one choice is from animals and the other is from vehicles and do a contrast. It is not the case that humans can still do that if they are asked to choose from two novel classes from animals. I would recommend the authors to reposition this experiment as testing the separatability of higher-level groups in the emerged tokens. Also, how dependent is this result to the particular reward of 0.75 for confusion within animal classes? Can other values also lead to a similar accuracy? Then a higher-level question follows: how would agents receive such rewards in real-world communication? 

Meta-review: 
There are two over-arching themes in the reviews that highlight areas for improvement. First, the motivation and framing of the use of a discrete set of continuous vectors for communication as opposed to one-hot vectors was a source of confusion. This makes sense, as this paper is essentially bucking the trend of how most work in emergent communication both approaches the problem and even motivates that approach. So, some amount of inertial resistance is to be expected. The authors explain their perspective in a couple of different ways in the rebuttals that I found quite intuitive. (Side note: How does the stated advantage in extrapolating from limited experience, in response to reviewer fU14, compare with an approach using a sequence of one-hot vectors? Continuous would seem to be something like a limit point of this approach as the sequence of one-hots increases.) That reviewers got stuck on this point suggests that the paper can be improved by building readers' intuition (as authors have done for reviewers in their rebuttals) before getting to the bulk of the paper.

Second, the setup and presentation surrounding the experimental results could be improved. Although (as one data point), I was quite enthusiastic to see the human-agent experiments, the presentation of the results was not the easiest to parse. I think the issue here, and this may readily apply to the other experimental sections, is that the structure in these sections is very uniform. That is, I think the clarity would be improved by beginning each experiment with a truly high-level description and motivation, what are we trying to understand with this experiment, what are the aspects that are really important, etc; then with readers' minds properly primed, we would be ready for the details. As is, but not in all cases, the reader is left to build this structure on the fly as we read the section.

Neither of these areas are show-stoppers that should inherently prevent publication, but it is my opinion that they are largely the cause of the lower scores from reviewers and if addressed would make acceptance upon resubmission much more likely. Obviously, this is an interpretation based upon my read, the reviews and rebuttals, and the specific comments and suggestions of the reviewers should be considered in subsequent versions of the paper. As some reviewers noted, the paper would be further improved by expanding the agent-agent experiments, and I agree with this assessment, but could also see the paper getting accepted without it. Finally, I want to encourage the authors here as this paper has a very interesting perspective that I would like to see published and influencing the field. However, given reviewers' reservations, I will be suggesting it not be accepted at this time.



Decision: accept