Emergent Discrete Communication in Semantic Spaces
Authors: Mycal Tucker
Huao Li
Siddharth Agrawal
Dana Hughes
Katia P. Sycara
Michael Lewis
Julie Shah
Venue: NeurIPS 2021 Poster (2021)
Keywords: Emergent communication
zero-shot understanding
reference games

Abstract: Neural agents trained in reinforcement learning settings can learn to communicate among themselves via discrete tokens, accomplishing as a team what agents would be unable to do alone. However, the current standard of using one-hot vectors as discrete communication tokens prevents agents from acquiring more desirable aspects of communication such as zero-shot understanding. Inspired by word embedding techniques from natural language processing, we propose neural agent architectures that enables them to communicate via discrete tokens derived from a learned, continuous space. We show in a decision theoretic framework that our technique optimizes communication over a wide range of scenarios, whereas one-hot tokens are only optimal under restrictive assumptions. In self-play experiments, we validate that our trained agents learn to cluster tokens in semantically-meaningful ways, allowing them communicate in noisy environments where other techniques fail. Lastly, we demonstrate both that agents using our method can effectively respond to novel human communication and that humans can understand unlabeled emergent agent communication, outperforming the use of one-hot communication.

Emergent Discrete Communication in Semantic Spaces
Mycal Tucker!, Huao Li², Siddharth Agrawal Dana Hughes Katia Sycara?, Michael Lewis?, and Julie Shah
Massachusetts Institute of Technology
University of Pittsburgh
3 Carnegie Mellon University
sycal@mit, edu, hul52@upitt.em edu, siddhara@cs. cmu, edu
Abstract
Neural agents trained in reinforcement learning settings can learn communicate among themselves via discrete tokens, accomplishing as team what agents would be unable do alone. However, the current standard of using one-hot vectors as discrete communication tokens prevents agents from acquiring more desirable aspects of communication such zero-shot understanding. Inspired by word embedding techniques from natural language processing, we propose neural agent architectures that enables them communicate via discrete tokens derived from a learned, continuous space. We show in a decision theoretic framework that our tech- nique optimizes communication over wide range of scenarios, whereas one-hot tokens are only optimal under restrictive assumptions. In self-play experiments, we validate that our trained agents learn to cluster tokens in semantically- meaninnfuul ways, allowing them communicate in noisy environments where other techniques fail. Lastly, we demonstrate both that agents using our method can effectively respond to novel human communication and that humans can understand unlabeled emergent agent communication, outperforming the use of one-hot communication.
Introduction
longstanding goal of AI has been develop agents that cooperate with other agents humans to accomplish tasks together. Often, communication is necessary to enable such cooperation: the study of emergent communication has recently shown great success producing agents that learn to communicate. In reinforcement learning settings, guided only by environment reward, neural agents can learn to communicate by broadcasting numerical vectors each other (9] 31 [8).
Given this success, and in part inspired by the discrete nature of words in natural language, some researchers have focused emergent discrete communication by forcing agents to broadcast one-hot vectors [ (0) These tokens in effect become a lexicon used by agents. Studying when these tokens emitted allows researchers uncover their meanings, as well study the broader questions of what environment or agent factors contribute to desirable aspects of learned communication (e.g., compositionality or, in continuous communication settings, zero-shot understanding)
[5]
19 5 4
We claim that discretizing messages by constraining them to conform to one-hot hot vectors fundamentally precludes agents from learning relationships between tokens because each one-hot vector orthogonal to and equally far away from all other vectors. Conversely, research from natural language processing and word embeddings has long established the importance of learning representations of discrete words within a continuous, semantic space 29 32
desirable properties of language. One-hot vectors establish
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In this work, we demonstrate the benefit of agents that employ discrete set of tokens within continuous space over agents that use the standard practice of communicating via one-hot vectors in discrete emergent communication settings. We present novel architecture and implementation for learning such communication and provide decision-theorette analysis of the value of such an approach the congruence of meaning and form of communications. Simulation experiments confirmed these results: our agents learned an arrangement of tokens that clustered in Human--understndabbi patterns. The arrangement of discrete tokens within the learned communication space produced team performance that was robust to environment noise and enabled agents to effectively utilize novel communication vectors. In human-agers experiments, agents aligned their tokens with natural language embeddings and responded appropriately to novel English phrases. Lastly, we showed that humans capably interpreted unlabeled emergent communication tokens in reference game
2
Related Work
We propose a technique within emergent communication literature, drawing inspiration from work on word embeddings in natural language processing (NLP) and zero-shot classification.
2.1 Emergent Communication
Researchers of emergent communication study techniques to enable agents to learn to communicate among themselves, enabling high task performance in reinforcement learning settings (see 36 10 18
among others). These settings, such as reference games or Lewis signalling games [25], are designed such that agents must communicate to perform the task successfully; in "cheap-talk" scenarios, agents often learn successful communication strategies by sending real-valued vectors to each other. We focus on discrete communication emerging among decentralized agents that may communicate by sending one of a finite set of vectors to each other, but may not access other agents' weights or gradients during training or execution [8]. Previously, such discrete communication has often taken the form of one-hot messages: Foerster et al. [9] proposed binary discrete messages, but subsequent works seem to have reverted to one-hot vectors 13 III
Even when agents learn to communicate, they often fail to learn protocol that humans or separately- trained agents can understand; that is, they fail at the "zero-shot" learning problem [2]. To address this gap, several recent works have found properties of environments that encourage zero-shot continuous communication (5 Other research addresses gumman-undersandaan
communication
in RL settings. Using bottom-up approaches, researchers align emergent communication tokens with human language [2 26 Lazaridou al. (20 even demonstrated that humans can understand agents' tokens in zero-shot settings (e.g., interpreting the agent's 's token for "dolphin" as referring to photo of water). Complementary. "top-down" approaches leverage pre-trained task-specific language models for natural language communication 211.
We position our work within the field of emergent discrete communication: we seek to have agents learn to communicate via a finite set of possible messages, without access to a task-specific language model. Unlike prior work, we relax the assumption that discrete tokens must take the form of one-hot vectors and demonstrate that our proposed architecture enables agent utilization of novel communication, and supports human understanding of learned communications.
2.2 Natural Language Word Embeddings
While early neural NLP research used one-hot representations of words, most modern techniques rely upon dense word embeddings such as Word2 or GLoVE for faster training and greater general- ization power 2 29 32 I. In essence, these embedding techniques learn discrete representations of words within a semantically meaningful space; words representing similar meanings are often embedded in similar locations. We argue that current research emergent discrete communication that relies upon one-hot tokens is similar to the early one-hot word embedding techniques that suffered from limited generalizability. Instead, we propose that discrete emergent communication should be leameed within continuous, semantically meaningful space, like modern word embeddings.
Anonymized code available http:///wannnnooo..ooo..sseeeeea

---

2.3 Zero-Shot Classification
Research in zero and few-shot classification seeks to train classifiers that are able to correctly classify inputs belonging new classes despite seeing few training examples of that class [38 28 7 Often, techniques employ "side-information" to enable high performance, for example using labeled data from different domain (28 information from language (35] We focus emergent communication rather than classification, but in experiments aligning communication tokens with word embeddings, we similarly exploit side information for zero-shot understanding.
3 Technical Approach
3.1 Emergent Communication in Multi Agent Reinforcement Learning
We adopt standard methods for training multiple agents in decentralized partially-observable Markov decision process [3] The Dec-POMDP is defined by the (S, A, R. O, (2,7) tuple. is the set of states; A,Vi E [1 are the sets of actions, including communication, for each of N agents; S is the probabilistic transition between states due to joint actions. We focus settings to encourage communication among agents. S2 defines the set of
T: S x AN...Av + on partially-observa possible observations; O; AN...AN S - maps from joint actions and the state to distributions of observations for each agent. Lastly, and R define the discount factor and reward function, respectively. The goal is to find the policies for all agents, Ti o E Ω + A, that maximize the expected discounted reward. Multi-agent existing techniques for solving these problems in multi-agent reinforcement learning settings, using agent architecture develop.
well studied (30) 27 [34]; we borrow
3.2 Decision-Thean Noisy Discrete Communication
In this section, we formalize a noisy channel reference game, an instance of a Dec-POMDP. We derive that one-hot vectors optimal tokens under restrictive assumptions, and relaxation of those assumptions may lead to suboptimality of one-hot communication.
Consider two agents, speaker and listener, that communicate via dimensional noisy channel, Each episode composed two timesteps. the first timestep, a speaker emits token, 4, chosen from distribution over columns in matrix of tokens, T; Texx- In the second timestep. the listener observes a message, m E R a noisy version of t, corrupted by additive zero-mean noise. We constrain all tokens to fall within the unit hypercube (i.e., 0 T I 1; 00, cl, [0, =)), which constrains the relative scales of tokens and noise.
The listener's s task is to predict which token the speaker emitted, given the observed noisy message. assume there exists some reward matrix, Rzx= that specifies the shared reward among agents, where entry R[i,j] j] corresponds to the reward if the speaker emits 7[i] and the listener predicts T[i]: standard reward matrices resemble the identity matrix, potentially with some small, positive values off the diagonal for "reasonable" mistakes. From seccision-thoorete framework, the speaker and listener wish to maximize expected reward. We seek to find the optimal set of tokens, T", that maximizes this expected value, shown below:
P(T (TTj)I)))((T) Exer P(T m.) P(T(K)
1 arg max T
EP(TTU)
max R(i J] JES
(1)
(Tm)
P[] mm))
arg max
E PP
Evez
P(T[k]]mm))
Uniform prior; R
(2)
i6z
= =arg min Σ EP(m//T())
Σ P(T(ilmm)
(3)
iE3
JE:;$$$
arg max
Σ
[T]|] - ?
Gaussian noise
(4)
T
3

---

Equation I specifies the expected reward using a rational listener model: taking the expectation over tokens (the outer sum), the listener chooses the token T[j] that maximizes the expected reward given the observed message, my. Equation 2 follows under assumptions of (an equal number of tokens and communication dimensions), Equation 3 follows, deriving that the optimal tokens minimize the likelihood of confusion of any pair of tokens. Finally, assuming a Gaussian noise model with a variance matrix equal to ol for some constant o, Equation 4 states that the optimal tokens maximize the mean squared euclidean distance between tokens. If tokens are constrained within a unit hypercube, one-hot encodings are an optimal solution. A proof of optimality of one-hot tokens is included in Appendix A
= T=x2, and a uniform prior over emitted tokens.
While one-hot encodings may be optimal in some scenarios, the above analysis also reveals when they may not be: when the cost of emrrr or prior over tokens are not uniform, or when the number of tokens and communication dimensions are not equal. Examples of suboptimality of one-hot tokens when these assumptions are relaxed are included in Appendix [ Thus, although many academic experiments respect these assumptions, a more powerful framework for learning tokens is needed in the general case.
3.3 Discrete Prototype Communication
We present our method for prototype-based communication to address the limitations of one-hot communication by endowing agents with learnalee tokens. We create neural agents with communi- cation head instantiated as a multi-layer perceptron with a penultimate softmax layer (where we use the gumbel softmax trick (I4), which then is multiplied by a learnable matrix, Taxes for the final output. More formally, an agent, parametrized by weights O and T, produces communication token I according to Equation 5
= fo(O)
d one hargarg max(g, log mT))
(5)
1 d Taxe
Equation 5 states that agents produce logits, т, from an observation, O, convert the logits to one-hot vector, d, by using sampled 9i drawn i.i.d. from a Gumbel(O. 1) distribution, and then multiply d by T to produce the token Although the argmax operation is not differentiable, we use it in the forward pass (that is, actually using "hard" argmax) but use the gumbel softmax operation for calculating gradients, used in the backwards pass. We refer readers to Jang et al. [14 for more details on the gumbel softmax approximation.
We note that our technique introduces an additional set of learnable parameters compared to one-hot communicative agents, which would merely output d. This increased flexibility may enable improved performance but may also increase learning difficulty; experiment results, reported in Sections 5 and o indicated net benefits derived from our technique.
3.4 Opportunities for Behavior Shaping
The previous section illustrates how T may be learned using traditional reinforcement learning techniques; using a policy gradient method, the tokens will evolve during training to enable high task performance. While this emergent behavior is desirable for its flexibility, we also examined two ways of constraining learned behavior: -exiicon-etttin" and grounding.
The first constraint, which we call "lexicon- "lexicon-setting." consisted of manually specifying a non non-trainable T. During training, gradients passed through T but did not update its elements. This allowed designer to specify the set of tokens agents used to communicate, in other words setting the "lexicon."
The second constraint consisted of teaching grounding via small set of supervised data. Even with hand-specified tokens, it is unclear a priori what meanings agents will associate with tokens. To enable human-agent communication, we require common understanding, or grounding, of tokens.
In experiments, we constrained the tokens to the unit hypercube by passing t through sigmoid activation enable fair comparisons to one-hot encodings, but tokens need not generally obey such constraints.

---

Inspired by prior art in learning social conventions in RL settings 24, 26), we used small set of data mapping observations desired communications, The gradient the supervised loss (mean squared error of predicted communication vs. desired communication) was added to the policy loss from the RI, environment, leading the agent to use communication that matched the grounding data and enabled high task performance.
Experiment Preliminaries
4.1 Baselines
We compared our technique to two emergent communication baselines: continuous and one-hot communication. The continuous communication method (henceforth, cont) allowed agents to output real-valued vectors in the communication space, Rº, bounded along each dimension within (-1, 1) by using a tanh activation. The discrete one-hot baseline (henceforth, one-hot) allowed agents output one-hot vector within the communication space using the gumbel softmax trick. By definition, this constrained one-hot agents to output one of tokens.
All techniques were trained using the multi-agent deep deterministic policy gradient (MADDPG)
method proposed by Lowe et al. [27). a common policy-gradiee method that updates neural network weights maximize the expected discounted reward.
4.2 Hypotheses
Throughout experiments. assessed trained agents by measuring environmeent-specific reward under different scenarios. First, we measured self-play reward for agents trained together, as the environment reward they received during 500 evaluation episodes. Second, we measured aspects of zero-shot behavior and performance in reference game (introduced in Section 6). Measurements included listener agent's 's accuracy in selecting a target image when using novel communication generated by humans, as well as human participant accuracy in identifying target image based on unlabeled agent communication.
Using these metrics, we formulated the following hypotheses:
HI: noisy environments, prototype-based communication will enable agents to achieve higher self-play reward than one-hot tokens.
H2: Prototype-based agents, trained with word embeddings as tokens, will identify target
images in reference games based on novel, numan-generaan communication, whereas one-hot encodings will fail outperform random chance.
H3: Human participants will accurately identify target images in reference games based on tokens learned by prototype- based agents.
Our hypotheses were informed by prototype- based communication design, which we expected to clus- ter tokens with similar meanings, thereby providing robustness noisy environments and benefiting human understanding of the communication space. We expected this improved semantic understand- ing of the communication space to bolster zero-shot understanding novel communication.
5 Agent Self-Play Experiments
We first evaluated our technique in particle-world environments, inspired by the environments introduced by Lowe et al. (27 and Mordatch and Abbeel [3]] Agents were trained and tested in fixed teams; thus, these experiments were used to measure robustness noise and assess characteristics of learned tokens. Depictions of our environments are shown in Figure Full details of the environments included in Appendix(B [
In the first environment, triangle, speaker and listener agents were spawned at the origin, and three landmarks were located at the points of equilateral triangle. In each episode, landmark was designated as the target, and noise was drawn from a e = dimensional zero-mean Gaussian with E 0.9 Only the speaker observed the position of the target; thus, to maximize the reward
5

---

t
(a) Triangle
(b) Points
Uniform
Figure The particle environments used for agent-only testing. In the triangle and Points environments, the landmarks (colored circles) were in fixed locations; a target was chosen among the possible landmarks. In the uniform environment, the landmarks were in locations chosen uniformly at random within the game.
Table 1: Median (standard erorr) self-play reward over runs. Enabling Iearnable tokens endowed our agents with greater robustness to environmental noise.
Env. Dist.
Rewards
Proto
One Hot
Unif.
0.9
9
-124 (12)
-201 (10)
Tri.
Skew.
0.9
9
-100 (4)
-136 (9)
Figure 2:
The learned prototypes (left) reflect structure in the 9 points environment (right), with similar prototypes encoding similar locations. Pro- 2 9; a 0.3 exper-
Unif.
0.0
3
-60 (7)
-96 (10)
9-P.
Unif
0.0
9
-65 (2)
-66 (6)
Unif
0.5
9
-100 (10)
-132 (2)
totypes taken from iment conducted for visualization.
(negative distance from the listener to the target), the speaker must communicate to the listener, via 9D vector, and the listener must move the target.
We investigated the effect of different priors over tokens by changing the distribution over which landmark chosen as the target: in setting, targets were drawn uniformly. but in the other, targets were drawn from a skewed categorical distribution of [0.49, 0.49, 0.02]. Three learnable prototypes were defined for our technique (= 3). Note that this meant that our prototype technique had fewer tokens than the 9 for one-hol: we selected = 9 to study scenarios in which < c.
In second environment, 9 points, speaker and listener agents were spawned at the origin, with the speaker observing the location of the target and the listener observing its own state. Reward was similarly defined the negative distance squared from the listener to the target. 9 landmarks were spawned in fixed locations, depicted in Figure Db. This arrangement landmarks produced non-identity reward matrix: the listener confusing confusing two landmarks that were further apart. We used the 9 points environment in settings: two settings with = 0 and 3 our technique used 9 prototypes. These settings allowed us to study cases when 2 c. Results from the first environments, assessing robustness to noise, shown in Table 1
nearby landmarks was less costly than
9, and one noisy setting with 0.5 and 9. In 9 points,
The triangle environment results reveal two benefits our technique over one-hor: robustness to noise when than communication dimensions, agents used the additional dimensions for robust communication, as shown in the top row of Table T The three prototypes spread into distant corners of the 9-dimensional unit hypercube, creating a larger distance between tokens than one-hot encodings. Next, the second row shows that technique yielding greater robustness for tokens denoting the more likely landmarks. The one-hot technique, in contrast, learned to always go the location between the most likely landmarks.
and the ability to exploit non-uniform prior. First, when there were fewer tokens
able benefit from the non-unin non-uniform prior over landmarks by
The 9 points environment enabled us to study two other attributes learned communication tokens: the benefits of greater expressivity in channel-limine -imiteed domain (2 c) and clustering behavior due

---

0.5a + 0.5g
Table 2: Median (std. error) reward over 5 runs in Uniform. Continuous communication out- performed both discrete methods, which converged to similar dis- cretization behaviors the con- tinuous space.
C
3
-54
300
Method
Reward
Cont. -9..8 (3.8)
Tracing seeker paths (black lines) and final locations (red dots) revealed the discretization im- posed by prototypes.
Figure
Figure 4: Mcan destination of the seeker agent when given pairwise in- terpolations of prototypes from 0% (red) to 100% (purple).
Proto -13.66 (3.8)
One Hot
-111.8 (3.4)
to non-identity reward matrix when First, we confirmed in the third row of Table[] in an environment with e = 3 but 9 landmarks, one-hot lacked the expressivity to uniquely denote each target; our technique decoupled the number of tokens and the communication dimension, allowing discrete tokens in a 3D communication channel. The baseline of = == 9, in the fourth row, confirmed that the 9 prototypes in 3D achieved the same reward 9 prototypes in 9D or 9 one-hot tokens. Lastly, the fifth row, we observed that increasing environment noise led to a graceful degradation of performance for prototype-bad bbssdeeeeeedddd--eoe agents, but worse failures for one-hot. Inspection of the learned prototypes reveals why: tokens denoting landmarks in the same cluster of 3 landmarks converged into clusters in the communication space, allowing the clusters to separate from each other, as shown in Figure 2 Thus, intra-cluster errors became more likely, but inter-cluster error likelihood decreased. In fact, the mean reward of -10 achieved in the noisy environment similar to the reward of -96 achieved by the 3D one-hot communication method: both of these techniques devolved to the behavior of treating clusters single points.
In our final environment, uniform, a seeker and observer agent were spawned at random locations in the world; two landmarks (target and distractor) were also spawned at locations drawn from uniform distribution over the world bounds. The observer agent observed the location of both landmarks; the seeker agent observed its own position and the id of the target (i.e., the first second landmark). Each agent communicated to the other at each timestep. To achieve high reward, designated as the negative distance from the seeker to the target, both agents had communicate: the seeker telling the observer which landmark was the target, and the observer responding with the location of the target. This environment, unlike the prior two, allowed communicate about continuous data. Prototype and cont agents communicated with c = 3; one-hot used 5, prototype agents used
assess how agents used discrete vocabulary
In the uniform environment, we found that our method discretized continuous data in a reasonable way and that the emergent tokens enabled some measure of interpolation between representations. First, our method and one-hot performed similarly to each other and worse than cons (Table 2 Given that the observer agent had to communicate about continuous values (the target's location) with a finite, discrete vocabulary, this is unsurprising. Plotting trajectories of a seeker agent over 200 episodes, the discrete pattern of locations emerges (Figure 3).
Lastly, we tested prototype-base agents with interpolations of learned tokens by overriding the ob- server agent's 's communication with pairwise interpolations of all tokens at 10% intervals. An example of the resulting behavior is plotted in Figure 4 (We note that interpolation for continuous agents was impossible because of the lack of discrete tokens, and one-hot precludes communication other than one-hot vectors.) Interestingly, we found that prototype-based seeker agents often responded to the interpolation of tokens with interpolation of behavior: for example, when observing the mean of two tokens, a and B, the seeker agent went to the mean of the locations it would have gone to when observing just just B. These initial findings motivated work in the next section zero-shot understanding of natural language.
Together, these three experiments provided support for our first hypothesis, H1. Our technique enabled agents learn tokens that provided greater robustness to noise the triangle and 9 points environments by separating and clustering tokens, confirmed visually in Figure
7

---

Table 3: CIFARIO zero-shot reference game mean reward (std. error). Prototype-based models the only
discrete communication method that enabled betler-than-random zero-shot understanding.
Method
Label In
Label Out
Self In
Self Out
AMT In
AMT Out
Cont (BERT)
91% (0.7)
55% (1.0)
91% (0.7)
68% (1.0)
73% (0.4)
53% (0.6)
Proto (BERT)
93% (0.7)
58% (1.0)
93% (0.7)
66% (0.9)
70% (0.4)
56% (0.6)
Proto (Hand)
75% (1.1)
62% (1.2)
80% (0.6)
58% (1.0)
One-hot
96% (0.2)
50% (0.8)
96% (0.2)
69% (1.0)
6 Human-Agent Experiments
Lastly, we tested how well our technique supported aspects of human- human-agent agent interaction in zero-shot settings. Results from the prior section already indicated that prototype-based agents learned a sort of semantic communication space: in 9-points, for example, similar target locations were encoded via similar prototypes, and in uniform, the seeker agent responded to interpolations of prototypes with interpolations of behavior. In order to enable suman-agenn interaction in experiments in this section, we aligned parts of human and emergent communication. We found that this alignment, in conjunction with semantically-men menningffll communication spaces, supported two forms of zero-shot understanding. First, agents were able understand novel human communication at test time (H2). Second, humans could understand emergent communication embeddings that agents learned (H3).
We conducted these experiments via reference game, a standard type of environment for emergent communication 12 In our environment, a speaker observed an image, drawn from the CIFARIO dataset, and emitted communication vector [16]. A listener agent observed the original image and a "distractor" image that belonged to different class, as well as the communication emitted by the speaker. The listener predicted in the second, final timestep which of the two images was the "target" observed by the speaker. Reward was defined via a matrix, R10x10. where R[i,]] set the shared reward if the target image belonged to class and the listener agent predicted We penalized some errors more than others according to high-level groupings of classes: correct predictions got reward 1, confusion within the set of 6 animal classes or 4 vehicle classes got 0.75, and predictions that belonged to the wrong group got O. We transformed this basic reference game into two zero-shot experiments.
image of class j.
6.1 Zero-Shot Agent Understanding
In our first zero-shot experiment, we assessed agents' ability to understand novel communication by withholding two image classes (one animal and one vehicle) from training. We trained teams from scratch for each of the 24 possible vehicle excluded combinations, resulting in 120 teams. All agents were trained in environments with o = 0.05; cont and prototype agents used c = 3, prototype agents used z = 10, and one-hot used 10.
To align human and agent communication, we augmented self-play training with 100 supervised examples of image-commn conee-communication pairs. These data were interleaved with self-play in training agents, biasing them match the desired policy, as done in prior social convention literature [24 37 26).
For cont and prototype agents, we used BERT-like model to create communication by embedding the class label names 33]; one-hot used the classification label. An additional set prototype agents was trained with hand-designed word embeddings that clustered vehicle and animal embeddings. See Appendix D for details on embeddings and image feature extraction. For the prototype agents, we used the lexicon-setting approach from Section 3.4 to set T' equal to the label embeddings.
During evaluation, we measured listener prediction accuracy over 500 trials when using different types of communication or images. In "in-distribution" experiments, we evaluated models using the 8 image classes seen during training: in -iat-----istiiutiin experiments, we used only the image classes never seen in training. Furthermore, we tested three types of communication vectors: those produced by the trained speaker agent (Self), by embedding the class label (Label), or by using embeddings of captions generated by human annotators on Amazon Mechanical Turk (AMT). Details of the user study for generating AMT descriptions are included in Appendix [F] in essence, participants were asked to provide descriptions of images, replicating the role of the speaker agent.
8

---

Results, shown in Table supported hypothesis H2: prototype listener agents supported two types of zero-shot communication. AMT In experiments reflected one type of zero-shot understanding by forcing listeners to interpret novel communication at test time. Despite training with a small, finite vocabulary, prototype listeners performed almost on par with cont agents. Conversely, one-hot encoding precludes embedding novel communication. AMT Out and Label Out results reflected second type of zero-shot understanding: listeners simultaneously observed novel images and novel communication. In these settings, our method outperformed cont and the random performance of one-hot. By testing prototype agents trained using BERT or hand-generated embeddings, we found that different embedding methods played an important role in agent performance. Lastly, in ablation studies, we confirmed that both the reward matrix and environment noise contributed to agents' zero-shot understanding (Appendix D
6.2 Zero-Shot Human Understanding of Tokens
In our final experiment, we studied whether humans could understand agent communication in zero-shot settings, is essence treating humans listeners in the reference game. We trained prototype agents in the standard reference game, communicating in 2 dimensions with all 10 classes and no explicit guidance on the tokens. See Appendix [ for full training details. In evaluation, we showed participants 8 of the 10 learned tokens, annotated with English class labels. Finally, we plotted one of the held-out tokens with no label and showed participants two images, one for each of the two held out classes, and asked participants to select the image they thought the new communication most likely referred to. We recorded participants' selection accuracies among animal-vehicle image pairs for 5 prototype-based models trained from scratch, and a 2D PCA visualization of BERT-baasd embeddings of labels.
Averaged across the 5 prototype models, with an average of 300 responses per model, participants selected the correct image 70% of the time (standard error of 5%). This compares favorably to the 66% achieved using embeddings derived from the BERT-based model (x (1, N = 2093) = 3.21, p .073). While prior art has studied how humans interpret labeled agent tokens in new contexts (e.g., interpreting a one-hot token for "dolphin" as referring to a photo of water in Lazaridou al. (20), we believe this is the first demonstration of human participants understanding novel tokens. Ultimately, these results provided support for our final hypothesis, H3.
7 Contributions
In this work, we proposed technique to address current gap in existing discrete emergent com- munication literature. Prior work in this domain relies upon one-hot tokens for communication, but insights from NLP, as well as scheme is inadequate in many settings. Using a technique we developed to enable learnable, discrete tokens, we demonstrated benefits in self-play, including increased robustness to environment noise. In human- human-subject subject experiments, we showed improvements over the random performance of prior art in zero-shot understanding of human communication. Lastly, we demonstrated that prototype agents learned emergent communication tokens that were tuman-interepretable and semantically meaningful. Overall our experiments provided support that our prototype-based communication method produced semantically meaningful communication spaces in noisy environments with non-identity reward matrices, which in turn enabled zero-shot understanding of communication.
analysis. demonstrate why such an encoding
Our work provides a first step towards connecting theories of emergent communication and word embeddings. While we have taken step this direction through our proposed technique, future work leveraging insights from discrete representation learning could likely further improve upon our results. In addition, many of the techniques used to examine word embeddings could be used to analyze emergent communication tokens. Specifically, we note that prior literature has found reinforced biases in word embeddings; similar care of encoding desirable properties in emergent tokens must be taken. Lastly, further experiments examining environmental factors that contribute to zero-shot understanding of discrete communication could yield important insight into effective agent training and language evolution.

---

Acknowledgments and Disclosure of Funding
We thank Jacob Andreas for early discussions about embeddings in NLP, and the NeurIPS reviewers for their careful feedback. This work has been funded by ARL award W91
References
[1] Felipe Almeida and Geraldo Xexéo. 1rXiv:190..00669 2019.
Word embeddings:
survey.
arXiv preprint
[2] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. The journal of machine learning research, 3:1 137-1155, 2003.
[3] Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4):819-840, 2002.
[4] Kalesha Bullard, Douwe Kiela, Joelle Pineau, and Jakob Foerster. Quasi-equivalence discovery for zero-shot emergent communication. arXiv preprint arXiv:2/03.08067, 2021.
[5] Kalesha Bullard, Franziska Meier, Douwe Kiela, Joelle Pincau, and Jakob Foerster. Exploring zero-shot emergent communication in embodied multi-agent populations. arXiv preprint arXiv:2010.15896 2020.
[6] Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encoding in emergent communication. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[7] Long Chen, Hanwang Zhang, Jun Xiao, Wei Liu, and Shih-Fu Chang. Zero-shot visual recognition using semantics-preserve adversarial embedding networks. In Proceedings of the IFFE Conference on Computer Vision and Pattern Recognition, pages 1043-1052, 2018
[8] Tom Eccles, Yoram Bachrach, Guy Lever, Angeliki Lazaridou, and Thore Graepel. Biases for emergent communication in multi-agent reinforcement learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. A'lch--Buc E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[9] Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[10] Matthew John Hausknecht. Cooperation and communication multiagent deep reinforcement learning. PhD thesis, 2016.
[11] Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[12] Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "Other-play" for zero-shot coordination. In International Conference Machine Learning, pages 4399-4410. PMLR, 2020.
[13] Shariq Iqbal.
Maddpg-pytorch.
https: http:s///gttuub..comm//arggggggp
maddpg pytorch] 2018
[14] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, 2017.
10

---

[15] Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge 'naturally' in multi-agent dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2962-2967, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
[16] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 (canadian institute for advanced research).
[17] H. W. Kuhn and A. W. Tucker. Nonlinear programming. In Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability, 1950, pages 481-492, Berkeley and Los Angeles, 1951. University of California Press.
[18] Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. arXiv preprint arXiv:2006. 02419, 2020.
[19] Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In 6th Interna- fional Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
[20] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In 5th International Conference on Learning Rep- resentations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings. OpenReview.net, 2017.
[21] Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7663-7674, Online, July 2020. Association for Computational Linguistics.
[22] Jason Lee, Kyunghyun Cho,, and Douwe Kiela. Countering language drift via visual grounding. In EMNLP-IJCNLP 2019 2019 Conference Empirical Methods in Natural Language Pro- cessing and 9th International Joint Conference on Natural Language Processing, Proceedings the Conference, EMNLP-IJCNEL 2019- 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, pages 4385 4385-4395 Association for Computational Linguistics, 2020.
[23] Jason Lee, Kyunghyun Cho, Jason Weston, and Douwe Kiela. Emergent translation in multi- agent communication. January 2018. 6th International Conference Learning Representations, ICLR 2018; Conference date: 30-04-2018 Through 03-05-2018.
[24] Adam Lerer and Alexander Peysakhovich. Learning existing social conventions via observa- tionally augmented self-play, aaai. In ACM conference Artificial Intelligence, Ethics, and Society, 2019.
[25] David Lewis.
Convention: A philosophical study. John Wiley & Sons, 2008
[26] Ryan Lowe, Abhinav Gupta, Jakob N. Foerster, Douwe Kiela, and Joelle Pineau. On the interac- tion between supervision and self- self-play in emergent communication. In ICLR. OpenReview.net, 2020.
[27] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettiivee environments. Neural Information Processing Systems (NIPS), 2017
(28) Qingjie Meng, Daniel Rueckert, and Bernhard Kainz. Unsupervised cross-domain image classification by distance metric guided feature alignment. In Medical Ultrasound, and Preterm, Perinatal and Paediatric Image Analysis, pages 146-157. Springer, 2020.
[29] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed rep- resentations of words and phrases and their compositionality. In C. J. C. Burges, L. Botton, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
11

---

[30] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 1928-1937. PMIR, 2016.
[31] Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi- agent populations. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018.
[32] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. Proceedings of the 2014 conference on empirical methods natural language processing (EMNLP), pages 1532-1543, 2014.
[33] Nils Reimers and Iryna Gurevych. Sentence-bert Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019.
[34] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707 7077006477 2017.
[35] Richard Socher, Milind Ganjoo, Christopher D. Manning, and Andrew Y. Ng. Zero-shot learning through cross-modal transfer. In Proceedings of the 26th International Conference on Neural Information Processing Systems- Volume NIPS' 13, page 935-943, Red Hook, NY, USA,
2013. Curran Associates Inc.
[36] Sainbayar Sukhbaatar, arthur szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[37] Mycal Tucker, Yilun Zhou, and Julie Shah. Adversarially guided self-play for adopting social conventions. arXiv preprint arXiv:2001.055: 055944, 2020.
[38] Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251 -2651,2226,2221 2018.
[39] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2403- 2412, 2018
[40] Lukasz Kuciński, Pawel Kotodziej, and Piotr Milos. Emergence of compositional language in communication through noisy channel. 2020.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 3.3 on learning, and
Section on broader limitations.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 7 on how word embeddings and our technique could reinforee biases.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] In our decision- theoretic analysis, analyze what assumptions lead to one-hot tokens being optimal communication tokens.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A
12

---

3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes] See footnote
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendices.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See standard error, reported in all tables.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix C
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section 4.1
(b) Did you mention the license of the assets? [Yes] See Section 4.1
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
No new assets
(d) Did you discuss whether and how consent was obtained from people whose data you'l re using/curating? [Yes] Data were not published; details on user studies are in Appendices.
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information offensive content? [Yes] Data identifying, as shown in user study details Appendices.
published and were
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [Yes] See Appendix IF]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [Yes] See Appendix [F]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] See Appendix
13


Reviews:
[{"review_id": "oZwrql0M3GE", "reviewer": "Reviewer_bPM2", "summary": "The paper explores how discrete latent space can be used in emergent communication instead of one-hot vectors. Indeed, the authors claim that one-hot vectors may be too restrictive, while continuous messages provide an interesting venue for research.\n\nMore precisely, they propose to replace the use of one-hot vectors by a set of tokens derived from a continuous space (embeddings), that are either learned or inherited from another model. They show that using embeddings leads to:\n - a communication optimisation\n - robustness in noisy environments\n - better understanding in machine-human settings\n\nThe main contribution of the paper is thus the introduction of a new model for the communication channel that leads to better communication performances compared to one-hot vectors. ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Authors accurately point out the limits of the standard use of one-hot vectors in language emergence. They provide a method that fairly improves communication performances with several scenarios. However, the paper lacks finer analysis to strongly complete language emergence literature. In particular, it could have been fruitful to add a descriptive approach, e.g, under which environmental constraints agents spontaneously develop discreteness within a continuous communication channel ? Finding such constraints would be a strong argument to use embeddings in language emergence as a standard lexicon. Here, the discrete communication of continuous vectors does not emerge per se. \n\n\n*Originality/Significance/Quality*\n\nChoice of model: The originality of the paper is to transfer tools inherited from NLP (word embeddings) in order to improve communication performances. Authors are right to notice that researchers  in emergent communication mostly promote constrained communication channels (eg. [1,2]). It is often motivated by:\n -  the sake of simplicity (eg. [3])\n - language emergence studies aim at studying core properties of languages without the need of reaching optimal performances (eg. [4,5])\n\nThe paper's goal is here to:\n- introduce the notion of continuous semantic space\n- show how it improves performances compared to one-hot vectors\n\nThe paper mostly considers 3 baselines (one-hot vectors (a), discrete set of continuous vectors (b), continuous vectors (c). Based on the different scenarios, authors show theoretically or empirically that either (a), (b) and (c) are optimal/more performant. Since (a) is a constrained version of (b), and (b) a constrained version of (c), a nice result would have been to show that, without directly constraining the communication channel, either case (a), (b) or (c) emerge only based on environmental constraints. In other words, does discreteness emerge by itself when the environment is structured in a way such that discreteness is optimal ? (eg. paper following this approach [6]) \n\nAuthors provide evidence that introducing a continuous semantic space improves the performances. However, it would be worthwhile adding a descriptive part on the spontaneous emergence of discreteness. \n\nAs baselines, authors chose two extreme cases:\n- messages as single one-hot vectors\n - messages as continuous vectors\n\nThe paper locates the method (message as continuous vectors taken from a discrete set) as an intersection of the other algorithms. For instance, it could have been interesting to add as a baseline sequences of one-hot vectors or sequences of digits (eg. ${0,1}^{d}$). Indeed, this communication channel is frequently used in emergent communication settings (eg. [5], [7], [8]). With the choice of baseline, it is difficult to analyze the causes explaining the gain of performance (are one-hot vectors limited because of the lack of relationship between tokens ? are continuous tokens more performant because of the continuity or just the dimensionality ?).\n\nConsequently, it could have been fruitful to complete the work with additional baselines.\n\n*Choice of environments*\n\nAuthors mainly introduce 2 types of environments:\ntriangle + 9 points: finite number of objects at fixed (continuous) locations\nuniform: infinite number of (continuous) locations\n\nIt is a strength of the paper to introduce several games. However, as a reader, it is difficult to understand why those environments have been chosen. First, environment presentations lack some short motivations. Second, the results are not aligned over all environments (in triangle + 9 points, proto shows an improvement compared to One Hot, but it is not the case for uniform). Therefore, adding uniform does not comfort the results. The main contribution of uniform is to introduce the alignment between messages and behaviour. As it is an initial observation for the next section, it could have been moved at the beginning of section 6.\n\nIn section 3, authors state assumptions under which one-hot vectors are optimal. Introducing an environment under which one-hots are optimal would have been appreciated to strengthen the theoretical results / showing a “baseline” experiment.\n\nEventually, it seems that there is a correspondence between the structure of the triangle+9 points and prototype-based models ; and between the structure of uniform and cont. model. It could therefore have been interesting to discuss or show that each method is optimal (or better than the others) when the environment is structurally close to it. \n\n*Clarity*\n\nThe introduction, related section and the experiment preliminaries are well organized. The rest of the paper is harder to read. To me, it is only due to the organization of the paper and small changes could ease the reading. \n\nThe proof (Part 3.2) comes too early in the paper. Since no game is presented at this time, it is difficult to understand the assumptions of the proof and it seems that the proof merely applies for a denoising problem. \nIn addition, since the core of the paper is to present the advantage of the presented method compared to one-hots, this paper would benefit from additional details on the suboptimality of one-hots under assumption relaxation. Sub-paragraph (l128-133) could be extended to provide a better intuition on the suboptimality of the one-hot tokens.\n\nSection 5 may be bit too monolithic. Authors alternate environment presentations, results and analysis within the same 2 pages section. To me, it would have been clearer to use a more standard organization (environment presentation / results / analysis). In particular, it would have naturally impose to:\n- motivate the choice of environments\n- leave more space for technical details such as reward, losses (personally, equations would have been appreciated for the sake of clarity).\n - link the results of the different experiments (especially those of triangle+9 points and uniform)\n- provide an overall analysis that summarizes the results of all experiments.\n\n*Details*\n\nTechnical approach\n - 3.1 / 3.2. Formalization is quite clear.\n - The proof in subsection 3.2 is not clear to me. Notations could be improved (for example by using a different notation to refer to speaker and listener policy).\n - The proof is provided before presenting any game or environment. It is hard to understand if the proof applies for any game/environment. Can you give additional insights?\n- Agent Self-Play Environment: Even if mentioned in the text, it would be nice to have 1 or 2 formal equations defining the inputs/output/messages/rewards in order to have a clear idea of the setting (and the potential biases that could derive from it…)\n- Table could be completed with score boundaries (randomness/optimality) in order to have a better sense of the range of the quantitative results\n- It could be advantageous to motivate the choice of noise values (even one line telling that the overall analysis are independent on the value of the noise)\n- Human-Agent Experiments: l302-309: protocole would benefit additional explanation. It is not very clear the interest of doing the supervision \n\n[1] Jakob Foester, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiiteson. Learning to communication with multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, 2016\n\n[2] Angeliki Lazaridou, Alexander Peysakhovic and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In 5th International Conference on Learning Representations. ICLR 2017\n\n[3] Miss Tools and Mr Fruit: Emergent communication in agents learning about object affordances. Diane Bouchacourt and Marco Baroni (2019)\n\n[4] Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, Stephen Clark Emergence of linguistic communication from referential games with symbolic and pixel input. In 6th International Conference on Learning Representations. ICLR 2018\n\n[5] Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux and Marco Baroni. Anti-efficient encoding in emergent communication. Advances in Neural Information Processing Systems, 2019\n\n[6] On the Spontaneous Emergence of Discrete and Compositional Signals. Nur Geffen Lan, Emmanuel Chemla, Shane Steinert-Threlked. In Proceedings of the 58th Annual Meeting of the Association for Computation Linguistics (ACL) 2020\n\n[7] Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and Generalization in Emergent Languages. In Proceedings of the 58th Annual Meeting of the Association for Computation Linguistics (ACL) 2020\n\n[8] Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. Emergent Communication in a multi-modal, multi-step referential game. In 6th International Conference on Learning Representations. ICLR 2018\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "cf above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5h"}, "extra_text": {}}, {"review_id": "S3xIHX_BnFw", "reviewer": "Reviewer_BeFo", "summary": "This paper proposes to learn word embedding along with a communication protocol in multiple communication games. More specifically, a word embedding is a bounded vector, acting as a token for communication. To find a point in the middle of discrete and complete continuous communication channels, the author introduced a designated architecture for the communication policy: prototype-based communication. Namely, the agents can only communicate with z tokens (prototypes) from a c-dim hypercube. In their experiments, the authors show that the proposed prototype-based communication protocols are more robust to noise than one-hot communication protocols. They also illustrate better generalizability and semantic alignment of prototype-based protocols. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The present work is novel in that it goes beyond communication with discrete tokens but does not lose tractability in a continuous space. This is achieved by a designated communication policy that employs a fixed number of learnable and continuous prototypes as bases. Such a special architecture echoes our intuition that in communication we convey discrete concepts which can be represented with continuous prototypes. It also hints a direction to emerge word embeddings that reflects the semantic space, an unexplored question in the emerging literature of emergent communication. \n\n2. This work is in general of good quality in both analysis and experimental design. For analysis, the authors formulate a sufficient decision-making condition for one-hot communication being optimal, which supports the claim that one-hot communication may not optimal be if the reward matrix is not Identity or the prior is not uniform. All experiments are designed in accord with this analysis. However, it is unclear how this analysis connects with the main focus of semantic spaces. The experiments are interesting, though with some flaws and confusion to be detailed in Limitations. The authors try to study couples of interesting problems including (1) if the agents can exploit additional dimensions to handle noises, (2) if agents can cluster concepts that are proximal in the return landscape with the prototype-based communication, (3) if the emerged tokens can zero-shot generalize to unseen classes with the guidance from outsourced word embeddings, (4) if humans can understand emerged tokens in zero-shot. \n\n3. There is considerable room for improvement in clarity. For example, by \"clustering tokens with similar meanings\" (P5 L195), it is confusing if the authors want to cluster tokens together or cluster similar communication intents around one token. Section 6 and Table 3 are also quite confusing, see Limitations. \n\n4. The proposed question to study, to emerge semantic word embedding, could be significant if investigated properly. However, this paper mostly works with synthetic tasks, limiting the significance with considerable particularity. I think the significance can be improved if the authors can (1) provide a clear definition of how semantic space should connect with the reward function, (2) discuss the impact of the number of prototypes, (3) find a more natural way to incorporate pretrained word embedding. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would consider increasing my rating if the authors could resolve my following concerns:\n\n1. The number of prototypes is a hyperparameter and seems to need handtuning with domain knowledge. In the experiments, this number is always set to be the number of clusters underlying the communication intents (goals or classes). In real communication scenario, it is not reasonable to let agents know the number of clusters a priori. I wonder how the presented results would be affected by changing this number to an arbitrary value. \n\n2. The results reported in Table 1 need further consideration. One may be curious why in the triangle experiments agents perform better in a skewed distribution of goals than in uniform distribution. Moreover, in the triangle experiment, the authors compare (the prototype-based communication with z tokens in c-dim continuous space) to (c tokens in one-hot communication), which occurs unfair to me. What would happen if the one-hot agents are endowed with a z-length one-hot vector and a c-length one-hot vector? Are they still prone to noise? For the 9 Points environment, it is not clear if the authors set all goal states as absorbing states or just the target one. It looks suspicious since the final returns are higher than in the Triangle experiment. The authors claim that this experiment demonstrates the clustering behavior of prototype-based communication. But we cannot figure from the reported final returns if the listener successfully distinguishes the target from its neighbors. Does the design of reward that encourages reaching proximal distractors also induce ambiguity? \n\n3. The human-agent experiments can be interesting but are largely limited by the current confusing presentation. First, to achieve \"zero-shot\" communication in unseen classes, the authors introduce an auxiliary training loss to align the emerged word embedding with embeddings from other sources. But it becomes unclear how much the communication game affects the learned tokens. Apparently, the receiver may choose correctly by fitting to the outsourced word embeddings and generalize with the inherent regularity of such embeddings. Second, Table 3 and the explanation in L314 in P8 are quite confusing to me. The authors stated that Self in Table 3 is the trained communication vector, but why one-hot setup has trained communication vectors? Shouldn't their communication vectors be one-hot? Why agents even achieve the best performance in unseen classes with one-hot communication? Third, there is a considerable degree of overclaim in \"zero-shot human understanding of tokens\". The experiment reported in 6.2 basically test if humans can tell an emerged 2-d vector is close to vector representations of prototypes of \"animals\" classes or \"vehicles\" classes. Human subjects can exploit the particular design that one choice is from animals and the other is from vehicles and do a contrast. It is not the case that humans can still do that if they are asked to choose from two novel classes from animals. I would recommend the authors to reposition this experiment as testing the separatability of higher-level groups in the emerged tokens. Also, how dependent is this result to the particular reward of 0.75 for confusion within animal classes? Can other values also lead to a similar accuracy? Then a higher-level question follows: how would agents receive such rewards in real-world communication? ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "QAU7wCmDI0g", "reviewer": "Reviewer_V8Ya", "summary": "This paper studies a communication game where a speaker has to provide instructions to a listener via a noisy communication channel; the listener then has to execute a navigation task in a particle world. The communication setup differs from the commonly used one by providing the speaker and listener with a shared token embedding table. The authors find that training with a shared embedding table leads to more accurate communication, or at least better task performance by the listener.\n\nThere is a second contribution in the form of a set of experiments where human-agent pairs play a referential game on the CIFAR-10 dataset. By training on a subset of classes and evaluating on the held-out ones, the authors evaluate generalization performance under distribution shift, and find that both agents and humans successfully interpret communication about unseen data.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The technique proposed in this paper interpolates in an interesting way between straightforward differentiable communication (DIAL-style) and non-differentiable symbolic communication (RIAL-style). Because the agents use the same embedding table, which is subject to weight updates from the RL loss, more information flows between the agents than in a pure RIAL-like case, but less than in the case of a continuously valued vector being sent, like with DIAL.\n\nA comment: The tokens being transmitted in natural language are less similar at the level of their physical realization than at the level of their semantic meaning. Your setup encourages similarity of the tokens, rather than similarity of the elicited responses (semantics). One-hot tokens, when processed by a language model, are typically multiplied by an input processing matrix. A one-hot token effectively selects a column from that matrix, which acts like a token embedding for the purposes of the LM, i.e. the purposes of the listening agent. Similarly, the generating agent generates a vector which is projected+softmaxed into a one-hot form. The projection matrix used there contains effective embeddings in a similar way. The difference with respect to your solution here is that the embeddings are not shared between the speaker and listener. This is not a criticism of the proposed method, but more to point out that a setup with one-hot tokens often effectively also involves embedding tables, and it might be good to make that clear in the text.\n\nAnother comment: the set-up with a shared codebook, with a speaker / encoder and a listener / decoder bears a lot of similarities to a VQ-VAE. It is obviously not exactly the same, e.g. because the listener doesn’t reconstruct the original input in this paper, but executes some other downstream task. Still, a discussion of the similarities and differences wouldn’t be out of place.\n\nThe paper has a number of strong points in its execution: the agent-agent experiments aren’t simply about communication but about doing a downstream task; there are human-agent experiments involving communication about held-out stimuli, and the analysis of the agents’ behavior and codebook is well done.\n\nThere are some points for improvement though. One thing that would make the paper stronger is more agent-agent experiments, ideally with similar codebook analysis, but at least with more complexity in the task, since the particle world tasks don't ask a lot from the communication system. An additional baseline method to consider might be VQ-VAE-based, given the similarities - e.g. let the speaker figure out the codebook on its own, in a VQ-VAE, and then see how well the listener can learn to work with that.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Potential negative societal impact is not really discussed, but is also not very directly relevant. The limitations are not very different from the exisiting emergent communication literature; if anything, by showing successful human communication, the authors indicate their method is less limiting.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "6Z-QkIi0nuH", "reviewer": "Reviewer_fU14", "summary": "This paper presents a study of emergent communication where the communication channel is, according to the authors, composed of a finite set of continuous embeddings, rather than one-hot tokens representing unrelated words, as is typically used in the literature to simulate linguistic transmission.\n\nThe authors first show that, in a noisy channel communication task where the speaker selects from a finite set of embeddings and the listener's goal is to identify the index of the sent embedding, one-hot vectors are only optimal under certain restrictive assumptions, one of them being when the number of embeddings (vocab size) equals their dimensionality. They use this finding to motivate a communication game where the speaker selects from a set of dense embeddings, and show both in a suite of multi-agent RL tasks an an image-based reference game that semantic embeddings outperform traditional one-hot communication channels in various ways, including:\n\n1. They achieve higher reward and communication success than one-hot tokens\n2. Embeddings are interpretable and cluster according to the semantics of their corresponding messages, unlike one-hot vectors that are orthogonal and have no predefined relationships between each otheer\n3. Communication channels with semantic embeddings allow a human to play the role of both speaker and listener in this communication task (by generating novel utterances which are then embedded and passed to the listener, or by asking humans to interpret messages sent by a speaker assuming some 2D visualization of the semantic space)", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I was very confused by this paper. Unless I am missing something obvious, I feel like this paper has a fundamental understanding of how communication via one-hot tokens is treated in the emergent communication literature. Like most NLP applications, in speaker/listener setups there is indeed a component in the model where distinct words are treated as one-hot tokens with no inherent semantics. However, these one-hot tokens are invariably *embedded* by the listener, where the one-hot tokens are used to lookup continuous semantic word embeddings, which are then provided for further computation in the listener (e.g. via an LSTM or Transformer architecture). Often the speaker has its own embedding matrix, which helps it understand the semantics of its generated message during autoregressive language generation. In other words, these are *precisely* the word embedding mechanisms that the authors claim in Section 2.2 constitute motivation for their present idea.\n\nIn other words, I see no difference between what the authors propose and what the approach that standard emergent communication setups already use. Speakers already produce \"discrete communication in semantic spaces\" insofar as the generated one-hot tokens are used to index into a listener's learned embedding matrix. And this embedding matrix presumably has all of the benefits that the authors propose in this paper: semantic clustering (Figure 2), interpolation (Figure 4), interpretability (6.2), potential for zero-shot novel communication (6.1), etc.\n\nThis made me very confused as to what the \"one-hot\" baselines are - it is really hard to imagine an architecture that does not have an implicit embedding matrix (e.g. even if the vectors are fed as binary features into an MLP, the first dense layer serves the role of a word embedding matrix and presumably learns some semantics). I was thus very interested in how these one-hot baselines are formulated, but unfortunately the paper offers almost no architectural details on the agents in both the RL and reference game experiments.\n\nI think some of the ideas in this paper have merit - if we assume a noisy channel then I can see why communicating only distinct tokens might be too corruptible of a communication channel, which motivates communication via more \"continuous\"-y measures (presumably this is why we have words composed of reusable/partially redundant parts like characters and phonemes). But noisy channel communication is also an assumption that is not often explored in emergent communication, and so the derivation of the suboptimality of one-hot tokens in Equations 1-4 feels a bit out of place: here, the communication task is presented as a Shannon-style encoder-decoder noisy-communication task, where the goal of the listener is to recover the message. But the literature tends to formulate the objective in terms of listener performance on some downstream task, rather than trying to reconstruct the message, because reconstructing the message is assumed to be trivial. The framing of the paper should be updated to take this into account.\n\nFinally, some of the conditions that must be violated for one-hot tokens to be suboptimal don't make much sense to me. For example, in L130, why would the number of tokens and communication dimensions not be equal in discrete communication? The way to specify a vocabulary of size N for the agents is literally to define exactly N unique one-hot vectors corresponding to each word? If there are more token dimensions, then one-hot tokens don't uniquely identify each token. Similarly, if the number of tokens is less than the token dimensions, the extra token dimensions are completely superfluous. Again this doesn't make much sense to me given my own priors on how discrete communication is treated in the literature.\n\nTo summarize, this paper left me very confused, as it seems to fundamentally misunderstand how discrete communication is treated in emergent communication. I don't understand how the presented prototype models differ at all from standard emergent communication models and am not sure what kinds of existing models the one-hot baselines in the paper instantiate, due to lack of architectural details. This means I can't recommend acceptance, as the experiments in the paper do not seem to support some novel contribution, but rather the status quo of emergent communication setups. Indeed, similar experiments into the interpretability of learned embeddings (Lazaridou et al. ICLR 2017), generalizability to human language (Lowe et al., ICLR 2020), etc. have already been explored in depth. I am open to revising my review, and look forward to the authors response, which may be able to clear up my confusion.\n\n## Minor\n\n- Can figure 1 have more labels (what is student, teacher, etc, can you give an example of what a goal looks like?) I assume t_i is the target but I didn't see it in the text.\n- \"Uniform\" as a term feels overloaded in Section 5. E.g. Table 1 Dist = Uniform, and the Uniform environment.\n- Can we have z in table 1? also this is minor but I had trouble remembering what z and c correspond to. Could instead e.g. use |C| to denote vocab size, and h_c or c_dim for dimensionality.\n- L305 why \"BERT-like\"? Can you specifically name the model?\n- My understanding is that the Appendix should be submitted as supplementary material, otherwise this is possibly a page limit violation.\n- Meaning of \"In\" and \"Out\" in table 3 could be made more clear. Maybe mention parenthetically in L311-312\n- The framing of the 6.1 experiment is a bit odd. I don't think it's \"novel communication\" if the listener model includes a pretrained BERT model which is trained on a massive amount of external data and can embed a variety of expressions into a semantically meaningful space (the agents are precisely communicating in BERT embedding space due to the lexicon setting technique). BERT must be treated as a component of the listener, and if human generated referring expressions aren't novel to BERT, they aren't really novel to the listener.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
There are two over-arching themes in the reviews that highlight areas for improvement. First, the motivation and framing of the use of a discrete set of continuous vectors for communication as opposed to one-hot vectors was a source of confusion. This makes sense, as this paper is essentially bucking the trend of how most work in emergent communication both approaches the problem and even motivates that approach. So, some amount of inertial resistance is to be expected. The authors explain their perspective in a couple of different ways in the rebuttals that I found quite intuitive. (Side note: How does the stated advantage in extrapolating from limited experience, in response to reviewer fU14, compare with an approach using a sequence of one-hot vectors? Continuous would seem to be something like a limit point of this approach as the sequence of one-hots increases.) That reviewers got stuck on this point suggests that the paper can be improved by building readers' intuition (as authors have done for reviewers in their rebuttals) before getting to the bulk of the paper.

Second, the setup and presentation surrounding the experimental results could be improved. Although (as one data point), I was quite enthusiastic to see the human-agent experiments, the presentation of the results was not the easiest to parse. I think the issue here, and this may readily apply to the other experimental sections, is that the structure in these sections is very uniform. That is, I think the clarity would be improved by beginning each experiment with a truly high-level description and motivation, what are we trying to understand with this experiment, what are the aspects that are really important, etc; then with readers' minds properly primed, we would be ready for the details. As is, but not in all cases, the reader is left to build this structure on the fly as we read the section.

Neither of these areas are show-stoppers that should inherently prevent publication, but it is my opinion that they are largely the cause of the lower scores from reviewers and if addressed would make acceptance upon resubmission much more likely. Obviously, this is an interpretation based upon my read, the reviews and rebuttals, and the specific comments and suggestions of the reviewers should be considered in subsequent versions of the paper. As some reviewers noted, the paper would be further improved by expanding the agent-agent experiments, and I agree with this assessment, but could also see the paper getting accepted without it. Finally, I want to encourage the authors here as this paper has a very interesting perspective that I would like to see published and influencing the field. However, given reviewers' reservations, I will be suggesting it not be accepted at this time.


Decision: Accept (Poster)