Emergent Discrete Communication in Semantic Spaces
Authors: Mycal Tucker
Huao Li
Siddharth Agrawal
Dana Hughes
Katia P. Sycara
Michael Lewis
Julie Shah
Venue: NeurIPS 2021 Poster (2021)
Keywords: Emergent communication
zero-shot understanding
reference games

Abstract: Neural agents trained in reinforcement learning settings can learn to communicate among themselves via discrete tokens, accomplishing as a team what agents would be unable to do alone. However, the current standard of using one-hot vectors as discrete communication tokens prevents agents from acquiring more desirable aspects of communication such as zero-shot understanding. Inspired by word embedding techniques from natural language processing, we propose neural agent architectures that enables them to communicate via discrete tokens derived from a learned, continuous space. We show in a decision theoretic framework that our technique optimizes communication over a wide range of scenarios, whereas one-hot tokens are only optimal under restrictive assumptions. In self-play experiments, we validate that our trained agents learn to cluster tokens in semantically-meaningful ways, allowing them communicate in noisy environments where other techniques fail. Lastly, we demonstrate both that agents using our method can effectively respond to novel human communication and that humans can understand unlabeled emergent agent communication, outperforming the use of one-hot communication.

Emergent Discrete Communication in Semantic Spaces
Mycal Tucker!, Huao Li², Siddharth Agrawal Dana Hughes Katia Sycara?, Michael Lewis?, and Julie Shah
Massachusetts Institute of Technology
University of Pittsburgh
3 Carnegie Mellon University
sycal@mit, edu, hul52@upitt.em edu, siddhara@cs. cmu, edu
Abstract
Neural agents trained in reinforcement learning settings can learn communicate among themselves via discrete tokens, accomplishing as team what agents would be unable do alone. However, the current standard of using one-hot vectors as discrete communication tokens prevents agents from acquiring more desirable aspects of communication such zero-shot understanding. Inspired by word embedding techniques from natural language processing, we propose neural agent architectures that enables them communicate via discrete tokens derived from a learned, continuous space. We show in a decision theoretic framework that our tech- nique optimizes communication over wide range of scenarios, whereas one-hot tokens are only optimal under restrictive assumptions. In self-play experiments, we validate that our trained agents learn to cluster tokens in semantically- meaninnfuul ways, allowing them communicate in noisy environments where other techniques fail. Lastly, we demonstrate both that agents using our method can effectively respond to novel human communication and that humans can understand unlabeled emergent agent communication, outperforming the use of one-hot communication.
Introduction
longstanding goal of AI has been develop agents that cooperate with other agents humans to accomplish tasks together. Often, communication is necessary to enable such cooperation: the study of emergent communication has recently shown great success producing agents that learn to communicate. In reinforcement learning settings, guided only by environment reward, neural agents can learn to communicate by broadcasting numerical vectors each other (9] 31 [8).
Given this success, and in part inspired by the discrete nature of words in natural language, some researchers have focused emergent discrete communication by forcing agents to broadcast one-hot vectors [ (0) These tokens in effect become a lexicon used by agents. Studying when these tokens emitted allows researchers uncover their meanings, as well study the broader questions of what environment or agent factors contribute to desirable aspects of learned communication (e.g., compositionality or, in continuous communication settings, zero-shot understanding)
[5]
19 5 4
We claim that discretizing messages by constraining them to conform to one-hot hot vectors fundamentally precludes agents from learning relationships between tokens because each one-hot vector orthogonal to and equally far away from all other vectors. Conversely, research from natural language processing and word embeddings has long established the importance of learning representations of discrete words within a continuous, semantic space 29 32
desirable properties of language. One-hot vectors establish
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In this work, we demonstrate the benefit of agents that employ discrete set of tokens within continuous space over agents that use the standard practice of communicating via one-hot vectors in discrete emergent communication settings. We present novel architecture and implementation for learning such communication and provide decision-theorette analysis of the value of such an approach the congruence of meaning and form of communications. Simulation experiments confirmed these results: our agents learned an arrangement of tokens that clustered in Human--understndabbi patterns. The arrangement of discrete tokens within the learned communication space produced team performance that was robust to environment noise and enabled agents to effectively utilize novel communication vectors. In human-agert experiments, agents aligned their tokens with natural language embeddings and responded appropriately to novel English phrases. Lastly, we showed that humans capably interpreted unlabeled emergent communication tokens in reference game
2
Related Work
We propose a technique within emergent communication literature, drawing inspiration from work on word embeddings in natural language processing (NLP) and zero-shot classification.
2.1 Emergent Communication
Researchers of emergent communication study techniques to enable agents to learn to communicate among themselves, enabling high task performance in reinforcement learning settings (see 36 10 18
among others). These settings, such as reference games or Lewis signalling games [25], are designed such that agents must communicate to perform the task successfully; in "cheap-talk" scenarios, agents often learn successful communication strategies by sending real-valued vectors to each other. We focus on discrete communication emerging among decentralized agents that may communicate by sending one of a finite set of vectors to each other, but may not access other agents' weights or gradients during training or execution [8]. Previously, such discrete communication has often taken the form of one-hot messages: Foerster et al. [9] proposed binary discrete messages, but subsequent works seem to have reverted to one-hot vectors 13 III
Even when agents learn to communicate, they often fail to learn protocol that humans or separately- trained agents can understand; that is, they fail at the "zero-shot" learning problem [2]. To address this gap, several recent works have found properties of environments that encourage zero-shot continuous communication (5 Other research addresses gumman-undersandaan
communication
in RL settings. Using bottom-up approaches, researchers align emergent communication tokens with human language [2 26 Lazaridou al. (20 even demonstrated that humans can understand agents' tokens in zero-shot settings (e.g., interpreting the agent's 's token for "dolphin" as referring to photo of water). Complementary. "top-down" approaches leverage pre-trained task-specific language models for natural language communication 211.
We position our work within the field of emergent discrete communication: we seek to have agents learn to communicate via a finite set of possible messages, without access to a task-specific language model. Unlike prior work, we relax the assumption that discrete tokens must take the form of one-hot vectors and demonstrate that our proposed architecture enables agent utilization of novel communication, and supports human understanding of learned communications.
2.2 Natural Language Word Embeddings
While early neural NLP research used one-hot representations of words, most modern techniques rely upon dense word embeddings such as Word2 or GLoVE for faster training and greater general- ization power 2 29 32 I. In essence, these embedding techniques learn discrete representations of words within a semantically meaningful space; words representing similar meanings are often embedded in similar locations. We argue that current research emergent discrete communication that relies upon one-hot tokens is similar to the early one-hot word embedding techniques that suffered from limited generalizability. Instead, we propose that discrete emergent communication should be leameed within continuous, semantically meaningful space, like modern word embeddings.
Anonymized code available http:///wannnnooo..ooo..sseeeeea

---

2.3 Zero-Shot Classification
Research in zero and few-shot classification seeks to train classifiers that are able to correctly classify inputs belonging new classes despite seeing few training examples of that class [38 28 7 Often, techniques employ "side-information" to enable high performance, for example using labeled data from different domain (28 information from language (35] We focus emergent communication rather than classification, but in experiments aligning communication tokens with word embeddings, we similarly exploit side information for zero-shot understanding.
3 Technical Approach
3.1 Emergent Communication in Multi Agent Reinforcement Learning
We adopt standard methods for training multiple agents in decentralized partially-observable Markov decision process [3] The Dec-POMDP is defined by the (S, A, R. O, (2,7) tuple. is the set of states; A,Vi E [1 are the sets of actions, including communication, for each of N agents; S is the probabilistic transition between states due to joint actions. We focus settings to encourage communication among agents. S2 defines the set of
T: x AN...Av + on partially-observa possible observations; O; AN...AN S - maps from joint actions and the state to distributions of observations for each agent. Lastly, and R define the discount factor and reward function, respectively. The goal is to find the policies for all agents, Ti o E Ω A, that maximize the expected discounted reward. Multi-agent existing techniques for solving these problems in multi-agent reinforcement learning settings, using agent architecture develop.
well studied (30) 27 [34]; we borrow
3.2 Decision-Thean Noisy Discrete Communication
In this section, we formalize a noisy channel reference game, an instance of a Dec-POMDP. We derive that one-hot vectors optimal tokens under restrictive assumptions, and relaxation of those assumptions may lead to suboptimality of one-hot communication.
Consider two agents, speaker and listener, that communicate via dimensional noisy channel, Each episode composed two timesteps. the first timestep, a speaker emits token, 4, chosen from distribution over columns in matrix of tokens, T; Texx- In the second timestep. the listener observes a message, m E R a noisy version of t, corrupted by additive zero-mean noise. We constrain all tokens to fall within the unit hypercube (i.e., 0 T I 1; 00, cl, [0, =)), which constrains the relative scales of tokens and noise.
The listener's s task is to predict which token the speaker emitted, given the observed noisy message. assume there exists some reward matrix, Rzx= that specifies the shared reward among agents, where entry R[i,j] j] corresponds to the reward if the speaker emits 7[i] and the listener predicts T[i]: standard reward matrices resemble the identity matrix, potentially with some small, positive values off the diagonal for "reasonable" mistakes. From seccision-thoorete framework, the speaker and listener wish to maximize expected reward. We seek to find the optimal set of tokens, T", that maximizes this expected value, shown below:
P(T (TTj)I)))((T) Exer P(T m.) P(T(K)
1 arg max T
EP(TTU)
max Rli J] JES
(1)
(Tm)
P[] mm))
arg max
E PP
Evez
P(T[k]]mm))
Uniform prior; R
(2)
i6z
= =arg min Σ EP(m//T())
Σ P(T(ilmm)
(3)
iE3
JE:;$$$
arg max
Σ
[T]|] - ?
Gaussian noise
(4)
T
3

---

Equation I specifies the expected reward using a rational listener model: taking the expectation over tokens (the outer sum), the listener chooses the token T[j] that maximizes the expected reward given the observed message, my. Equation 2 follows under assumptions of (an equal number of tokens and communication dimensions), Equation 3 follows, deriving that the optimal tokens minimize the likelihood of confusion of any pair of tokens. Finally, assuming a Gaussian noise model with a variance matrix equal to ol for some constant o, Equation 4 states that the optimal tokens maximize the mean squared euclidean distance between tokens. If tokens are constrained within a unit hypercube, one-hot encodings are an optimal solution. A proof of optimality of one-hot tokens is included in Appendix A
= T=x2, and a uniform prior over emitted tokens.
While one-hot encodings may be optimal in some scenarios, the above analysis also reveals when they may not be: when the cost of emrrr or prior over tokens are not uniform, or when the number of tokens and communication dimensions are not equal. Examples of suboptimality of one-hot tokens when these assumptions are relaxed are included in Appendix [ Thus, although many academic experiments respect these assumptions, a more powerful framework for learning tokens is needed in the general case.
3.3 Discrete Prototype Communication
We present our method for prototype-based communication to address the limitations of one-hot communication by endowing agents with learnalee tokens. We create neural agents with communi- cation head instantiated as a multi-layer perceptron with a penultimate softmax layer (where we use the gumbel softmax trick (I4), which then is multiplied by a learnable matrix, Taxes for the final output. More formally, an agent, parametrized by weights O and T, produces communication token I according to Equation 5
= fo(O)
d one hargarg max(g, log mT))
(5)
1 d Taxe
Equation 5 states that agents produce logits, т, from an observation, O, convert the logits to one-hot vector, d, by using sampled 9i drawn i.i.d. from a Gumbel(O. 1) distribution, and then multiply d by T to produce the token Although the argmax operation is not differentiable, we use it in the forward pass (that is, actually using "hard" argmax) but use the gumbel softmax operation for calculating gradients, used in the backwards pass. We refer readers to Jang et al. [14 for more details on the gumbel softmax approximation.
We note that our technique introduces an additional set of learnable parameters compared to one-hot communicative agents, which would merely output d. This increased flexibility may enable improved performance but may also increase learning difficulty; experiment results, reported in Sections 5 and o indicated net benefits derived from our technique.
3.4 Opportunities for Behavior Shaping
The previous section illustrates how T may be learned using traditional reinforcement learning techniques; using a policy gradient method, the tokens will evolve during training to enable high task performance. While this emergent behavior is desirable for its flexibility, we also examined two ways of constraining learned behavior: -exiicon-etttin" and grounding.
The first constraint, which we call "lexicon- "lexicon-setting." consisted of manually specifying a non non-trainable T. During training, gradients passed through T but did not update its elements. This allowed designer to specify the set of tokens agents used to communicate, in other words setting the "lexicon."
The second constraint consisted of teaching grounding via small set of supervised data. Even with hand-specified tokens, it is unclear a priori what meanings agents will associate with tokens. To enable human-agent communication, we require common understanding, or grounding, of tokens.
In experiments, we constrained the tokens to the unit hypercube by passing t through sigmoid activation enable fair comparisons to one-hot encodings, but tokens need not generally obey such constraints.

---

Inspired by prior art in learning social conventions in RL settings 24, 26), we used small set of data mapping observations desired communications, The gradient the supervised loss (mean squared error of predicted communication vs. desired communication) was added to the policy loss from the RI, environment, leading the agent to use communication that matched the grounding data and enabled high task performance.
Experiment Preliminaries
4.1 Baselines
We compared our technique to two emergent communication baselines: continuous and one-hot communication. The continuous communication method (henceforth, cont) allowed agents to output real-valued vectors in the communication space, Rº, bounded along each dimension within (-1, 1) by using a tanh activation. The discrete one-hot baseline (henceforth, one-hot) allowed agents output one-hot vector within the communication space using the gumbel softmax trick. By definition, this constrained one-hot agents to output one of tokens.
All techniques were trained using the multi-agent deep deterministic policy gradient (MADDPG)
method proposed by Lowe et al. [27). a common policy-gradiee method that updates neural network weights maximize the expected discounted reward.
4.2 Hypotheses
Throughout experiments. assessed trained agents by measuring environmeent-specific reward under different scenarios. First, we measured self-play reward for agents trained together, as the environment reward they received during 500 evaluation episodes. Second, we measured aspects of zero-shot behavior and performance in reference game (introduced in Section 6). Measurements included listener agent's 's accuracy in selecting a target image when using novel communication generated by humans, as well as human participant accuracy in identifying target image based on unlabeled agent communication.
Using these metrics, we formulated the following hypotheses:
HI: noisy environments, prototype-based communication will enable agents to achieve higher self-play reward than one-hot tokens.
H2: Prototype-based agents, trained with word embeddings as tokens, will identify target
images in reference games based on novel, numan-generaan communication, whereas one-hot encodings will fail outperform random chance.
H3: Human participants will accurately identify target images in reference games based on tokens learned by prototype- based agents.
Our hypotheses were informed by prototype- based communication design, which we expected to clus- ter tokens with similar meanings, thereby providing robustness noisy environments and benefiting human understanding of the communication space. We expected this improved semantic understand- ing of the communication space to bolster zero-shot understanding novel communication.
5 Agent Self-Play Experiments
We first evaluated our technique in particle-world environments, inspired by the environments introduced by Lowe et al. (27 and Mordatch and Abbeel [3]] Agents were trained and tested in fixed teams; thus, these experiments were used to measure robustness noise and assess characteristics of learned tokens. Depictions of our environments are shown in Figure Full details of the environments included in Appendix(B [
In the first environment, triangle, speaker and listener agents were spawned at the origin, and three landmarks were located at the points of equilateral triangle. In each episode, landmark was designated as the target, and noise was drawn from a e = dimensional zero-mean Gaussian with E 0.9 Only the speaker observed the position of the target; thus, to maximize the reward
5

---

t
(a) Triangle
(b) Points
Uniform
Figure The particle environments used for agent-only testing. In the triangle and Points environments, the landmarks (colored circles) were in fixed locations; a target was chosen among the possible landmarks. In the uniform environment, the landmarks were in locations chosen uniformly at random within the game.
Table 1: Median (standard erorr) self-play reward over runs. Enabling Iearnable tokens endowed our agents with greater robustness to environmental noise.
Env. Dist.
Rewards
Proto
One Hot
Unif.
0.9
9
-124 (12)
-201 (10)
Tri.
Skew.
0.9
9
-100 (4)
-136 (9)
Figure 2:
The learned prototypes (left) reflect structure in the 9 points environment (right), with similar prototypes encoding similar locations. Pro- 2 9; a 0.3 exper-
Unif.
0.0
3
-60 (7)
-96 (10)
9-P.
Unif
0.0
9
-65 (2)
-66 (6)
Unif
0.5
9
-100 (10)
-132 (2)
totypes taken from iment conducted for visualization.
(negative distance from the listener to the target), the speaker must communicate to the listener, via 9D vector, and the listener must move the target.
We investigated the effect of different priors over tokens by changing the distribution over which landmark chosen as the target: in setting, targets were drawn uniformly. but in the other, targets were drawn from a skewed categorical distribution of [0.49, 0.49, 0.02]. Three learnable prototypes were defined for our technique (= 3). Note that this meant that our prototype technique had fewer tokens than the 9 for one-hol: we selected = 9 to study scenarios in which < c.
In second environment, 9 points, speaker and listener agents were spawned at the origin, with the speaker observing the location of the target and the listener observing its own state. Reward was similarly defined the negative distance squared from the listener to the target. 9 landmarks were spawned in fixed locations, depicted in Figure Db. This arrangement landmarks produced non-identity reward matrix: the listener confusing confusing two landmarks that were further apart. We used the 9 points environment in settings: two settings with = 0 and 3 our technique used 9 prototypes. These settings allowed us to study cases when 2 c. Results from the first environments, assessing robustness to noise, shown in Table 1
nearby landmarks was less costly than
9, and one noisy setting with 0.5 and 9. In 9 points,
The triangle environment results reveal two benefits our technique over one-hor: robustness to noise when than communication dimensions, agents used the additional dimensions for robust communication, as shown in the top row of Table I The three prototypes spread into distant corners of the 9-dimensional unit hypercube, creating a larger distance between tokens than one-hot encodings. Next, the second row shows that technique yielding greater robustness for tokens denoting the more likely landmarks. The one-hot technique, in contrast, learned to always go the location between the most likely landmarks.
and the ability to exploit non-uniform prior. First, when there were fewer tokens
able benefit from the non-unin non-uniform prior over landmarks by
The 9 points environment enabled us to study two other attributes learned communication tokens: the benefits of greater expressivity in channel-limine -imiteed domain (2 c) and clustering behavior due

---

0.5a + 0.5g
Table 2: Median (std. error) reward over 5 runs in Uniform. Continuous communication out- performed both discrete methods, which converged to similar dis- cretization behaviors the con- tinuous space.
C
3
-54
300
Method
Reward
Cont. -9..8 (3.8)
Tracing seeker paths (black lines) and final locations (red dots) revealed the discretization im- posed by prototypes.
Figure
Figure 4: Mcan destination of the seeker agent when given pairwise in- terpolations of prototypes from 0% (red) to 100% (purple).
Proto -13.66 (3.8)
One Hot
-111.8 (3.4)
to non-identity reward matrix when First, we confirmed in the third row of [Tble[]]haa in an environment with e = 3 but 9 landmarks, one-hot lacked the expressivity to uniquely denote each target; our technique decoupled the number of tokens and the communication dimension, allowing discrete tokens in a 3D communication channel. The baseline of = == 9, in the fourth row, confirmed that the 9 prototypes in 3D achieved the same reward 9 prototypes in 9D or 9 one-hot tokens. Lastly, the fifth row, we observed that increasing environment noise led to a graceful degradation of performance for prototype-bad bbssdeeeeeedddd--eoe agents, but worse failures for one-hot. Inspection of the learned prototypes reveals why: tokens denoting landmarks in the same cluster of 3 landmarks converged into clusters in the communication space, allowing the clusters to separate from each other, as shown in Figure 2 Thus, intra-cluster errors became more likely, but inter-cluster error likelihood decreased. In fact, the mean reward of -10 achieved in the noisy environment similar to the reward of -96 achieved by the 3D one-hot communication method: both of these techniques devolved to the behavior of treating clusters single points.
In our final environment, uniform, a seeker and observer agent were spawned at random locations in the world; two landmarks (target and distractor) were also spawned at locations drawn from uniform distribution over the world bounds. The observer agent observed the location of both landmarks; the seeker agent observed its own position and the id of the target (i.e., the first second landmark). Each agent communicated to the other at each timestep. To achieve high reward, designated as the negative distance from the seeker to the target, both agents had communicate: the seeker telling the observer which landmark was the target, and the observer responding with the location of the target. This environment, unlike the prior two, allowed communicate about continuous data. Prototype and cont agents communicated with c = 3; one-hot used 5, prototype agents used
assess how agents used discrete vocabulary
In the uniform environment, we found that our method discretized continuous data in a reasonable way and that the emergent tokens enabled some measure of interpolation between representations. First, our method and one-hot performed similarly to each other and worse than cons (Table 2 Given that the observer agent had to communicate about continuous values (the target's location) with a finite, discrete vocabulary, this is unsurprising. Plotting trajectories of a seeker agent over 200 episodes, the discrete pattern of locations emerges (Figure 3).
Lastly, we tested prototype-base agents with interpolations of learned tokens by overriding the ob- server agent's 's communication with pairwise interpolations of all tokens at 10% intervals. An example of the resulting behavior is plotted in Figure 4 (We note that interpolation for continuous agents was impossible because of the lack of discrete tokens, and one-hot precludes communication other than one-hot vectors.) Interestingly, we found that prototype-based seeker agents often responded to the interpolation of tokens with interpolation of behavior: for example, when observing the mean of two tokens, a and B, the seeker agent went to the mean of the locations it would have gone to when observing just just B. These initial findings motivated work in the next section zero-shot understanding of natural language.
Together, these three experiments provided support for our first hypothesis, H1. Our technique enabled agents learn tokens that provided greater robustness to noise the triangle and 9 points environments by separating and clustering tokens, confirmed visually in Figure
7

---

Table 3: CIFARIO zero-shot reference game mean reward (std. error). Prototype-based models the only
discrete communication method that enabled betler-than-random zero-shot understanding.
Method
Label In
Label Out
Self In
Self Out
AMT In
AMT Out
Cont (BERT)
91% (0.7)
55% (1.0)
91% (0.7)
68% (1.0)
73% (0.4)
53% (0.6)
Proto (BERT)
93% (0.7)
58% (1.0)
93% (0.7)
66% (0.9)
70% (0.4)
56% (0.6)
Proto (Hand)
75% (1.1)
62% (1.2)
80% (0.6)
58% (1.0)
One-hot
96% (0.2)
50% (0.8)
96% (0.2)
69% (1.0)
6 Human-Agent Experiments
Lastly, we tested how well our technique supported aspects of human- human-agent agent interaction in zero-shot settings. Results from the prior section already indicated that prototype-based agents learned a sort of semantic communication space: in 9-points, for example, similar target locations were encoded via similar prototypes, and in uniform, the seeker agent responded to interpolations of prototypes with interpolations of behavior. In order to enable suman-agenn interaction in experiments in this section, we aligned parts of human and emergent communication. We found that this alignment, in conjunction with semantically-men menningffll communication spaces, supported two forms of zero-shot understanding. First, agents were able understand novel human communication at test time (H2). Second, humans could understand emergent communication embeddings that agents learned (H3).
We conducted these experiments via reference game, a standard type of environment for emergent communication 12 In our environment, a speaker observed an image, drawn from the CIFARIO dataset, and emitted communication vector [16]. A listener agent observed the original image and a "distractor" image that belonged to different class, as well as the communication emitted by the speaker. The listener predicted in the second, final timestep which of the two images was the "target" observed by the speaker. Reward was defined via a matrix, R10x10. where R[i,]] set the shared reward if the target image belonged to class and the listener agent predicted We penalized some errors more than others according to high-level groupings of classes: correct predictions got reward 1, confusion within the set of 6 animal classes or 4 vehicle classes got 0.75, and predictions that belonged to the wrong group got O. We transformed this basic reference game into two zero-shot experiments.
image of class j.
6.1 Zero-Shot Agent Understanding
In our first zero-shot experiment, we assessed agents' ability to understand novel communication by withholding two image classes (one animal and one vehicle) from training. We trained teams from scratch for each of the 24 possible vehicle excluded combinations, resulting in 120 teams. All agents were trained in environments with o = 0.05; cont and prototype agents used c = 3, prototype agents used z = 10, and one-hot used 10.
To align human and agent communication, we augmented self-play training with 100 supervised examples of image-commn conee-communication pairs. These data were interleaved with self-play in training agents, biasing them match the desired policy, as done in prior social convention literature [24 37 26).
For cont and prototype agents, we used BERT-like model to create communication by embedding the class label names 33]; one-hot used the classification label. An additional set prototype agents was trained with hand-designed word embeddings that clustered vehicle and animal embeddings. See Appendix D for details on embeddings and image feature extraction. For the prototype agents, we used the lexicon-setting approach from Section 3.4 to set T' equal to the label embeddings.
During evaluation, we measured listener prediction accuracy over 500 trials when using different types of communication or images. In "in-distribution" experiments, we evaluated models using the 8 image classes seen during training: in -iat-----istiiutiin experiments, we used only the image classes never seen in training. Furthermore, we tested three types of communication vectors: those produced by the trained speaker agent (Self), by embedding the class label (Label), or by using embeddings of captions generated by human annotators on Amazon Mechanical Turk (AMT). Details of the user study for generating AMT descriptions are included in Appendix [F] in essence, participants were asked to provide descriptions of images, replicating the role of the speaker agent.
8

---

Results, shown in Table supported hypothesis H2: prototype listener agents supported two types of zero-shot communication. AMT In experiments reflected one type of zero-shot understanding by forcing listeners to interpret novel communication at test time. Despite training with a small, finite vocabulary, prototype listeners performed almost on par with cont agents. Conversely, one-hot encoding precludes embedding novel communication. AMT Out and Label Out results reflected second type of zero-shot understanding: listeners simultaneously observed novel images and novel communication. In these settings, our method outperformed cont and the random performance of one-hot. By testing prototype agents trained using BERT or hand-generated embeddings, we found that different embedding methods played an important role in agent performance. Lastly, in ablation studies, we confirmed that both the reward matrix and environment noise contributed to agents' zero-shot understanding (Appendix D
6.2 Zero-Shot Human Understanding of Tokens
In our final experiment, we studied whether humans could understand agent communication in zero-shot settings, is essence treating humans listeners in the reference game. We trained prototype agents in the standard reference game, communicating in 2 dimensions with all 10 classes and no explicit guidance on the tokens. See Appendix [ for full training details. In evaluation, we showed participants 8 of the 10 learned tokens, annotated with English class labels. Finally, we plotted one of the held-out tokens with no label and showed participants two images, one for each of the two held out classes, and asked participants to select the image they thought the new communication most likely referred to. We recorded participants' selection accuracies among animal-vehicle image pairs for 5 prototype-based models trained from scratch, and a 2D PCA visualization of BERT-baasd embeddings of labels.
Averaged across the 5 prototype models, with an average of 300 responses per model, participants selected the correct image 70% of the time (standard error of 5%). This compares favorably to the 66% achieved using embeddings derived from the BERT-based model (x (1, N = 2093) = 3.21, p .073). While prior art has studied how humans interpret labeled agent tokens in new contexts (e.g., interpreting a one-hot token for "dolphin" as referring to a photo of water in Lazaridou al. (20), we believe this is the first demonstration of human participants understanding novel tokens. Ultimately, these results provided support for our final hypothesis, H3.
7 Contributions
In this work, we proposed technique to address current gap in existing discrete emergent com- munication literature. Prior work in this domain relies upon one-hot tokens for communication, but insights from NLP, as well as scheme is inadequate in many settings. Using a technique we developed to enable learnable, discrete tokens, we demonstrated benefits in self-play, including increased robustness to environment noise. In human- human-subject subject experiments, we showed improvements over the random performance of prior art in zero-shot understanding of human communication. Lastly, we demonstrated that prototype agents learned emergent communication tokens that were tuman-interepretable and semantically meaningful. Overall our experiments provided support that our prototype-based communication method produced semantically meaningful communication spaces in noisy environments with non-identity reward matrices, which in turn enabled zero-shot understanding of communication.
analysis. demonstrate why such an encoding
Our work provides a first step towards connecting theories of emergent communication and word embeddings. While we have taken step this direction through our proposed technique, future work leveraging insights from discrete representation learning could likely further improve upon our results. In addition, many of the techniques used to examine word embeddings could be used to analyze emergent communication tokens. Specifically, we note that prior literature has found reinforced biases in word embeddings; similar care of encoding desirable properties in emergent tokens must be taken. Lastly, further experiments examining environmental factors that contribute to zero-shot understanding of discrete communication could yield important insight into effective agent training and language evolution.

---

Acknowledgments and Disclosure of Funding
We thank Jacob Andreas for early discussions about embeddings in NLP, and the NeurIPS reviewers for their careful feedback. This work has been funded by ARL award W91
References
[1] Felipe Almeida and Geraldo Xexéo. Word embeddings: 1rXiv:190..00669 2019.
survey.
arXiv preprint
[2] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. The journal of machine learning research, 3:1 137-1155, 2003.
[3] Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4):819-840, 2002.
[4] Kalesha Bullard, Douwe Kiela, Joelle Pineau, and Jakob Foerster. Quasi-equivalence discovery for zero-shot emergent communication. arXiv preprint arXiv:2/03.08067, 2021.
[5] Kalesha Bullard, Franziska Meier, Douwe Kiela, Joelle Pincau, and Jakob Foerster. Exploring zero-shot emergent communication in embodied multi-agent populations. arXiv preprint arXiv:2010.15896 2020.
[6] Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encoding in emergent communication. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[7] Long Chen, Hanwang Zhang, Jun Xiao, Wei Liu, and Shih-Fu Chang. Zero-shot visual recognition using semantics-preserve adversarial embedding networks. In Proceedings of the IFFE Conference on Computer Vision and Pattern Recognition, pages 1043-1052, 2018
[8] Tom Eccles, Yoram Bachrach, Guy Lever, Angeliki Lazaridou, and Thore Graepel. Biases for emergent communication in multi-agent reinforcement learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. A'lch--Buc E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[9] Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[10] Matthew John Hausknecht. Cooperation and communication multiagent deep reinforcement learning. PhD thesis, 2016.
[11] Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning to communicate with sequences of symbols. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[12] Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "Other-play" for zero-shot coordination. In International Conference Machine Learning, pages 4399-4410. PMLR, 2020.
[13] Shariq Iqbal.
Maddpg-pytorch.
https: http:s///gttuub..comm//arggggggp
maddpg pytorch] 2018
[14] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, 2017.
10

---

[15] Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge 'naturally' in multi-agent dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2962-2967, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
[16] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 (canadian institute for advanced research).
[17] H. W. Kuhn and A. W. Tucker. Nonlinear programming. In Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability, 1950, pages 481-492, Berkeley and Los Angeles, 1951. University of California Press.
[18] Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. arXiv preprint arXiv:2006. 02419, 2020.
[19] Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In 6th Interna- fional Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
[20] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In 5th International Conference on Learning Rep- resentations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings. OpenReview.net, 2017.
[21] Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7663-7674, Online, July 2020. Association for Computational Linguistics.
[22] Jason Lee, Kyunghyun Cho,, and Douwe Kiela. Countering language drift via visual grounding. In EMNLP-IJCNLP 2019 2019 Conference Empirical Methods in Natural Language Pro- cessing and 9th International Joint Conference on Natural Language Processing, Proceedings the Conference, EMNLP-IJCNEL 2019- 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, pages 4385 4385-4395 Association for Computational Linguistics, 2020.
[23] Jason Lee, Kyunghyun Cho, Jason Weston, and Douwe Kiela. Emergent translation in multi- agent communication. January 2018. 6th International Conference Learning Representations, ICLR 2018; Conference date: 30-04-2018 Through 03-05-2018.
[24] Adam Lerer and Alexander Peysakhovich. Learning existing social conventions via observa- tionally augmented self-play, aaai. In ACM conference Artificial Intelligence, Ethics, and Society, 2019.
[25] David Lewis.
Convention: A philosophical study. John Wiley & Sons, 2008
[26] Ryan Lowe, Abhinav Gupta, Jakob N. Foerster, Douwe Kiela, and Joelle Pineau. On the interac- tion between supervision and self- self-play in emergent communication. In ICLR. OpenReview.net, 2020.
[27] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compettiivee environments. Neural Information Processing Systems (NIPS), 2017
(28) Qingjie Meng, Daniel Rueckert, and Bernhard Kainz. Unsupervised cross-domain image classification by distance metric guided feature alignment. In Medical Ultrasound, and Preterm, Perinatal and Paediatric Image Analysis, pages 146-157. Springer, 2020.
[29] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed rep- resentations of words and phrases and their compositionality. In C. J. C. Burges, L. Botton, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
11

---

[30] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 1928-1937. PMIR, 2016.
[31] Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi- agent populations. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018.
[32] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. Proceedings of the 2014 conference on empirical methods natural language processing (EMNLP), pages 1532-1543, 2014.
[33] Nils Reimers and Iryna Gurevych. Sentence-bert Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019.
[34] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707 7077003477 2017.
[35] Richard Socher, Milind Ganjoo, Christopher D. Manning, and Andrew Y. Ng. Zero-shot learning through cross-modal transfer. In Proceedings of the 26th International Conference on Neural Information Processing Systems- Volume NIPS' 13, page 935-943, Red Hook, NY, USA,
2013. Curran Associates Inc.
[36] Sainbayar Sukhbaatar, arthur szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[37] Mycal Tucker, Yilun Zhou, and Julie Shah. Adversarially guided self-play for adopting social conventions. arXiv preprint arXiv:2001.055: 055944, 2020.
[38] Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251 -2651,2226,2221 2018.
[39] Fisher Yu, Dequan Wang, Evan Shelhamer, and Trevor Darrell. Deep layer aggregation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2403- 2412, 2018
[40] Lukasz Kuciński, Pawel Kotodziej, and Piotr Milos. Emergence of compositional language in communication through noisy channel. 2020.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 3.3 on learning, and
Section on broader limitations.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 7 on how word embeddings and our technique could reinforee biases.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] In our decision- theoretic analysis, analyze what assumptions lead to one-hot tokens being optimal communication tokens.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A
12

---

3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes] See footnote
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendices.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See standard error, reported in all tables.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix C
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section 4.1
(b) Did you mention the license of the assets? [Yes] See Section 4.1
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
No new assets
(d) Did you discuss whether and how consent was obtained from people whose data you'l re using/curating? [Yes] Data were not published; details on user studies are in Appendices.
(e) Did you discuss whether the data you are using/curating contains personally identifi- able information offensive content? [Yes] Data identifying, as shown in user study details Appendices.
published and were
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [Yes] See Appendix IF]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [Yes] See Appendix [F]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] See Appendix
13


Reviews:
[{"review_id": "X1hkr072zjp", "reviewer": "Reviewer_HBxw", "summary": "This paper extends research in multi-agent emergent communication using discrete tokens. The agents learn to communicate across a noisy channel. The main idea is to have the agents learn to embed discrete tokens into a continuous space, similar to NLP techniques such as word2vec. The authors show that the learned tokens are interpretable to other agents in self-play. They also show that these messages can be adapted to understand embeddings produced by BERT or human labels, and that humans can in turn interpret the embeddings.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I enjoyed reading this paper. The interpretability the actions and communications of an RL agent is an important problem which the paper clearly and thoroughly studies. The main idea is simple to understand and well motivated. The results are in simple settings.\n\nPros: \n1) Equations are explained clearly and the paper is well written.\n2) Experimental results show clear improvement over the baselines in the first two environments.\n3) Experiments and ablations are thorough. I particularly liked the results on human-interpretability of the agent messages. \n\nCons:\n1) It would be nice to see some agent-only environments beyond particle worlds, and tasks beyond navigating to the correct -landmark. Similarly, it would be interesting to see performance on a slightly larger set of image classes. \n2) The prototype agent doesn't seem significantly different from the cont agent on the zero-shot reference game. \n3) 100 supervised examples in the training set is not quite the same as zero-shot learning. \n\nQuestions:\n1) Do the agents communicate once per episode, or continuously? If the latter, do the messages change? \n2) What are the one hot agent messages when c < z? \n3) In Section 5 Figure 3, why are the final locations only roughly clustered? Given one of five messages, the seeker agent should reach one location and stop to maximize expected reward. Figure 4 seems to support this. \n4) In Appendix D, it would be interesting to see the non-identity reward matrix. \n5) For the zero-shot human understanding experiments, are the held out classes always 1 animal and 1 vehicle? Is there any interesting pattern in human errors e.g. classes that are conflated consistently? Was the same experiment done with the cont model? \n\nDetails:\n- (Section 5 Table 1) - Some additional noise ablations would be interesting.\n- (Section 5 Table 2) - Some additional experiments varying z would be interesting. \n- (Section 6.2 Line 331) - is essence -> in essence.\n- (Section 6.2) - Add reference to the results in Appendix F/Table 8. A reference to Figure 8 is also helpful. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors address the limitations and social impact of this work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "OQCU0iu1Ztm", "reviewer": "Reviewer_nATd", "summary": "The paper argues that using discrete tokens in emergent communication fundamentally limits what agents can learn.\n\nMethod: Instead, they propose to learn an embedding matrix in addition to a token selection policy for use in emergent communication. The embeddings are trained in 3 different ways: supervised, learned from scratch, and manual specification. This approach is corroborated by analyzing a noisy channel reference game in an information-theoretic manner.\n\nTheir self-play tasks are in a particle-world environment:\n- In _triangle_ a speaker must communicate which point of the triangle the listener should go to.\n- In _9 points_, a speaker must communicate which point of the triangle the listener should go to. But, the 9 points are grouped into 3 groups of 3. The agent is rewarded based on distance, so going to a point in the same group as the target is less costly.\n- In _uniform_, the positions of two landmarks are sampled randomly. A seeker observes the id of the target and an observer observes their locations. They must communicate to guide the seeker to the target.\n\nThey find that their proposed discrete prototype model outperforms a one-hot representation. In the _9 point_ task, the embedding space also reflects the point grouping.\n\nIn their zero-shot human communication experiments, they train speakers and listeners on a referential game over pairs of CIFAR images. They compare learned embeddings performance against class-label embeddings, and embeddings trained on sentences generated by humans for the task. The aim of these experiments was to show that agents were able to generalize to human communication and to generalize to novel communication settings (ie to images unseen before).", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Adding embedding tokens to an emergent communication task is a sensible contribution. The creativity of this paper shines in the way the experiments were designed. E.g., the 3 evaluation tasks were well designed to understand how the agents change communication strategies as imprecision becomes less costly (triangle -> points) and how agents use discrete tokens to reason about continuous phenomenon. Changing out the different kinds of embeddings in the CIFAR experiments was also a clever way of understanding how the communication protocol changes across different tokens.\n\nQuality: the experiments were thorough and well executed (e.g., median and standard error reported, experiment design and rationale was highly detailed.)\n\nClarity: The paper was very straightforward to follow. From the experimental section, I felt I could reproduce the results. Specifically, there visualizations communicated the key concepts and environments very effectively.\n\nSignificance: Adding embeddings as tokens in emergent communication could open up many avenues for analysis going forward. E.g., the paper already demonstrated how embeddings can better demonstrate how humans understand emergent communication protocols and how communication can be transferred to new settings (such as differentiating between new images).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, they address limitations under the contributions section.", "review_text": "", "extra_scores": {"ethical_concerns": "The authors exhibited good general ethical conduct when conducting human experiments.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "76C-WyzB7eZ", "reviewer": "Reviewer_AvnV", "summary": "The authors present theoretical and experimental results that support the use of employing discrete tokens derived from continuous word embeddings in place of the standard practice of using one-hot vectors as symbols ('words') in emergent communication. The new approach is evaluated using particle-world environments (Lowe et al 2017), in terms of zero-shot reward in a standard referential game on CIFAR10, and in human evaluation. The main contributions are 1) a practical implementation of the technically sound idea of using discretised embeddings to reflect semantic similarities in language, instead of orthogonal one-hot vectors 2) new theoretical results on the optimality of one-hot tokens as vocabulary, and a theoretical analysis of the cases when one-hot tokens might be suboptimal (when the cost of errors or prior over tokens are not uniform, or when the number of tokens and communication dimensions are not equal) 3) experimental results of using those embedding-based tokens, in terms of robustness in noisy environments, interpretability and zero-shot generalization in agent-to-agent and human-to-agent communication.  The authors examine two behaviour shaping constraints that can be used to learn those tokens: 'lexicon-setting' (manually specified and non-trainable tokens) and token grounding using a small supervised set.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths:\n\nThe authors expand on the existing work on bridging emergent communication and NLP (Lazaridou et al 2020) and propose a sound implementation of word embeddings in multi-agent communication with a discrete channel.\n\nThe results are promising in terms of practical human-agent communication.\n\nThe tokens are shown to be meaningfully related (e.g, a mean of the tokens representing locations $\\alpha$ and $\\beta$ represents the actual mean of the two locations on a grid), which opens new possibilities in terms of interpretable communication protocols and zero-shot generalization in multi-agent tasks.\n\nThe authors include anonymised code in the submission. \n\nIn terms of writing and structure, the submission is significantly above average: for example, the authors clearly state the hypotheses which motivate their experiments. The appendix is detailed and the figures are of (above average) quality. Well done!\n\nThe authors motivated their answers in the Checklist.\n\nThe writing is quite dense in terms of the amount of technical content, but it is clear and coherent, with no unjustified claims.\n\n## Originality:\n\nThe contributions are original and the previous work in multi-agent communication is appropriately cited.\n\n## Quality:\n\nAs mentioned in the Strengths section, both the content and the presentation are of high quality.\n\n## Questions to the authors:\n\nIn Table 3, how many runs did you use to compute the mean reward and standard errors?\n\nHave you observed structural, semantic or pragmatic language drift (following Lazaridou et al 2020) in your experiments?\n\n## Significance:\n\nThe paper proposes a way of deriving a semantically interconnected yet discrete vocabulary in multi-agent communication, which might influence the research on discrete, interpretable communication channels. Zero-shot generalization, interpetability and various means of incorporating a priori knowledge in the agents' protocol based on the task are of interest to the research communities (emergent communication, deep RL) working on multi-agent communication.\n\n## Notation:\n\nLine 108: ‘ in a matrix of $z$ tokens’ suggests that $z$ is a number whereas in Equations (1)-(4) it is used as a set ($i \\in z$). Please change the limits of the sums and the maxima to something like $1 \\leq i \\leq z$.\n\nLine 124: The usual notation for variance is $\\sigma^2$, $\\sigma$ suggests standard deviation. \n\n## Typos: \n\nLine 7: ‘architectures that enables’ -> ‘architectures that enable’\n\nLine 12: ‘allowing them communicate’ -> ‘allowing them to communicate’\n\nLine 138: ‘$T_{z \\times c}$’ -> ‘$T_{c \\times z}$’\n\nLine 244 and line 245: a hyphen should be replaced with a minus -> $-100$, $-96$\n\n## Latex:\n\nLine 93 $\\forall i \\in [1,N]$, $A_i$ is a set of actions\n\n'...' in equations should be replaced with '$\\ldots$'\n\n## Linguistic comments:\n\nLine 117 ‘We seek to find’ -> ‘We seek’\n\nLine 138 ‘which then is multiplied’ -> ‘which is then multiplied’\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper lacks the Broader Impact section. Suggestions on how the authors can comment on the broader and societal impact to strengthen their paper:\n\nDiscuss the examples of real-life applications of agent-to-agent communication and human-to-agent communication, and the benefits of human-interpretable protocols (refer to the Section 6.2 Zero-Shot Human Understanding of Tokens: the token-based approach achieves higher results than a BERT-based model, which is widely used).\n\nRegarding social impact, the authors could discuss the possible dangers of using the work of human annotators on Amazon Mechanical Turk (AMT) (Section 6.1.).\n\nThe authors briefly commented on the biases found in the existing word embeddings, in the future work paragraph. This point could be developed in the Broader Impact section to strengthen the paper. \n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "11 hours"}, "extra_text": {}}, {"review_id": "3v5i0psJJN", "reviewer": "Reviewer_SUhS", "summary": "The authors extend previous work on communication in multi-agent reinforcement learning in 2-agent games involving a speaker and a listener, where the speaker generates a token and the listener performs an action informed by the token it receives. The contribution of the present paper is to replace the usual one-hot encoding of tokens with learned tokens in a richer, emergent encoding space. A mathematical justification for this step is given, and its advantages are evaluated in a variety of settings, including settings where humans take the part of either listeners or speakers. These experiments, which show interpolation and zero-shot generalisation, vindicate three hypotheses the authors set out concerning the advantages of the proposed learned encodings over one-hot encodings.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper offers a well motivated contribution to the sub-field of multi-agent RL agent communication, and presents a solid evaluation of the resulting model that demonstrates its advantages. The paper is mostly well written and clear. Overall the paper looks good to me, and deserves to be accepted.\n\nFor me, section 3.2 (the mathematical motivation) didn’t add very much to the paper, and the main findings stand for themselves. But other readers might take something useful away from the maths here.\n\nI have a few questions and minor issues.\n\nThe authors describe their method as using \"prototypes\". But what exactly is a \"prototype\"? Is this the same as a learned token (so there are z prototypes)? I assume this is the case. But then “prototype” seems like a strange word to choose, and I'm not sure what the intuition is. Either way, the term isn’t defined anywhere, as far as I can see. Maybe the authors could remedy this.\n\nIn Tables 1 and 2, why do you report the median rather than the mean?\n\nLine 200: “Agents were trained and tested in fixed teams” - Presumably a “team” is a pair of agents: a listener and a speaker? Maybe the authors should make this explicit.\n\nThe nature of the listener agent in the particle environment could be described briefly in the main text (if there's space). I gather from the Appendix that it moves according to (somewhat) realistic physics (as opposed to making discrete steps in a grid world). Do its actions comprise a continuously-valued force-direction pair?\n\nWords in human language don’t typically cluster according to semantic in the way advocated here (the “arbitrariness of the sign”). Perhaps, if there's space in the final paper, the authors could discuss this issue. Maybe the richer encodings here are analogous to multiple words rather than single words? (Eg: \"fluffy animal\" and \"dangerous animal\" are both semantically close and lexically similar.)\n\nTYPOS\n\nLine 139: upper-case theta shoud be lower-case\n\nEquation 5: Should T_{c x z} be T_{z x c} as in line 138 of the text? Or is it the text that has them the wrong way around? (I don’t think you need those subscripts in the equation anyway)\n\nLine 331: “is essence” -> “in essence”", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In several of the experiments, the \"continual\" encoding method out-performs the \"prototype\" method. Perhaps the authors could discuss this.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work studies referential games in between a speaker and a listener and proposes to replace the commonly used one-hot encoding of tokens with learned tokens in an encoding space. Moreover, the authors designed a good experimental setup to help them test and understand the effect of their approach and a thorough evaluation, including evaluation with humans.

All reviewers agree that this is a solid contribution to the field of emergent communication, positively commenting not only about the general structure of the paper on putting forward a hypotheses and then thoroughly testing it with targeted experiments, bit also commenting on the creative experimental setup but also the interpretability aspects of their proposed method.

This is a good contribution to the field of emergent communication and hence I suggest acceptance of this work.
Decision: Accept (Poster)