INTERNAL-CONSISTENCY CONSTRAINTS FOR EMERGENT COMMUNICATION
Authors: Charles Lovering
Ellie Pavlick
Venue:  (2020)
Keywords: Emergent Communication
Speaker-Listener Models

Abstract: When communicating, humans rely on internally-consistent language representations. That is, as speakers, we expect listeners to behave the same way we do when we listen. This work proposes several methods for encouraging such internal consistency in dialog agents in an emergent communication setting. We consider two hypotheses about the effect of internal-consistency constraints: 1) that they improve agents’ ability to refer to unseen referents, and 2) that they improve agents’ ability to generalize across communicative roles (e.g. performing as a speaker de- spite only being trained as a listener). While we do not find evidence in favor of the former, our results show significant support for the latter.

Under review as a conference paper at ICLR 2020
INTERNAL-CONSISTEENCY CONSTRAINTS FOR EMER- GENT COMMUNICATION
Anonymous authors Paper under double-bling review
ABSTRACT
When communicating, humans rely on internally-consistent language representa- tions. That is, speakers, we expect listeners to behave the same way do when we listen. This work proposes several methods for encouraging such internal- consistency in dialog agents in an emergent communication setting: shared em- beddings, symmetric encoding/decoding, and self-play. We consider two hy- potheses about the effect of internal-consistency constraints: 1) that they improve agents' ability to refer to unseen referents, and 2) that they improve agents' ability generalize across communicative roles (e.g. performing as a speaker despite only being trained as listener). While we do not find evidence in favor of the former, our results show significant support for the latter. In particular, we show that, when self coliisnnsst is enforced via self-play, agents are able to perform in novel roles as well as if they were trained with direct supervision those roles.
INTRODUCTION
Emergent communication is the study of how linguistic protocols evolve when agents are tasked to cooperate. For example, agents engaged in a simple object retrieval task learn communicate with one another in order to get the items they want (Lazaridou et al., 2018). To date, work of this type has each agent assume a conversational role. Thus, agents are often trained only to speak or only to listen (Lazaridou et al., 2018), or similarily trained to speak using vocabulary disjoint from the vocabulary is understands as a listener-e.g. speaking only to ask questions ("what color?") and listening only to comprehend the answer ("blue") (Kottur et al., 2017; Das et al., 2017).
These assumptions are misaligned with how we think about human communication, and with the way we't d like computational models work in practice. As humans, not only between roles, we also know that there is inherent symmetry between these roles: we expect others to speak (or listen) similarly to the way do, and we know that others expect the same of us.
easily shift
We test if dialog agents that incorporate the symmetry between themselves and their communicative partners learn more generalizable representations than those which do not. We introduce three mod- ifications to the agents to encourage that they abide by the "golden rule": speak/listen as you would want to be spoken/listened to. Specifically, these modifications include self-play training objectives, shared embedding spaces, and symmetric decoding and encoding mechanisms that share parameters. We test two hypotheses about the effect of the proposed modifications on emergent communication:
1. Internal-consistencyy constraints improve agents' ability to generalize unseen items-e.g. training on "red square" and "blue circle" and then testing on "blue square".
2. Internal-consistency constraints improve agents' ability to generalize across communica- tive roles-c.g. training "blue" as listener, and using "blue" speaker when testing.
We evaluate the effect of each of the proposed modifications with two reference game datasets and two model architectures, an RNN model used by Lazaridou et al. (2018) and a Transformer model. improves generalization to unseen items
We find no evidence to support that internal-consistens (Hypothesis 1), but significant evidence that these proposed constraints enable models to generalize leamned representations across communicative roles (Hypothesis 2), even in the case of where the agent receives no direct training in the target (test) role. All of our code and data are available at bit. 1y/ nttenal letternall-oone i stencc--emee
commun cation.

---

Under review as a conference paper at ICLR 2020
"blue triangle"
Alic
BoB
Alice
BoB
Alice
BoB
Alice
BoB
Figure 1: Reference game. Alice wants the blue triangle, and asks Bob to get it for her. He selects the green square, and Alice tells him "No" and shows him the item she wanted.
EMERGENT COMMUNICATION VIA REFERENCE GAMES
Following past work emergent communication (Lazaridou et al., 2018), evaluate our models in cooperative reference game (depicted in Fig. ()Lewss 2008). A basic reference game consists of two agents and set of items (each vector of attribute vallue pairs). One agent (the "speaker") has a target item in mind, and must describe to the other agent (the "listener") ) After training, we primarily interested in the properties and the generalizability of the resulting lexicon: the mapping from symbols in the vocabulary onto items and/or their attributes.
Implemented naively, agents converge trivial lexicon which assigns unique symbolic name each item in the training set (Kottur et al., 2017). Thus, a challenge in emergent communication is build agents which learn lexicon that maps symbols attributes other primitive concepts, so the agents can generalize to novel combinations of attributes at test time. For example, trained on items {[ [[sp circle, color: I red], [shape: square, color bbuue]] an ideal lex- icon would map each attribute-value pair to a unique word (e.g. wordl=blue, word2=ci rele) so the agents could refer to an unseen combination ([shape: cireee,,rile,,.eee color: bluu)) at test time.
Like past work, this paper focuses on evaluating inductive biases that result in lexicons which gen- eralize to novel items. addition, we focus on different type of generalization: generalization across communicative roles. Prior work has tended to treat these roles as distinct, such that an agent may learn to comprehend "blue" and "circle" perfectly as a listener, and yet unable to refer to [shape: circle, color: blue] as a speaker. Thus, we test whether models that are trained in both roles (either with direct supervision, or only via self-play) learn lexicons that not only generalize to novel items but also which can be transferred across roles.
Notation.
The space of possible references is parameterized by the number attributes ny that deseribe each item (e.g. color) and the number of values n. each attribute take (e.g. (red, blue)). Each item o is a bag-of-feature, vector o E {0, 1} where o if o expresses the ith feature value. The speaker produces message with symbols from vocabulary V with length L. For comparison, we use the best-performing setting [V] 100 and 10 from previous work (Lazaridou et al., 2018). Symbols in V are represented 1-hot vectors. In each round of the reference game, we construct (C,r.)) where C' is the context (set of item column vectors stacked into matrix), r is a vector representing the referent, and r is the index of the referent in C. We uniformly sample k e-1 items as distractors to form C (01,.... u The distractors are sampled randomly each round (in every epoch).
Ny R.p Each index
MODELS
We begin with a general architecture and training objective to underly all of our models (Sections 3.1 and 3.2). We then introduce three modifications which can be used to encourage internally- consistent representations: a self-play training objective, shared embedding space, and symmetric decoding and encoding mechanism with shared parameters (Section 3.3)
Forr implementation details, refer to Appendices A.I, A.2, A.3, source code bit. 1yt internal c stency- www.nncceemrrgnttnnoomm catmmmniiiiiionoommmm cation.
2

---

Under review as a conference paper at ICLR 2020
3.1
ARCHITECTURE
Agents contain four modules. Embedding modules 1) Eitem E RNXDD Emessage E RD (xx) embed items and messages. When speaking, the decoder module 2) consumes the em- bedded referent Estem (r) and produces a discrete message M E VL. Next, when listening, the encoder module 3) consumes embedded messages Emessage (MM) RWxD and then produces a representation of the referent Á RD. Finally, non-parametric pointing module 4) produces distribution P(C) over the context by matrix multiplying e with the embedded context Litem (C').
The decoders emit one symbol at a time, auto-regressively producing fixed-length messages. The messages are discretized with the traighh-throuuge Gumbel Softmax (Jang et al., 2016) as in Mor- datch Abbeel (2018). This converts a distribution to a one-hot vector while permitting gradients to flow through the operation and enables the agents to be optimized without reinforcement methods.
The Recurrent model uses a LSTM (Hochreiter Schmidhuber, 1997) decoder when speaking and a LSTM encoder when listening, as in (Lazaridou et al., 2018). The Transformer model (Vaswani et al., 2017) uses a Transformer Decoder when speaking and a Transformer Encoder to encode when listening. See Appendix A for implementation details and hyperparameters.
3.2 TRAINING OBJECTIVES
Speaking 9: : r. - M and listening L: C,M, - P(C)) are both functions where M y = (0. 1)_x(v] VI is a discrete-valued message with length I., P(C) is a distribution over the items in context, and 0 are optimizable parameters. We optimize the parameters 0 0 of agents A, B over dataset D to select each referent r from among the distractors in its context C' by minimizing the negative log likelihood of selecting the correct referent in context (Eq 1). In our experiments, the speaker modules and the listener modules instantiate the function P and L respectively.
CA-B
Σ
- log (r: 0B)r.
(C,rF)jD
3.3 IMPOSING NYTERNAI-CONSISTENGO CONSTRAINTS
We investigate three piternal-onniitency constraints, that encourage internally-consistent represen- tations. Baseline agents consist of separate sets of parameters, one for listening and one for speaking. For example, the baseline recurrent model consists of two recurrent models. This corre- sponds the scenario where agents either speak or listen, but not both (Lazaridou et al., 2018).
We introduce a 1) self-play loss for both agents of the same form as Eq. 1, except the given agent fulfills both roles, encouraging it to speak/listen to others the same way it speaks/listens to itself. When we use the self-play training objective, we use it for both agents. Next, 2) shared embedding agents use the same item embeddings and the same message embeddings when listening and speak- ing. Finally, 3) symmetric encoding and decoding agents use the same parameters (but different mechanisms) to decode a message when speaking as does to encode a message when listening. Parameters are only ever shared between roles within an agent, and never between agents.
4 DATASETS
Our evaluation is based on the simple reference game² as described in Section 2, played across two datasets. The datasets, summarized in Table 1, target different aspects of lexical reference. The first, Visual Attributes for Concepts (CONCEPTS) Silberer et al. (2013), is derived from anno- tated images of actual items (animals, tools, etc). Thus, the data contains realistic co-occurance patterns: groups of attributes like headhead, has- and has-snout appears together several times, whereas has- seedd,, has -mouth, made-ot oeeeeeoo-mttttte never co-occur. The intu- ition is that good lexicon will exploit the structure by developing words which refer to groups of frequently co-occurring attributes (e.g. "mammal") and will describe unseen referents in terms of
We use "context- hhieeettt-iddd"" game: the context is visible to the listener but not the speaker. This is to ensure that the speaker does not have access additional attributes (e.g. position of item within the context) that could enable the agents find degenerate solutions to the game.
3

---

Under review as a conference paper at ICLR 2020
these primitive concepts. The second dataset, SHAPES, is create ourselves in contrast with the CONCEPTS data. In SHAPES, items correspond to abstract shapes which are exactly describable by their attributes (e.g. blue, shaded, hexagon). All the attributes are independent and there is no co-occurence structure to exploit, so a good lexicon should ideally provide a means for uniquely specifying each attribute's value independently of the others.
Name
Features (nz)
Values (n.)
Context Size (k)
Train
Val
Test
SHAPES
10
50
5
1000
200
100
CONCEPTS
597
1
5
375
50
85
Table 1: Dataset statistics.
RESULTS
We present experimental results aimed at testing the hypotheses stated in Section 1. To provide intuition, we frame experiments in terms of two agents, "Alice" (Agent A) and "Bob" (Agent B), who are taking turns asking for and fetching toys.
5.1 HYPOTHESIS 1: GENERALIZATION TO NEW ITEMS
5.1.1 SETUP
We first test whether any of the proposed internal-consiseency constraints improve the agents' ability to generalize to novel items-i.e. items which consist of unseen combinations of features. Here, we focus on the performance when models are trained and tested within the same com- municative role. This corresponds to the setting that has typically been used in prior work on emergent communication: Alice always speaks in order to ask for toys, Bob always responds by fetching them, and the pair's success is evaluated in terms of Alice's ability to describe new toys such that Bob correctly gets them. For evaluation, we hold out a subset of the value combina- tions from each dataset to use for testing. For example, the agents might be trained to refer to [[shape: circee, color: red]:: [shape: :tthapessqurre, square, color : blluuee bel} and then tested on its ability refer dchape:cciccle
color:blue]. We compute validation and test accuracies.
5.1.2 COMPARISON OF MODEL ARCHITECTURES
Before introducing our internal-consistency constraints, we first evaluate our various architectures in this vanilla setting. Thus. these results serve chiefly to calibrate differences between model ar- chitectures. If we see differences between the test conditions, these should be attributed to general architectural advantages associated with implementing these internal cnternalcoonsstencyy constraints, rather than interpreted as evidence for/against enternal-oonistencyy per se. Table 2 shows the results. Over- all, the transformer architecture outperforms the recurrent architecture, and the transformer outper- forms the previously best-reported result for this task (which is only available on the CONCEPTS data). We observe no clear trend associated with the shared embedding module (sometimes it helps, sometimes it hurts), but do see a slight positive effect associated with the symmetric transformer.
5.1.3 EFFECTS OF INTERNAL- CONSISTENCY CONSTRAINTS.
We next look at whether there is an advantage to using the the conesaal-onnsittncyy constraints, even when agents remain in fixed conversational roles. Intuitively, this corresponds to scenario in which both Alice and Bob are capable of performing either role (speaking or listening), but nonetheless, they only ever interact within the same routine: Alice only every speaks and Bob only ever listens and fetches items in response. However, both Alice and Bob can imagine how they would behave if they were to assume the opposite role, and thus, via self-play, each can enforce diternal-consiteen between their own actions as speaker (listener) and the way they imagine they would respond as listener (speaker). this manner, although Alice and Bob only ever receive direct feedback in
We additionally tried computing Tree Reconstruction Error as introduced by Andreas (2019). See Ap- pendix A.5 for implementation details. However, we did not find an interesting differences across conditions. Therefore, for compactness, omit these results.
4

---

Under review as a conference paper at ICLR 2020
SHAPES
CONCEPTS
Val
Test
Val
Test
Random (Baseline)
20.0
20.0
20.0
20.0
Prior SOTA (Lazaridou et al., 2018)
81.6
Recurrent Model
56.8 + 0.8
49.0 + 1.9
67.9 + 1.6
70.1 + 2.6
Transformer Model
86.5 + 1.5
89.5 + 1.2
86.2 + 1.7
87.0 + 0.7
Symmetric Transformer Model
86.7 + 1.2
89.3 + 1.4
92.7 + 0.7
89.3 + 1.6
Table 2: Results when agents trained and tested in single role, before any interna--onniisenccy constraints. These scores are mean accuracy with 95% confidence range averaged over runs over different test samplings (the distractors change).
one role, they can still impose internally consistent behavior across both roles. It is conceivable that doing so might improve performance even though each agent remains a fixed role, either by providing the model with additional information about the task structure, or simply by acting regularizer. Thus, for completeness, we assess whether the internl-consiseent constraints provide any advantage to the models in the vanilla emergent communication setting.
Table 3 shows the effect of adding the self-play objective in the fixed-role setting, across architec- tures and datasets. The trends are mixed: it appears the additional signal only noises the baseline and symmetric models, whereas the shared embeddings models are able to leverage it effectively. Thus, the effect is not clear enough to establish conclusively that the internal-consistency constraints help the agents generalize in this fixed-role setting, and in fact it may hurt.
Baseline %
-Self-play
+Shared Emb.
Symmetric
%
A
%
A
%
A
RNN, Shapes
49.0
52.6 + 0.8
+3.6
69.0 + 2.4
+20.0
20.1 + 1.5
-28.9
RNN, Concepts
70.1
68.6
+ 2.2
-1.5
81.4 + 1.9
+11.3
20.6 + 1.3
-49.5
Trans, Shapes
89.5
78.7
+ 1.7
-10.8
86.3
+ 1.0
-3.2
87.4
+ 0.4
-2.1
Trans, Concepts
87.0
85.3 + 1.4
-1.7
89.0 + 1.3
+2.0
90.1 + 0.4
+3.1
Table 3: Performance on task of referring to/fetching unseen items for baseline model compared against models with the interna--consstenne constraints. To highlight the difference of each con- straint compared the baseline performance, each delta compares the performance of the modi- fied model to the buseline model. In this setting, we see no clear advantage to enforcing internal- consistency via self-play. These scores are mean accuracy with 95% confidence interval averaged over 5 runs different test set samplings (the distractors change).
5.2 HYPOTHESIS 2: GENERALIZATION TO NEW ROLES
5.2. SETUP
We now look at whether internal-consiseency improves the agents' ability to generalize linguistic knowledge across roles. For example, we can picture the following scenario: Alice is speaking to Bob, and asks for the "truck". Bob hands her the doll, and Alice replies negatively, indicating that what she actually wanted was the truck. Now, without additional direct supervision, when Bob wants the truck, will he know do use the word "truck"? Such setting is particularly relevant in practical settings, for example when robotic agents must reach high accuracy despite only limited access to human interaction for training. We consider two versions of this setting, involving different levels of direct supervision (i.c. interaction with the other agent) as described below.
Training in one role. Our first experimental setting assumes that Alice and Bob each only receive direct training in one role, e.g. Alice only ever speaks to Bob, so Alice only receives feedback on speaker, and Bob on how he is performing
how she is performing Alice and Bob are able to practice in the opposite role via self-play. This setup is analogous to the experiment just discussed in Section 5.1.3. However, unlike before, Alice and Bob will be tested in
listener. However, both
5

---

Under review as a conference paper at ICLR 2020
the roles opposite of those in which they were trained. That is, if Alice trained as speaker, then she will be tested as a listener (on her ability to correctly identify items to which Bob refers).
Training in both roles. In second experimental setting, assume Alice and Bob enjoy healthy friendship, which both take turns speaking and listening to each other, and thus both receive direct supervision in both roles. However, they do not necessarily receive equal training on every vocabulary item. Rather, there are some contexts in which Alice only speaks and other contexts in which she only listens. Intuitively, this corresponds to a scenario in which Alice speaks exclusively about food (while Bob listens), while Bob speaks exclusively about toys (while Alice listens). We are interested in testing how well Alice is able to speak about toys test time.
We use the SHAPES dataset* to create two training splits, each having the same attributes but covering disjoint sets of values. For example, the first training split (train-1) might have colore (blue, red, yellow) whereas the second training split (train-2) has colore green, orange, purple). We use train-1 to train Alice as speaker and Bob as listener and train-2 to train them in the reverse roles. We then report performance with Alice as listener and Bob as speaker using test set that uses the same attribute values as train-1.
5.2.2 RESULTS
Our results for both training conditions shown in Table 4. The baseline model (which includes no -ntesmal-cosssseneyy constraints) performs, unsurprisingly, at chance. Adding the self-play objec- tive gives improvements across the board. Again, while seemingly straight forward, this result has promising practical interpretations in settings in which model has access to only small amount of interaction. For example, a human may be willing to train robot via speaking (pointing and naming items), but not patient enough to train it via listening (responding to the robot's 's noisy commands). In such a setting, the ability to massively augment performance via self-play is significant. to the self-play objective, we see that enforcing shared-embedding spaces yields further significant performance gains (in the range of 30 percentage points in some cases). The symmetric constraints on top of self-play and shared embeddings seem to hurt performance in general.
addition
Baseline
+ Self Self-play
Shared Emb.
+ +Symmetric
One Role
%
%
A
A
%
A
RNN, Shapes
19.8 + 1.3
50.7
1.3
+33.77
70.7
1.8
+50.9
20.1
+ 1.9
-49.9
RNN, Concepts
21.3 + 2.3
62.9
+ 0.9
+42.9
81.7
+ 0.8
+18.8
19.6
+ 1.5
-62.1
Trans, Shapes
19.5 + 0.8
82.1
1.5
+62.6
79.5
+ 0.8
-2.6
83.9
+ 1.8
+4.4
Trans, Concepts
19.6 + 2.9
52.8 + 1.7
+32.6
80.3 - 1.1
+27.5
77.3 - 2.1
-30
Both Roles
%
%
A
%
A
%
A
RNN, Shapes
20.4 + 1.0
40.1
0.6
+19.7
18.8
1.5
-21.3
69.4 + 1.9
+50.6
Trans, Shapes
20.9 + 2.5 61.7 + 2.2
+40.8
70.3 + 2.0
+8.6
74.7 + 1.5 +4.4
Table 4: Performance for tasks that requires agents to generalize across roles-e.g. training on the word "blue" " listener, but then having produce "blue" as a speaker. "One Role" refers to when agents receive direct feedback in single role (i.e. their training on the other roles is only via self-play). "Both Roles" refers when agents receive direct feedback in both roles, but only see the test vocabulary in the role opposite that in which they are tested. To inspect the additive differences between the internal-consistency constraints, each delta compares the performance of the current column to the previous column. These scores mean accuracy with 95% confidence range averaged over 5 with different test set samplings (the distractors change).
5.3 TAKEAWAYS
Overall, when agents can be trained directly in the role in which they are tested, there is clear evidence that adding internal-consistency constraints improves the ability of agents to generalize
cannot construct analogous splits with CONCEPTS since the train set is small.
We found that, in absence of self-play, neither symmetric encoding/lecoding nor shared embeddings per- formed competitively. Therefore, for compactness, we only report the results of the constraints in combination.

---

Under review as a conference paper at ICLR 2020
to new items. However, internal-consistenn constraints improve performance significantly when agents have limited ability to train in given role. Specifically, models which are equipped with self- play training objectives and shared embedding spaces show superior ability to generalize learned representations across roles, and perform about as well as if they had been trained on the target role.
6 ANALYSIS
In this section we provide additional analyses to highlight the effects of internal-consistrnt (in particular, self-play) on training efficiency and the emerged protocol. Here, we use a smaller SHAPES dataset (see B.1), and reduce the vocabulary size and message length ((V) 10, L = 3).
6.1 CAN SELF-PLAY REPLACE DIRECTION SUPERVISION?
baseline
shared
Here we inspect self play supplants direct su- pervision between agents. We consider the set- ting which Alice trains with full data as speaker, but vary the amount of data she has ac- cess to as listener. We then test Alice's 's perfor- mance listener (and vice-versa for Bob as speaker). Fig. shows the results, with fraction of the full training data that Alice (Bob) sees as a speaker (listener) shown along the x-axis. The self-play models without direct supervision perform well: it appears their protocol trans- fers across roles with out drifting. This sheds some light on the performance drop between the "one role" and "two role" settings in Sec- tion 5.2.2. where the additional experience in the "two role" setting did not help. Fig. 2 shows that additional training in the primary role is un- necessary, so is not surprising that training on disjoint features (train-2) is helpful.
test accuracy
recurrent
paradigmm default selfplay
100
transformer
0
50
100
100
% data
% data
Models trained with self-play per- form as well in novel roles as do vanilla mod- els trained with direct supervision in the tar- get role.
Figure 2:
6.2 DOES SELF-PLAN PLAY ENCOURAGE BETTER PROTOCOLS TO EMERGE?
We measure whether self-play leads to better communication protocols general. First, we mea- sure the agents' speaking and listening capacities separately, using measures proposed by Lowe et al. (2019); Eccles et al. (2019). Specifically, positive signaling (S*) measures the speaker's messages depend the features of the referent, and positive listening (L*) measures if the listener's ''s actions depend on the message Table 5 shows that self-play improves the agents' communication as mea- sured by accuracy well these orthogonal metrics. We also find that the model architectures and
Baseline
+ Self Selff-play
%
L
S+
%
A
L+
S+
A
RNN
95.4
4.0
6.1
96.2
+0.8
6.4
+1.4
6.4
+0.3
Trans
95.1
2.6
5.4
96.3
+1.2
3.1
+0.5
5.5
+0.1
Table 5:
Self-play improves the model's communication: the agents' respective positive listening and positive signaling metries both improve in the vanilla setting.
self-play impact the agents' lexicons. The recurrent models produces fewer unique messages than the transformer models (on average 110 versus 300), and often neglect to use all the vocabulary. Fig. 3 shows that self- ellf helps the recurrent model use more of the vocabulary, and leads both the recurrent and transformer models to develop sparser mappings from symbols onto features.
"See Appendix B.2 for details.

---

Under review as a conference paper at ICLR 2020
defauit RNN
selfplay RNN
default trans
selfplay trans
250
T_0.2
F_0_4
F_0_6-
200
FO8.8
F.1.2-
150
F.1.4
F_1_6
F_1_8
1__0
F_2_2
F_2.6
F_2_8
0
0123456789
0123456789
0123456789
012 345
Figure 3: Emerged lexicons. This figure show features along the y-axis and symbols along the x-axis. Each index counts the co-occurence of the symbol and the feature. For example, the trans- former self-play model appears to map a symbol to each distinct value of the first attribute, ignoring other attributes and word order. See more lexicons over other random seeds in B.3.
7 RELATED WORK
Work in emergent communication (Das et al., 2017; Lazaridou et al., 2018) analyzes agents that develop a shared protocol by playing reference games (Lewis, 2008). Kottur et al. (2017) presented results showing that computational models do not learn compositional protocols by default. Instead, the agents tend to develop brittle protocols that have trouble generalizing to novel items. Several approaches have been proposed which could encourage models to learn more generalizable repre- sentations of language, including compression (Kirby et al., 2015), efficiency (Gibson et al., 2019), memory constraints (Kottur et al., 2017), pragmatic constraints (Tomlin & Pavlick, 2018), and pos- itive biases to respond to other agents (Jaques et al., 2018; Eccles et al., 2019). Some work, like ours, assumes access to symbolic representations of referents and their attributes, whereas others' set in pixel-based multi-agent games (Jaques al., 2018 8: Eccles et al., 2019; Das et al., 2018) or multi-agent grid-worlds (Sukhbaatar et al., 2016; Lowe et al., 2019).
broader body of work speaker-listener models, specifically pragmatically-infone -infoomme models in which speakers reason recursively about listeners' beliefs (and vice-versa) (Frank & Goodman, 2012; Goodman & Frank, 2016). Such models have been used in applications such image captioning (Andreas & Klein, 2016: Yu et al., 2017: Monroe & Potts, 2015), and robotics (Vogel & Jurafsky, 2010: Vogel al., 2013; Fried et al., 2018), as well in linguistics and psychology in order to explain complex linguistic inferences (Tessler & Goodman, 2016; Mon- roe et al., 2017). Conceptually, our proposed common with these neural speaker-listener models developed outside of emergent communication. However, again, past work has tended assume that a speaker's mental model of their listener not necessarily consistent-in fact, it is often assumed explicitly to be inconsistent (Frank & Good- man, 2012)-with the way the speaker themself would behave as listener. We note, however, that proposed model architecture (because it lacks the recursion typical in other pragmatics models) likely unable to handle the types of higher-level inferences (e.g. implicatures) targeted by the mentioned prior work computational pragmatics, though this an interesting avenue to explore.
Our work also relates
constraints share something in
8 CONCLUSION
We propose three methods for encouraging dialog agents to follow "the golden rule": speak/listen to others as you would expect to be spoken/listened to. In the emergent communication setting, we find that the internal-consiteen to novel items, but both the self-play objective and shared embeddings significantly improve perfor- mance when agents tested roles they were not directly trained for. In fact, when trained in one role and tested on another, these internal ineemaa-conssstence constraints allow the agents to perform about well if they had been trained in the target role.
constraints do not systematically improve models' generalization
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jacob Andreas, Measuring compositionality in representation learning. arXiv:1902.07181, 2019.
arXiv preprint
Jacob Andreas and Dan Klein. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp- 1173-1182, Austin, Texas, November 2016. Association for Computational Linguistics. URL /ttp///lll...crgaattooollloo///e
D1 6 1125.
Abhishek Das, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. Learning cooperative visual dialog agents with deep reinforcement learning. Proceedings of the IEFE International Conference Computer Vision, pp. 2951-2960, 2017.
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Michael Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. arXiv preprint arXiv: 11187, 2018.
Tom Eccles, Yoram Bachrach, Guy Lever, Angeliki Lazaridou, and Thore Graepel. Biases for emergent communication in multi-agent reinforcement learning. 2019.
Michael C Frank and Noah D Goodman. Predicting pragmatic reasoning in language games. Sci- ence, 336(60884:998-9988 2012.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg- Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. speaker-follower models for vision- aision-and-laad and-language navigation. arXiv preprint arXiv: (rXXvv:800.02777 2018.
Edward Gibson, Richard Futrell, Steven T Piandadosi, Isabelle Dautriche, Kyle Mahowald, Leon Bergen, and Roger Levy. How efficiency shapes human language. Trends in cognitive sciences, 2019.
Noah D Goodman and Michael C Frank. Pragmatic language interpretation as probabilistic infer- ence. Trends in Cognitive Sciences, 20(1 1):818 01-888882, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizztioon with gumbel-softmax. arXiv preprint
2016.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro A Ortega, DJ Strouse, Joel Leibo, and Nando de Freitas. Intrinsic social motivation via causal influence multi-agent 2018.
rl. arXiv preprint urXiv:/810.066
Simon Kirby, Monica Tamariz, Hannah Cornish, and Kenny Smith. Compression and communica- tion in the cultural evolution of linguistic structure. Cognition, 141:87-102, 2015.
Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge 'nat- urally' in multi-agent dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2962-2967, Copenhagen, Denmark, September 2017. As- sociation for Computational Linguistics. doi: 10. 1/111663////11111211... aclweb. /rg/ antaanoogo/tnla1
URL https: www.
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of lin- guistic communication from referential games with symbolic and pixel input. arXiv preprint arXiv:1804.03984, 2018
David Lewis. Convention: philosophical study. John Wiley & Sons, 2008
Ryan Lowe, Jakob Foerster, Y-Lan Boureau, Joelle Pineau, and Yann Dauphin. On the pitfalls of measuring emergent communication. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 693-701. International Foundation for Au- tonomous Agents and Multiagent Systems, 2019.

---

Under review as a conference paper at ICLR 2020
Will Monroe and Christopher Potts. Learning in the rational speech acts model. arXiv preprint arXiv:1510.06807, 2015.
Will Monroe, Robert X. D. Hawkins, N. D. Goodman, and Christopher Potts. Colors context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics, 2017.
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Carina Silberer, Vittorio Ferrari, and Mirella Lapata. Models of semantic representation with vi- sual attributes. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 572-582, 2013.
Sainbayar Sukhbaatar, Rob Fergus, al. Learning multiagent communication with backpropaga- tion. In Advances in Neural Information Processing Systems, pp. 2244-2252, 2016.
Michael Henry Tessler and Noah D Goodman. A pragmatic theory of generic language. arXiv preprint arXiv: 16080026, 2016.
Nicholas Tomlin and Ellie Pavlick. Incremental pragmatics and emergent communication. In Emer- gent Communication Workshop at NeurIPS, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-60088 6008, 2017.
Adam Vogel and Dan Jurafsky. Learning follow navigational directions. In Proceedings of the 48th Anmual Meeting of the Association for Computational Linguistics, pp. 806-814. Association for Computational Linguistics, 2010.
Adam Vogel, Max Bodoia, Christopher Potts, and Daniel Jurafsky. Emergence of gricean maxims from multi-agen, decision theory. In Proceedings of the 2013 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1072-1081, Atlanta, Georgia, June 2013. Association for Computational Linguistics. URL http:: / www. ac web www.brg.....ee.e................ wowrg//antha ogy /N1 3-1127.
Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. A joint speakerlistener-reiffoeer model for referring expressions. In Computer Vision and Pattern Recognition (CVPR), volume 2, 2017.
A IMPLEMENTATION DETAILS
We use the deep learning framework Pytorch? (v1.2.0) to implement our models (and Python 3.7.3). For reproducibility, all random seeds (random, numpy, torch) are arbitrarily set to 42.
The general architecture, the four modules that comprise each agent are shown in Figure 4.
A.1 RECURRENT IMPLEMENTATION
The recurrent model decodes and encodes message follows: to generate message, the first input is the embedding of a SOS sfart-of-sentence symbol and the imitial hidden state is set as the embedded referent (and the cell memory is all zeroes). From here, at each step, the outputted hidden state (E P) is projected by the transposed word embeddings (ET (Einsssage Dx)V), and the next word is sampled from this resulting distribution across the vocabulary. Moving forward, the next input is the embedding of the sampled word, and the hidden state and cell memory are set those emitted at the previous step. We produce words until the maximum length reached.
When encoding message, learned embedding is the first hidden state, and the input each time step is the corresponding embedded word. The last hidden state is set as the encoding. In the symmetric variant of this model, the LSTM cell used for encoding and decoding is the same.
10

---

Under review as a conference paper at ICLR 2020
M
P(C)
Gumbel
Soft tmax
Pointing Module le
Embed_messape T
Decoder Module
Encoder Module
Embed item
Embed mes message
Embed i tem
M
(a) Speaker Modules.
(b) Listener Modules.
Figure 4: General architecture. This architecture underlies each model we use; only the imple- mentation of the Decoder and Encoder modules vary between models. In the baseline models no parameters shared within agent. In shared embedding models, the embeddings (purple) are shared across roles. In symmetric models, the encoder and decoder (pink) are shared across both roles. The blue modules are non paraamaeee.ic.
A.2 TRANSFORMER IMPLEMENTATION
The transformer model decodes and encodes message as follows: to generate a message describing the referent rego-segressively. all the embeddings of the words produced far M and the embed- ding NEXT symbol are concatenated together into a matrix X (Eq. 2). Next, the transformer decoder consumes this matrix and the referent embedding and produces contextualized represen- tation of the input matrix (Eq. 3). The last column vector X.,.. which corresponds the NEXT embedding, the internal representation of the next word. The next word m sampled from the projection of this representation with the transposed word embedding. More words produced in this way until the maximum length message is formed. Producing the + 1th word (so M consists of the first words), works as follows:
X Emesssge (M); EMEXT ((++1)xDD
X = Transforrmrreecoeer X, Fatem Freem()) R(W+1)
(3)
M GumbelSoffmaat X x ET Peeseage
(4)
To encode incoming messages when listening, the embeddings of the words message plus the embedding of ITEM symbol are concatenated together into matrix X (Eq. 5). Then, the transformer encoder is used to produce contextualized embedding X (Eq. 6). The last column vector X. which corresponds to the ITEM embedding, is set the message encoding. Note, W the number of words in each message (and the length of M).
(Emessage(Mi) Erru]] RW++1xD
(5)
X
Transformer(hcoder(() B (W++))D
(6)
X.W
A.3 SYMMETRIC AGENTS
For the Symmetric Recurrent Model, a LSTM cell is shared between the encoder and decoder. Oth- erwise, the recurrent model is unchanged. For the Symmetric Transformer Model, Transformer En- coders and Transformer Decoders have different structures, to share parameters between them, have change either how the transformer agent speaks or how it listens. We opt to replace the Trans- former Decoder with Transformer Encoder, and use it decode messages in-place when speaking.
11

---

Under review as a conference paper at ICLR 2020
A.3.1 SYMMETRIC TRANSFORMER IMPLEMENTATION
The Symmetric Transformer the same mechanism for encoding messages when listening as the default transformer model (described directly above). However, it uses a Transformer Encoder when speaking instead of a Transformer Decode. When speaking, produce the next symbol, the embeddings of the i words produced so far, the referent embedding, and the embedding of a NEXT symbol are concatenated together into a matrix X (Eq. 8). The Transformer Encoder then maps X contextualized representation X (Eq. 9). Finally, the column vector in X that corresponds to NEXT, is used to sample the next symbol:
X Eitem (r); Emessage M); Erresl.
X e R(+2)xD X e ((+2))D
(8)
TransformmerDecoderrXX
(9)
M
Gumbel So Sofbeexx X x FT
meesage)
M4LeVe 10.1)
(10)
A.4 HYPER PARAMETERS
Search strategy
uniform sampling (25 samples)
Hyperparameter
Search Space
optimizer
RMSProp
early stopping patience
100
batch size
64
number of layers
I
hidden dimensionality
choice[ 32, 64, 128]
learning rate scheduler
choice[0.0001, 0.001, 0.01]
choice(None, ReduceLLRnnllaeee CyclicLR]
Table 6: Recurrent Model Hyperparameter Search Results.
Search strategy
uniform sampling (25 samples)
Hyperparameter
Search Space
optimizer
RMSProp
early stopping patience
100
batch size
64
number of layers
hidden dimensionality
choice[16, 16, 32, 64, 128]
number of attention heads dropout
choice 1, 2, 8]
choice(O., 0.2, 0.5]
learning rate scheduler
choicef0.000/ 0.001, 0.01]
choice[None, ReduceLLROnPlaeauu CyclicLR]
Table 7: Transformer Model Hyperparameter Search Space.
We uniformly sampled 25 hyperparameter configurations for each model architecture, experiment, and dataset split. In every case, we fixed the hidden size dimensionality and embedding dimen- sionality to be the same. We searched over three different learning rate schedulers: (1) None (or no scheduler, (2) ReduceLLROnllaeeuu with patience 25, reduction factor 1. and uses validation accuracy as its measure of progress, and (3) CyclicLR rising from 0.00001 to the given learning rate over 500 batches and then declines towards 0.00001 for the rest of training (10,000 batches). This is similar the Noam Update in Vaswani et al. (2017). To save space, relegate the hyper-parameter parameter selections in our code bit htt.interennaaaaaennee www.eraaaccosssttssr..... wtw.:ooisssstttccceeeeeeeeeee gent communt cat ion / b htt/hyperparamterr... py.
12

---

Under review as a conference paper at ICLR 2020
A.5 TREE RECONSTRUCTION ERROR
TRE takes two important hyperparameters, an error function and a composition function. We select the same choices the author, for what amounts the same task (producing dis- crete message) as detailed the original paper. This method requires structured feature rep- resentations, so we assume that the features in each item are entirely right-branching. The composition function is learned, and set the number of update steps to 1000. The orig- and our modifi- cow.errgggetccmmmmmaaaat...o cowmunicatiinnno in the file
wwwpa///wwwaaneaaaa/aceooocaaeee inal implementation is at https: / wtw.ttttwwooocccaeeaeeeettttttee cation is at bit bit.:L//anterral-consssteee /ttpb//copps.tiiinnniitt...../// py.
B FURTHER ANALYSIS
B.I SHAPES SMALL
We simplify the SHAPES dataset in order to be able to empirically compute the positive listening and signaling scores, which requires iterating overall possible messages. In the original version of SHAPES this is impractical as there would be 50 possible messages. The details of this smaller version of SHAPES detailed in Table 9.
We fix the settings for the recurrent and transformer models found that majority of models across experiments used the same parameters. See Tables 10, 11. Furthermore, all results in Sec. averaged over 5 arbitrary random seeds (both trained and tested) (43, 44, 45, 46, 46).
B.2 POSITIVE SIGNALING AND LISTENING
Positive listening was introduced by Lowe et al. (2019) Causal Influence of Communication, and the precursor to positive signaling was introduced by Jaques et al. (2018).
We use the definitions Eccles et al. (2019) modified for one step game for both metrics:
Positive Listening ( Dxz(Pr(amm) Pri(a)),
Positive Speaking (S-) = I(m, .) = H(m) H(ml2z)
where m is a given message, a are the actions the agent can take, r is state, DKL L the Kuller-bach divergence, I is the mutual information, H. is the entropy. We can compute these quantities without sampling messages because the number of messages is tractable.
In setting, the game is single step, and the average policy actions independent of the message converges to the uniform distribution over actions, as the order of referents randomized. Thus have: L+ (Kr((P((u (Pr(a) (f).
The positive speaking metric is computed over sampling of the dataset (the distractors are random) follows in (Eccles al., 2019):
S+ (()) H(m(x),
= Σ ((m) log (()) + E E T(m| ()) log {mm[z)],
where the summations are over all possible messages, v is the given referent, and ((m) is empir- ically likelihood of the message being produced irrespective of ær, and π (m)2)) is the likelihood of the given message being produced given the referent æ. We report empirical averages L and 5 over all items in the dataset, also averaged over 5 arbitrary random seeds.
B.3 ANALYSIS DETAILS
Figs. 5, 6, 7, 8 show lexicons across random seeds. See Table 8 for additional results in the vanilla setting for SHAPES-Smi small that were elided for space.
13

---

Under review as a conference paper at ICLR 2020
F_0_0
250
F_0_3
F_0_6
200
F_0_9
F_1_2
150
F_1_5
F_1_8
100
F_2_1
F_2_4
50
F_2_7
2
92468 Figure Recurrent Default.
4
F_0_0
250
F_0_3
F_0_6
200
F_0_9
F_1_2
150
F_1_5
F_1_8
100
F_2_1
F_2_4
50
F_2_7
0
02468
0 2
2 4 6
Figure 6: Recurrent self-play.
F_O_0
250
F_0
F_0_6
200
F_0.9
F_1_2
150
1
F_1_8
100
F_2_1
F_2_4
50
F_2_7
4
6
6
6.8
Figure 7: Transformer Default.
F_0
250
F_0
F_0.6
200
F_0_9
F_1_2
150
F
F
100
F
F
50
F_2_7
2
0 22468
2 6.8
Figure 8: Transformer self-play.
14

---

Under review as a conference paper at ICLR 2020
Baseline
+Self-play
Shared Emb.
%
I,*
St
.A
I+,4 A
St A
% A
L A
S A
RNN
95.4
4.0
6.1
96.2,
+0.8
6.4, +14
6.4, +0.3
95.6, +0,2
5.2, +1.2
6.3, +0.2
Trans
95.1
2.6
5.4
96.3,
+ +.2
3.1, +0.5
5.5, +0.1
96.2, +1.1
3.4 0.8
5.6,+0.4 +0.2
Table 8:
Deltas are compared to the baseline value for each row. shared embeddings well as self-play.
Here we report the results for
Name
Features (ny)
Values (n,)
Context Size (k)
Train
Val
Test
SHAPES SMALL
5
800
100
100
Table 9: Dataset statisties.
Hyperparameter
Search Space
optimizer
RMSProp
early stopping patience
100
batch size number of layers
64
I
hidden dimensionality
128
learning rate scheduler
0.001
None
Table 10: Best Recurrent Model Hyperparameter Search Settings.
Hyperparameter
Search Space
optimizer
RMSProp
early stopping patience
100
batch size
number of layers
1
hidden dimensionality
128
number of attention heads
1
dropout
0.
learning rate scheduler
0.0001
Reduccel.OOmPlateeuuS5ee. epochs)
Table 11: Best Transformer Model Hyperparameter Settings.
15


Reviews:
[{"review_id": "Skg9fxAyqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In an emergent communication setting, the paper proposes three methods for encouraging internal-consistency for dialog agents --- an agent speaks/listens to others as it would expect to be spoken/listened to. The paper concludes that the internal-consistency constraints do not improve models’ generalization to novel items, but improve performance when agents are tested on roles they were not trained for. \n\nThe experiments support the above conclusions on the small datasets used in the experiments. The paper is in general well structured and is clear and easy to follow. \n\nThe contributions of the paper are rather incremental. The main methods are described in section 3.3, which employ a self-play objective,  use shared embedding, and include sysmetric encoding and decoding. The extension is new but does not contain enough content for the conference. I do not think this paper presents enough contributions. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkxM3i8CYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the question of internal consistency in emergent communication. In other words, the paper aims to answer the question ‘how is emergent communication improved if we enforce the constraint that an agent must speak in the same way that it listens?’ The paper explores three methods of enforcing internal consistency: self-play, shared embeddings, and symmetric encoding/decoding. They find that, while internal consistency does not help generalization to unseen objects, it does allow agents to generalize over conversational roles (i.e. to perform well as a speaker despite only being trained as a listener).\n\nI have been eagerly anticipating a paper on this topic – it seems silly that the listening and speaking modules in traditional emergent communication research are completely disjoint. I think coming up with methods/ architectures that combine these two capabilities is an important research direction. I also think paper is very well-written and structured, and it reads very well.\n\nHowever, I have several concerns with the paper. First, half of the results center around the hypothesis ‘internal consistency helps agents generalize to unseen items’. While ultimately this hypothesis is disproven, it’s unclear as to why this might be expected in the first place. The only justification of this hypothesis I could find in the paper is the sentence “It is conceivable that <internal consistency> might improve performance even though each <agent> remains in a fixed role.” In my view, the fact that this hypothesis is ‘conceivable’ is a rather weak argument for it to be such a prominent part of the paper. Thus, I don’t think this half of the results add much to the overall paper. \n\nI also have mixed feelings about the use of ‘self-play’ to enforce internal consistency, and how it relates to the core result of the paper: “the proposed constraints enable models to generalize learned language representations across communicative roles, even in the case of where the agent receives no direct training in the target (test) role”. In short, I think the phrase “no direct training” is misleading, as the self-play itself is almost a form of direct training, and thus the result isn’t very surprising.  \n\n More specifically, each agent ‘Alice and Bob’ are composed of two modules: a speaking module and a listening module. During normal training (say, in the shape environment), Alice speaks and Bob listens (achieving a reward if Bob selects the right shape, which is backpropagated to Alice), and thus the Alice’s speaker module and Bob’s listener module are updated. Alice’s listening capabilities will be equivalent to a random agent, as her listening module is still randomly initialized. Now, during self-play, Alice’s speaker module is trained with her listener module (I believe in the same way as it is trained with Bob’s listener module) to achieve high reward. This listener module is then tested against Bob’s speaker module (which is also trained via self-play). To me, this process isn’t the same as the paper’s narrative of ‘we only train Alice to be a speaker, and she learns to listen!’ This is especially true since, without parameter tying, the choice of saying that listener module ‘belongs to Alice’ is arbitrary (since it’s completely separate). An equivalent way of framing this result would be “language learning is transitive: if we train agent A to speak to agent B who listens, then train agent C to listen to agent A and agent D to speak to agent B, then agents D can perform well with agent C”. With this framing, the result is much less surprising (in fact, it would be surprising if this weren’t true). \n\n\nFinally, the three methods of enforcing internal consistency are not tested independently --- shared embeddings are only tested on top of self-play, and symmetric encoding/decoding is only tested on top of the other two. While this does make the paper more concise, I suspect another reason for this is that the self-play is the core driver of performance, and without it the other two methods don’t do much. I’d like this to be explained more explicitly in the paper.\n\nOverall, I really like the problem the paper is tackling, however I have some issues with the framing of the paper in relation to the self-play constraint, and subsequently with the importance of the results. Thus, I do not recommend acceptance in the paper’s current form. \n\n\nQuestions:\n-\t“We set shared embedding agents to always use the self-play objective, because otherwise its equivalent to the baseline agent” -> it’s not clear to me why this is the case. Can this be elaborate on?\n-\tShouldn’t the final row of Table 4 read ‘Trans, shapes’? \n\nSmall fixes:\n-\tThis assumption is a reasonable -> is reasonable\n-\tSection 5.1.2: “We observe a no clear trend associated with the shared embedding module (sometimes it helps, sometimes it hurts)” -> I don’t see any results on shared embeddings in Table 2? \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJePWIB0Kr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper analyzes if enforcing internal-consistency for speaker-listener setup can (i) improve the ability of the agents to refer to unseen referents (ii) generalize for different communicative roles. The paper evaluates a transformer and arecurrent model modified with various sharing strategies on a single-turn reference game. Finally, the paper claims that results with self-play suggest that internal consistency doesn’t help (i) but improves cross-role performance for (ii).\n\nAs a reader, the paper doesn’t provide me a concrete finding which can help in designing future multi-agent systems. Most of the results for the experiments (except self-play) don’t have a uniform signal across the board to deduce whether the internal-consistency works in all of the cases. Most of the speaker-listener scenarios emerge in dialog based settings which are multi-turn and require agents to act both as speaker and listener. Though paper advocates through some of its results that self-play is helpful in generalization across roles via internal-consistency, without multi-step experiments, qualitative and quantitative analysis of what is happening and why there is so much variation, the paper is weak in its current form. Therefore, I recommend weak reject as my rating. Below I discuss some of the concerns in detail:\n\nWithout multi-step evaluation, it is hard to gauge the extent to which self-play for internal consistency help in generalization of the roles. For e.g., task from Das et. al. (2017) [1] provides a clear signal on how well the agents are able to communicate through dialog evaluation. So in 5.2.1, the setup which requires training in both roles can provide better signal overall if it was trained to do multi-step conversation.\n\nPaper is missing any kind of quantitative or qualitative analysis. What are the differences between the embeddings of the agent that learned via self-play and the one which learned directly. It also be interesting to see how the shared embeddings and symmetric encoding and decoding affect these embedding and might help explain the drop and randomness. In Table 4., the results on symmetric encoding suggest that the claim of generalization through internal consistency might not hold everywhere. For Shared Embedding results, on RNN shapes, it is surprising that training in one role improves performance through internal consistency while in both roles it drops. These require further analysis to solidify the claim. Given the flaky results, to boost the claim, have authors tried other settings to test internal-consistency like Predator-Prey?\n\nThings that didn’t affect the score:\n\nRelated work section is missing the relevant discussion on continuous communication work and discussion on why internal consistency wasn’t tested on those settings as well. (See Singh et.al [2]., Sukhbaatar et.al. [3], Das et.al. [4] etc)\n\nThe number of pages are above eight, you should reduce the redundancy between table descriptions and text and maybe squeeze Section 2, decrease setup explanation.\n\nThe setup for training and test sets explained at the end of page 7 isn’t very clear to me and needs to be rephrased.\n\n[1] Das, Abhishek, Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. \"Learning cooperative visual dialog agents with deep reinforcement learning.\" In Proceedings of the IEEE International Conference on Computer Vision, pp. 2951-2960. 2017.\n[2] Sukhbaatar, Sainbayar, and Rob Fergus. \"Learning multiagent communication with backpropagation.\" In Advances in Neural Information Processing Systems, pp. 2244-2252. 2016.\n[3] Singh, Amanpreet, Tushar Jain, and Sainbayar Sukhbaatar. \"Learning when to communicate at scale in multiagent cooperative and competitive tasks.\" arXiv preprint arXiv:1812.09755 (2018).\n[4] Das, Abhishek, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Michael Rabbat, and Joelle Pineau. \"Tarmac: Targeted multi-agent communication.\" arXiv preprint arXiv:1810.11187 (2018).\n\n=========\nPost-rebuttal Comments\n=========\nThanks for updating the manuscript to resolve my and R2's concerns. The new analysis section does provide good insights into what exactly is happening.\n\nWhen I was talking about actionable insights, I was talking about both negative and positive insights. Currently, the only take-away is that self-play helps in generalizing to listener roles as well. For the other negative insight that internal consistency doesn't help with generalization, as R2 suggested, it is unclear why that would be case in the first place (I read the pscyhology arguments, but I am not still not convinced). I still believe that without multi-step communication, the work is as useful as it can be in current form. In real world, no meaningful conversation is usually one step.\n\nFor Predator-Prey setup, I was talking about OpenAI https://github.com/openai/multiagent-particle-envs in which multiple tasks can be setup. For e.g. if prey thinks of what action predator might take, does internal consistency help prey to perform better?\n\nI think most of what you got is correct for multi-step, see second para for more details in this response.\n\nThanks for bringing the manuscript under 8 pages. [3] is still missing from references.\n\nFinal comments: I would like to see multi-step experiments due to the reasons I explained above. The scheme of internal-consistency should be applicable beyond conversation to Predator-Prey setups also, thus, I feel experiments are not enough (only on 1 setting) to claim generalization of the hypothesis. Beyond these comments, I feel this is still a step in right direction and I would like to update my rating to weak accept while hoping that authors try to address these issues in camera-ready version if paper gets accepted.\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work examines how internal consistency objectives can help emergent communication, namely through possibly improving ability to refer to unseen referents and to generalize across communicative roles. Experimental results support the second hypothesis but not the first.
Reviewers agree that this is an exciting object of study, but had reservations about the rationale for the first hypothesis (which was ultimately disproven), and for how the second hypothesis was investigated (lack of ablations to tease apart which part was most responsible for improvement, unsatisfactory framing). These concerns were not fully addressed by the response.
While the paper is very promising and the direction quite interesting, this cannot in its current form be recommended for acceptance. We encourage authors to carefully examine reviewers' suggestions to improve their work for submission to another venue.
Decision: Reject