Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents an analysis of the communication systems that arose when neural network based agents played simple referential games. The set up is that a speaker and a listener engage in a game where both can see a set of possible referents (either represented symbolically in terms of features, or represented as simple images) and the speaker produces a message consisting of a sequence of numbers while the listener has to make the choice of which referent the speaker intends. This is a set up that has been used in a large amount of previous work, and the authors summarize some of this work. The main novelty in this paper is the choice of models to be used by speaker and listener, which are based on LSTMs and convolutional neural networks. The results show that the agents generate effective communication systems, and some analysis is given of the extent to which these communications systems develop compositional properties – a question that is currently being explored in the literature on language creation.

This is an interesting question, and it is nice to see worker playing modern neural network models to his question and exploring the properties of the solutions of the phone. However, there are also a number of issues with the work.

1. One of the key question is the extent to which the constructed communication systems demonstrate compositionality. The authors note that there is not a good quantitative measure of this. However, this is been the topic of much research of the literature and language evolution. This work has resulted in some measures that could be applied here, see for example Carr et al. (2016): http://www.research.ed.ac.uk/portal/files/25091325/Carr_et_al_2016_Cognitive_Science.pdf

2. In general the results occurred be more quantitative. In section 3.3.2 it would be nice to see statistical tests used to evaluate the claims. Minimally I think it is necessary to calculate a null distribution for the statistics that are reported.

3. As noted above the main novelty of this work is the use of contemporary network models. One of the advantages of this is that it makes it possible to work with more complex data stimuli, such as images. However, unfortunately the image example that is used is still very artificial being based on a small set of synthetically generated images.

Overall, I see this as an interesting piece of work that may be of interest to researchers exploring questions around language creation and language evolution, but I think the results require more careful analysis and the novelty is relatively limited, at least in the way that the results are presented here.
review_title: Explores interesting issues,  but needs more quantitative analysis and has limited novelty

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper presents a set of studies on emergent communication protocols in referential games that use either symbolic object representations or pixel-level representations of generated images as input. The work is extremely creative and packed with interesting experiments.

I have three main comments.

* CLARITY OF EXPOSITION

The paper was rather hard to read. I'll provide some suggestions for improvement in the minor-comments section below, but one thing that could help a lot is to establish terminology at the beginning, and be consistent with it throughout the paper: what is a word, a message, a protocol, a vocabulary, a lexicon? etc.

* RELATION BETWEEN VOCABULARY SIZE AND PROTOCOL SIZE

In the compositional setup considered by the authors, agents can choose how many basic symbols to use and the length of the "words" they will form with these symbols. There is virtually no discussion of this interesting interplay in the paper. Also, there is no information about the length distribution of words (in basic symbols), and no discussion of whether the latter was meaningful in any way.

* RELATION BETWEEN CONCEPT-PROPERTY AND RAW-PIXEL STUDIES

The two studies rely on different analyses, and it is difficult to compare them. I realize that it would be impossible to report perfectly comparable analyses, but the authors could at least apply the "topographic" analysis of compositionality in the raw-pixel study as well, either by correlating the CNN-based representational similarities of the Speaker with its message similarities, or computing similarity of the inputs in discretized, symbolic terms (or both?).

* MINOR/DETAILED COMMENTS

Section 1

How do you think emergent communication experiments can shed light on language acquisition?

Section 2

In figure 1, the two agents point at nothing.

\mathbf{v} is a set, but it's denoted as a vector. Right below that, h^S is probably h^L?

all candidates c \in C: or rather their representations \mathbf{v}?

Give intuition for the reward function.

Section 3

We use the dataset of Visual Attributes...: drop "dataset"

I think the pre-linguistic objects are not represented by 1-hot, but binary vectors.

do care some inherent structure: carry

Note that symbols in V have no pre-defined semantics...: This is repeated multiple times.

Section 3

I couldn't find simulation details: how many training elements, and how is training accuracy computed? Also, "training data", "training accuracy" are probably misleading terms, as I suppose you measured performance on new combinations of objects.

I find "Protocol Size" to be a rather counterintuitive term: maybe call Vocabulary Size "Alphabet Size", and Protocol Size "Lexicon Size"?

State in Table 1 caption that the topographic measure will be explained in a later section. Also, the -1 is confusing: you can briefly mention when you introduce the measure that since you correlate a distance with a similarity you expect an inverse relation? Also, you mention in the caption that all Spearman rhos are significant, but where are they presented again?

Section 3.2

Does the paragraph starting with "Note that the distractor" refer to a figure or table that is not there? If not, it should be there, since it's not clear what are the data that support your claims there. Also, you should explain what the degenerate strategy the agents find is.

Next paragraph:

- I find the usage of "obtaining" to refer to the relation between messages and objects strange.

- in which space are the reported pairwise similarities computed?

- make clear that in the non-uniform case confusability is less influenced by similarity since the agents must learn to distinguish between similar objects that naturally co-occur (sheep and goats)

- what is the expected effect on the naturalness of the emerged language?

Section 3.3

adhere to, the ability to: "such as" missing?

Is the unigram chimera distribution inferred from the statistics over the distribution of properties across all concepts or what? (please clarify.)

In Tables 2 and 3, why is vocabulary size missing?

In Table 2, say that the protocol size columns report novel message percentage **for the "test" conditions***

Figure 2: spelling of Levensthein

Section 3.3.2

while for languages (c,d)... something missing.

with a randomly initialized...: no a

More importantly, I don't understand this "random" setup: if architecture was fixed and randomly initialized, how could something be learned about the structure of the data?

Section 4

Refer to the images the agents must communicate about as "scenes", since objects are just a component of them.

What are the absolute sizes of train and test splits?

Section 4.1

we do not address this issue: the issue

Section 4.2

at least in the game C&D: games

Why is Appendix A containing information that logically follows that in Appendix B?

review_title: Very interesting paper, writeup could be clearer

final_score: rating: 7: Good paper, accept
review_content: --------------
Summary:
--------------
This paper presents a series of experiments on language emergence through referential games between two agents. They ground these experiments in both fully-specified symbolic worlds and through raw, entangled, visual observations of simple synthetic scenes. They provide rich analysis of the emergent languages the agents produce under different experimental conditions. This analysis (especially on raw pixel images) make up the primary contribution of this work.


--------------
Evaluation:
--------------
Overall I think the paper makes some interesting contributions with respect to the line of recent 'language emergence' papers. The authors provide novel analysis of the learned languages and perceptual system across a number of environmental settings, coming to the (perhaps uncontroversial) finding that varying the environment and restrictions on language result in variations in the learned communication protocols. 

In the context of existing literature, the novelty of this work is somewhat limited -- consisting primarily of the extension of multi-agent reference games to raw-pixel inputs. While this is a non-trivial extension, other works have demonstrated language learning in similar referring-expression contexts (essentially modeling only the listener model [Hermann et.al 2017]). 

I have a number of requests for clarification in the weaknesses section which I think would improve my understanding of this work and result in a stronger submission if included by the authors.  

--------------
Strengths:
--------------
- Clear writing and document structure. 


- Extensive experimental setting tweaks which ablate the information and regularity available to the agents. The discussion of the resulting languages is appropriate and provides some interesting insights.


- A number of novel analyses are presented to evaluate the learned languages and perceptual systems. 


--------------
Weaknesses:
--------------
- How stable are the reported trends / languages across multiple runs within the same experimental setting? The variance of REINFORCE policy gradients (especially without a baseline) plus the general stochasticity of SGD on randomly initialized networks leads me to believe that multiple training runs of these agents might result is significantly different codes / performance. I am interested in hearing the author's experiences in this regard and if multiple runs present similar quantitative and qualitative results. I admit that expecting identical codes is unrealistic, but the form of the codes (i.e. primarily encoding position) might be consistent even if the individual mappings are not).


- I don't recall seeing descriptions of the inference-time procedure used to evaluate training / test accuracy. I will assume argmax decoding for both speaker and listener. Please clarify or let me know if I missed something.


- There is ambiguity in how the "protocol size" metric is computed. In Table 1, it is defined as 'the effective number of unique message used'. This comes back to my question about decoding I suppose, but does this count the 'inference-time' messages or those produced during training? 
Furthermore, Table 2 redefines "protocol size" as the percentage of novel message. I assume this is an editing error given the values presented and take these columns as counts. It also seems "protocol size" is replaced with the term "lexicon" from 4.1 onward.

- I'm surprised by how well the agents generalize in the raw pixel data experiments. In fact, it seems that across all games the test accuracy remains very close to the train accuracy. 

Given the dataset is created by taking all combinations of color / shape and then sampling 100 location / floor color variations, it is unlikely that a shape / color combo has not been seen in training. Such that the only novel variations are likely location and floor color. However, taking Game A as an example, the probe classifiers are relatively poor at these attributes -- indicating the speaker's representation is not capturing these attributes well. Then how do the agents effectively differentiate so well between 20 images leveraging primarily color and shape?

I think some additional analysis of this setting might shed some light on this issue. One thought is to compute upper-bounds based on ground truth attributes. Consider a model which knows shape perfectly, but cannot predict other attributes beyond chance. To compute the performance of such a model, you could take the candidate set, remove any instances not matching the ground truth shape, and then pick randomly from the remaining instances. Something similar could be repeated for all attributes independently as well as their combinations -- obviously culminating in 100% accuracy given all 4. It could be that by dataset construction, object location and shape are sufficient to achieve high accuracy because the odds of seeing the same shape at the same location (but different color) is very low. 

Given these are operations on annotations and don't require time-consuming model training, I hope to see this analysis in the rebuttal to put the results into appropriate context.


- What is random chance for the position and floor color probe classifiers? I don't think it is mentioned how many locations / floor colors are used in generation.  


- Relatively minor complaint: Both agents are trained via the REINFORCE policy gradient update rule; however, the listener agent makes a fairly standard classification decision and could be trained with a standard cross-entropy loss. That is to say, the listener policy need not make intermediate discrete policy decisions. This decision to withhold available supervision is not discussed in the paper (as far as I noticed), could the authors speak to this point?



--------------
Curiosities:
--------------
- I got the impression from the results (specifically the lack of discussion about message length) that in these experiments agents always issued full length messages even though they did not need to do so. If true, could the authors give some intuition as to why? If untrue, what sort of distribution of lengths do you observe?

- There is no long term planning involved in this problem, so why use reinforcement learning over some sort of differentiable sampler? With some re-parameterization (i.e. Gumbel-Softmax), this model could be end-to-end differentiable.


--------------
Minor errors:
--------------
[2.2 paragraph 1] LSTM citation should not be in inline form.
[3 paragraph 1] 'Note that these representations do care some' -> carry
[3.3.1 last paragraph] 'still able comprehend' --> to


-------
Edit
-------
Updating rating from 6 to 7.
review_title: Review

Meta-review: Important problem (analyzing the properties of emergent languages in multi-agent reference games), a number of interesting analyses (both with symbolic and pixel inputs), reaching a finding that varying the environment and restrictions on language result in variations in the learned communication protocols (which in hindsight is that not surprising, but that's hindsight). While the pixel experiments are not done with real images, it's an interesting addition the literature nonetheless.  
Decision: Accept (Oral)