Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this manuscript, the authors connect psychological experimental methods to understand how the black box of the mind solves problems with current issues in understanding how the black box of deep learning methods solves problems. The authors used situated versions of human language learning tasks as simulation environments to test a CNN + LSTM deep learning network. They examined a few key phenomena: shape/color bias, learning negation concepts, incremental learning, and how learning affects the representation of objects via attention-like processes. They illustrated conditions in which their deep learning network acts similarly to people in simulations.
Developing methods that enable humans to understand how deep learning models solve problems is an important problem for many reasons (e.g., usability of models for science, ethical concerns) that has captured the interest of a wide range of researchers. By adapting experimental methodology from psychology to test that have been used to understand and explain the internal workings of the mind, the authors approach the problem in a novel and innovative manner. I was impressed by the range of phenomena they tackled and their analyses were informative in understanding the behavior of deep learning models 
I found the analogy persuasive in theory, but I was not convinced that the current manuscript really demonstrates its value. In particular, I did not see the value of situating their model in a grounded environment. One analysis that would have helped convince me is a comparison to an equivalent non-grounded deep learning model (e.g., a CNN trained to make equivalent classifications), and show how this would not help us understand human behavior. However, the more I thought about the logic of this type of analysis, the more concerned I became about the logic of their approach. 
What would it mean if the equivalent non-situated model does not show the phenomena? If it does not, it could illustrate the efficacy of using situated environments. But, it also could mean that their technique acts differently for equivalent situated and non-situated models. In this case though, what would we learn about the more general non-situated case then? It does not seem like we would learn much, which would defeat the purpose of the technique. Alternatively, if the equivalent non-situated model does show the phenomena, then using the situated version would not be useful because the model acts equivalently in both cases. I am not fully convinced by the argument I just sketched, but leaves me very concerned about the usefulness of their approach. (note that the “controlled” vs. “naturalistic” analyses in the word learning section did not convince me. This argues for the importance of using naturalistic statistics – not necessarily cross-modal, situated environments as the authors argue for).
Additionally, I was unconvinced that simpler models could not be used to examine the phenomena that they analyzed. Although combining LSTM with CNN via a “mixing” module was interesting, it added another layer of complexity that made it more difficult to assess what the results meant. This left me less convinced of the usefulness of their paradigm. If we need to create a novel deep learning method to illustrate its efficacy, how will it be useful for solving the problem that motivated everything: understanding how pre-existing deep learning methods solve problems. 

review_title: An innovating and interesting approach, but there are serious, lingering concerns about the approach 

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents an analysis of an agent trained to follow linguistic commands in a 3D environment.  The behaviour of the agent is analyzed by means of a set of "psycholinguistic" experiments probing what it learned, and by inspection of its visual component through an attentional mechanism.

On the positive side, it is nice to read a paper that focuses on understanding what an agent is learning. On the negative side, I did not get many new insights from the analyses presented in the study.

3 A situated language learning agent

I can't make up the chair from the refrigerator in the figure.

4.1 Word learning biases

This experiment shows that, when an agent is trained on shapes only, it will exhibit a shape bias when tested on new shapes and colors. Conversely, when it is exposed to colors only, it will have a color bias. When the training set is balanced, the agent shows a mild bias for the simpler color property. How is this interesting or surprising? The crucial question, here, would be whether, when an agent is trained in a naturalistic environment (i.e., where distributions of colors, shapes and other properties reflect those encountered by biological agents), it would show a human-like shape bias. This, however, is not addressed in the paper.

Minor comments about this section:

- Was there noise also in shape generation, or were all object instances identical?

- propensity to select o_2: rather o_1?

- I did not follow the paragraph starting with "This effect provides".

4.2 The problem of learning negation

I found this experiment very interesting.

Perhaps, the authors could be more explicit about the usage of negation here. The meaning of commands containing negation are, I think, conjunctions of the form "pick something and do not pick X" (as opposed to the more natural "do not pick X").

modifiation: modification

4.3 Curriculum learning

Perhaps the difference in curriculum effectiveness in language modeling vs grounded language learning simulations is due to the fact that the former operates on large amounts of natural data, where it's hard to define the curriculum, while the latter are typically grounded in toy worlds with a controlled language, where it's easier to construct the curriculum.

4.4 Processing and representation differences

There is virtually no discussion of what makes the naturalistic setup naturalistic, and thus it's not clear which conclusions we should derive from the corresponding experiments. Also, I don't see what we should learn from Figure 5 (besides the fact that in the controlled condition shapes are easier than categories). For the naturalistic condition, the current figure is misleading, since different classes contain different numbers of instances. It would be better to report proportions.

Concerning the attention analysis, it seems to me that all it's saying is that lower layers of a CNN detect lower-level properties such as colors, higher layers detect more complex properties, such as shapes characterizing objects. What is novel here?

Also, since introducing attention changes the architecture, shouldn't the paper report the learning behaviour of the attention-augmented network?

The explanation of the attention mechanism is dense, and perhaps could be aided by a diagram (in the supplementary materials?). I think the description uses "length" when "dimensional(ity)" is meant.

6. Supplementary material

It would be good to have an explicit description of the architecture, including number of layers of the various components, structure of the CNN, non-linearities, dimensionality of the layers, etc. (some of this information is inconsistently provided in the paper).

It's interesting that the encoder is actually a BOW model. This should be discussed in the paper, as it raises concerns about the linguistic interest of the controlled language that was used.

Table 3: indicates is: indicates if

review_title: Worthy goal, but implementation felt a bit underwhelming

final_score: rating: 7: Good paper, accept
review_content: This paper presents an analysis of the properties of agents who learn grounded language through reinforcement learning in a simple environment that combines verbal instruction with visual information. The analyses are motivated by results from cognitive and developmental psychology, exploring questions such as whether agents develop biases for shape/color, the difficulty of learning negation, the impact of curriculum format, and how representations at different levels of abstraction are acquired. I think this is a nice example of a detailed analysis of the representations acquired by a reinforcement learning agent. The extent to which it provides us with insight into human cognition depends on the degree to which we believe the structure of the agent and the task have a correspondence to the human case, which is ultimately probably quite limited. Nonetheless the paper takes on an ambitious goal of relating questions in machine learning in cognitive science and does a reasonably good job of analyzing the results.

Comments:

1. The results on word learning biases are not particularly surprising given previous work in this area, much of which has used similar neural network models. Linda Smith and Eliana Colunga have published a series of papers that explore these questions in detail:

http://www.iub.edu/~cogdev/labwork/kinds.pdf
http://www.iub.edu/~cogdev/labwork/Ontology2003.pdf

2. In figure 2 and the associated analyses, why were 20 shape terms used rather than 8 to parallel the other cases? It seems like there is a strong basic color bias. This seems like one of the most novel findings in the paper and is worth highlighting.

This figure and the corresponding analysis could be made more systematic by mapping out the degree of shape versus color bias as a function of the number of shape and color terms in a 2D plot. The resulting plot would show the degree of bias towards color.

3. The section on curriculum learning does not mention relevant work on “starting small”  and the “less is more" hypothesis in language development by Jeff Elman and Elissa Newport:

https://pdfs.semanticscholar.org/371b/240bebcaa68921aa87db4cd3a5d4e2a3a36b.pdf
http://www.sciencedirect.com/science/article/pii/0388000188900101

4. The section on learning speeds could include more information on the actual patterns that are found with human learners, for example the color words are typically acquired later. I found these human results hard to reconcile with the results from the models. I also found it hard to understand why colors were hard to learn given the bias towards colors shown earlier in the paper.

5. The section on layerwise attention claims to give a “computational level” explanation, but this is a misleading term to use — it is not a computational level explanation in the sense introduced by David Marr which is the standard use of this term in cognitive science. The explanation of layerwise attention could be clearer.

Minor:

“analagous” -> “analogous”

The paper runs longer than eight pages, and it is not obvious that the extra space is warranted.

review_title: Interesting work that could be grounded more strongly in cognitive science

Meta-review: This paper resulted in significant discussion -- both between R2 and the authors, and between the AC, PCs, and other solicited experts.

The problem of language grounding (and instruction following) in virtual environments is clearly important, this work was one of the first in the recent resurgence, and the goal of understand what the agents have learned is clearly noble and important. In terms of raw recommendations, the majority reviewer recommendation is negative, but since concerns raised by R2 seemed subjective (which in principle is not a problem), out of abundance of caution, we solicited additional input. Unfortunately, we received feedback consistent with the concerns raised here:

-- The lack of generality of the behavior found. Even if we ignore the difficult question of why the agent prefers what it does, it's unclear how the conclusions here generalize much farther than the model and environment used; the manuscript does not provide any novel or transferable principals of the form "this kind of bias in the environment leads to this kind of bias in models with these properties".

-- We realize even providing that concrete a statement might be hard, but also missing are thorough comparisons to other kinds of models (e.g. non-deep, as asked by R1) to establish that this is a general phenomenon.

Ultimately, there is a sense that this is too narrow an analysis, too soon. If there was one architecture for learning embodied agents in 3d environments that was clearly successful and useful, then studying its properties might be interesting (even crucial).  But the dust in this space isn't settled. Our current agents are fairly poor, and so the impact of understanding the biases of a specific model trained in a specific environment seems fairly low.

Finally -- this not taken into consideration in making the decision -- it is not okay to list personal homepage domains (that may reveal author identity to ACs) as conflict domains; those are meant for institutional conflicts/domains. 
Decision: Reject