Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This work studies factors which promote combinatorial generalization in a "neural network agent" embodied in a 3d simulation environment. The authors present interesting experiments and some insightful empirical findings on how a richer environment and a first-person egocentric perspective can aid a simple neural net to generalize better over previously unseen tasks. While I truly commend the effort undertaken to perform the experiments, I have several concerns which I explain below and would be happy to raise my score if they can all be addressed satisfactorily:

1) While the authors interpret the experiment results in sec 4.1 in a positive way, the results don't seem to necessarily indicate good systematic generalization. For instance, after learning with 40 words the agent only achieves 60% test accuracy. While the accuracy increases to 78% on training with 100 words, the training and test accuracy gap indicates that the performance is still far from any kind of systematic generalization. The results instead seems to be hinting that neural nets don't indeed perform combinatorial generalization on their own, but can be forced towards it by supplying them huge amounts of diverse data (which is not true for humans). Also, the fact that increasing the number of words helps in generalizing better is true for most ML models and does not come as a surprise. So the results in this subsection are somewhat trivial and do not necessarily contribute any new understanding.

2) For the experiments regarding egocentric frame in sec 4.3, I feel that the results are not really conclusive (even including the control exps in appendix D). Could it be that if one uses any frame rigidly attached (i.e. fixed displacement and rotational coordinates) to the agent's egocentric frame, one would achieve the same generalization performance? It is also possible that as suggested by authors in sec 4.4, it is just the motion of the egocentric frame which might be giving diverse views of the environment to the agent. So the frame might not even need to be egocentric, but just a moving frame which gives richer and diverse views whenever the agent moves. Please include experiments to test for these possibilities.

3) In section 4.4, the authors have trained the non-embodied classifier with just a single image frame. But this does not necessarily justify the conclusion that active perception helps in generalization. This is because the motion of the RL agent gives it both a varied set of views AND also control over what views to obtain by taking actions. In order to better understand which of these factors (or perhaps both) aid in generalization, another set of experiments is required which shows the classifier agent more images while keeping the desired object in view. In one experiment, these images should be chosen with random movements but the number of such images provided to the classifier should be increased in sub-experiments to gauge if giving more varied views bridges the performance gap between the classifier and the RL agent's generalization performance. In a second experiment, one might want to first train the RL agent, then extract a few (say 10) frames out of its enacted policy for all pairs of objects and use these frames as a part of the training set for the classifier agent. This would allow one to gauge if both varied views and actively selecting to interact with the environment can help bridge the generalization gap.

4) Lastly, sec 4.5 seems to be hinting at a potentially very incorrect conclusion: "language is not a large factor (and certainly not a necessary cause) of the systematic generalisation...". This cannot be said from the small single experiment presented in sec 4.5. For instance, that experiment has been devised in a way that an optimal policy can be found with/without language. However, if a language input is provided to explicitly state the desired object, that might speed up the training of the RL agents significantly. In such a case, it might be helpful to see if learning the policy with the language input is being accomplished with a much lower number of frames during training, as opposed to when no language input is provided. Please provide the training error plots. But regardless of the plots, the experiments can still be quite inconclusive since language helps in systematic generalization in a variety of other ways apart from what has been tested for. In general, language starts helping humans once it has been acquired to a sufficient extent since one needs noun-concept linkages, verb-action linkages etc. to have been acquired a priori before the benefits of language emerge in combinatorial generalization. Training an LSTM to understand the language commands in tandem with learning policies for picking desired objects could lead to sub-optimal or heavily over-fitted language models which may not help in generalization. Testing for the true role of language will require many more experiments, which may be somewhat out of scope for this paper given the space constraints for a single paper. But, I would advise the authors to refrain from drawing hasty inferences about the role of language without thorough experimentation.

Minor issues:

1) What are the 26 actions in the Unity 3D environment in section 3? It is important to know the action space to understand how easy or hard it is for the agent to learn generalizable policies.
2) The x-axis of Figure 2 is not readable at all. Please rectify those graphs and reduce the number of ticks.


-------------------------- Update after interaction during author feedback period -------------------------------
I appreciate the efforts that the authors have undertaken to address my concerns. While the paper is far from perfect, it is still a very thought provoking work and I believe that it would make a valuable contribution to the line of works on systematic generalization in embodied agents. I am updating my score to reflect the same.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: 
=============================== Update after revisions =====================================================

In my initial review, I had raised some issues with the interpretation of the results and suggested some control experiments to tighten the conclusions. The authors chose to weaken their initial claims by rephrasing their conclusions instead. I understand that there may not have been enough time to run many of the experiments I suggested, but I still think they are worth considering for the future. I'm mostly satisfied with the rephrasing of the conclusions in the revised paper, so as promised, I'm happy to increase my score and recommend acceptance. 

I spotted several typos in the revised paper, however: section 4.1: "we choose to consider negation ...", p. 5: "for for ...", a citation on p. 5 is not compiled correctly. There may be more. For the final version please make sure to go through the paper thoroughly a couple of times and fix all the typos.

========================================================================================================

The authors present a systematic study of generalization in agents embedded in a simulated 3d environment. I think there are some interesting results in this paper that might be useful for people to know about. I appreciate the thoroughness of the experiments, in particular. I have, however, some issues with the interpretation of several of the main results. I would be happy to increase my score if we can resolve some of these issues. Here are my main concerns:

1) In the experiments in section 3, only a limited test set is used. How is the train/test split decided in these experiments? Table 6 suggests that you have a much larger repository of objects. Why not use all possible objects in the test set? It is a bit premature to declare your results as systematic generalization if you can’t show that it actually works for a much larger set of test objects (ideally for all possible objects). 

2) Section 4.1: in these experiments, the training set size is increased, but the test set size is kept constant (and small), so the train/test size ratio also increases. So, an alternative explanation of the results in this section is that the model behaves largely according to visual similarity and as the training set size is increased, it becomes easier to find a training set object that is visually similar to any test set object. I think the authors should run an experiment where both training and test set sizes increase by the same amount so that the train/test set size ratio stays constant. If the model can’t achieve systematic generalization in that case, it would be wrong to conclude, as the authors do now, that increasing the training set size itself improves systematic generalization. The correct conclusion would rather be that increasing the train/test size ratio improves generalization, which is a weaker conclusion. Please note that the results in this section are quite similar to those in Lake & Baroni (2018) and in Bahdanau et al. (2018) (see their Figure 3). Bahdanau et al. (2018), for example, also show that increasing train/test set size ratio (their “rhs/lhs” ratio) improves generalization in generic neural networks. It is interesting to note, however, that neither Lake & Baroni (2018) nor Bahdanau et al. (2018) interpret these results positively (i.e., these results don’t show systematicity), whereas the current paper seems to put a more positive spin on essentially the same result. I think these earlier results should be explicitly discussed here and the authors should justify why they are interpreting the results differently (if they are). It should also be noted that in the real world the train/test size ratio for humans is presumably very small, perhaps zero (given the compositional abilities of humans).

3) Section 4.3: I don’t think the results in this section are sufficient to establish the egocentric frame per se as the key factor. One possibility is that perhaps the frame doesn’t have to be centered on the agent, but as long as it has some systematic relationship to the agent’s location (for example, the center of the visibility window could be some distance away from the agent, and the agent itself may or may not be inside this window), that’s good enough to get generalization improvements. An even weaker possibility is that simply a moving frame is enough for improved generalization. In this case, the reference frame doesn’t even need to have a systematic relationship to the agent’s location. For example, the frame could be relative to a fictitious agent that randomly explores the environment. I think the authors should run some experiments to rule out these possibilities if they want to claim that the egocentric frame itself is responsible for generalization improvements. 

4) Section 4.4: In the experiments in this section, I think there are two relevant factors that need to be better disentangled: 1) the number and variability of image frames experienced by the two models; 2) the active perception aspect (the fact that the agent interacts with the environment and affects its own perceptual experience in one case). The authors claim the second factor as the key aspect enabling better generalization, but 1) is equally likely (this would be more in line with a standard data augmentation type result). A good control experiment here would be to not just use the first frame but a larger number of more variable frames for training the non-situated agent (for example, one can use image frames that would be seen by a camera that more or less randomly moves in front of the objects perhaps with the constraint that both objects are always at least partially visible). If the classification model generalizes as well as the situated agent in this control condition, you cannot claim active perception as the key factor.

5) As a more general point, it’s a bit frustrating to have to judge systematic generalization by only looking at the results of some limited set of experiments. How do I interpret the results if the agent achieves only 84% accuracy in some experiment (as opposed to 100%)? It would be much better if the authors could somehow more rigorously prove systematicity. Here, I don’t necessarily mean “prove” in a mathematical sense, but just analyzing the learned representations a bit more rigorously and being able to say something along the lines of: here’s exactly how the trained model represents “lift”; because of reason X, Y, Z, this representation is completely disentangled from all object representations in the dataset (and ideally from all possible object representations, because that’s really what true systematicity entails, although I highly doubt that any generic model of the type studied in this paper will be able to achieve this, regardless of the amount and type of input it receives).   

More minor issues:

6) In Table 5, “table lamp” appears both in training and test sets. Is this a typo?

7) Some results are presented in the appendix without any mention in the main text (Appendix D. 2). I think this is not a good practice in general. In the main text, please make sure to mention, however briefly, every result that appears in the appendix (something along the lines of "This result could not be explained by confound X or Y (Appendix Z)" would suffice).

8) Font size in Figure 2 is tiny (axis labels are impossible to read), please make it bigger. You don’t need that many ticks on the axes.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper studies systematic generalization in a situated agent. The authors examine the degree to which various factors influence systematic generalization, including 2D vs. 3D environments, egocentric vision,  active perception, and language. The experiments reveal that the first three factors, but not language, promote systematic generalization. The experiments are well-done and worthwhile, and identifying the key factors that affect generalization is a strength of the paper.

I have two main criticisms. First, the model's abilities for systematic generalization are overstated. Second, critical details about the experiments are omitted that make them difficult to evaluate.

Let's start with the abilities of the model. The title of the paper is "Emergent systematic generalization in a situated agent," which of course implies that the agent has "systematic generalization." The authors go on to say, in the abstract, that "we demonstrate strong emergent systematic generalisation in a neural network agent". The results, however, fall short of these statements.

The strongest results pertain to generalizing a highly-practiced action such as "lifting" or "putting" to novel objects. In this case, highly-practiced means that the actions have been trained on 31 unique objects for millions of steps. However the paper does not study whether or not the agent can learn a novel action (e.g. "lifting" or "putting" with only a few examples) and generalize it systematically to familiar objects. Nor does it study whether novel actions can be combined systematically in new ways using relations and modifiers such as "finding the toothbrush ABOVE the hat" or "finding AND putting" or "putting to the right of." Benchmarks for systematic generalization such as SQOOP and SCAN include these types of generalizations, and an agent with systematic generalization should handle them as well. To be clear, I don't think it's necessary to add additional experiments to the paper, but the current results should not be overstated in their generality.

Even within the reported experiments, the results suggest that systematicity is lacking in several places. In the negation task, where chance is 50% accuracy, the agent achieves only 60% correct after learning from 40 unique words and 78% performance with 100 unique words (doesn't systematic generalization imply 100%?) For the putting tasks, the agent achieves 90% correct in one experiment (section 3.2) and then only achieves 63% correct in another (section 4.2). Again, the generalization abilities seem far from systematic.

Critical details about the action space and the simulation parameters are needed. The action-space has 26 actions, but the paper does not say what these actions are. These details are crucial to understanding what is required to generalize "lift" or "put" to new objects -- instead the paper only says that "in particular the process required to lift an object does not depend on the shape of that object (only its extent, to a degree)" and that "shape is somewhat more important for placement" compared to lifting.

I would consider updating my evaluation if the authors make revisions to ensure that the evidence supports their conclusions. The paper's title should also be supported by the findings; to offer a suggestion, something like "Richer environments promote systematic generalization in situated agents".

Other suggestions
- The axis on Fig. 2 is too small to read. Also, it should be mentioned in text that the network is trained for 100 million+ steps (also, what is a step? how many episodes was it trained for?)
- The number of objects in sets X_1 and X_2 is important and should be mentioned in the main text.

------

** Update to review **

Thanks for your response to my review. It's clear that the authors have made considerable effort to improve the paper. In particular, the revised title, abstract, and introduction now more accurately reflect the contributions of the paper. It's not perfect, but the paper is improved and I revised my score accordingly.

While it did not affect my final score, not all my suggestions were incorporated and I hope the authors will make further improvements in their revisions. The number of objects in sets X_1 and X_2  (Sections 3.1 and 3.2) are not mentioned and are tucked away in the appendix' this should be in the main text. Thanks for providing the list of 26 actions, but it's still not completely clear what makes a successful "lift" or "put" in terms of the sequence of actions. Finally, rather than simply saying that your agent "in no way exhibits complete systematicity" (Discussion), I hope the authors will expand on this and discuss the limitations of their experiments, and the kinds of systematicity not addressed and which could be the focus of future work.
review_title: Official Blind Review #3

Meta-review: The paper studies out-of-sample generalisation that require an agent to respond to never-seen-before instructions by manipulating and positioning objects in a 3D Unity simulated room, and analyzes factors which promote combinatorial generalization in such environment. 

The paper is a very thought provoking work, and would make a valuable contribution to the line of works on systematic generalization in embodied agents. The draft has been improved significantly after the rebuttal. After the discussion, we agree that it is worthwhile presenting at ICLR.  
Decision: Accept (Poster)