Environmental drivers of systematicity and generalization in a situated agent
Authors: Felix Hill
Andrew Lampinen
Rosalia Schneider
Stephen Clark
Matthew Botvinick
James L. McClelland
Adam Santoro
Venue:  (2020)
Keywords: systematicitiy
systematic
generalization
combinatorial
agent
policy
language
compositionality

Abstract: The question of whether deep neural networks are good at generalising beyond their immediate training experience is of critical importance for learning-based approaches to AI. Here, we consider tests of out-of-sample generalisation that require an agent to respond to never-seen-before instructions by manipulating and positioning objects in a 3D Unity simulated room. We first describe a comparatively generic agent architecture that exhibits strong performance on these tests. We then identify three aspects of the training regime and environment that make a significant difference to its performance: (a) the number of object/word experiences in the training set; (b) the visual invariances afforded by the agent's perspective, or frame of reference; and (c) the variety of visual input inherent in the perceptual aspect of the agent's perception. Our findings indicate that the degree of generalisation that networks exhibit can depend critically on particulars of the environment in which a given task is instantiated. They further suggest that the propensity for neural networks to generalise in systematic ways may increase if, like human children, those networks have access to many frames of richly varying, multi-modal observations as they learn.

Published as a conference paper at ICLR 2020
ENVIRONMENTAL DRIVERS OF SYSTEMATICITY AND GENERALISATION IN A SITUATED AGENT
Felix Hill', Andrew Lampinen Rosalia Schneider', Stephen Clark!
Matthew Botvinick! James L. McClelland & Adam Santoro
DeepMind, London
2 Gatsby Computational Neuroscience Unit, University College London
3 Dept. of Psychology, Stanford University
ABSTRACT
The question of whether deep neural networks are good at generalising beyond their immediate training experience is of critical importance for bearning-based approaches to AI. Here, we consider tests of out-of-sample generalisation that re- quire an agent to respond to sever-seen-before instructions by manipulating and positioning objects in a 3D Unity simulated room. We first describe a compara- tively generic agent architecture that exhibits strong performance on these tests. We then identify three aspects of the training regime and environment that make significant difference to its performance: (a) the number of object/word expe- riences in the training set; (b) the visual invariances afforded by the agent's 's per- spective, frame of reference; and (c) the variety of visual input inherent in the perceptual aspect of the agent's 's perception. Our findings indicate that the degree of generalisation that networks exhibit can depend critically on particulars of the environment in which given task is instantiated. They further suggest that the propensity for neural networks to generalise in systematic ways may increase if, like human children, those networks have access to many frames of richly varying, multi-modal observations as they learn.
INTRODUCTION
Since the earliest days of research on neural networks, a recurring point of debate is whether neural networks cxhibit generalisation beyond their training experience, in a systematic way (Smolensky, 1988; Fodor & Pylyshyn, 1988; Marcus, 1998; McClelland et al., 1987). This debate has been re-energized over the past few years, given resurgence in neural network research overall (Lake & Baroni, 2017; Bahdanau et al., 2018; Lake, 2019). Generalisation in neural networks is not a binary question; since there cases where networks generalise well and others where they do not, the pertinent research question is when and under what conditions neural networks are able to generalise. Here, we establish that a conventional neural-network-bassee agent exposed to raw visual input and symbolic [language-like instructions readily learns to exhibit generalisation that approaches systematic behaviour, and we explore the conditions supporting its emergence. First, we show in 3D simulated room that an agent trained to find all objects from a set and lift only some of them can lift withheld test objects never lifted during training. Second, we show that the same agent trained to lift all of the objects and put only some of them during training can put withheld test objects, zero-shot, in the correct location. That is, the model learns to re-compose known concepts (verbs and nouns) in novel combinations.
In order to better understand this generalisation, we conduct several experiments to isolate its con- tributing factors. We find three to be critical: (a) the number of words and objects experienced during training; (b) a bounded frame of reference or perspective; and (c) the diversity of perceptual input afforded by the temporal aspect of the agent's perspective. Crucially, these factors can be enhanced by situating an agent in a realistic environment, rather than an abstract, simplified setting. These results serve to explain differences between our findings and studies showing poor generalisation, where networks were typically trained in a supervised fashion on abstract or idealised stimuli from a
Work carried during internship DeepMind.

---

Published as a conference paper at ICLR 2020
single modality (e.g. Lake & Baroni, 2017). They also suggest that the human capacity to exploit the compositionality of the world, when learning to generalise in systematic ways, might be replicated in artificial neural networks if those networks
afforded access to a rich, interactive, multimodal stream of stimuli that better matches the experience of an embodied human learner (Clerkin et al., 2017; Kellman & Arterberry, 2000; James et al., 2014; Yurovsky
al., 2013; Anderson, 2003). Our results suggest that robust systematic generalisation may be an emergent property of an agent interacting with rich, situated environment (c.f. McClelland al., 2010).
1.1 SYSTEMATICITY AND GENERALISATION
Systematicity is the property of human cognition whereby "the ability to entertain a given thought implies the ability to entertain thoughts with semantically related contents" (Fodor & Pylyshyn, 1988). As an example of systematic thinking, Fodor & Pylyshyn (1988) point out that any human who can understand John loves Mary will also understand the phrase Mary loves John, whether or not they have heard the latter phrase before. Systematic generalisation (Plaut, 1999; Bahdanau al., 2018; Lake et al., 2019) is desirable characteristic for computational model because suggests that if the model can understand components (or words) in certain combinations, it should also understand the same components different combinations. Note that systematic generalisation is also sometimes referred to as "combinatorial generalisation' (O (O'Reilly, 2001; Battaglia et al., 2018). However, even human reasoning is often not perfectly systematic (O D'Rellly et al., 2013). We therefore consider this issue from the perspective of generalisation, and ask to what degree a system generalises in accordance with the systematic structure implied by language.
Recent discussions around systematicity and neural networks have focused on the issue of how best to encourage this behaviour in trained models. Many recent contributions argue that generalising systematically requires inductive biases that are specifically designed support some form of sym- bolic computation (such as graphs (Battaglia et al., 2018), modular-components defined by symbolic parses (Andreas et al., 2016; Bahdanau et al., 2018), explicit latent variables (Higgins et al., 2017) or other neuro-symboli hybrid methods (Mao et al., 2019)). On the other hand, some recent work has reported instances of strong generalisation in the absence of such specific inductive biases (Chaplot et al., 2018; Yu et al., 2018; 00;8: Lake, 2019). In the following sections, we first add to this latter literature by reporting several novel cases of emergent generalisation. Unlike previous work, the examples that we present here involve tasks involving the manipulation of objects via motor-policies, well as language and vision. This is followed by an in-depth empirical analysis the environmental conditions that stimulate generalisation in these cases.
2 A MINIMAL MULTI-MODAL AGENT
All our experiments use the same agent archi- tecture, a minimal set of components for visual (pixels) perception, language (strings) percep- tion, and policy prediction contingent on cur- rent and past observations. The simplicity of the architecture is intended to emphasize the generality of the findings; for details see App C.
Policy
Multi-modal Integration
LSTM
Language Processing
Visual
LSTM
CNN
Processing
"Put the airplane on bed"
Visual processing Visual observations at each timestep come in the form of a W x real-valued tensor (W and H depend on the particular environment), which processed by 3-layer convolutional network with 64, 64, 32 channels in the first, second and third layers re- spectively. The flattened output of this network is concatenated with an embedding of the lan- guage observation.
Language
Visual Observation
3
Observation
Environment
Figure 1: Schematic of the architecture used in all ex- periments. The blue components show some critical differences that differentiate from more abstract stud- ies that reported failures in systematic generalization.
Language processing Language instructions are received at every timestep as a string. The agent splits these on whitespace and processes them with (word-level) LSTM network with hidden state
2

---

Published as a conference paper at ICLR 2020
size 128. The final hidden state is concatenated with the output of the visual processor to yield multimodal representation of the stimulus at each timestep.
Memory, action and value prediction The multimodal representation is passed to 128-umit LSTM. At each timestep, the state of this LSTM is multiplied by a weight matrix containing x 128 weights; the output of this operation is passed through softmax to yield distribution over actions (A is the enviromment-dependen size of the action set). The memory state is also multiplied by a 1 X 128 weight matrix yield value prediction.
Training algorithm We train the agent using an importance-weiighted actor-critic algorithm with a central learner and distributed actors (Espeholt al., 2018).
3 DEMONSTRATING GENERALISATION
A key aspect of language understanding is the ability to flexibly combine predicates with arguments; verb-noun binding is perhaps the canonical example. Verbs typically refer to processes and actions, so we study this phenomenon in a 3D room built in the Unity game engine. In this environment, the agent observes the world from a first-person perspective, the Unity objects are 3D renderings of ev- eryday objects, the environment has simulated physics enabling objects to be picked-up, moved and stacked, and the agent's 's action-space consists of 26 actions that allow the agent to move its location, its field of vision, and to grip, lift, lower and manipulate objects. Executing simple instruction like f find a toothbrush (which can be accomplished on average in six actions by a well-trained agent in our corresponding grid world) requires an average of around 20 action decisions.
3.1 A GENERAL NOTION OF LIFTING
Lifting is simple motor process that corresponds to a verb and is easily studied in this environment. We consider the example instruction 1ift helicopter to be successfully executed if the agent picks up and raises the helicopter above a height of 0.5m for 2 seconds, a sequence which requires multiple actions once the helicopter is located. Similarly, the instruction find helicopter is realised the agent moves within two metres of a helicopter and fixes its gaze (as determined by a ray cast from the agent's visual field) while remaining within the two metre proximity for five timesteps without lifting the object during this time.
To measure how general the acquired notion of lifting is, we trained the agent find each of set X = X1 U X2 of different objects (allowing the agent to learn to ground objects to their names) and to lift subset X1 of those objects, in trials in small room containing two objects positioned at random (one 'correct' according to the instruction, and one 'incorrect).. The agent receives a positive reward if finds or lifts the correct object, and the episode ends with no reward if the agent finds or lifts the incorrect object. We then evaluate its ability to extend its notion of lifting (zero- shot) to objects x E X2. In a variant of this experiment, we trained the agent to lift all of the objects in X when referring to them by their color (lift green object), and to find all of the objects in X according to either shape or color (find a pencil or find a blue object). We again tested whether the agent lifted objects x E X2 according variants are the same). As shown in Fig 2(a), in both variants the agent generalises with near- cear-peefect accuracy. The agent therefore learned notion of what it is to lift an object (and how this binds to the word lift) with sufficient generality that it can, without further training, apply it to novel objects, or to familiar objects with novel modes of linguistic reference.
their shape (so, the test trials in both
3.2 A GENERAL NOTION OF PUTTING
We took our study of predicate-object binding further by randomly placing two large, flat objects (a bed and tray) in the room and training agent place one of three smaller objects top. As before, the agent received a positive reward if placed the correct small object on the bed or the tray according to the instruction. If it placed incorrect object on the bed or tray, the episode ended with no reward. To test generalisation in this case, we trained the agent to lift each of a set X X1 U X2 of smaller objects (as in the previous experiment) and then to put some
http:///nnit3dd.ccmm
3

---

Published as a conference paper at ICLR 2020
14-
10
0..
0.9-
0.8
Variant
03
Veriant
0.-
a -shaps on bedftrey (training)
a shape, (triiingg)
0.6
Lift cubo, (triiing) Lift Shape (testing)
66-
Lift a aasaaapp__eee_ (tesing)
0.-
04-
aat__shapa, on bedflry (tesling)
0.4
02-
0.3
033
Trinigg atgp (140)
0.2
Figure 2: Test and training performance agent learns zero-shot predicate-argumenn binding for (a) 'lifting' in two variants: (I) training instructions are e.g. find a spaceship, find/lift pencil, testing instruc- tions are c.g. lift a spaceship. (2) training instructions are e.g. find spaceship/lift object, testing instructions are e.g. lift a spaceship. (b) "putting": training instructions are e.g. put a spaceship on the bed, lift a pencil, testing instructions are e.g. put pencil on the bed.
green
subset X1 C X of those objects on both the bed and the tray as instructed. then measured its performance (with no further learning) in trials where it was instructed to put objects from X2 onto either the bed or the tray. Surprisingly, we found that the agent was able to place objects with over 90% accuracy onto the bed or the tray zero-shot as instructed (Fig 2(b)). This generalisation requires the agent to bind its knowledge of objects (referred to by nouns, and acquired in the lifting trials) to complex control process (acquired in the training putting trials) involving locating the correct target receptacle, moving the object above it (avoiding obstacles like the bed-head) and dropping gently. An important caveat here is that control in this environment, while finer-grained than in many common synthetic environments, is far simpler than the real world; in particular the process required lift an object does not depend on the shape of that object (only its extent, Once the objects are held by the agent, however, their shape becomes somewhat more important and placing something on top of something else, avoiding possible obstacles that could knock the object from the agent's s grasp, is somewhat object-dependent process.
degree).
4 UNDERSTANDING THE DRIVERS OF GENERALISATION
4.1 NUMBER OF TRAINING INSTRUCTIONS
To emphasize most explicitly the role of the diversity of training experience in the emergence of systematic generalisation, showed that a two-layer feedforward network with sufficient hidden units can learn to predict the negation of binary patterns. Here, adapt this experiment to embodied agent with continuous visual input and, importantly, focus not only learning, but also generalisation. We choose to consider negation since is an example of an operator for which we found that, for our standard environment configuration, our agent unequivocally fails to exhibit an ability to generalize in systematic way.
consider the abstract notion of negation. Rumelhart et al. (1986)
To this, we generated trials in the Unity room with two different objects positioned random and instructions of the form find a box (in which case the agent was rewarded for locating and fixing its gaze on the box) and find [something that is] not a box (in which case there was a box in the room but the agent was rewarded for locating the other object). Like Rumelhart et al. (1986), we found that was unproblematic to train an agent to respond correctly to both positive and negative training inputs. To explore generalisation, we then trained the agent to follow instructions find x for all x E X and negated instructions find a not æ for only x E X1 (where X = X1 U X2. and X, X2 = 0), and tested how it interpreted negative commands for X²-
When X2 contained only a few objects ([X,l= = 6). the agent interpreted negated instructions (in- volving objects from X2) with below chance accuracy. In other words, for objects 22 E X2, in response to the instruction find a not Iz. the agent was more likely to find the object 22 than the correct referent of the instruction. This is an entirely un-systematic interpretation of the negation
4

---

Published as a conference paper at ICLR 2020
predicate." Interestingly, however, for ([Xg] 40) the agent achieved above chance interpretation of held-out negated instructions, and for [[X1 100) performance on held-out negated instructions increased to 0.78 (Table 1).
These results show that, for a test set of fixed size, the degree of systematicity exhibited by agent when generalizing can grow with the va- riety of words/objects experienced in the train- ing set, even for highly non-compositional op- erators such as negation. Of course, the mere fact that larger training sets yield better gener- alisation in neural networks not novel or un- expected. On the other hand, we find the emer- gence of logical operator like negation in the agent in reasonably systematic way (noting that adult humans are far from perfectly sys- tematic (Lake al., 2019))), given experience of 100 objects (again, not orders of magnitude dif- ferent from typical human experience), to be notable, particularly given the history of research into learning logical operators in connectionist models and the importance of negation in language processing (Steedman, 1999; 1). In what follows, we complement these observations by estab- lishing that not only the amount, but also the type of training data agent experience) can have significant impact on emergent generalisation.
Training set
Train accuracy
Test accuracy
6 words
1.00
0.40
40 words
0.97
0.60
100 words
0.91
0.78
Table
Accuracy extending a negation predicate to novel arguments (test accuracy) when agents are trained negate different numbers of words/objects.
4.2 3D VERSUS 2D ENVIRONMENT
(b)
Figure 3: Screenshots from the agent's perspective of an episode in the grid-world and Unity environments. In both the instruction put the picture frame the bed.
Our first observation is that the specifics of the environment (irrespective of the logic of the task) can play an important role in emergent generalisation. To show this most explicitly we consider a simple color-shape generalisation task. Prior studies in both 2D and 3D environments have shown that
agents can correctly interpret instructions referring to both the color and shape of objects (find a red ball) zero-shot when they have never encountered that particular com- bination during training (Chaplot et al., 2018: Hill et al., 2017; Yu et al., 2018). We replicate the demonstration of Hill al. (2017) the 3D DeepMind-Lat environment (Beattie et al., 2016), and for comparison implement an analogous task in a 2D grid-world (compare Fig 3 (a) and Fig 4).
As in the original experiments, we split the available colors and shapes in the environment into sets S = s US and C' = c U é. We then train the agent on episodes with instructions sampled from one of the sets c x s, x
X š, and, once trained, we evaluate its performance on instructions from è x š. All episodes involve the agent in small room faced with two objects, one of which matches the description in the instruction. Importantly, both the color and the shape word the instruction needed to resolve the task, and during both training and testing the agent faces trials in which
We suspect in such cases that the agent is simply learning non-compositional interpretation during train- ing along the lines of not referring the set {y E X1 : x v)-
5

---

Published as a conference paper at ICLR 2020
the confounding object either matches the color of the target object or the shape of the target object. While it was not possible to have exactly the same shapes in set S in both the grid world and the 3D environment, the size of C' and S and all other aspects of the environment engine were the same in both conditions. As shown in Table 2 (top), we found that training performance was marginally worse on the 3D environment, but that test performance in 3D (M 0.97, SD = 0.04) was six percentage points higher than in 2D (M = 0.91, SD = 0.08) (a suggestive but not significant difference given the small sample of agents; ((8) 1.38, p 0.20).
To probe this effect further, we devised an analogue of the "putting" task (Section 3.2) in the 2D grid-world. In both the 3D and 2D environments, the agent was trained on 'lifting' trials, in which had to visit specific object, and on 'putting' trials, in which it had to pick up a specific object and move to the bed. To match the constraints of the grid-world, we reduced the total global object set in the Unity room to ten, allocating three to both lifting and putting trials during training, and the remaining seven only to lifting trials. The evaluation trials then involved putting the remaining seven objects on the bed (see Figure 3 for an illustration of the two environments). While the grid- world and the 3D room tasks are identical at a conceptual (and linguistic) level, the experience of an agent is quite different in the two environments. In the grid world, the agent observes the entire environment (including itself) from above at every timestep, and can move to the 81 = 9 x 9 possible locations by choosing to move up, down, left or right. To lift an object in the grid-world, the agent simply has to move to the square occupied by that object, while 'putting' requires the agent to lift the object and then walk to the square occupied by the white (striped) bed.
As shown in Table 2 (bottom), agents in both conditions achieved near perfect performance in train- ing. However, on test trials, per- formance of the agent in the Unity room (M = 0.63, SD = 0.06) was significantly better than the agent in the grid world (M- = 0.40, SD = 3,48,p < 0.005.
Train accuracy
Test accuracy
Color-shape task Grid world 3D room (DM-Lab)
0.99
0.91 +0.09
0.98
0.97 10.04
Putting task
Grid world Grid world, scrolling
0.99
0.40 10.14
0.14); t(8)
0.93
0.60 10.14
In failure cases, the agent in the grid world can be observed exhibit- ing less certainty, putting the wrong object on the bed or running out of time without acting decisively with the objects.
3D room (Unity)
0.99
0.63 10.06
Table 2:
Tests of systematic generalisation in 2D and 3D environ- ments; five randomly-inittalized agent replicas in each condition.
4.3 VISUAL INVARIANCES IN AGENTS' PERSPECTIVES
To understand more precisely why agents trained in 3D worlds generalise better, we ran a further condition in which we gave the agent in the grid-world an ego-centric perspective, as it has in the 3D world. Specifically, we adapted the grid-world so that the agent's ' field of view was 5 x 5 and centred on the agent (rather than 9x9 and fixed). While this egocentric constraint made it harder for the agent to learn the training task, performance on the test set improved significantly t(8) = 2.35, p < 0.05, accounting for most of the difference in generalisation performance between the 3D world and the (original) 2D world. This suggests that the visual invariances introduced when bounding the agent's s perspective, which in our case was implemented using an ego-centric perspective, may improve an agent's ability to factorise experience and behaviour into chunks that can be re-used effectively in novel situations. The relevant difference is likely that the agent is always in an invariant location in the visual input, which reduces some difficulty of perception, rather than the fact that the agent centred per se. See Appendix D for control experiments showing that partial-observabiiiiyy alone does not induce the same boost to generalisation, nor does difference in the information available from a single frame.
4.4 TEMPORAL ASPECT OF PERCEPTION
Egocentric vs. allocentric vision is not the only difference between a grid-world and a 3D world. Another difference is that, in the 3D world, the agent experiences much richer variety of (highly

---

Published as a conference paper at ICLR 2020
(b)
Instr: magenta ball
Inst: magenta guitar
Inetr. brown horse
Instr: red horse
Instr: red ball
Figure 4: (a) Four (independent) training inputs from training data for classifier model. (b) Four timesteps of input from a single episode for the situated agent.
correlated) visual stimuli in any particular episode. To isolate the effect of this factor on general- isation, we return to the color-shape task, which is appropriate for the present experiment because single train/test experiment (defined in terms of the instructions and objects that the agent ex- periences) can be constructed either as an interactive MDP for a situated agent or as a supervised classification problem.
In the vision+language classifier con- dition, supervised model must predict either left right in response to still image (the first frame of an episode) two objects and language instruction of the form find a red ball. In the agent condition, situated RI agent begins an episode facing two objects and is trained move towards and bump into the object specified by the instruction. Importantly, the architecture for the vision+language classifer and the agent were identical except that the final (action and salue-prediction) layer in the agent replaced by single linear layer and softmax over two possible outcomes in the classifier.
Regime Classifier Agent
Train accuracy
Test accuracy
0.95
0.80 #0.05
1.00
1.00 +0.00
Table 3: generalisation accuracy achieved by vision-and- language classifier trained on single screenshots versus a uated agent trained in the DMLaa environment.
On the same set of training instructions, we trained the classifier to maximise the likelihood of its object predictions, and the agent to maximise the return from selecting the correct object. As shown in Table the accuracy of the classifier on the training instructions converged 0.95 compared to 1.0 for the agent (which may be explained by greater difficulty in detecting differences between objects given more distant viewpoint). More importantly, performance episodes (M 0.80, SD 0.05) was significantly worse than that of the agent (M 1.00, SD = 0.00); t(8) = 8.61, < 0.0001. We conclude that an agent that can move its location and its gaze (receiving richer variety views for given set of object stimuli) learns not only to recognize those objects better, but also to generalise understanding of their shape and color with greater systematicity than agent that only observe single snapshot for each instruction." This richer experience can be thought of as effectively an implicit form of data augmentation.
the classifier on test
4.5 THE ROLE OF LANGUAGE
Thus far, we have considered tasks that involve both (synthetic) language stimuli and visual ob- servations, which makes it possible pose diverse and interesting challenges requiring systematic behaviour. However, this approach raises the question of whether the highly regular language expe- rienced by our agents during training contributes the consistent systematic generalisation that observe at test time. Language can provide a form of supervision for how to break down the world
See Yurovsky al. (2013) for discussion of the importance of this factor in children's 's visual and word- learning development.

---

Published as a conference paper at ICLR 2020
and/or learned behaviours into meaningful sub-parts, which in turn might stimulate systematicity and generalisation. Indeed, prior work has found that language can serve to promote compositional behaviour in deep RL agents (Andreas et al., 2017).
To explore this hypothesis, we devised a simple task in the grid world that can be solved either with without relying on language. The agent begins each episode in random position in the grid objects, four of one color and shape and four of another. In each episode, one of the object types was randomly designated
containing eight sandomly-positionas
'correct' and the agent received positive +I reward for collecting objects of this type. The episode ended if the agent collected all of the correct objects (return +4) or if it collected single incorrect object (reward 0). Without access to language, the optimal policy is to simply select an object type at random and then (if pos- sible) the remainder that type (which returns on average). the language condition, however, the target object type (e.g. 'red square") was named explicitly, so the agent could learn to achieve return of all episodes. To test generalisation both conditions, as in the color-shape reference expressions described above, we trained the agent episodes involving half of the possible color- shape combinations (as correct or distractor objects) and tested it on episodes involving the other half. Note that, both during training and testing, in all episodes the incorrect object differed from the correct object in terms of either color or shape, but not both (so that awareness of both color and shape was necessary to succeed in general).
For fair comparison between conditions, when measuring agent performance we consider perfor- mance only episodes in which the first object selected by the agent correct. As shown in Figure 5, the non-linguistic agent performed worse on the training set (when excluding the 50% of trails in which it failed on the first object). Performance on the episodes was also worse overall for the non-linguistic agent, but by a similar amount to the discrepancy on the training set (thus the test error was similar in both conditions). With increasing training, we observed this discrepancy to diminish to a negligible amount (Figure 5, right). Importantly, both linguistic and non- non-linguistic agents exhibited test generalisation that was substantially above chance. While not conclusive, this analysis raises the possibility that language may not be playing a significant role (and is certainly not the unique cause) of the systematic generalisation that we have observed emerging in other experiments.
1.0
TRAIN Red circle Green cross Blue square Orange triange
0.9
Normalised retumn
0.8
47
language trials)
No languege (train trials) With language (test trials)
No language (test trials)
90
TEST Orange circle Green square Blue triangle
0.5
Experted nendom
300
0.4
Traning steps (Ic9)
130
0.3
0.0
0.2
0.4
0.6
Figure 5: Left An episode of the grid-workd task that can be posed with or without language. Right Nor- malised episode returns on training and evaluation trials as the agent learns, adjusted to ignore episodes where the first object that the agent collects is incorrect.
DISCUSSION
We have shown that neural-network bebserd agent with standard architectural components can learn to execute goal-directed motor behaviours in response to novel instructions zero-shot. This suggests that, during training, the agent learns not only how to follow training instructions, but also general information about how word-like symbols compose and how the combination of those words af- fects what the agent should do in its world. With respect to generalisation, our findings contrast with other recent contributions suggesting that neural networks do not generalise well in these set- tings (Lake, 2019; Bahdanau et al., 2018), and are more aligned with older empirical analyses of neural networks (McClelland et al., 1987; Frank, 2006). Our work builds those earlier studies
8

---

Published as a conference paper at ICLR 2020
by considering not only the patterns functions that neural networks can lear, but also how they compose familiar patterss to interpret entirely novel stimuli. Relative to more recent experiments on color-shape generalisation (Higgins et al., 2017; Chaplot et al., 2018; Yu et al., 2018), we study a wider range of phenomena, involving abstract modifying predicates (negation) and verbs ('to lift', 'to put') corresponding to complex behaviours requiring approximately 50 movement or manipu- lation actions and awareness of objects and their relations. By careful experimentation, we further establish that visual invariances afforded by a bounded frame of reference, in our case implemented using a first-person perspective of an agent acting over time, plays an important role in the emer- gence of this generalisation.
An obvious limitation of our work, which technology precludes us from resolving at present, is that we demonstrate the impact of realism on generalisation by comparing synthetic environments of differing degrees of realism, while even our most realistic environment is considerably more simple and regular than the world itself. This is in keeping with a long history of using simulation for the scientific analysis of learning machines (Minsky & Papert, 1969; Marcus, 1998). Nonetheless, the possibilities for learning in situated robots develop, future work should explore whether the general- isation that we observe here is even closer to perfectly systematic under even more realistic learning conditions. An additional limitation of our work is that much like closely-related work (Lake & Baroni, 2017; Bahdanau al., 2018) - we do not provide nomological explanation for the gen- eralisation (or lack thereof) that we observe. Depending on the desired level of analysis, such an explanation may ultimately come from thcoretical work on neural network generalisation (Arora et al., 2018; Lampinen& & Ganguli, 2018; Allen-Zhu et al., 2018; Arora et al., 2019). Our focus here, however, is on providing careful experiments that isolate the significance of environmental factors on systematicity and to provide measures of the robustness (i.e. variance) of these effects.
We also emphasize that our agent in no way exhibits complete systematicity. The forms of gen- eralisation it exhibits do not encompass the full range of systematicity of thought/behaviour that one might expect of mature adult human. However, this is unsurprising given that our network has substantially less experience than an adult human, and learns in a substantially simpler environment. is also worth noting that none of our experiments reflect the human ability to learn (rather than extend existing knowledge) quickly (as in, e.g. Lake et al. (2015); Botvinick et al. (2019)). Finally, depending on one's perspective, it is always possible to characterise the generalisation we describe here as interpolation. Rather than engage with this thomy philosophical distinction (which touches upon, e.g., the problem of induction (Vickers, 2009)), we simply emphasize that in our experiments there is a categorical difference between the data which the agent was trained and test stimuli to which they respond. Finally, our experiments concur with a message expressed most clearly by Anderson's 's famous contribution in Science (Anderson, 1972). Conclusions that are reached when experimenting with pared-down or idealised stimuli may be different from those reached when con- sidering more complex or naturalistic data, since the simplicity of the stimuli can stifle potentially important emergent phenomena. We suggest that strong generalisation may emerge when an agent is situated in a rich environment, rather than a simple, abstract setting.
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparameter- ized neural networks, going beyond two layers. arXiv preprint 1.00//9//1 2018.
Michael L Anderson. Embodied cognition: A field guide. Artificial intelligence, 149(1):91-13, 2003.
Philip W Anderson. More is different. Science, 17774047):393-336, 1972.
Jacob Andreas, Mareus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 39-48, 2016.
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 166-175. JMLR. org, 2017.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. arXiv preprint arXiv:/802.00 180200529. 2018

---

Published as a conference paper at ICLR 2020
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine- line-grained analysis of optimization and generalization for overparameterized two-layer neural networks. arXiv preprint arXiv:1901.08584, 2019.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What is required and can it be learned? arXiv preprint arXiv:1811.1288 2018.
Peter Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Gonzale,, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arxiv Ak0iv:800.00 206.01261, 2018.
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Mareus Wainwright, Heinrich Küttler, Andrew Lefrancq. Simon Green, Víctor Valdés, Amir Sadik, Julian Schrittwieser, Keith Ander- son, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab. arXiv preprini arXiv: /w/////22800000 2016.
Mathew Botvinick, Sam Ritter, Jane X Wang, Zeb Kurth-Nelson, Charles Blundell, and Demis Hassabis. Reinforcement learning, fast and slow. Trends cognitive sciences, 2019.
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Ra- jagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018
Elizabeth M Clerkin, Elizabeth Hart, James M Rehg, Chen Yu, and Linda B Smith. Real-world visual statistics and infants' first-leamed object names. Philosophical Transactions of the Royal Society B: Biological Sciences, (72(1711):2016 60055, 2017.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint 1rXiv:1802.001 2018.
Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: A critical analy-
sis. Cognition, 28(1-2):3-71, 1988.
Stefan L Frank. Learn more by training less: systematicity sentence processing by recurrent networks. Connection Science, 18(3):287-302. 2006.
Irina Higgins, Nicolas Sonnerat, Loie Matthey, Arka Pal, Christopher P Burgess, Matko Bosnjak, Murray Shanahan, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. Scan: Learning hierarchical compositional visual concepts. arXiv preprint arXiv: 1707.033870 2017.
Felix Hill, Karl Moritz Hermann, Phil Blunsom, and Stephen Clark. Understanding grounded lan- guage learning agents. arXiv preprint arXiv:1710.09860 2017.
Karin H James, Susan S Jones, Linda B Smith, and Shelley N Swain. Young children's self- generated object views and object recognition. Journal of Cognition and Development, 15(3): 393-401, 2014.
Philip Kellman and Martha E Arterberry. The cradle of knowledge: Development of perception in infancy. MIT press, 2000.
Brenden M Lake. Compositional generalization through meta sequence-to-seqqueece learning. arXiv preprint arXiv: 05381, 2019.
Brenden M Lake and Marco Baroni. Generalization without systematicity: On the compositional recurrent networks. arXiv preprint arXiv:1711.00350 2017.
skills of sequence-to-sequees
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266): 133--1133, 2015,
Brenden M Lake, Tal Linzen, and Marco Baroni. Human few-shot learning of compositional in- structions. arXiv preprint arXiv:1901.000 1901.0457. 2019.
10

---

Published as a conference paper at ICLR 2020
Andrew K Lampinen and Surya Ganguli. An analytic theory of generalization dynamies and transfer learning in deep linear networks. arXiv preprint 3x.x::8809.107777 2018.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv:1904. 2019.
Gary F Marcus. Rethinking eliminative connectionism. Cognitive psychology. 37(3):243-3333) 1998.
James L McClelland, David E Rumelhart, PDP Research Group, et al. Parallel distributed process- ing, volume 2. MIT press Cambridge, MA:, 1987.
James L. McClelland, Matthew M Botvinick, David C Noelle, David C Plaut, Timothy Rogers, Mark Seidenberg, and Linda Smith. Letting structure emerge: connectionist and dynamical systems approaches to cognition. Trends in cogmitive sciences, 14(8):348- -3566,,, 2010.
Marvin Minsky and Seymour Papert. Perceptron: introduction to computational geometry. The MIT Prexs, Cambridge, expunded edition, 19(88):2, 1969.
Randall O' O'Reilly. Generalization in interactive networks: The benefits of inhibitory competition and hebbian learning. Neural computation, 13(6):1199 1900:1199--21111 1241,, 2001.
Randall C O'Reilly, Alex Petrov, Jonathan D Cohen, Christian J Lebiere, Seth A Herd, Trent
Kriete, In Paco Calvo, and John Symons. How limited systematicity emerges: A computational cognitive neuroscience approach. The architecture of cognition: Rethinking fodor and Pylyshyn' systematicity challenge, 2013.
David C Plaut. Systematicity and specialization in semanties. In Connectionist Models in Cognitive Neuroscience, pp. 184-193, Springer, 1999.
D. F. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error propagation. In L McClelland, D E Rumelhart, PDP Research Group, al. (eds.), Parallel distributed processing, volume 2, chapter 8, pp. 318-362. The MIT Press, 1986.
Paul Smolensky. On the proper treatment of connectionism. Behavioral and Brain Sciences, 11: 1-23, 1988.
Mark Steedman. Connectionist sentence processing in perspective. Cognitive Science, 23(4):615- 634, 1999.
Mark Steedman. Taking Scope The Natural Semantics of Quantifiers. The MIT Press, 2011.
John Vickers. The problem of induction. Stanford encyclopedia of philosophy, 2009.
Haonan Yu, Haichao Zhang, and Wei Xu. Interactive grounded language acquisition and general- ization in 2d world. arXiv preprint arXiv:1802.011433, 2018.
Daniel Yurovsky, Linda B Smith, and Chen Yu. Statistical word learning scale: The baby's view is better. Developmental science, 16(6)9959-96,, 2013.
A LIMITATIONS
Like a much recent Al research, we experiment with an agent learning and behaving in a simu- lated world, with the obvious limitation that the data that the agent experiences are not 'real'. Our paradigm does not directly reflect challenges in robotics, natural language processing, or indeed human learning and cognition. On the other hand, we note the long history of extrapolating from re- sults on models in simulation to draw conclusions about AI general (Minsky Papert, 1969). The simulated experience of our agents in this study is in many ways more realistic than the idealised, symbolic data employed in those influential studies. Moreover, our results suggest that systematic generalization emerges most readily in the consider, which is at least suggestive that the class of model that we consider here might behave in similar ways when moving towards learning in the real world.
realistic of the (simulated) environments that
11

---

Published as a conference paper at ICLR 2020
Another clear limitation of our work is that much like recent work that emphasises the failures of neural networks - we do not provide a nomological explanation for the generalization (or lack thereof) that we observe. That is, do not provide give precise answer to the question of why? systematic generalisation necessarily emerges. Depending on the desired level of explanation, such question may ultimately be resolved by promising theoretical analyses of neural (Arora et al., 2018; Lampinen & Ganguli. 2018; Allen-Zhu et al., 2018: Arora et al., 2019), and some more general intuitions can be gleaned from developmental psychology (Clerkin et al., 2017; Kell- man & Arterberry, 2000; James et al.. 2014; Yurovsky et al., 2013). Our focus here, however, is on providing careful experiments that isolate the significance of environmental factors on systematicity and to provide measures of the robustness (i.e. variance) of these effects.
generalization
B EXPERIMENT DETAILS
B.1 ENVIRONMENT DETAILS
At each timestep the agent can take one 26 actions that allow it to move its body or head (direction of gaze), to grab objects that in range, and An object is held while the agent executes actions with the GRAB prefix, and is dropped once the agent emits action without the GRAB prefix. A visual aid (a white box) appears in the agent's view around any object that is within range of being grabbed, and when an object is held and lifted, vertical line projected below the object assist in placing (the limits of depth perception with monocular vision make placing objects very challenging otherwise).
rotate held objects in various directions.
Body movement actions
Movement and grip actions
Object manipulation
NOOP
GRAB
GRAB SPIN OBJECT RIGHT
NOVE_FORWARD
GRAB + MOVE_FORWARE
GRAB + SPIN OBJECT_LEFT
HOVE.BACKWARD
GRAB BOYKKAR
GRAB SPIN. S-I-S-IIN.BIIEI .OBJECT.UP
MOVE RIGHT
GRAB + MOVE RIGHT
GRAB SPIN OBJECT DOWN
MOVE LEFT
GRAB MOVE LEFT
GRAB SPIN OBJECT FORWARD
LOOK RIGHT
GRAB LOOK_RIGHT
GRAB
LOOK LEFT
GRAB LLOK
GRAB PUSH.OBBECT.WWAY
LOOK_UP
GRAB LOOK_UP
GRAB PULL PULL.OBIICTECLOSE
LOWK
GRAB -LOOO
Table 4: Action space of the agent in the 3D room.
B.2 LIFTING, PUTTING AND NEGATION EXPERIMENTS
For lifting, putting and negation experiments, we divided the shapes into two sets, X, and X2, to test generalisation unseen combinations. These sets vary according to the experiment see Tables and 6.
In the negation experiment, we tested three sets of different sizes for X,, and keeping X2 always the same size. Table 7 shows the set division in each experiment.
B.3 2D 3D EXPERIMENTS
As explained in Sections 4.2 and 4.4, we considered a color-shape generalization task in DeepMind Lab (Beattie et al., 2016). Table details the separation of objects in the sets S U and C' = CUG. C.
B.4 GRID-WORLD FINDING & PUTTING COMPARISON, AND LANGUAGE LESION
The grid-world environment consisted of 9 x 9 room surrounded by a wall of width 1 square, for a total size of 11 x 11. These squares were rendered at 9 9 pixel resolution, for total visual input size 99 x 99. The agent had 4 actions available, corresponding to moving one square in each the 4 cardinal directions. The agent was given time limit of 40 steps for each episode, which sufficed
12

---

Published as a conference paper at ICLR 2020
Sets
11
basketball, book, bottle, candle, comb, cube block, cuboid block, cushion fork, football, glass, grinder, hairdryer, headphones, knife, mug, napkin, pen, pencil, picture frame, pillar block, plate, potted plant, roof block, rubber duck, scissors, soap, soap dispenser, sponge, spoon, table lamp
X2 boat, bus, car, carriage, helicopter, keyboard, plane, robot, rocket, train, racket, vase
Color
aquamarine, blue, green, magenta, orange, purple, pink, red, white, yellow
Table 5: Object/Color sets in the Lifting experiment.
Sets
X1
soap, picture frame, comb, mug, headphones, candle, cushion, potted plant, glass, tape dispenser, basketball, mirror, book, bottle, candle, grinder, rubber duck, soap dispenser, glass, pencil
X₂
toy robot, teddy, hairdryer, plate, sponge, table lamp, toilet roll, vase, napkin, keyboard aquamarine, blue, green, magenta, orange, purple, pink, red, white, yellow
Color
Table Object/Color in the Putting experiment.
to solve any task we presented. For the objects we used simple, easily-distinguisabbee shapes, such circles, squares, triangles, plus signs and shapes etc.
Finding & lifting: For the comparisons to 3D (section 4.2). for the finding comparison, and the lifting trials in the putting task, the agent was rewarded if it moved onto the correct object (i.e. the one that matched the instruction). and the episode was terminated without reward it moved onto the wrong object. For the putting trials, when the agent moved onto the object, it would pick the object up and place it on its head, that the object was seen in place of the agent, analogously to how the background is obscured by an object carried the 3D world. the agent moved onto an object while carrying another, the object it was currently carrying would be dropped the previous square, and replaced with the new object. the agent moved onto the bed (striped lines in the screenshot) with object, the episode terminated, and the agent was rewarded if the object was correct. See appendix D for some variations on this, including making the agent visible behind the object it carrying, to ensure that this lack of information was not the factor underlying the results.
Language lesion: For the language vs. no-language comparison (section 4.5), if the agent moved onto a correct object, it received reward of This reward was both used as a training signal, and given as input on the next time-step. to help the agent figure out which objects to pick up. In addition, if the agent picked up an incorrect object for the current episode, the episode would terminate immediately, without any additional reward.
C AGENT DETAILS
C.1 ARCHITECTURE PARAMETERS
The agent visual processor is residual convolutional network with 64, 64, 32 channels in the first, second and third layers respectively and residual blocks in each layer.
13

---

Published as a conference paper at ICLR 2020
Sets
X1 (small)
basketball, book, bottle, candle, comb, cuboid block
X1 (medium)
basketball, book, bottle, candle, comb, cuboid block, fork, football, glass, hairdryer, headphones, knife, mug, napkin, pen, pencil, picture frame, pillar block, plate, potted plant, roof block, scissors, soap, soap dispenser, sponge, spoon, table lamp, tape dispenser, toothbrush, toothpaste, boat, bus, car, carriage, helicopter, keyboard, plane, robot, rocket, train, vase
X1 (large)
chest of drawers, table, chair, sofa, television receiver, lamp, desk box, vase, bed, floor lamp, cabinet, book, pencil, laptop, monitor, ashcan, coffee table, bench, picture, painting, armoire, rug, shelf, plant, switch, sconce, stool, bottle, loudspeaker, electric refrigerator, stand, toilet, buffet, dining table, hanging, holder, rack, poster, wall clock, cellular telephone, person, cup, desktop computer, mirror, stapler, toilet tissue, table lamp, entertainment center, flag, printer, armchair, cupboard, bag, bookshelf, pen, soda can, sword, curtain, fireplace, memory device, battery, bookcasc, pizza, game, hammer, chaise longue, food, dishwasher, mattress, pencil sharpener, candle, glass, ipod, microwave, room, screen, lamppost, model, oven, bible, camera, ed player, file, globe, target, videodisk, wardrobe, calculator, chandelier, chessboard, mug, pillow, basket, booth, bowl, clock, coat hanger, cereal box, wine bottle,
X2
tennisball, toilet roll, teddy, rubber duck, cube block, cushion, grinder, racket
Color
aquamarine, blue, green, magenta, orange, purple, pink, red, white, yellow
Table 7: Object/Color sets in the Negation experiment.
Attribute
Sets
Shape (S) s
$
tv, ball, balloon, cake, can, cassette, chair, guitar
hat, ice lolly
Color (C)
C
è
black, magenta, blue, cyan, yellow, gray, pink, orange red, green
Table 8: Object sets the 2D x 3D experiment. The same sets of objects were used for DeepMind Lab and our 2D environment.
In the learning algorithm, actors carry replicas of the latest learner network, interact the with environment independently and send trajectories (observations and agent actions) back to a central leamne. The learning algorithm modifies the learner weights to optimize actor-critic objective, with updates importance- weighted to correct for differences between the actor policy and the current state of the learner policy.
14

---

Published as a conference paper at ICLR 2020
D SOME CONTROL EXPERIMENTS FOR THE 2D VS. 3D PUTTING TASK
We outline some control experiments we ran to more carefully examine what was causing the differ- ence in generalization performance on the putting task between the 3D environment and the different conditions in the 2D gridworld.
D.1 GRID-WORLD EGOCENTRIC SCROLLING EFFECT VS. PARTIAL OBSERVABILITY
In the main text, we noted that introducing egocentric scrolling to the wooldworld putting task resulted in better generalization performance (although still worse than generalization performance in the 3D world). We attributed this effect to the benefits of an egocentric frame of reference. However, there is another change that is introduced if egocentric frame of reference is introduced by the size of the visual input is kept the same - the world becomes partially observable. If the agent is in one corner of the room, and the bed is in the opposite corner, the bed will be outside the agent's view in the egocentric framework. This increases the challenge of learning this task, and indeed the agent trained with egocentric reference took longer to achieve good training performance, despite generalizing better. Thus, an alternate interpretation of the increase in generalization performance would be that this partial observability is actually contributing to robustness by forcing the agent to more robustly remember things it has seen.
In order to examine this, we ran two additional experiments, one with an egocentric scrolling window that was twice large the room (in order to keep everything in view at all times), and another with fixed frame of reference of this same larger size. The latter was a control because, for memory reasons, we had to decrease the resolution of the visual input slightly with this larger view window (7 x 7 pixels per grid square instead of x 9). Test episode performance in the case of the fully- observable egocentric window (M 0.64, SD 0.04) was not significantly different from that in the ego-centric and partially-observable case (M = 0.60, SD = 0.14), t(8) = 0.64, p > 0.05. We therefore conclude that it is invariant frame of reference afforded by the egocentrie perspective, rather than partial observability per se, that leads to better generalization in this case.
D.2 SCROLLING AND AGENT VISIBILITY WHEN HOLDDIG OBJECTS IN THE GRID WORLD
In the experiments reported in the main text, when an agent picks up an object in the grid world, it places the object on its head, and so the object obscures the agent's location. This was meant to mimic the way that an object held front of the agent in the 3D world blocks a substantial portion of its field of view. However, this could also make the task more difficult for the agent, because it is required to use its memory (i.c. its LSTM) to recall what its location is when is carrying an object- it is no longer visible directly from single observation, since the agent could be under any of the objects on screen. By contrast, in the egocentric scrolling perspective, the agent is always in the center of the screen, so this ambiguity is removed. This could provide yet another explanation for the difference between the fixed view and scrolling view.
To examine this, ran an additional experiment, where we made the agent visible in the back- ground behind the object it was carrying. That is, since the agent is a white square, effectively an object it is carrying now appears to be on a white background. However, test episode performance in this case (M- = 0.38, SD = 0.07) was. if anything, worse than when the agent is completely hidden behind the object (M 0.42, SD 0.14). This somewhat surprising at first glance, since in the 3D world the agent seems to be able to generalize to e.g. finding the bed when objects it has never carried to the bed are obscuring part of view. However, it reflects the broad strokes of our argument- in the grid-world, seeing examples of carrying only three objects on top of the agent may not suffice to induce a general understanding of how to parse an agent the background from the object top of it. By contrast, in the 3D environment the agent will see many perspectives on each object it carries, and will see the background obscured in many different ways from many different perspectives over the course of its training episodes. We suggest that this leads more robust representations.
15


Reviews:
[{"review_id": "rJxuMyU1ir", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work studies factors which promote combinatorial generalization in a \"neural network agent\" embodied in a 3d simulation environment. The authors present interesting experiments and some insightful empirical findings on how a richer environment and a first-person egocentric perspective can aid a simple neural net to generalize better over previously unseen tasks. While I truly commend the effort undertaken to perform the experiments, I have several concerns which I explain below and would be happy to raise my score if they can all be addressed satisfactorily:\n\n1) While the authors interpret the experiment results in sec 4.1 in a positive way, the results don't seem to necessarily indicate good systematic generalization. For instance, after learning with 40 words the agent only achieves 60% test accuracy. While the accuracy increases to 78% on training with 100 words, the training and test accuracy gap indicates that the performance is still far from any kind of systematic generalization. The results instead seems to be hinting that neural nets don't indeed perform combinatorial generalization on their own, but can be forced towards it by supplying them huge amounts of diverse data (which is not true for humans). Also, the fact that increasing the number of words helps in generalizing better is true for most ML models and does not come as a surprise. So the results in this subsection are somewhat trivial and do not necessarily contribute any new understanding.\n\n2) For the experiments regarding egocentric frame in sec 4.3, I feel that the results are not really conclusive (even including the control exps in appendix D). Could it be that if one uses any frame rigidly attached (i.e. fixed displacement and rotational coordinates) to the agent's egocentric frame, one would achieve the same generalization performance? It is also possible that as suggested by authors in sec 4.4, it is just the motion of the egocentric frame which might be giving diverse views of the environment to the agent. So the frame might not even need to be egocentric, but just a moving frame which gives richer and diverse views whenever the agent moves. Please include experiments to test for these possibilities.\n\n3) In section 4.4, the authors have trained the non-embodied classifier with just a single image frame. But this does not necessarily justify the conclusion that active perception helps in generalization. This is because the motion of the RL agent gives it both a varied set of views AND also control over what views to obtain by taking actions. In order to better understand which of these factors (or perhaps both) aid in generalization, another set of experiments is required which shows the classifier agent more images while keeping the desired object in view. In one experiment, these images should be chosen with random movements but the number of such images provided to the classifier should be increased in sub-experiments to gauge if giving more varied views bridges the performance gap between the classifier and the RL agent's generalization performance. In a second experiment, one might want to first train the RL agent, then extract a few (say 10) frames out of its enacted policy for all pairs of objects and use these frames as a part of the training set for the classifier agent. This would allow one to gauge if both varied views and actively selecting to interact with the environment can help bridge the generalization gap.\n\n4) Lastly, sec 4.5 seems to be hinting at a potentially very incorrect conclusion: \"language is not a large factor (and certainly not a necessary cause) of the systematic generalisation...\". This cannot be said from the small single experiment presented in sec 4.5. For instance, that experiment has been devised in a way that an optimal policy can be found with/without language. However, if a language input is provided to explicitly state the desired object, that might speed up the training of the RL agents significantly. In such a case, it might be helpful to see if learning the policy with the language input is being accomplished with a much lower number of frames during training, as opposed to when no language input is provided. Please provide the training error plots. But regardless of the plots, the experiments can still be quite inconclusive since language helps in systematic generalization in a variety of other ways apart from what has been tested for. In general, language starts helping humans once it has been acquired to a sufficient extent since one needs noun-concept linkages, verb-action linkages etc. to have been acquired a priori before the benefits of language emerge in combinatorial generalization. Training an LSTM to understand the language commands in tandem with learning policies for picking desired objects could lead to sub-optimal or heavily over-fitted language models which may not help in generalization. Testing for the true role of language will require many more experiments, which may be somewhat out of scope for this paper given the space constraints for a single paper. But, I would advise the authors to refrain from drawing hasty inferences about the role of language without thorough experimentation.\n\nMinor issues:\n\n1) What are the 26 actions in the Unity 3D environment in section 3? It is important to know the action space to understand how easy or hard it is for the agent to learn generalizable policies.\n2) The x-axis of Figure 2 is not readable at all. Please rectify those graphs and reduce the number of ticks.\n\n\n-------------------------- Update after interaction during author feedback period -------------------------------\nI appreciate the efforts that the authors have undertaken to address my concerns. While the paper is far from perfect, it is still a very thought provoking work and I believe that it would make a valuable contribution to the line of works on systematic generalization in embodied agents. I am updating my score to reflect the same.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HyeAa6v9Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies systematic generalization in a situated agent. The authors examine the degree to which various factors influence systematic generalization, including 2D vs. 3D environments, egocentric vision,  active perception, and language. The experiments reveal that the first three factors, but not language, promote systematic generalization. The experiments are well-done and worthwhile, and identifying the key factors that affect generalization is a strength of the paper.\n\nI have two main criticisms. First, the model's abilities for systematic generalization are overstated. Second, critical details about the experiments are omitted that make them difficult to evaluate.\n\nLet's start with the abilities of the model. The title of the paper is \"Emergent systematic generalization in a situated agent,\" which of course implies that the agent has \"systematic generalization.\" The authors go on to say, in the abstract, that \"we demonstrate strong emergent systematic generalisation in a neural network agent\". The results, however, fall short of these statements.\n\nThe strongest results pertain to generalizing a highly-practiced action such as \"lifting\" or \"putting\" to novel objects. In this case, highly-practiced means that the actions have been trained on 31 unique objects for millions of steps. However the paper does not study whether or not the agent can learn a novel action (e.g. \"lifting\" or \"putting\" with only a few examples) and generalize it systematically to familiar objects. Nor does it study whether novel actions can be combined systematically in new ways using relations and modifiers such as \"finding the toothbrush ABOVE the hat\" or \"finding AND putting\" or \"putting to the right of.\" Benchmarks for systematic generalization such as SQOOP and SCAN include these types of generalizations, and an agent with systematic generalization should handle them as well. To be clear, I don't think it's necessary to add additional experiments to the paper, but the current results should not be overstated in their generality.\n\nEven within the reported experiments, the results suggest that systematicity is lacking in several places. In the negation task, where chance is 50% accuracy, the agent achieves only 60% correct after learning from 40 unique words and 78% performance with 100 unique words (doesn't systematic generalization imply 100%?) For the putting tasks, the agent achieves 90% correct in one experiment (section 3.2) and then only achieves 63% correct in another (section 4.2). Again, the generalization abilities seem far from systematic.\n\nCritical details about the action space and the simulation parameters are needed. The action-space has 26 actions, but the paper does not say what these actions are. These details are crucial to understanding what is required to generalize \"lift\" or \"put\" to new objects -- instead the paper only says that \"in particular the process required to lift an object does not depend on the shape of that object (only its extent, to a degree)\" and that \"shape is somewhat more important for placement\" compared to lifting.\n\nI would consider updating my evaluation if the authors make revisions to ensure that the evidence supports their conclusions. The paper's title should also be supported by the findings; to offer a suggestion, something like \"Richer environments promote systematic generalization in situated agents\".\n\nOther suggestions\n- The axis on Fig. 2 is too small to read. Also, it should be mentioned in text that the network is trained for 100 million+ steps (also, what is a step? how many episodes was it trained for?)\n- The number of objects in sets X_1 and X_2 is important and should be mentioned in the main text.\n\n------\n\n** Update to review **\n\nThanks for your response to my review. It's clear that the authors have made considerable effort to improve the paper. In particular, the revised title, abstract, and introduction now more accurately reflect the contributions of the paper. It's not perfect, but the paper is improved and I revised my score accordingly.\n\nWhile it did not affect my final score, not all my suggestions were incorporated and I hope the authors will make further improvements in their revisions. The number of objects in sets X_1 and X_2  (Sections 3.1 and 3.2) are not mentioned and are tucked away in the appendix' this should be in the main text. Thanks for providing the list of 26 actions, but it's still not completely clear what makes a successful \"lift\" or \"put\" in terms of the sequence of actions. Finally, rather than simply saying that your agent \"in no way exhibits complete systematicity\" (Discussion), I hope the authors will expand on this and discuss the limitations of their experiments, and the kinds of systematicity not addressed and which could be the focus of future work.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Byeq7rzTYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n=============================== Update after revisions =====================================================\n\nIn my initial review, I had raised some issues with the interpretation of the results and suggested some control experiments to tighten the conclusions. The authors chose to weaken their initial claims by rephrasing their conclusions instead. I understand that there may not have been enough time to run many of the experiments I suggested, but I still think they are worth considering for the future. I'm mostly satisfied with the rephrasing of the conclusions in the revised paper, so as promised, I'm happy to increase my score and recommend acceptance. \n\nI spotted several typos in the revised paper, however: section 4.1: \"we choose to consider negation ...\", p. 5: \"for for ...\", a citation on p. 5 is not compiled correctly. There may be more. For the final version please make sure to go through the paper thoroughly a couple of times and fix all the typos.\n\n========================================================================================================\n\nThe authors present a systematic study of generalization in agents embedded in a simulated 3d environment. I think there are some interesting results in this paper that might be useful for people to know about. I appreciate the thoroughness of the experiments, in particular. I have, however, some issues with the interpretation of several of the main results. I would be happy to increase my score if we can resolve some of these issues. Here are my main concerns:\n\n1) In the experiments in section 3, only a limited test set is used. How is the train/test split decided in these experiments? Table 6 suggests that you have a much larger repository of objects. Why not use all possible objects in the test set? It is a bit premature to declare your results as systematic generalization if you can’t show that it actually works for a much larger set of test objects (ideally for all possible objects). \n\n2) Section 4.1: in these experiments, the training set size is increased, but the test set size is kept constant (and small), so the train/test size ratio also increases. So, an alternative explanation of the results in this section is that the model behaves largely according to visual similarity and as the training set size is increased, it becomes easier to find a training set object that is visually similar to any test set object. I think the authors should run an experiment where both training and test set sizes increase by the same amount so that the train/test set size ratio stays constant. If the model can’t achieve systematic generalization in that case, it would be wrong to conclude, as the authors do now, that increasing the training set size itself improves systematic generalization. The correct conclusion would rather be that increasing the train/test size ratio improves generalization, which is a weaker conclusion. Please note that the results in this section are quite similar to those in Lake & Baroni (2018) and in Bahdanau et al. (2018) (see their Figure 3). Bahdanau et al. (2018), for example, also show that increasing train/test set size ratio (their “rhs/lhs” ratio) improves generalization in generic neural networks. It is interesting to note, however, that neither Lake & Baroni (2018) nor Bahdanau et al. (2018) interpret these results positively (i.e., these results don’t show systematicity), whereas the current paper seems to put a more positive spin on essentially the same result. I think these earlier results should be explicitly discussed here and the authors should justify why they are interpreting the results differently (if they are). It should also be noted that in the real world the train/test size ratio for humans is presumably very small, perhaps zero (given the compositional abilities of humans).\n\n3) Section 4.3: I don’t think the results in this section are sufficient to establish the egocentric frame per se as the key factor. One possibility is that perhaps the frame doesn’t have to be centered on the agent, but as long as it has some systematic relationship to the agent’s location (for example, the center of the visibility window could be some distance away from the agent, and the agent itself may or may not be inside this window), that’s good enough to get generalization improvements. An even weaker possibility is that simply a moving frame is enough for improved generalization. In this case, the reference frame doesn’t even need to have a systematic relationship to the agent’s location. For example, the frame could be relative to a fictitious agent that randomly explores the environment. I think the authors should run some experiments to rule out these possibilities if they want to claim that the egocentric frame itself is responsible for generalization improvements. \n\n4) Section 4.4: In the experiments in this section, I think there are two relevant factors that need to be better disentangled: 1) the number and variability of image frames experienced by the two models; 2) the active perception aspect (the fact that the agent interacts with the environment and affects its own perceptual experience in one case). The authors claim the second factor as the key aspect enabling better generalization, but 1) is equally likely (this would be more in line with a standard data augmentation type result). A good control experiment here would be to not just use the first frame but a larger number of more variable frames for training the non-situated agent (for example, one can use image frames that would be seen by a camera that more or less randomly moves in front of the objects perhaps with the constraint that both objects are always at least partially visible). If the classification model generalizes as well as the situated agent in this control condition, you cannot claim active perception as the key factor.\n\n5) As a more general point, it’s a bit frustrating to have to judge systematic generalization by only looking at the results of some limited set of experiments. How do I interpret the results if the agent achieves only 84% accuracy in some experiment (as opposed to 100%)? It would be much better if the authors could somehow more rigorously prove systematicity. Here, I don’t necessarily mean “prove” in a mathematical sense, but just analyzing the learned representations a bit more rigorously and being able to say something along the lines of: here’s exactly how the trained model represents “lift”; because of reason X, Y, Z, this representation is completely disentangled from all object representations in the dataset (and ideally from all possible object representations, because that’s really what true systematicity entails, although I highly doubt that any generic model of the type studied in this paper will be able to achieve this, regardless of the amount and type of input it receives).   \n\nMore minor issues:\n\n6) In Table 5, “table lamp” appears both in training and test sets. Is this a typo?\n\n7) Some results are presented in the appendix without any mention in the main text (Appendix D. 2). I think this is not a good practice in general. In the main text, please make sure to mention, however briefly, every result that appears in the appendix (something along the lines of \"This result could not be explained by confound X or Y (Appendix Z)\" would suffice).\n\n8) Font size in Figure 2 is tiny (axis labels are impossible to read), please make it bigger. You don’t need that many ticks on the axes.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies out-of-sample generalisation that require an agent to respond to never-seen-before instructions by manipulating and positioning objects in a 3D Unity simulated room, and analyzes factors which promote combinatorial generalization in such environment. 

The paper is a very thought provoking work, and would make a valuable contribution to the line of works on systematic generalization in embodied agents. The draft has been improved significantly after the rebuttal. After the discussion, we agree that it is worthwhile presenting at ICLR.  
Decision: Accept (Poster)