Environmental drivers of systematicity and generalization in a situated agent

1 INTRODUCTION . Since the earliest days of research on neural networks , a recurring point of debate is whether neural networks exhibit generalisation beyond their training experience , in a systematic way ( Smolensky , 1988 ; Fodor & Pylyshyn , 1988 ; Marcus , 1998 ; McClelland et al. , 1987 ) . This debate has been re-energized over the past few years , given a resurgence in neural network research overall ( Lake & Baroni , 2017 ; Bahdanau et al. , 2018 ; Lake , 2019 ) . Generalisation in neural networks is not a binary question ; since there are cases where networks generalise well and others where they do not , the pertinent research question is when and under what conditions neural networks are able to generalise . Here , we establish that a conventional neural-network-based agent exposed to raw visual input and symbolic ( language-like ) instructions readily learns to exhibit generalisation that approaches systematic behaviour , and we explore the conditions supporting its emergence . First , we show in a 3D simulated room that an agent trained to find all objects from a set and lift only some of them can lift withheld test objects never lifted during training . Second , we show that the same agent trained to lift all of the objects and put only some of them during training can put withheld test objects , zero-shot , in the correct location . That is , the model learns to re-compose known concepts ( verbs and nouns ) in novel combinations . In order to better understand this generalisation , we conduct several experiments to isolate its contributing factors . We find three to be critical : ( a ) the number of words and objects experienced during training ; ( b ) a bounded frame of reference or perspective ; and ( c ) the diversity of perceptual input afforded by the temporal aspect of the agent ’ s perspective . Crucially , these factors can be enhanced by situating an agent in a realistic environment , rather than an abstract , simplified setting . These results serve to explain differences between our findings and studies showing poor generalisation , where networks were typically trained in a supervised fashion on abstract or idealised stimuli from a ∗Work carried out during internship at DeepMind . single modality ( e.g . Lake & Baroni , 2017 ) . They also suggest that the human capacity to exploit the compositionality of the world , when learning to generalise in systematic ways , might be replicated in artificial neural networks if those networks are afforded access to a rich , interactive , multimodal stream of stimuli that better matches the experience of an embodied human learner ( Clerkin et al. , 2017 ; Kellman & Arterberry , 2000 ; James et al. , 2014 ; Yurovsky et al. , 2013 ; Anderson , 2003 ) . Our results suggest that robust systematic generalisation may be an emergent property of an agent interacting with a rich , situated environment ( c.f . McClelland et al. , 2010 ) . 1.1 SYSTEMATICITY AND GENERALISATION . Systematicity is the property of human cognition whereby “ the ability to entertain a given thought implies the ability to entertain thoughts with semantically related contents ” ( Fodor & Pylyshyn , 1988 ) . As an example of systematic thinking , Fodor & Pylyshyn ( 1988 ) point out that any human who can understand John loves Mary will also understand the phrase Mary loves John , whether or not they have heard the latter phrase before . Systematic generalisation ( Plaut , 1999 ; Bahdanau et al. , 2018 ; Lake et al. , 2019 ) is a desirable characteristic for a computational model because it suggests that if the model can understand components ( or words ) in certain combinations , it should also understand the same components in different combinations . Note that systematic generalisation is also sometimes referred to as ‘ combinatorial generalisation ’ ( O ’ Reilly , 2001 ; Battaglia et al. , 2018 ) . However , even human reasoning is often not perfectly systematic ( O ’ Reilly et al. , 2013 ) . We therefore consider this issue from the perspective of generalisation , and ask to what degree a system generalises in accordance with the systematic structure implied by language . Recent discussions around systematicity and neural networks have focused on the issue of how best to encourage this behaviour in trained models . Many recent contributions argue that generalising systematically requires inductive biases that are specifically designed to support some form of symbolic computation ( such as graphs ( Battaglia et al. , 2018 ) , modular-components defined by symbolic parses ( Andreas et al. , 2016 ; Bahdanau et al. , 2018 ) , explicit latent variables ( Higgins et al. , 2017 ) or other neuro-symbolic hybrid methods ( Mao et al. , 2019 ) ) . On the other hand , some recent work has reported instances of strong generalisation in the absence of such specific inductive biases ( Chaplot et al. , 2018 ; Yu et al. , 2018 ; Lake , 2019 ) . In the following sections , we first add to this latter literature by reporting several novel cases of emergent generalisation . Unlike in previous work , the examples that we present here involve tasks involving the manipulation of objects via motor-policies , as well as language and vision . This is followed by an in-depth empirical analysis of the environmental conditions that stimulate generalisation in these cases . 2 A MINIMAL MULTI-MODAL AGENT . All our experiments use the same agent architecture , a minimal set of components for visual ( pixels ) perception , language ( strings ) perception , and policy prediction contingent on current and past observations . The simplicity of the architecture is intended to emphasize the generality of the findings ; for details see App C. Visual processing Visual observations at each timestep come in the form of a W × H × 3 real-valued tensor ( W and H depend on the particular environment ) , which is processed by a 3-layer convolutional network with 64 , 64 , 32 channels in the first , second and third layers respectively . The flattened output of this network is concatenated with an embedding of the language observation . Language processing Language instructions are received at every timestep as a string . The agent splits these on whitespace and processes them with a ( word-level ) LSTM network with hidden state size 128 . The final hidden state is concatenated with the output of the visual processor to yield a multimodal representation of the stimulus at each timestep . Memory , action and value prediction The multimodal representation is passed to a 128-unit LSTM . At each timestep , the state of this LSTM is multiplied by a weight matrix containing A×128 weights ; the output of this operation is passed through a softmax to yield a distribution over actions ( A is the environment-dependent size of the action set ) . The memory state is also multiplied by a 1× 128 weight matrix to yield a value prediction . Training algorithm We train the agent using an importance-weighted actor-critic algorithm with a central learner and distributed actors ( Espeholt et al. , 2018 ) . 3 DEMONSTRATING GENERALISATION . A key aspect of language understanding is the ability to flexibly combine predicates with arguments ; verb-noun binding is perhaps the canonical example . Verbs typically refer to processes and actions , so we study this phenomenon in a 3D room built in the Unity game engine.1 In this environment , the agent observes the world from a first-person perspective , the Unity objects are 3D renderings of everyday objects , the environment has simulated physics enabling objects to be picked-up , moved and stacked , and the agent ’ s action-space consists of 26 actions that allow the agent to move its location , its field of vision , and to grip , lift , lower and manipulate objects . Executing a simple instruction like find a toothbrush ( which can be accomplished on average in six actions by a well-trained agent in our corresponding grid world ) requires an average of around 20 action decisions . 3.1 A GENERAL NOTION OF LIFTING . Lifting is a simple motor process that corresponds to a verb and is easily studied in this environment . We consider the example instruction lift a helicopter to be successfully executed if the agent picks up and raises the helicopter above a height of 0.5m for 2 seconds , a sequence which requires multiple actions once the helicopter is located . Similarly , the instruction find a helicopter is realised if the agent moves within two metres of a helicopter and fixes its gaze ( as determined by a ray cast from the agent ’ s visual field ) while remaining within the two metre proximity for five timesteps without lifting the object during this time . To measure how general the acquired notion of lifting is , we trained the agent to find each of a set X = X1 ∪ X2 of different objects ( allowing the agent to learn to ground objects to their names ) and to lift a subset X1 of those objects , in trials in a small room containing two objects positioned at random ( one ‘ correct ’ according to the instruction , and one ‘ incorrect ’ ) . The agent receives a positive reward if it finds or lifts the correct object , and the episode ends with no reward if the agent finds or lifts the incorrect object . We then evaluate its ability to extend its notion of lifting ( zeroshot ) to objects x ∈ X2 . In a variant of this experiment , we trained the agent to lift all of the objects in X when referring to them by their color ( lift a green object ) , and to find all of the objects in X according to either shape or color ( find a pencil or find a blue object ) . We again tested on whether the agent lifted objects x ∈ X2 according to their shape ( so , the test trials in both variants are the same ) . As shown in Fig 2 ( a ) , in both variants the agent generalises with near-perfect accuracy . The agent therefore learned a notion of what it is to lift an object ( and how this binds to the word lift ) with sufficient generality that it can , without further training , apply it to novel objects , or to familiar objects with novel modes of linguistic reference . 3.2 A GENERAL NOTION OF PUTTING . We took our study of predicate-object binding further by randomly placing two large , flat objects ( a bed and a tray ) in the room and training an agent to place one of three smaller objects on top . As before , the agent received a positive reward if it placed the correct small object on the bed or the tray according to the instruction . If it placed an incorrect object on the bed or tray , the episode ended with no reward . To test generalisation in this case , we trained the agent to lift each of a set X = X1 ∪ X2 of smaller objects ( as in the previous experiment ) and then to put some 1http : //unity3d.com subset X1 ⊂ X of those objects on both the bed and the tray as instructed . We then measured its performance ( with no further learning ) in trials where it was instructed to put objects from X2 onto either the bed or the tray . Surprisingly , we found that the agent was able to place objects with over 90 % accuracy onto the bed or the tray zero-shot as instructed ( Fig 2 ( b ) ) . This generalisation requires the agent to bind its knowledge of objects ( referred to by nouns , and acquired in the lifting trials ) to a complex control process ( acquired in the training putting trials ) – involving locating the correct target receptacle , moving the object above it ( avoiding obstacles like the bed-head ) and dropping it gently . An important caveat here is that control in this environment , while finer-grained than in many common synthetic environments , is far simpler than the real world ; in particular the process required to lift an object does not depend on the shape of that object ( only its extent , to a degree ) . Once the objects are held by the agent , however , their shape becomes somewhat more important and placing something on top of something else , avoiding possible obstacles that could knock the object from the agent ’ s grasp , is a somewhat object-dependent process .

Summary: This paper studies systematic generalization in a situated agent. The authors examine the degree to which various factors influence systematic generalization, including 2D vs. 3D environments, egocentric vision,  active perception, and language. The experiments reveal that the first three factors, but not language, promote systematic generalization. The experiments are well-done and worthwhile, and identifying the key factors that affect generalization is a strength of the paper.