Grounded Language Learning Fast and Slow

1 INTRODUCTION . Language models that exhibit one- or few-shot learning are of growing interest in machine learning applications because they can adapt their knowledge to new information ( Brown et al. , 2020 ; Yin , 2020 ) . One-shot language learning in the physical world is also of interest to developmental psychologists ; fast-mapping , the ability to bind a new word to an unfamiliar object after a single exposure , is a much studied facet of child language learning ( Carey & Bartlett , 1978 ) . Our goal is to enable an embodied learning system to perform fast-mapping , and we take a step towards this goal by developing an embodied agent situated in a 3D game environment that can learn the names of entirely unfamiliar objects in a single exposure , and immediately apply this knowledge to carry out instructions based on those objects . The agent observes the world via active perception of raw pixels , and learns to respond to linguistic stimuli by executing sequences of motor actions . It is trained by a combination of conventional RL and predictive ( semi-supervised ) learning . We find that an agent architecture consisting of standard neural network components is sufficient to follow language instructions whose meaning is preserved across episodes . However , learning to fast-map novel names to novel objects in a single episode relies on semi-supervised prediction mechanisms and a novel form of external memory , inspired by the dual-coding theory of knowledge representation ( Paivio , 1969 ) . With these components , an agent can exhibit both slow word learning and fast-mapping . Moreover , the agent exhibits an emergent propensity to integrate both fast-mapped and slowly acquired word meanings in a single episode , successfully executing instructions such as “ put the dax in the box ” that depend on both slow-learned ( “ put ” , “ box ” ) and fast-mapped ( “ dax ” ) word meanings . Via controlled generalization experiments , we find that the agent is reasonably robust to a degree of variation in the number of objects involved in a given fast-mapping task at test time . The agent also exhibits above-chance success when presented with the name for a particular object in the ShapeNet ar X iv :2 00 9 . 01 71 9v 4 [ cs .C L ] 1 4 O ct 2 02 0 taxonomy ( Chang et al. , 2015 ) and then instructed ( using that name ) to interact with a different exemplar from the same object class , and this propensity can be further enhanced by specific metatraining . We find that both the number of unique objects observed by the agent during training and the temporal aspect of its perceptual experience of those objects contribute critically to its ability to generalize , particularly its ability to execute fast-mapping with entirely novel objects . Finally , we show that a dual-coding memory schema can provide a more effective basis to derive a signal for intrinsic motivation than a more conventional ( unimodal ) memory . 2 AN ENVIRONMENT FOR FAST WORD LEARNING . We conduct experiments in a 3D room built with the Unity game engine . In a typical episode , the room contains a pre-specified number N of everyday 3D rendered objects from a global set G. In all training and evaluation episodes , the initial positions of the objects and agent are randomized . The objects include everyday household items such as kitchenware ( cup , glass ) , toys ( teddy bear , football ) , homeware ( cushion , vase ) , and so on . Episodes consist of two phases : a discovery phase , followed by an instruction phase ( see Figure 1 ) .1 In the discovery phase , the agent must explore the room and fixate on each of the objects in turn . When it fixates on an object , the environment returns a string with the name of the object ( which is a nonsense word ) , for example “ This is a dax ” or “ This is a blicket ” . Once the environment has returned the name of each of the objects ( or if a time limit of 30s is reached ) , the positions of all the objects and the agent are re-randomized and the instruction phase begins . The environment then emits an instruction , for example “ Pick up a dax ” or “ Pick up a blicket ” . To succeed , the agent must then lift up the specified object and hold it above 0.25m for 3 consecutive timesteps , at which point the episode ends , and a new episode begins with a discovery phase and a fresh sample of objects from the global set G. If the agent first lifts up an incorrect object , the episode also ends ( so it is not possible to pick up more than one object in the instruction phase ) . To provide a signal for the agent to learn from , it receives a scalar reward of 1.0 if it picks up the correct object in the instruction phase . In the default training setting , to encourage the necessary information-seeking behaviour , a smaller shaping reward of 0.1 is provided for visiting each of the objects in the discovery phase . Given this two-phase episode structure , two distinct learning challenges can be posed to the agent . In a slow-learning regime , the environment can assign the permanent name ( e.g . “ cup ” , “ chair ” ) to objects in the environment whenever they are sampled . By contrast , in the fast-mapping regime , 1Rendered images are higher resolution than those passed to the agent . which is the principal focus of this work , the environment assigns a unique nonsense word to each of the objects in the room at random on a per-episode basis . The only way to consistently solve the task is to record the connections between words and objects in the discovery phase , and apply this ( episode-specific ) knowledge in the instruction phase to determine which object to pick up . 3 MEMORY ARCHITECTURES FOR AGENTS WITH VISION AND LANGUAGE . The agents that we consider build on a standard architecture for reinforcement learning in multimodal ( vision + language ) environments ( see e.g . ( Chaplot et al. , 2018 ; Hermann et al. , 2017 ; Hill et al. , 2020 ) ) . The visual input ( raw pixels ) is processed at every timestep by a convolutional network with residual connections ( a ResNet ) . The language input is passed through an embedding lookup layer plus self-attention layer for processing . Finally , a core memory integrates the information from the two input sources over time . A fully-connected plus softmax layer maps the state of this core memory to a distribution over 46 actions , which are discretizations of a 9-DoF continuous agent avatar . A separate layer predicts a value function for computing a baseline for optimization according to the IMPALA algorithm ( Espeholt et al. , 2018 ) . We replicated previous studies by verifying that a baseline architecture with LSTM core memory ( Hochreiter & Schmidhuber , 1997 ) could learn to follow language instructions when trained in the slow-learning regime . However , the failure of this architecture to reliably learn to perform abovechance in the fast-learning regime motivated investigation of architectures involving explicit external memory modules . Given the two observation channels from language and vision , there are various ways in which observations can be represented and retrieved in external memory . Differentiable Neural Computer ( DNC ) In the DNC ( Wayne et al. , 2018 ) , at each timestep t a latent vector et = w ( ht−1 , rt−1 , xt ) , computed from the previous hidden state ht−1 of the agent ’ s core memory LSTM , the previous memory read-out rt−1 , and the current inputs xt , is written to a slot-based external memory . In our setting , the input xt is a simple concatenation [ vt , lt ] of the output of the vision network and the embedding returned by the language network . Before writing to memory , the latent vector et is also passed to the core memory LSTM to produce the current state ht . The agent reads from memory by producing a query vector q ( ht ) and read strength β ( ht ) , and computing the cosine similarity between the query and all embeddings currently stored in memory ei ( i < t ) . The external memory returns only the k most similar entries in the memory ( where k is a hyperparameter ) , and corresponding scalar similarities . The returned embeddings are then aggregated into a single vector r̂t by normalizing the similarities and taking a weighted average of the embeddings . This reading procedure is performed simultaneously by n independent read heads , and the results [ r̂1t , . . . , r̂ n t ] are concatenated to form the current memory read-out rt . The vectors et and ht are output to the policy and value networks . Dual-coding Episodic Memory ( DCEM ) We propose an alternative external key-value memory architecture inspired by the Dual-Coding theory of human memory ( Paivio , 1969 ) . The key idea is to allow different modalities ( language and vision ) to determine either the keys ( and queries ) or the values . In the present work , because of the structure of the tasks we consider , we align the keys and queries with language and the values with vision . However , for different problems ( such as those requiring language production ) the converse alignment could be made , or a single memory system could implement both alignments . In our implementation , at each timestep the agent writes the current linguistic observation embeddings lt to the keys of the memory and the current visual embedding vt to its values . To read from the memory , a query q ( vt , lt , ht−1 ) is computed and compared to the keys by cosine similarity . The k values whose keys are most similar to the query , [ mj ] j≤k , are returned together with similarities [ sj ] j≤k . To aggregate the returned memories into a single vector rt , the similarities are first normalized into a distribution { ŝj } and then applied to weight the memories m̂j = ŝjmj . These k weighted memories are then passed through a self-attention layer and summed elementwise to produce rt . As before this is repeated for n read heads , and the results concatenated to form the current memory read-out rt . rt is then concatenated with ht−1 and new inputs xt to compute a latent vector et = w ( ht−1 , rt , xt ) , which is passed to the core memory LSTM to produce the subsequent state ht , and finally et and ht are output to the policy and value networks . Mean ( S.D ) accuracy Architecture 1e9 training steps Gated Transformer ( XL ) We also consider an architecture where the agent ’ s core memory is a Transformer ( Vaswani et al. , 2017 ) , including the gating mechanism from Parisotto et al . ( 2019 ) . The only difference from Parisotto et al . ( 2019 ) is that we consider a multi-modal environment , where the observations xt passed to the core memory are the concatenation of visual and language embeddings . We use a 4-layer Transformer with a principal embedding size of 256 ( 8 parallel heads with query , key and value size of 32 per layer ) . These parameters are chosen to give a comparable number of total learnable parameters to the DCEM architecture . Policy learning The agent ’ s policy is trained by minimizing the standard V-trace off-policy actorcritic loss ( Espeholt et al. , 2018 ) . Gradients flow through the policy layer and the core LSTM to the memory ’ s query network and the embedding ResNet and self-attention language encoder . We also use a policy entropy loss as in ( Mnih et al. , 2016 ; Espeholt et al. , 2018 ) to encourage random-action exploration . For more details and hyperparameters see Appendix A.4 . Observation reconstruction In order to provide a stronger representation-shaping signal , we make use of a reconstruction loss in addition to the standard V-trace setup . The latent vector et is passed to a ResNet g that is the transpose of the image encoder , and outputs a reconstruction of the image input dimt = g ( et ) . The image reconstruction loss is the cross entropy between the input and reconstructed images : limt = −ximt logdimt − ( 1 − ximt ) log ( 1 − dimt ) . The language decoder is a simple LSTM , which also takes the latent vector et as input and produces a sequence of output vectors that are projected and softmaxed into classifications over the vocabulary dlangt . The loss is the cross entropy between the classification produced and the one-hot vocabulary indices of the input words : llangt = −x lang t logd lang t − ( 1− x lang t ) log ( 1− d lang t ) . For more details regarding the flow of information and gradients see Appendix A.4 .

Summary: The authors use a 3D world to explore grounded language learning, in which an agent uses RL to combine novel word-learning with stably acquired meanings to successfully identify and manipulate objects.  They show that a novel, psychologically-inspired memory mechanism is more memory-efficient than Transformers (both of which outperform plain LSTMs) and that it exhibits surprisingly robust generalization to novel action-object pairs.  The results should be of interest to many working in grounded language / multimodal representation learning, and the experiments are thorough and well-motivated.