Grounded Language Learning Fast and Slow
Authors: Felix Hill
Olivier Tieleman
Tamara von Glehn
Nathaniel Wong
Hamza Merzic
Stephen Clark
Venue: ICLR 2021 Spotlight (2021)
Keywords: language
cognition
fast-mapping
grounding
word-learning
memory
meta-learning

Abstract: Recent work has shown that large text-based neural language models acquire a surprising propensity for one-shot learning. Here, we show that an agent situated in a simulated 3D world, and endowed with a novel dual-coding external memory, can exhibit similar one-shot word learning when trained with conventional RL algorithms. After a single introduction to a novel object via visual perception and language ("This is a dax"), the agent can manipulate the object as instructed ("Put the dax on the bed"), combining short-term, within-episode knowledge of the nonsense word with long-term lexical and motor knowledge. We find that, under certain training conditions and with a particular memory writing mechanism, the agent's one-shot word-object binding generalizes to novel exemplars within the same ShapeNet category, and is effective in settings with unfamiliar numbers of objects. We further show how dual-coding memory can be exploited as a signal for intrinsic motivation, stimulating the agent to seek names for objects that may be useful later. Together, the results demonstrate that deep neural networks can exploit meta-learning, episodic memory and an explicitly multi-modal environment to account for 'fast-mapping', a fundamental pillar of human cognitive development and a potentially transformative capacity for artificial agents.   

GROUNDED LANGUAGE LEARNING FAST AND SLOW
Felix Hill, Olivier Tieleman, Tamara von Glehn, Nathaniel Wong, Hamza Merzic, Stephen Clark DeepMind London, UK
[felixhill, tieleman, tamaravg, nathanielwong, hamzamerzic,
2XXVV22009.1171944 [Cs 14 14tct. 2020
ABSTRACT
Recent work has shown that large text-based neural language models acquire a surprising propensity for one-shot learning. Here, we show that an agent situated in a simulated 3D world, and endowed with a novel dual-coding external mem- ory, can exhibit similar one-shot word learning when trained with conventional RL algorithms. After a single introduction to a novel object via visual perception and language ("This is dax"), the agent manipulate the object as instructed ("Put the dax on the bed"), combining short-term, within-episode knowledge the nonsense word with long-term lexical and motor knowledge. We find that, der certain training conditions and with a particular memory writing mechanism, the agent's one-shot word-object binding generalizes to novel exemplars within the same ShapeNet category. and is effective in settings with unfamiliar numbers of objects. We further show how dual-coding memory can be exploited as a signal for intrinsic motivation, stimulating the agent to seek names for objects that may be useful later. Together, the results demonstrate that deep neural networks can ex- ploit meta-learning, episodic memory and an explicitly multi-modal environment to account for faxt-mupping, fundamental pillar of human cognitive development and potentially transformative capacity for artificial agents.
INTRODUCTION
Language models that exhibit one- or few-shot learning are of growing interest in machine learn- ing applications because they can adapt their knowledge to new information (Brown et al. 2020) Yin. 2020). One-shot language learning in the physical world is also of interest to developmental psychologists; fast-mapping, the ability to bind new word to unfamiliar object after a single exposure, is a much studied facet of child language learning (Carey & Bartlett 1978). Our goal is to enable an embodied learning system to perform fast-mapping, and take step towards this goal by developing an embodied agent situated in a 3D game environment that can learn the names of entirely unfamiliar objects in single exposure, and immediately apply this knowledge carry out instructions based on those objects. The agent observes the world via active perception of raw pixels, and learns to respond to linguistic stimuli by executing sequences of motor actions. It is trained by a combination of conventional RL and predictive (semi-supervised) learning.
We find that an agent architecture consisting of standard neural network components is sufficient to follow language instructions whose meaning is preserved across episodes. However, learning fast-map novel names to novel objects in single episode relies on semi-supervised prediction mechanisms and a novel form of external memory, inspired by the dual-coding theory of knowl- edge representation (Paivio, 1969). With these components, an agent can exhibit both slow word learning and fast- fast-mapping Moreover, the agent exhibits an emergent propensity to integrate both fasp-d and slowly acquired word meanings in a single episode, successfully executing in- structions such "put the dax in the box" that depend both slow-learned ("put", "box") and fast-mapped ("dax") word meanings.
Via controlled generalization experiments, we find that the agent is reasonably robust to degree of variation in the number of objects involved in a given fast- maapiig task at test time. The agent also exhibits above-chance success when presented with the name for a particular object in the ShapeNet

---

Discovery phase
Instruction phase
*Pick up biicket"
Agent
"Thi is dax"
Randomize positions
Thie blicket"
This bilibak
2914 18 thele
Figure 1: Top: The two phases of fast-mapping episode. Bottom: Screenshots of the task from the agent's perspective at important moments (including the contents of the language channel).
laxonomy (Chang al. 2015) and then instructed (using that name) interact with different exemplar from the same object class, and this propensity can be further enhanced by specific meta- training. We find that both the number of unique objects observed by the agent during training and the temporal aspect of its perceptual experience of those objects contribute critically to its ability to generalize, particularly its ability to execute fast-mapping with entirely novel objects. Finally, show that a dual-coding memory schema can provide a more effective basis to derive a signal for intrinsic motivation than more conventional (unimodal) memory.
AN ENVIRONMENT FOR FAST WORD LEARNING
We conduct experiments in 3D room built with the Unity game engine. In typical episode, the room contains pre-specified number N of everyday 3D rendered objects from a global set G. In all training and evaluation episodes, the initial positions of the objects and agent randomized. The objects include everyday household items such as kitchenware (cup, glass), toys (teddy bear, foothall), homeware (cushion, vase), and
Episodes consist of two phases: a discovery phase, followed by an instruction phase (see Figure In the discovery phase, the agent must explore the room and fixate on each of the objects in turn. When it fixates on an object, the environment returns a string with the name of the object (which nonsense word), for example "This dax" or "This is blicket". Once the environment has returned the name of each of the objects (or if a time limit of 30s is reached). the positions of all the objects and the agent are re-randomized and the instruction phase begins. The environment then emits an instruction, for example "Pick up dax" or "Pick up blicket". To succeed, the agent must then lift up the specified object and hold it above 0.25m for consecutive timesteps, at which point the episode ends, and new episode begins with discovery phase and fresh sample of objects from the global set G. If the agent first lifts up an incorrect object, the episode also ends (so not possible to pick up more than object in the instruction phase). To provide a signal for the agent to learn from, it receives a scalar reward 1.0 if it picks up the correct object in the instruction phase. In the default training setting, encourage the necessary information-sseeing behaviour, smaller shaping reward of 0.1 1 is provided for visiting each of the objects in the discovery phase.
Given this two-phase episode structure, two distinct learning challenges In a slow-lcarning regime, the environment can assign the permanent name (e.g. "cup", "chair") objects in the environment whenever they are sampled. By contrast, in the fast-mapping regime,
be posed the agent.
Rendered images higher resolution than those passed the agent.
2

---

which is the principal focus of this work, the environment assigns unique nonsense word each of the objects in the room at random on a per-episode basis. The only way to consistently solve the task is to record the connections between words and objects in the discovery phase, and apply this (episode-specific) knowledge in the instruction phase to determine which object to pick up.
MEMORY ARCHITECTURES FOR AGENTS WITH VISION AND LANGUAGE
The agents that we consider build on standard architecture for reinforcement learning in multi- modal (vision + language) environments (see e.g. (Chaplot et al., 2018 Hermann et al. 2017 7 Hill et al. 2020). ). The visual input (raw pixels) is processed at every timestep by convolutional network with residual connections (a ResNet). The language input is passed through an embedding lookup layer plus self-attention layer for processing. Finally, a core memory integrates the information from the two input sources over time. A fully-connected plus softmax layer maps the state of this core memory
distribution over 46 actions, which are discretizations of 9-DoF continuous agent avatar. A separate layer predicts a value function for computing a baseline for optimization according to the IMPALA algorithm (Espeholt al., 2018).
We replicated previous studies by verifying that a baseline architecture with LSTM core memory (Hochreiter Schmidhuber 1997) could learn follow language instructions when trained in the slow-learning regime. However, the failure of this architecture to reliably learn to perform above- chance in the fast-learning regime motivated investigation of architectures involving explicit external memory modules. Given the two observation channels from language and vision, there are various ways in which observations can be represented and retrieved in external memory.
Differentiable Neural Computer (DNC) In the DNC (Wayne et al. 2018), at each timestep a latent vector e₁ w(h,1,, 1, xt), computed from the previous hidden state h of the agent's core memory LSTM, the previous memory read-out 1 and the current inputs xt, is written to a slot-based external memory. In our setting, the input x; is a simple concatenation [v,, 1r] of the output of the vision network and the embedding returned by the language network. Before writing to memory, the latent vector e, is also passed to the core memory LSTM to produce the current state h, The agent reads from memory by producing query vector 9 gh,) and read strength ((,)), and computing the cosine similarity between the query and all embeddings currently stored memory e, Gi l). The external memory returns only the & most similar entries in the memory (where k is a hyperparameter), and corresponding scalar similarities. The returned embeddings are then aggregated into a single vector i, by normalizing the similarities and taking a weighted average of the embeddings, This reading procedure is performed simultancously by n independent read heads, are concatenated form the current memory read-out rt- The vectors
and the results iFl, "fl.... +?]] o₁ and h, are output to the policy and value networks.
Dual-coding Episodic Memory (DCEM) We propose an alternative external key-value memory architecture inspired by the Dual-Coding theory of human memory (Paivio 1969). The key idea is to allow different modalities (language and vision) to determine either the keys (and queries) or the values. In the present work, because of the structure of the tasks we consider, we align the keys and queries with language and the values with vision. However, for different problems (such as those requiring language production) the converse alignment could be made, single memory system could implement both alignments.
In our implementation, at each timestep the agent writes the current linguistic observation embed- dings I to the keys of the memory and the current visual embedding v, to its values. To read from the memory, a query q(v,, 1, h,-1) is computed and compared to the keys by cosine similarity. The k values whose keys are most similar to the query, [m]]isk, are returned together with similarities [s']]sk- To aggregate the returned memories into a single vector r,, the similarities are first nor- malized into a distribution ( and then applied to weight the memories m³ á m². These weighted memories are then passed through a self-altention layer and summed elementwise to pro- duce rt. As before this is repeated for n read heads, and the results concatenated to form the curret memory read-out rt. It is then concatenated with h4-1 and new inputs x, compute a latent vector e, = w wohh-11 Fe, xx), which is passed to the core memory LSTM to produce the subsequent state h and finally e, and h, are output to the policy and value networks.
3

---

Mean (S.D) D) accuracy 1e9 training steps
Architecture
LSTM
0.33 (0.05)
LSTM +
0.61 (0.27)
Transtormer
DNC
DNC mem=1024
0.34 (0.01)
DNC mem=1024
0,64 (0.27)
Accuracy
TransformerXL. mem=1024
0.32 (0.02)
0.6
TransformerXI. mem=
0,98 (0.01)
DCEM 1e2=1024
0.33 (0.02)
DCEM mem=1024 +
0.98 (0.01)
TransformerXT. 100em=100++
0.73 (0.35)
DCEM mem= 100
0.98 (0.01)
0.0
Random object selection
0.33
Table 1: Left: Performance when training on three-object fast-mapping task with |G| 30. mem: size of memory buffer/window R: with reconstruction loss. Right: Learning curves, each showing mean S.D. over random seeds.
Gated Transformer (XL) We also consider architecture where the agent's ' memory is Transformer (Vaswani et al., 2017). including the gating mechanism from Parisotto et al.] (2019). The only difference from Parisotto et al. (2019) that we consider multi-modal environment, where the observations X₁ passed to the core memory are the concatenation of visual and language embeddings. We use 4-layer Transformer with principal embedding size of 256 parallel heads with query, key and value size of 32 per layer). These parameters are chosen to give a comparable number of total learnable parameters to the DCEM architecture.
Policy learning The agent's policy is trained by minimizing the standard V-trace off-polic actor- critic loss (Espeholt et al., 20188). Gradients flow through the policy layer the core LSTM to the memory's 's query network and the embedding ResNet and self-attentic language encoder. We also use a policy entropy loss as (Mnih et al.] 2016) Espehhlt et al. 2018) to encourage random-action exploration. For more details and hyperparameters Appendix A.4
Observation reconstruction In order to provide a stronger representatioonssaaing signal, we make of reconstruction loss in addition the standard V-trace setup. The latent vector e, is passed to ResNet 9 that is the transpose of the image encoder, and outputs reconstruction of the image input d! g(e,). The image reconstruction loss the cross entropy between the input and reconstructed images: 1im = x!! log d! (1 x!m!) log(1 d(m). The language decoder a simple ISTT,, which also takes the latent vector e; as input and produces sequence of output
vectors that are projected and softmaxed into classifications over the vocabulary d The loss the cross entropy between the classification produced and the one-hot vocabulary indices of the
input words: !.an -x, log d (1 Jang) log(1 d. For more details regarding the flow of information and gradients see Appendix A.4
4 EXPERIMENTS
We compared the different memory architectures with and without semi-supervised reconstruction loss on version of the fast-mapping task involving three objects (N 3) sampled from global set of 30 (lG) 30). As shown in Table I only the DCEM and Transformer architectures reliably solve the task after x 10 timesteps of training.
DCEM vs. TransformerXI. Importantly, the Transformer and DCEM are the two architectures that can exploit the principle of dual-coding. Since the inputs to the Transformer are the concate- nation of visual and language codes, this model can recover the dual-coding aspect of the DCEM by learning self-attention weights Wt and Wg that project the language code to keys and queries, and weights W. project the visual code to values. Learning in the DCEM marginally more sample-efficiens but this is perhaps expected given it was designed with fast-mapping tasks in mind. In light of this, it really worth pursing memory systems with explicit episodic memories?

---

(a) Trials with 3 objects
(b) Trials with 5 objects
(c) Triils with objects
Evaluation accuracy
0.4
0.
0.4
0.3
0.0
0.
03
04
2.6
0.3
10
0.0
122
0.4
0.
10
1.2
0.0
0.2
04
3.5
0.2
3.0
Trained on:
object trials
5 abbjcct trials
8 &bbject trials
Figure 2: Accuracy of agents trained on probe trials involving different number of total objects for agents meta-trained with different numbers of total objects.
To show one clear justification for external memory architectures, we conducted an additional com- parison in which the memory windows of both the DCEM and the Transformer agents were limited to 100 timesteps (from 1024 in the original experiment), approximately the length an episode if an agent is well-trained to the optimal policy. With memory span of 100, the Transformer is forced to use the XL window-recurrent 2019), while any capacity to retain episodic information beyond 100 timesteps in the DCEM must be managed by by the LSTM controller. In this setting we observed that the DCEM was substantially more effective (Table [ in real-world tasks working memory will always be at a premium. These results suggest that DCEM more workking-mmmory-eee
mechanism to pass information across context windows (Dai et al,
left, bottom). While this imposed memory constraint may seem arbitrary,
than the Transformer agent. Indeed, by employing a simple heuristic by which the agent only writes to its external memory when the language observation changes from one timestep to the next, the DCEM agent with only 20 memory slots could solve the task with similar efficiency to a Transformer agent with 1024-slot memory. See Appendix A. for these results and details of the selective writing heuristic.
4.1 GENERALIZATION
To explore the generalization capabilities of our agents, subjected trained agents to various be- havioural probes, and measured performance across thousands of episodes without updating their weights. Unless stated otherwise, all experiments in this section involve the DCEM+Recons agent.
Number of objects We first probed the robustness of the agent to fast-mapping episodes with different numbers of objects. In all conditions, the same objects appear in both the discovery and instruction phases of the episode, and the objects sampled from the same global set G' (IG) = 30).
As shown in Figures bb) and (c) (red curves), with the (default) meta-training setting involving episodes involving five objects is approximately 70%, and with eight objects around 50%. This sub-optimal performance suggests that, with this meta- training regime, the agent does tend overfit, to some degree, to the "three-ness" of its experience. Figure (bb) shows, however, that the overfitting of the agent can be alleviated by increasing the number of objects during meta training. Finally, Figure 2[a) confirms, perhaps unsurprisingly. that fewer objects than it was trained on.
three objects in each episode, performance
the agent has no problem generalizing to episodes fiwh
Novel objects To probe the ability of the agents to quickly learn about any arbitrary new object, we instrumented trials with objects sampled from a global test set of novel objects H H n 0. | 10. As shown in Figure we found that an agent meta-trained on 20 objects (i.e. |G| = 20) was almost perfectly robust to novel objects. As may be expected, this robustness degraded to some degree with decreasing GG), which is symptomatic of the agent specializing (and overfitting) to the particular features and distinctions
the objects in its environment. However, we only observed a substantial reduction in robustness to objects when |G] was reduced as low three - i.e. meta- training experience in which all episodes contain the same three objects (the first three elements of G alphabetically, i.e. boat, book and bottle).
5

---

Training
Testing: novel objects
0.0
088
0.6
0.2
52
0.5 1.0
3.5
2.0
255
3.0
160
Figure 3: Accuracy during training and evaluation trials involving unfamiliar objects, for different sizes of global training set G. Curves show mean + S.E. over 3 agent seeds each condition.
Training Exect recall condition
1.0
This is dax.
Fick up dox
Exact recall recall condition Extension training set
Teating accuracy
Extension within training
0.6
This is dax.
Pick up dex"
0.4
Testing
0.2
Extension of novel callegories
"This it disx"
"Pick up
0.0
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1c9
Figure 4: Accuracy of agents in fast-mapping trials requiring the extension of ShapeNet categories from a single exemplar. Curves show the mean + S.E. over three agent seeds in each condition.
Fast category extension Children aged between three and four can acquire one shot not only bindings between new words and specific unfamiliar objects, but also bindings between new words and categories (Behrend et al. 2001; Waxman & Booth 2000) Vlach & Sandhofer, 2012). We conducted an analogous experiment by exploiting the category structure in ShapeNet (Chang al. 2013). In a test trial, in the discovery phase the agent is presented with exemplars from three novel (held-out) ShapeNet categories (together with nonsense names). In the instruction phase, the agent must then pick up a different and unseen exemplar from one of these three new categories as in- structed. As shown in Figure 4 when trained described previously, the agent achieves around 55% accuracy on test trials, which is above chance (33%) but still substantial error rate. However, this performance can be improved by requiring the agent extend the training object categories as it learns. In this regime, three ShapeNet exemplars from distinct classes are encountered by the agent the discovery phase of training episodes, and the instruction phase involves different exem- plars from the same three classes. When trained in this way (which share similarities with matching networks (Vinyals et al. 20166), performance on extending novel categories increases to 88%.
Role of temporal aspect category extension relied on the agent reading multiple values from memory for each query. See A.2 a discussion these results, which suggest that the temporal aspect of the agent's experience (and learning from multiple views of the same object) is an important driver of generalization.
Through ablations we found that both novel objects generalization and
4.2 INTRINSIC MOTIVATION
The default version of the fast-mapping task includes shaping reward to encourage the agent to visit all objects in the room. Without this reward, the credit assignment problem of a fast-mapping episode too challenging. However, we found that the DCEM agent able to solve the task

---

1.0
1.0
DOEM, Atang 16-3. Alm
DCEM, Alanz =0, Aim =0
0.8
EM (DNC). A= 10-4
0.6
0.6
reward
0.4
0.4
DCEM, Alane 1033, =0
0.2
0.2
DCEM, Alang" 'e-3, Amm 3e-5
DCEM. Atang"
0.0
0,0
0.5
1.0
1.5
2.0 Steps
2.5
3.0
3.5
0.0
0.5
1.0
1.5
2.0 steps
2.5
3.0
3.5
1e8
Figure 5:
Accuracy agents trained without shaping reward on the 3-object fast-mapping task with GG 30. Curves show mean + S.E. across three seeds in each condition.
without shaping rewards by employing a memory-based algorithm for intrinsic motivation (NGU; Badia et al. (2020)). NGU computes 'surprise" score for observations by computing its distance to other observations in the episodic memory, as described Appendix AA.4.3) The surprise score is applied as reward signal ,.NGU which is added the environment reward encourage the agent to seek new experiences. We compared the effect doing this the DNC and the DCEM agents. For DCEM, the NGU computation (vision) column, or both. In the former case, the agent seeks novelty in the language space Tang and in the latter, in the visual space. The final reward is ext Atang NGG + Aim N.N As shown in Figure we found that the DCEM agent (with Alang 100 and Xim 3 10-5) was able to solve the Tast-mapping tasks without any shaping reward. This was not the case for the DNC agent, presumably because required signal for llanuaagenoovlty is not approximated well by the surprise score of the merged visual- lisuul-lnguuage codes in the episodic memory.
be applied the memory's 's keys (language) column, its values
NGG
4.3 INTEGRATING FAST AND SLOW LEARNING
To test whether our agents can integrate new information with existing lexical (and perceptual and motor) knowledge, we combined fast-mapping task with more conventional instruction-followingg task. In the discovery phase, the agent must explore to find the names of three unfamiliar objects, but in this case the room also contains large box and large bed, both of which are immovable. The positions of all objects and the agent are then re-randomized as before. In the instruction phase, agent is then instructed to put one of the three movable objects (chosen random) on either bed or in the box (again chosen at random). As shown in Figure if the training regime consisted of conventional lifting and putting tasks, together with a fast-mapping lifting task and a fast- mapping putting task, the agent learned to execute the evaluation trials with near peer-ee accuracy. Notably, we also found that aobstantilll--hhove-haaa trials without needing to train the agent the evaluation task in any form. If we trained the agent on conventional lifting and putting tasks, and a fast-mapping task involving lifting only, the agent could recombine the knowledge acquired during this training resolve the evaluation trials novel (zero-shot) task with tess-ehan-perfect but substantially-aboe--hhae accuracy.
performance could be achieved on the evaluation
4.4 RESULTS WITH ANOTHER ENVIRONMENT
To verify that the observed effects hold beyond our specific Unity environment, we added a new task to the DeepMind Lab suite (Beattie et al. 2016). Results for this task are given in Appendix A.3
RELATED WORK
Meta-learning, of the sort observed in agent, has been applied to train matching networks: image classifiers that can assign the correct label to a novel image, given a small support set of (image, label) pairs that includes the correct target label (Vinyals al. 2016). Our work is also inspired by

---

Task name
Train/test relationship
Evalustion aecurney fast-mapping putting
up
Fartitrr objects, familar task
0.97
up
objects, unfamilar taok
0.96
Minimal training regime
Liting
Cafirilia
0.71
Thds car
Put cer the
Cafariliar objeets, unfumiliar usk
objects at nenkm (stance)
0.17
Thes
Put
Fast-maspigg +putting (novel objects))
Evalustion
Right: The accuracy of the agent (accuracy +.S.E.) on evaluation trials when exposed to different training regimes. Left: Schematic of the most impoverished training regime.
Figure 6:
Snell al. (2017), who propose more efficient way integrate small support of experience into a coherent space of image 'concepts' for improved fast learning, and [Santoro et al. (2016), who show that the successful meta-training of image classifiers can benefit substantially from external memory architectures such as Memory Networks (Weston et al. 2014) or DNC (Graves et al. 2016
In NLP, meta-learning has been used to train few-shot classifiers for various tasks (see Yin 2020)
for a recent survey). Meta-learning has also previously been observed in reinforcement learning agents trained with conventional policy-gra saaide-gradient algorithms (Duan al., [0166] |Wan et al. 2019).
In Model-Agnostic Meta Learning (Finn et al. 2017). models are (meta) trained to be easily tunable (by any gradient algorithm) given a small number of novel data points. When combined with policy- gradient algorithms, this technique yields fast learning tasks. cognitive tasks where fast learning is not explicitly required, external memories have proven to help goal-directed agents (Fortunato al. 2019), and can be particularly powerful when combined with an observation reconstruction loss (Wayne et al. 2018).
both 2D navigation and 3D locomotion
Recent work at the intersection of psychology and machine learning is also relevant in that it shows how the noisy. first-person perspective of child can support the acquisition robust visual cat- egories in artificial neural networks (Bambach al. 2018). When deep networks are trained data recorded from children's head cameras, unsupervised or semi-supervised learning objectives can substantially improve the quality of the resulting representations (Orhan al. 2020).
CONCLUSION
Our experiments have highlighted various benefits of having an explicitly multi-modal episodic memory system. First, mechanisms that allow the agent query its memory in modality-speciio way (either within or across modalities) can better allow them to rapidly infer and exploit connec- tions between perceptual experience and words, and therefore to realize fust-mapping, a notable aspect of human learning. Second, external (read-write) memories can achieve better performance for the same number of memory 'slots" than Transformer-bii efficiency' may be increasingly important as agents are applied to real-world tasks with very long episodic horizons. Third, in cases where it is useful estimate the degree of novelty or "surprise" in the current state of the environment (for instance to derive a signal for intrinsic motivation), more informative signal may be obtained by separately estimating novelty based on each modality and aggregating the result. Finally, an episodic memory system may ultimately be essential for fast knowledge consolidation. The potential for memory buffers and offline learning processes such as experience replay to support knowledge consolidation is not Mnih et al., 2016] Lillicrap et al.) 2016)) McClelland et al., 2020). For language learning agents, the need to both rapidly acquire and retain multi-modal knowledge may further motivate explicit
memories. This greater 'memory-
idea (McClelland et al.) 1995.
8

---

external memories. Retaining in memory visual experiences together with aligned (and hopefully pertinent) language (i.e. a dual-coding schema) may facilitate something akin to offline 'supervised' language learning. We leave this possibility for future investigations, which will facilitate by releasing publicly the environments and tasks described in this paper.
REFERENCES
Adrià Puigdomenech Badia, P. Sprechmann, Alex Vitvitskyi, Daniel Guo, B. Piot, Steven Kaptur- owski, O. Tieleman, Martín Arjovsky, A. Pritzel, Andew Bolt, and Charles Blundell. Never give up: Learning directed exploration strategies. ArXiv, abs/2002.06038, 2020.
Sven Bambach, David Crandall, Linda Smith, and Chen Yu. Toddler-inspired visual object learning. In Advances in neural information processing systems, pp. 1201-1210 2018.
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, Julian Schritwieser,, Keith Ander- son, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab, CoRR, abs/1612.03801 2016. URL http://arxiv..om wrg. 1612 03801
Douglas Behrend, Jason Scofield, and Erica E Kleinknecht. Beyond fast mapping: Young chil- dren's s extensions of novel words and novel facts. Developmental Psychology, 37(5);698, 2001.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Susan Carey and Elsa Bartletr. Acquiring single new word. Papers and Reports Child Language Development, 1978.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-Fes 3D model repository. arXiv preprint 1521::1512.030122.200
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Ra- jagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978-2988. 2019.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RL2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint orXiv:1611.02777 2016.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted learner architectures. arXiv preprint (00X:v:8002.0111 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine Model-agnostic meta-learning for fast adaptation of deep networks. arXiv preprint arXiv: /003.v::700.3340000 2017.
Meire Fortunato, Melissa Tan, Ryan Faulkner, Steven Hansen, Adrià Puigdomènech Badia, Gavin Buttimore, Charles Deck, Joel Z Leibo, and Charles Blundell. Generalization of reinforcement learners with working and episodic memory. In Advances in Neural Information Processing Systems, pp. 12469-122 12478, 2019.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwinska, Sergio Gómez Colmenarejo, Edward Grefenstette. Tiago Ramalho, John Agapiou, et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538 (7626):471-77,, 2016.

---

Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer. David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg. Denis Teplyashin, et al. Grounded lan- guage learning in simulated 3D world. arXiv preprint 1rXiv 06551, 2017.
Felix Hill, Stephen Clark, Karl Moritz Hermann, and Phil Blunsom. Understanding early word learning in situated artificial agents. Proceedings of CogSci, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
Timothy P Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In ICLR (Poster), 2016.
James L McClelland, Bruce L McNaughton, and Randall C C''Reilly, Why there are complementary learning systems the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
James McClelland, Felix Hill, Maja Rudolph, Jason Baldridge, and Hinrich Schütze. Extending machine language models toward human-level language understanding. PNAS (to appear), 2020.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, 2016.
A. Emin Orhan, Vaibhav V. Gupta, and Brenden M. Lake. Self-supervised learning through the eyes of a child, 2020.
Allan Paivio. Mental imagery in associative learning and memory. Psychological review, 76(3):241, 1969.
Emilio Parisotto, H. Francis Song, Jack W. Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant M. Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, Matthew M. Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing transformers for reinforcement learn- ing, 2019.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with eemmmory-augmnnee neural networks. In International conference machine learn- ing. pp. 1842-1850, 2016.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learn- ing. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 4077-4087. Curran Associates, Inc., 2017. URL http:://ppppt 69g6-prototyppccl--eetttttttteee wew.callnetwwwss.... for- few shot learning. pdf
nips pss.p............ cc/paper/
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 3630-3638. Curran Associates, Inc., 2016. URL bttp:://pappp http://ppprranssscccc.cc.c nwpp.cc/pappp... n soee leannttterrnggppp..ee
Haley Vlach and Catherine M Sandhofer. Fast mapping across time: Memory processes support children's retention of learned words. Frontiers in psychology, 3:46, 2012.
Xin Wang, Qiuyuan Huang. Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang. and Lei Zhang. Reinforeed cross moos- matching and self-supervised imita- tion learning for uision-languse navigation. In Proceedings of the IEÉE Conference on Computer Vision and Pattern Recognition, pp. 6629-66388 2019.
10

---

Sandra R Waxman and Amy E Booth. Principles that are invoked in the acquisition of words, but not facts. Cognition, 77(2)BB33 B43, 2000.
Greg Wayne, Chia-Chun Hung, David Amos, Mehdi Mirza, Arun Ahuja, Agnieszka Grabska- Barwinska, Jack Rae, Piotr Mirowski, Joel Z Leibo, Adam Santoro, al. Unsupervised predictive memory in a goal-directed agent. arXiv preprint 1803.10003.0060 2018.
Jason Weston, Sumit Chopra, and Antoine Bordes.
Memory networks. arXiv preprint
arXiv:1410.3916, 2014.
Wenpeng Yin. Meta-learning for few-shot natural language processing: A survey, arXiv preprint arXiv:2007.09604, 2020.

---

A APPENDICES
A.1 COMPARING TRANSFORMERXL TO DCEM WHEN MEMORY IS LIMITED
Both the TransformerX and DCEM/DNC agents have a hyperparameter that determines the effec- tive size of their explicit working memory. In a vanilla Transformer it determines the size of the window of timesteps that the network can be applied for each forward (and backward) pass. To give models some chance of passing information beyond this hard constraint, TransformerXL. ar- chitecture conditions each forward pass also on representations computed the previous window, which establishes a form of recurrence over time from window window. In our original exper- iments, this mechanism was not tested, because we set the window size to 1024 timesteps, which is longer than most episodes of the fast-mapping task, which are typically 80- 10-120 timesteps for well-trained agent.
To examine the performance of the TransformerXL in cases where it is required to pass information across context windows, we reduced the size of the window. For a fair comparison, we similarly reduced the equivalent parameter (the capacity in rows in the FIFO external memory) for the DCEM agent. As shown in Figure 7 for cases where the memory window size (or buffer) is reduced to (100) or below (50, 20) the normal episode length, the DCEM performs better than the TransformerXL agent. This suggests that the TransformerXL has difficulty making the necessary (visual and lin- guistic) information available to policy head when that information must be passed between context windows. Surprisingly, the DCEM was able to learn the tasks efficiently with a memory size of 50, which suggests that must exploit its LSTM controller retain sufficient information when its external memory begins to overflow. Both architectures fail when the memory size is reduced to 20, that case a DCEM agent can in fact learn the optimal policy a simple heuristic for selective writing, described below, employed. This highlights advantage of explicitly read-write external memories; information can managed via the reading or the writing process. not immediately obvious how the same strategy could be applied with window-basee memory architectecture like TransformerXL.
Memory size 20
Memory size 50
Memory size 100
1.9
0.6
0.4
02
60
0.3
0.0
60
02
34
0.6
0.8
0.3
0.2
0.5
06
0.5
0.2
0.1
0.6
DCEM
DCEM Selective wriling
TransformerXL
Figure 7: Training success comparison between DCEM, DCEM with selective writing and Trans- formerXL for different sizes of memory-buffer (DCEM) or window (TransformerXL).
A.1.1 SELECTIVE WRITING HEURISTIC
One advantage of explicit external (read-write) memories is that the flow of information to the agent's policy can be influenced by the writing process as well as the reading function. To ver- ify this fact, implemented simple non-parametric heuristic writing condition in the DCEM architecture, whereby observations are written to the external memory when there is a change to the observation in the language channel. This heuristic aligns with the principle of dual-coding exploited elsewhere in the paper: while visual observations change continuously every timestep, changes to observed language are events that might signal some important change in the environment.
More formally, our heuristic relies on a window size parameter w that we set to 3 in all cases. For observation 2; {vz, 4 }, language change indicator T E N is set Tn
12

---

Novel heldout objects
Shapenet category extension
Accurey
Frames read mamory query
Frames per mamory query
Train
Test holdout
Figure 8: Training and test accuracy two types of generalization tasks for agents that read different numbers frames from their memory per query.
t, 4-11
if = 4-1 otherwise.
Then, the content C written memory at
[v ift 4, w 0.
Ct =
otherwise,
where, before, Vz, the agent embeddings of the visual and linguistic observations. Thus, memories are written for w timesteps proceeding a change in the language observation.
A.2 ROLE OF TEMPORAL ASPECT IN GENERALIZATION
In seeking to understand the mechanisms that support the generalization effects reported in the main paper, we found that the parameter lt was an important factor, where the top-k memories are returned to the agent policy head per memory read. As the agent explores during the discovery phase of episodes, it writes multiple perspectives of the same object to memory. As shown in Figure g both its robustness to entirely novel objects (left) and its ability to extend categories from novel exemplars (right), as well as its ability to solve the training task, are enhanced when > 1; ie. when it determines which object visit in the instruction phase based memories written from more than one view of each object.
A.3 VERIFICATION IN DEEPMIND LAB
At a high level, the design of an episode is very similar to the default fast-mapping task in the Unity environment. The agent must move down corridor, bumping into (and collecting) three distinct objects. When an agent collects an object it is immediately presented with the (episode-specific) name for that object. After passing three objects, the corridor opens into room containing two of the three objects found in the corridor. Upon entering the room, the agent is presented with the name corresponding to one the two objects, and must bump into that object in order to receive reward of 1. As before, shaping reward of 0.1 is given as the agent collects each object in the corridor. Compared with the Unity environment, the DeepMind Lab action space smaller (8 vs. 46 actions), the objects are larger, and the agent has substantive way to interact with the objects (they disappear the moment the agent collides with them). Note also that the agent must choose between 2 (rather than 3) objects in the instruction phase, agent selecting objects random would achieve 50% accuracy.
To provide some the various agent architectures directly to this environment with no enviromment-specife tuning. As shown in Table 2 without any further tuning of the agent, observe similar pattern of results
of the robustness and generality of the effects observed thus far, applied
13

---

Tinn
Train. accuracy
Test
theble'
Architecture
(novel objects)
LSTM R
0.50 (0.02)
0.40 (0.06)
DNC + R
0.48 (0.04)
0.30 (0.15)
TransformerXL mem=100 R
0.70 (0.28)
0.55 (0.29)
DCEM mem=100- + R Random object selection
0.80 (0.26)
0.65 (0.32)
0.5
0.5
Table Left: Architectures compared on DeepMind Lab after 5e8 timesteps of training. Data show mean accuracy (S.D) across 5 seeds in each condition. mem: the agent's memory buffer size. R: with reconstruction loss. Right: Schematic of episode structure in the DeepMind Lab fast-binding tasks.
in DeepMind Lab as in the Unity room. As in that case, the Transformer and DCEM architectures performed best, with three and two seeds out of five (respectively) mastering the training task. As the Unity environment, we also observed above-chance ability to apply fast-mapping knowledge time.
zero-shot unseen objects
A.4 AGENT ARCHITECTURE DETAILS
A.4.1 ARCHITECTURE DIAGRAMS
Entropy loss
Reconstruction loss
V-trace loss
Baseline estimate
Policy
LSTM decoder
Transpose ResNet
Agent core
Memory
See figure
Latent embedding
Latent encoder
Language embedding
Image embedding
"This is a dax."
Language encoder
ResNet
Figure 9: Agent architecture. figure 10 for details the dual coding episodic memory compo- nent. Dashed lines correspond to connections across timesteps.
14

---

Memory buffer
'language'
'maage
3x, for heads
Self-attention
3x, for heads
aggregator
Compute similarity
Read-out per head
Query
Concatenate and project
Write new row after querying
Query network
Memory read-out
Language embedding
Image embedding
Latent encoder
Core state
Agent core
Figure 10: DCEM architecture. This corresponds to the 'memory' component in the agent archi- tecture, Figure 9 NB: the similarity computation and selection of nearest neighbours is replicated for each read head, but depicted here to avoid clutter. Dashed lines correspond connections across timesteps.
15

---

A.4.2
HYPERPARAMETERS
image width
96
image height
ResNet kernel size
3
convolutional layers per ResNet block
ResNet blocks
2,2,
ResNet strides between blocks
2,
ResNet number of channels
16, 32, 32
post-ResNet layer output size (visual embedding)
256
language encoder embedding size
32
language encoder attenttentoo key / query size
16
language encoder self attenttenion value size
16
language encoder output size (instruction embedding)
32
language decoder hidden size
32
number of memory read heads
memory aggregation self atlf-attention key / query size
256
memory aggregation self eelf-attention value size
256
latent representation size
256
core LSTM hidden size
512
policy latent size
256
value latent size
256
policy cost
0.1
entropy cost
10 10-4
reconstruction cost
1.0
return cost
discount factor
0.95
unroll length
128
batch size
64
Adam learning rate
10
Adam 31
Adam B3
0.95
Adam
x 10-8
Table 3:
Agent hyperparameters (independent of specific architecture). The return cost (not dis- cussed in the main text) is used to weight the baseline estimate term in the V-trace loss.
A.4.3 INTRINSIC MOTIVATION ALGORITHM
The intrinsic reward is based the similarity (Euclidean distance) between the new embedding and the nearest neighbors already present in memory {e,). The average distance ğ used below is lifetime average of all p that is updated with every computation.
P: e
e
+
max (pi - Pmins 0) +
1/2
ki
+
1/s
if Smax otherwise
INGU
0
The computation introduces the constants c, e, Pinin and mmaxe and the number neighbours [{e;}] used for the similarity estimate. Table lists the values used.
16

---

number nearest neighbours smoothing constant for inverse distance surprise
Roi
10
10
similarity kernel smoothing constant
10
cluster distance cut-off
Punin
8 x 10
maximal similarity cut-off
maax
2.0
Table 4: Hyperparameters for NGU.
A.5 ENVIRONMENT DETAILS
A.5.1 UNITY ACTION SPACE
The following discrete actions (and strengths) are available to the agent in all experiments except for those in DeepMind Lab. The scalar strengths translated into force and torque (for rotations) by the environment engine.
Movement without grip
Fine grained movements without grip
Movement with grip
NOOP,
MOVE (OVEE.RIGT(0005)..
GRAB,
MOVE FORWARD(1).
MOVE RIGHT(-0.05).
GRAB + MOVE FORWARD(1),
MOVE FORWARDDD,,,,))
LOOK.DOWNO DOWN(0.03),,
GRAB + MOVE. FOWNFROORDD))))
MOVE (IGHTR)
LOOK DOWN(-0.03),
GRAB + MOVE (I
MOVE RIGHT()),,-,,,
LOOK RIGHT((.22,
GRAB + MOVE (IVE_RIGHT((I);
LOOK LOOK_RIGHTO))
BOOK,RIGHT(-O...
GRAB + CLOOK_RIGHTIH
(IGHT))GHT--),,
LOOK-RIGHTON
GRAB + LOOK (IOKTRIGHH(-D),
LOOK_DOWNG
LOOK.RIGHT((D005)
GRAB + LOOK_DOWN(C)
LOOK DOWN(-1),
GRAB + LOOK DOWN(-1),
Fine grained movments with grip
Object manipulation
Fine grained object manipulation
GRAB + MOVE.RIGHTON
GRAB + SSINS (IGHTT),
GRAB + PULL(0.5), GRAB + PULL(-0.5), PU.L( PULL(-0.5),
GRAB + MOVE RIGHT(-0.05)
GRAB SPIN RIGHT(-1),
GRAB + LOOK DOWN(0.03))
GRAB + SPIN UP(1),
GRAB + LLOOKOOON(CCAA DOWN(--003),
GRAB SPIN -UP-),,
GRAB +
GRAB
GRAB + NI..RIGGH(--...)
GRAB SPIN_FORWARDCC
GRAB +
GRAB PULL(I).
GRAB + LOOK RIGHT(-0.05),
GRAB + PULL(-1),
A.5.2 SHAPENET
ShapeNet contains 3D models of objects with wide range of complexity and quality. To guarantee that the models are recognizable and of high quality, we manually filtered the ShapeNet Sem dataset, selecting subset of everyday semantic classes, ensuring that the selected models had reasonable number of vertices, and reasonable size and weight dimensions. The selected classes (number of models in each class) were as follows, with a total of 1,437 models across 31 different classes:
armoire (31), bag (11), bed (65), book (47). bookcase (13), bottle (26), box (37), bunk bed (9), chair
(150), chest of drawers (133), coffee table (43), computer (6), floor lamp (68), glass (11), hammer
(15), keyboard (5), lamp (100), loudspeaker (37), microwave (16), monitor (58), mug (12), piano (11), plant (31), printer (22), rug (36), soda can (20), sofa (145), stool (25), table (181), ), vase (66),
wine bottle (7).
17


Reviews:
[{"review_id": "wYLEzB_xveU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Overall review\n\nThe authors use a 3D world to explore grounded language learning, in which an agent uses RL to combine novel word-learning with stably acquired meanings to successfully identify and manipulate objects.  They show that a novel, psychologically-inspired memory mechanism is more memory-efficient than Transformers (both of which outperform plain LSTMs) and that it exhibits surprisingly robust generalization to novel action-object pairs.  The results should be of interest to many working in grounded language / multimodal representation learning, and the experiments are thorough and well-motivated. \n\nPros:\n* Interesting environment for combining fast-mapping with stable language learning in a grounded task.\n* Novel memory architecture, shown to improve memory-efficiency.\n* Psychologically-motivated and thorough experimentation, demonstrating surprising level of generalization.\n\nCons:\n* A figure describing the three memory architectures in addition to / instead of the plain text could have helped compare / contrast them.\n* More analysis of how the agent uses its memory would be welcome.\n\n\n# Minor comments\n\n* 4.1: \"Unless stated otherwise, all experiments in this section involve the DCEM+Recons agent\".  I think this means with full memory size (1024) and not the smaller one (100), but it would help to clarify.\n\n* Figure 2: while training with 3 objects and testing with more does not work, training with 5 and testing with 8 works as well as training with 8 and testing with 8.  What do the authors make of this?  Is there some kind of \"threshold of diversity\" beyond which the agent can generalize to more objects?  Such a threshold idea also seems consistent with the results in Figure 3.\n\n* Fast category extension: these results show that if the agents are trained to pick up different exemplars of a category, they can do so in testing.  During training, there was one exemplar from each of three distinct categories.  I was curious if the authors experimented with relaxing that to environments where more than one exemplar of a category could be present, with the agent being rewarded for picking any of the correct exemplars.  \n\n* The environments the authors use seem like they would allow for testing of mutual exclusivity phenomena (see Ghandi and Lake 2020 [https://arxiv.org/abs/1906.10197] and references therein), by providing instructions with a novel word in a setting with one unseen object (or category).  I would be curious to see if their memory architecture does better with these phenomena than existing ones.\n\n\n# Typographic comments\n\n* p 3, \"the current visual embedding l_t\": l_t should be v_t\n\n* Appendices were referenced in the text, but not included in the uploaded PDF (even though I believe this was allowed at ICLR).  These appendices seem very helpful for complete model/experiment details.\n\n\n# Update\n\nI thank the authors for their thoughtful reply and for incorporating the feedback.  The additional information is most welcome, and so I maintain my score of 8 for the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "wY2zl9oSCzM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:** An agent following instructions in a grounded world is a core task in AI. This paper studies agent that accomplish this using memory-based architecture. This paper presents an argument for a multi-modal memory-architecture called DCEM whose key/queries and values are dependent on language and vision modalities respectively (or vice versa). An argument is made that this will be helpful for generalizing to novel language at test-time. Results are presented in a simple 3D domain containing several objects randomly sampled each time from a set of 30 objects. Task contain two types of instructions: \"pick up an object\" and \"place an object on another object\". Interaction proceeds in episodes where each episode contains a discovery phase where the agent learns the phrase associated with each object, and an instruction phase where the agent solves a given instruction. The proposed DCEM model outperforms baselines on various metrics and ablation. Importantly, it is shown that the DCEM can generalize to novel object names. \n\n**Strengths:**\n- The proposed architecture learns a near-optimal policy, requires few memory slots than baseline and can generalize to novel object names (fast mapping).\n- It is argued that the language and observation key-value embedding can be used to compute a simple intrinsic reward that incentivizes reaching states (language+vision) whose distance in the embedding space is far from what has been encountered. Results show that this intrinsic reward removes the need for any reward shaping that is required by baselines. This claim is a bit more speculative since comparison with other exploration methods has not been provided. Distance in the observation space is also not a robust choice for defining intrinsic reward since similar observations can be actually very different in the latent state. E.g., consider observations which always contain a certain noise (say due to camera), then the intrinsic reward would never go down to 0 and the agent would keep exploring everywhere. In another example, walking down a long uniformly decorated corridor can result in a very small change in the observation but the agent would be making progress in the latent state. Such a path would not be adequately incentivized.\n\n**Weakness:**\n- The paper only investigates generalization of object names. This is kind of a low hanging fruit for generalization in language space. More difficult generalization include different textual styles, different choice of verbs, etc.\n- DCEM architecture cannot generalize to more objects. \n- Experiments are a bit underwhelming. The task is quite simple involving pick and place, with very simple language capabilities and a few objects. There is a plethora of complex environments now. E.g., see \n\n1. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Task. Shridhar et al., 2020\n2. Touchdown: Natural Language Navigation and Spatial Reasoning in Visual Street Environments. Chen et al., 2019\n3. Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments. Anderson et al., 2018.\n\nI understand the advantage of control that comes with synthetic tasks, but replicating some results to more realistic tasks makes the claim stronger.\n\n**Questions**\n1. Is the language description changing with time? If not, wouldn't the language embedding l_t always be the same?\n2. By \"slot-based external memory\" you mean memory with a fixed size of key-value pairs updated in a First-In-First-Out style? Section 3 can use some more explanation or an architecture Figure. \n3. How do \"independent read heads\" work? Do they work similarly to \"heads\" in a transformer?\n4. Did you try to generalize verbs? E.g., use a different word to refer to \"Pick\" during testing.\n5. Does the background of the environment change across episodes (e.g., layout, colour of the wall, etc.)", "extra_scores": {}, "extra_text": {}}, {"review_id": "mF6XsNFqFU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of fast mapping: in a 3d environment, learning to explore an environment to learn a (new) mapping between objects and names, and then during a new phase, learning to pick up objects by their name ('pick up a dax'). This problem is challenging because the names change every time, so a model cannot simply learn connections between a shape and a static name (that persist longer than one episode).\n\nStrengths:\n* The paper defines the task and situates it within the context of a 3d environment, which could be useful for future work to build off of.\n* The paper compares several sequence models for this task, and finds that a Transformer-XL and a a 'Dual-Coding Episodic Memory (DCEM)' model both do well. The DCEM model seems novel to this reviewer at least, and it seems to function like a hybrid LSTM and Transformer. It seems that the DCEM model performs better than the transformer-XL model when the number of timesteps is limited (so the transformer-Xl model needs to rely on its recurrence mechanism to propagate information).\n* The paper tries several different generalization experiments involving more objects, new objects, a different evironment, and reward ablations. These generalization experiments provide us some insight as to what models learn in this setup.\n\nWeaknesses: \nAt least to this reviewer, there are no major weaknesses (except perhaps that the dataset is a bit toy, which isn't a concern to me). However...\n* It's not quite clear how much the memory bank is learning as opposed to the LSTM hidden state (which is also something mentioned in the supplemental). It would be interesting to learn a probe to measure when the model learns to assign an object to a name (if this is indeed something measurable by the memory bank alone).\n* The reference to the 'DeepMind Lab Suite' is worded in a way that comes somewhat close to breaking anonymity...\n\n\n-----\n\nupdate: thanks for the clarification points + figure 9 :) I still recommend acceptance (score of 8).", "extra_scores": {}, "extra_text": {}}, {"review_id": "lVV12NMif6w", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents experiments for acquiring words via fast-mapping in an embodied environment. The technical contribution is interesting and solid, but the experiments fail to address some important questions that are yet scoped by the claims of the paper (namely, that learning is being done -both- fast and slow, as per the title). Notably, the paper is really well-written and readable, and the experiments on novel category + novel instance recognition are really convincing specifically for fast-mapping (4.1).\n\nExperiments That Would Really Strengthen the Paper [I would love to hear your thoughts on these for rebuttal]:\n\n- Fast mapping test where some object categories use the canonical name across trials, to demonstrate that the agent can still associate descriptive + instructive (not just instructive) langauge with both the fast and slow paradigm\n\n- Experiments where the nonsense words are randomly generated characters in every episode of training and evaluation, to exactly tease out that the new word is being memorized and associated, not that the agent is memorizing that \"dax\" per episode changes meaning, which is itself knowledge about the word that renders it not truly \"nonsense\" or unseen. Currently, its meaning just becomes instead \"a variable assigned to a new object per trial that I need to memorize\".\n\nQuestions:\n\n- Table 1 random object selection / LSTM perform at .33; does this mean there's a control module that decides how to do the navigation? Is the agent guided at test time to each object? This confuses me a lot! I thought it had to learn to go visit the objects and then to go get the one later, which seems unlikely to happen from an LSTM alone or random actions.\n\n\nImprovements:\n\n- I think it would be worth spending some time in the intro motivating why the agent is embodied. What does embodiment get us for the hypothesis that a network can be trained to do fast mapping? We can imagine just giving a sequence of images and associated descriptions (e.g., \"this is a dax\"), followed by the command \"pick up the dax\" together with all object images and the task of classification. What challenges does embodiment present that are missing from that paradigm? E.g., seeing the objects from non-canonical angles, having to find them, etc. There's some recent push in the NLP community in general for embodiment, so I think it's worth spending time on as motivation in this paper. [There's like a little bit? 4.1 mentions that learning from multiple views of the object is important, but this is sort of the entirety of what embodiment is buying us here versus a static image paradigm, so it's worth emphasizing, probably.]\n\n\n- 4.3 is cool but doesn't -quite- do the fast-mapping test we'd like to see, which is that a word used consistently applies consistently. That would look more like including a few object categories whose names are used \"correctly\" across episodes, rather than one of the random names that are used inconsistently. This brings up a larger question of the consequences of using the same nonsense words for every trial, such that they can be memorized as being vacuous and in need of memorizing. Alternatively, the nonsense word could have been a random string to really test the memorization of an entirely novel token.\n\n\n\nNits:\n- Figure 5 caption says these are accuracy curves, but the y axis is \"reward\"; is this task accuracy or the intrinsic reward values from surprisal?\n- 4.4 why include this if there are no results in the paper for it? The constant references to the appendix are generally jarring, but this one is just sort of silly. It could be a single sentence maybe but not a whole subsection.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the capacity for neural language models to perform fast-mapping word acquisition using a proposed multimodal external memory architecture. Much work exists that shows that neural models are capable of following instructions whose meaning persists across episodes (i.e., slow-learning), however much less attention has been paid to instruction-following in a one-shot learning context. Using a simulated 3D navigation/manipulation domain, the paper shows that the proposed multimodal memory network is capable of both slow and one-shot word learning when trained via standard RL.

The submission was reviewed by four knowledgable referees, who read the author feedback and engaged in discussion with the authors. The paper is topical---one-shot language learning for instruction-following using neural models is of significant interest of-late. The reviewers agree that the proposed multimodal memory architecture is both interesting and technically solid. The reviewers raised concerns about the experimental evaluation and the role of embodiment. The author feedback together with discussion with reviewers were helpful in resolving some of these issues. However, the authors are encouraged to ensure that the paper clearly motivates the importance of embodiment to slow learning and fast-mapping, particularly given the large body of work in language acquisition in robotics, a truly embodied domain, which is notably missing from the related work discussion.
Decision: Accept (Spotlight)