Robust Instruction-Following in a Situated Agent via Transfer-Learning from Text
Authors: Felix Hill
Sona Mokra
Nathaniel Wong
Tim Harley
Venue:  (2020)
Keywords: agent
language
3D
simulation
policy
instruction
transfer

Abstract: Recent work has described neural-network-based agents that are trained to execute language-like commands in simulated worlds, as a step towards an intelligent agent or robot that can be instructed by human users. However, the instructions that such agents are trained to follow are typically generated from templates (by an environment simulator), and do not reflect the varied or ambiguous expressions used by real people. We address this issue by integrating language encoders that are pretrained on large text corpora into a situated, instruction-following agent. In a procedurally-randomized first-person 3D world, we first train agents to follow synthetic instructions requiring the identification, manipulation and relative positioning of visually-realistic objects models. We then show how these abilities can transfer to a context where humans provide instructions in natural language, but only when agents are endowed with language encoding components that were pretrained on text-data. We explore techniques for integrating text-trained and environment-trained components into an agent, observing clear advantages for the fully-contextual phrase representations computed by the well-known BERT model, and additional gains by integrating a self-attention operation optimized to adapt BERT's representations for the agent's tasks and environment. These  results bridge the gap between two successful strands of recent AI research: agent-centric behavior optimization and text-based representation learning. 

Under review as a conference paper at ICLR 2020
ROBUST INSTRUCTION IN A SITUATED AGENT VIA TRANSFER-LEARNING FROM TEXT
Anonymous authors Paper under double-blind review
ABSTRACT
Recent work has described neural-network-haaee agents that are trained to exe- cute language-like commands in simulated worlds, as a step towards an intelligent agent or robot that can be instructed by human users. However, the instructions that such agents are trained to follow are typically generated from templates (by an environment simulator), and do not reflect the varied or ambiguous expressions used by real people. We address this issue by integrating language encoders that are pretrained on large text corpora into a situated, In procedurally-randomiinn randomized first-person 3D world, we first train agents to fol- low synthetic instructions requiring the identification, manipulation and relative positioning of coualll-rralistii object models. We then show how these abilities can transfer to context where humans provide instructions in natural language, but only when agents are endowed with language encoding components that pretrained on text-data. We explore techniques for integrating text-trained and anvironment-trained components into an agent, observing clear advantages for the fully-contextual phrase representations computed by the well-known BERT model, and additional gains by integrating a self-attention operation optimized adapt BERT's s representations for the agent's 's tasks and environment. These results bridge the gap between two successful strands of recent Al research: agent-centric behavior optimization and text-based representation learning.
cnorucctionfollowiini agent,
INTRODUCTION
Developing machines that can follow natural human commands, particularly those pertaining to an environment shared by both machine and human, is a long-standing and elusive goal of Al (Wino- grad 1972). Recent work has applied end-to-end, ixpresentation-leamiii baased methods to this challenge, where a neural-network-basse agent is optimized to process language input, perceive its surroundings and execute appropriate movements jointly (Oh et al. 2017 Hermann et al., 2017 Chaplot et al.) 2018). End-to-end learning promises way to deal flexibly with the complexity of the physical, visual and linguistic world without relying on (potentially brittle) hand-crafted fea- tures, rules or policies. Nevertheless, the cost of this flexibility is the large number of environment interactions (samples) required for a randomly-inittalizzd network to learn behaviour policies from raw experience. To make the approach feasible, many studies thus employ a synthetic language that is generated on demand from templates by the environment simulator (Chevalier-Boise 2018 Jiang al. 2019 Yu al. 2018b]a), a). The studies that do combine both end-to-end learning with natural-langa language data do so in less realistic grid-like environments (Misra et al. 2017) or grant agents access to privileged global observations make learning more tractable (Misra al. 2018).
et al.)
Here, we take a different learning- based approach to instruction-follont commands. We train agent policies to respond synthetic, template- based language but also endow them with powerful language encoders that are pretrained natural language text. The synthetic instructions that our agents are trained to follow require mastery of 26 fine-grained motor actions in order to identify and manipulate visually-realistic models from the ShapeNet dataset (Chang et al.) 2013) in a 3D room. The visual realism of the world makes it possible to elicit diverse natural human ways of instructing and/or referring to things, and to study the agents' robustness to this diversity. Unsurprisingly. we find that agents that are trained on template- bbasdd commands do not cope well with the diversity and variation of natural keyboard-typed language. In contrast, when powerful
that is robust to human

---

Under review as a conference paper at ICLR 2020
Figure An episode of the putting task, seen from above (left) and from the agent's perspec- tive (right). In each episode of training evaluation, there are three randomly-coloured moveable objects, plus pink bed and a white tray. Initial positions all objects, doors, windows and the agent random. To succeed in evaluation trial, the agent must process instruction given by a human (e.g. "Put the musical keyboard on the bed", or "Drop A Casio on the bed"), identify the intended object of reference, move towards it and lift it, rotating if necessary, and then lower into the specified location. Our best-performing agent succeeds at this evaluation with 72% accuracy.
Pixel observations
Memory and partial obs.
Object interaction
Object manipulation
Visual realism
Natural language
Oh 2017
Hermann et al.
2017
Chaplot 2018
Misra et al.
2018
Wu et al. 2018
Jiang et al. 2019
This work
Table 1: Recent work instruction-folloiing in 3D environments (not exhaustive). By transfer- learning from simulation, our best-performing agents overcome unique combination of policy- optimization challenges (partial observability, memory, raw perception, object manipulation and relations) and can also interpret natural human commands.
pretrained language encoders are integrated into our agent, we find that agents can satisfy human instructions with substantially above-chance accuracy.
To better analyze the generalization that supports this robustness, we probe trained agents with specific modifications their training instructions. We find that methods based conventional (context-free) word embeddings support generalization that is driven by lexical similarity (executing lift vehicle when trained to lift car), but capture phrasal equivalence less well (failing at put a plate on the container when trained to put the dish on the tray). In con- trast, methods that integrate contextual word representations support both types of generalization. We also find that robustness to synonyms can often be improved when pretrained encoders are com- bined with additional modal) self-attention tuned the agent's environmental objectives. Ablation experiments highlight the role that WordPiece tokenization (Schuster & Nakajima, (012) plays in robustness to human instructions. This motivates the addition typo noise to our training pipeline, which further improves the accuracy of the agent's 's responses.
Overall, our principal contributions are the following:
We train agent that can both interpret human language commands and overcome similar range of behavioural and environmental challenges to state-of-the-a policy-learning approaches (Table
2

---

Under review as a conference paper at ICLR 2020
We develop the techniques of transfer-learning from text representation-leamiin model to embodied agent. Note that this is very different from transfer between language classification tasks (Collobert & Weston 2008 Devlin et al.] 2018); in evaluations the agent must interpret unfamiliar instructions zero-shot (i.e. without additional learning steps), and must realize complex language-conditional behaviours (decoding 50 appropriate actions) in spite of this uncertainty.
1.1 HIGH-LEVEL APPROACH
We experiment in a simulated room in the Unity game engine (Fig Importantly the realism of the visual assets in this room enable human observers to recognize the objects (and hence refer to them in natural ways). Our experimental pipeline then involves the following steps:
1. Define a (language-dependen)) task, including initial conditions for the simulated room, template-based process for generating synthetic language instructions and a reward func- tion for checking game states and exposing reward if the agent succeeds.
2. Train agents with different language encoders to solve the task when conditioned on the synthetic language produced by the environment generator.
3. Evaluate the trained agent 1000 episodes of the task with the synthetic instructions replaced with synonyms or free-form commands typed by human annotators.
ARCHITECTURES FOR INSTRUCTION AGENTS
Since the object of our study is instruction-following we consider an agent architecture that is as conventional as possible in aspects not related to language processing. It consists of standard components for computing visual representations of pixels, embedding text strings and predicting actions contingent these and memory of past inputs
Visual processing real-valued tensors, which are processed by a 3-layer residual convnet.
The visual observations received by the agent at each timestep are 96 x 72 x 3
Memory core The output of the visual processing is combined with language information accord- ing to a particular encoding strategy, as described below. In all conditions, some combination of vision and language input each timestep passes into LSTM memory core with hidden dim. 128.
Action and value prediction The state of the memory core at each timestep is passed through linear layer and softmax to compute a distribution over 26 actions. Independently, the memory state is passed to linear layer to yield a scalar value prediction.
Training algorithm The agent is trained using importance-weighted actor-critic algorithm with a central learner and distributed actors (Espeholt et al. 2018).
2.1 LANGUAGE ENCODING
The agent must process both string observations in the simulated environment, during training, and human instructions, also encoded as strings, during evaluation. Its representations of language must be combined with visual information to make decisions about how to act. We compare various different ways of achieving this encoding. For methods that transfer knowledge from unsupervised text-based learning, we take weights from the well-known BERT model (Devlin at. 2018), specif- ically the uncased BERT model made available by the authors
BERT + mean pool For a given input of u words, BERT BASE returns w comtext-dependent (sub)- word representations of 768 units. In this condition, mean pooling operation is applied over the w dimension to yield a single representation of dimension 768, which is concatenated with the flattened output of the visual processing module. This multi-modal representation is passed through a single layer MLP with tanh activation and output dimension 128 before entering the memory core of the agent. We apply the standard BERTBASE WordPiece vocabulary (of size 30,000). Note
Unity. http://unity3d..com
"See Appendix C for details not included here. 'Available https: / fhub. dev/goog h/bbertuuncaaedeeeeeeeee
12/1
3

---

Under review as a conference paper at ICLR 2020
that WordPiece encodes language in terms of subwords (a mix of characters, common word chunks, morphemes and words) rather than the word-level vocabulary applied in more traditional neural language models.
BERT + self-attention layer When applying BERT to text classification, performance can some- times be improved by fine-tuning the weights in the BERT encoder suit the task. Because of the large number of gradient updates required to learn a complex behaviour policy, fine-tuning the BERT weights in this way would cause substantial overfitting to the synthetic environment language. We therefore keep the BERT weights frozen, but experiment with an additional learned self-attention layer (Vaswani et al.) 2017) to create a language encoder whose bottom layers are pretrained and whose final layer optimizes these representations to the present environment or tasks. This addi- tional layer has 4 attention heads, and uses 64 dimensional key and value embeddings.
BERT+ cross-modal self-attention We also consider a cross-modal self-attention layer of the type suggested by e.g. Tsai al. (2019); Lu al. (2019), which in our application provides an explicit pathway for the agent to bind visual experience to specific (contextual) word representations. In this case, we treat each of the output channels of the visual processing network as word-like entities, by passing them through a linear layer of output size 768 to match the BERT output. The embeddings for all words and the visual channels are then processed with a single self-attention layer, whose parameters are again learned with all other agent parameters.
Pretrained (sub)-word embeddings Language classification experiments with BERT show the value of highly context-dependent (sub)-word representations, but transfer in that context is also pos- sible with more conventional ccontextinndependerr To measure the effect of this distinction, we consider a simpler encoder based on the (context dependent) input (sub-)word embeddings from BERT, which are also of dimension 768 4 Taking a mean of these language. We therefore process them with single-layer transformer with 4 attention heads. As in BERT + mean pool, this output is averaged, reduced by single-layer MLP (from 768 128 units) and passed to the agent's 's core memory.
word embeddings (Collobert & Weston 2008).
indeeenenn vectors would yield a word-order-invariann representation of
Typo noise Finally, in one condition we introduce typo noise (each key character was replaced with that of an adjacent character on a standard keyboard, with a probability of 0.01) to the synthetic language strings produced by the environment, allowing the agent to learn to compensate during training. No modifications were made to the evaluation stimuli. Much previous work applies typing noise for language classifier robustness; see e.g. (Pruthi et al., 2019) for a recent survey.
We compare to various basclines designed to isolate specific components of these encoders.
Random mean pool As a direct baseline for BERT mean pool, we consider an identical architecture but in which the contextual BERT embeddings are replaced by fixed random (subword-speciffe vectors (also of dimension 768). The weights in the rest of the agent network are trained as in other conditions. We note that random sentence vectors classification tasks (Wieting& & Kiela 2019).
competitive baseline many language
Word-level and WordPiece transformers In addition to pre-trained weights, potentially impor- tant aspect of encoders based on BERT is WordPiece tokenization, which can afford greater ability words with familiar morphemes than in a more conventional word-
to make sense of typos level encoder. To isolate the effect of tokenization from that of pretrained weights, we compared two further encoders. In one, we split all input strings by whitespace and hash each word string to a unique index representing an imput to a single-layer transformer with 4 attention heads and em- bedding size 768 (chosen match BERTBASE), the output of which is processed identically to the Random mean pool condition. We contrast this with an otherwise identical condition in which the WordPiece tokenization from BERT is applied rather than splitting on white space.
EXPERIMENTS
We experiment with a lifting task, which focuses on object identification, and a putting task which focuses on object relations and manipulation. In both tasks the locations of all objects and the
'These embeddings capture lexical similarity much like conventional word embeddings; a cosine metric produces Spearman correlation of 0.49 with human ratings from the Simlex-999 dataset (Hill et al. 2015).
4

---

Under review as a conference paper at ICLR 2020
initial position and orientation of the agent are chosen at random floating point scales, so it is highly unlikely that any two episodes are spatially identical (whether during training or eval- uation). The action set consists of movements (move-(Forward, back, left, right)), turns (turn-(upp (up, down, 1eft, ri ght and object pick-up and manipulation with 4 DoF. The place- ment of objects is assisted by visible vertical line. See Appendix Cla full description.
3.1 LIFTING WITH HUMAN INSTRUCTIONS
In an episode of the lifting task, the environment selects two movable objects at random from a global object set, and generates a language instruction of the form Lift a X, where X is the correct name for (exactly) one of those objects. To achieve reward, the agent must locate the X, lift it more than Im above the ground and keep it there for 5 time steps. If the reward function in the environment determines that these conditions are met, a positive reward of +1 is emitted and the episode ends. lifting takes place with the incorrect object, the episode also ends with reward 0. For global set of recognisable objects, we use the ShapeNel dataset (Chang et al. 2013), which contains over 12,000 3D rendered models. For performance reasons we discard models with more than 8,000 vertices or an OBJ file size greater than MB and consider only those tagged into a synset in the WordNet laxonomy. We use the name of the first lemma of that synset as a proxy for the name of the object. From these names, we selected 80 to for the environment, each referring to set of object models with a minimum of 12 exemplars. See appendix D for more details.
For all language encoding strategies described above, the agent was able to learn the task, and a well-trained policy completed episodes in an average of x 20 timesteps with accuracy around 90%. The failure of agents to reach per- fect performance on the training set is unim- portant for the present study; suspect that the agent's s comparatively small convolutional network fails perceive important distinctions between the more intricate ShapeNet models.
Template lang. (training)
Natural
Synonym
referring expression
Lift the indian flag
Lift flag
Lift
Lift a flag
banner
Lift the flag.
Lift pillows
Lift pillow
Lift cushion
Lift cushion
Lift a paper
Table Example training (left) and test instruc- tions in the lifting task.
Template lang. (training)
D.O.
LO. synonym Put a mug on the box
D.O. & LO.
synonym a cup on the tray
synonym Put cup the box
Natural instruction
mug on the tray
place mug in the bosket
Keep the cup in atub container
place the mug
put the coffee mug in the box
Put train on the hed
Put a locomotive on the bed
Put a train on the bunk
Put locomotive
Put the tractor on the bed Move the train toy onto the bed Place toyvehicle on the bed place the rail on ttee bed
the bunk
Table 3:
Example training (left) and test instructions in the putting task. Underlined words examples of synonyms, italics indicate entire phrases provided by human annotators.
We consider two evaluation settings. In the synonym evaluation, we ran the environment for 1,000 episodes with the noun in the environment template instruction replaced by synonym (Lift a X becomes Lift a X* where X R X*). The synonyms were provided by native English speaking subjects. In the natural referring expression evaluation, gave 40 annotators access to a room containing a single ShapeNet model via a crowd-soureing platform. We asked them to write down what they found in the room and then hit button that restarted the environment with new object in the room F] As illustrated in Table 2 unlike the synonyms, the natural referring expressions involve variation in articles well nouns pencil might become the pen), may include spelling mistakes or typos, can refer entirely incorrectly to the intended object (if the subject fails to recognize the ShapeNet model), but may also match the training instruction exactly. Moreover, unlike the synonym test, there are 30- - 40 natural referring expressions for each of the 80 environment nouns, from which we sample randomly when evaluating the agent (again on 1,000 evaluation espisodes).
"See Appendix B for the full list of synonyms and annotator instructions.
5

---

Under review as a conference paper at ICLR 2020
3.2 PUTTING OBJECTS ON OTHER OBJECTS WITH HUMAN INSTRUCTIONS
The strength of models like BERT is their ability to combine lexical representations into phrasal or sentence representations. To study this capacity in the context of instruction-folloinng we devised a putting task involving the verb 'to put', which, in the imperative (put the cup on the tray) takes two arguments, the direct object (D.O.) cup and the indirect object (I.O.) tray. In terms of the behaviours required, the putting task focuses on manipulation and object relations rather than object identification or reference. The environment was configured to begin each episode with three randomly- moveable objects and two larger immovable objects (a bed and tray), each randomly positioned in the room. In each episode of this task, the agent receives an instruction Put D.O. on the I.O., where D. 0. is any of the three moveable objects (chosen from global set of ten) and I.O. is either bed or tray. The environment checks whether an instance of D.D. D. is at rest (and not held by the agent) the episode. If the object D.O. is placed on something other than a I.O., or if another movable object is placed on the bed or the tray then again the episode ends immediately with reward 0|
top of the I.O., returning positive reward + 1 if so and ending
As before, we first trained all agents on the putting task with synthetic environment language instruc- tions. Training a policy on this task with reinforcement learning required bespoke task curriculum (see Appendix actions/timesteps. To gather the evaluation stimuli, we again crowd-sourced humans to provide both natural synonyms for each of the 12 objects in the global set for this task and, in this case, entirely free-form natural human instructions. To obtain natural instructions, we instantiated an environment with only one of the global set of moveable objects, coloured red, and one of either the bed or the tray, coloured white, and asked subjects to ask somehody to place the red object on top of the white object without mentioning their color. From these instructions, we defined four evaluations, illus- trated in Table 3 D.O. synonym, L.O. synonym and D.O. I.O. synonym, in which particular parts of the original template command were replaced with synonyms, and Natural instruction, the fully free-form human instruction, which can include orthographic errors and misidentified objects.
C for details); a well-trained policy completes each episode in an average of R 50
3.3 DISCUSSION OF RESULTS
Template language (training)
Natural referring expression
Model
Synonym
Random 'lifting' act Random--mbeddding + MP
0.5
0.50
0.50
0.86
0.49
0.48
Word-level Transf.
0.91
0.58
0.61
WordPiece Transf.
0.89
0.57
0.66
Word-level Transf + TN
0.86
0.62
0.70
WordPiece Transf. + TN
0.86
0.64
0.69
Word embeddings Transf.
0.89
0.62
0.64
BERT MP
0.93
0.77
0.76
BERT SA
0.91
0.70
0.70
BERT CMSA
0.88
0.68
0.67
BERT CMSA + TN
0.89
0.74
0.73
Multitask BERT + CMSA + TN
0.87
0.74
0.73
Table 4: Accuracy of different models training instructions, the synonym evaluation and the natural referring expression evaluation. MP: mean pool, SA: self-attention layer, CMSA: cross- modal self-attention layer, TN: typo-noise. Multitask: single agent trained on both 'lifting' and "putting' tasks. Scores show mean across 1,000 episodes (with instructions randomly chosen by the environment generator).
The accuracies for both lifting tasks presented in Table 4 and for putting tasks in Table The results reveal the following main effects of language encoding on model performance:
Substantial transfer from text requires contextual encoders Agents with weights that are pre- trained on text data exhibit substantially higher accuracy on both the lifting and the putting tasks. This effect is greatest in the more focused synonym evaluations, but also holds for the the free- form human instructions. A small transfer effect can be seen by comparing the word embeddings Transformer condition (62% accuracy on the synonym evaluation, lifting task and 57% accuracy
'We found that ending the episode in such cases made learning much faster.

---

Under review as a conference paper at ICLR 2020
Template lang (training)
D.O.
1.0.
D.O. T.O.
Natural
Model
synonym
synonym
synonym
instruction
Random 'putting' act Random-embedding MP Word-level Transf.
0.17
0.17
0.17
0.17
0.17
0.99
0.59
0.22
0.15
0.39
0.99
0.48
0.09
0.03
0.36
WordPiece Transf.
0.98
0.30
0.24
0.10
0.42
Word-level Transf. + TN
0.97
0.61
0.27
0.17
0.45
WordPiece Transf. TN
0.97
0.36
0.36
0.13
0,56
Word embeddings Transf.
0.99
0.57
0.47
0.36
0.55
BERT + MP
0.99
0.94
0.74
0.57
0.49
BERT + SA
0.99
0.96
0.69
0.56
0.54
BERT CMSA
0.99
0.93
0.67
0.47
0.43
BERT + CMSA
0.98
0.88
0.79
0.70
0.70
Multitask BERT + CMSA + TN
0.98
0.96
0.75
0.68
0.66
Table 5: Accuracy of different models on the putting task when different parts of instructions of the form "Put [D.0.] on the [I [I. o.]" are replaced with synonyms. Underlined words in model names indicate pre-trained weights from text-based training. MP: mean pool, SA: self-attention layer, CMSA: cross-modal self-attention layer, TN: typo-noise. Multitask: single agent trained on both 'lifting' and 'putting'' tasks. The evaluation task covers episodes in which [I.O.] is either 'bed' or 'tray'. Scores show mean across 1,000 random episodes.
on the D.O. synonym evaluation, putting task) with the WordPiece Transformer (57% and 20%).. However, overall the transfer effect is much stronger in the case of the full context-dependent BERT representations. On the same two evaluations, BERT + mean pool achieves 77% and 94% accuracy respectively. The gains from transferring via BERT representations vs. just (sub)word embeddings are greatest for the (longer) putting instructions than for the lifting instructions, and greatest of all in the D.O & L.O. synonym evaluation. These are cases where one would expect the marginal value of powerful sentential (rather than just lexical) representations should be greatest.
Tuning via self-attention layers (with typo noise) helps Interestingly, we find that tuned self-attention layers do not improve generaliza- tion performance over using BERT and mean pooling. This may be simply because the addi- tional layers cause degree of overfitting to the template environment language during training. However, typo noise mitigates this issue, that the strongest evaluation performance on the putting tasks overall is observed with com- bination of a tuned cross-modal self-attention layer and typo-noise training Indeed, the value of typo noise as a regularizer can be seen by the fact that it improves the robustness of agents with tuned self-attention layers even in the synonym evaluations (for both lifting and putting), which do not involve any typos. Thus, the BERT + CMSA + TN model performs bet- ter than others on two of the three synonym evaluations in the putting task.
dista
Dei wents attentinn
bel wonds to rid uttento
specry mh rensom atlention
frmquencyy
cosine distance
Figure 2: Sell-attention layers learn to pull-apart both training nouns and their synonyms the agent learns the putting task with synthetic environment language.
One way in which the appropriately-tunel self-attention layer might make the agent more robust to synonyms in this case is by spreading out task-relevant object-nouns in the agent's language representation space (leaving those words closer synonyms than to potential confounding words). The degree to which this happens when the object-nouns and their synonymns in our environments passed first through BERT and then through (BERT SA) agent-tuned self-attention layer (compared to passing through the same layer but with random weights) is shown in Fig
WordPiece tokenization adds robustness In the two evaluations involving natural language in- structions from humans, a comparison of Word-level Transformer and WordPiece Transformer shows that robustness is obtained simply from WordPiece encoding, which in turn must play
See Appendix A to compare BERT-based architectures with and without typo-noise.

---

Under review as a conference paper at ICLR 2020
some part in all BERT-based conditions (e.g. improving from 61% to 66% in the natural refer- ring expression evaluation and from 36% to 42% in the natural instruction putting evaluation). As mentioned above, BERT-based agents with WordPiece encoding are particularly robust to human instructions when trained with typo noise, and this is most effective when combined with tuned self-attention layers. This makes intuitive sense, as learnable self-attention layer should provide the agent with more flexibility to learn to correct for typos during training. Indeed, on the natural instruction evaluation of the putting task, where typos or spelling errors are most common, the cross- modal attention agent trained with typo noise achieves 70% accuracy. Note that 100% accuracy on this evaluation may be impossible, even for humans, because visual ambiguity or human error mean that instructions can sometimes refer in entirely mistaken ways to the objects in the room.
Multitask learning is possible The performance of an agent trained on both the putting and lifting tasks is not substantially lower than agents that are specialized to each of the tasks individually.
RELATED WORK
Most closely related to our work is an experiment by Chan et al. (2019). who showed how an agent trained with InferLite sentence representations (Kiros & Chan 2018) can be robust to synonym placements in template instructions. The task itself involves object identification in the VizDoom environment (Kempka al. 2016), which requires only 3 motor actions. Our work develops this insight substantially, applying a similar approach to a visually-realistic environment requiring fine- grained object manipulation, integrating context-dependent pre-trained models with subword tok- enization (BERT), analysing architectures and training strategies for integrating such models and extending from synonym replacements free-form instructions typed by humans.
Much recent work applies deep learning and policy optimization in end-to-end approaches to learn- ing instruction following (Chaplot et al. 20188] Oh et al. 2017 Bahdanau al.) 2018 Chevalier Boisvert et al. [2018] Yu et 2208 a Jiang et al. 2019). As noted in the introduction, these studies do not typically involve natural language. Our work differs from studies that do (Misra et al. 2017 2018) in several ways. First, consider generalization and robustness (rather than performance when trained on a set of natural language commands), Second, we apply our method to learn much more complex policies requiring object manipulation and positioning and mastery of fine- aee-graieed action sets. Our approach makes learning such policies possible because we combine (potentially infinite) template-bar based RL training with text-based pretraining and transfer.
A limitation of all studies above the reliance on simulation. Both object identification and manip- ulation are likely far harder in reality, and it remains to be seen whether our methods scale robot language understanding (Tellex et al.) 2011 2012] Walter et al.) 2014). See also (Anderson et al., 2018 Wang et al. 2019) for recent improvements visual realism in simulated environments.
Finally, there is a long history of building in knowledge about the structure of language and/or its environment into instruction-following systems rather than learning it end-to-end. In Winograd (1972)'s 's SHRDLU, syntactic modules parsed the language input into a logical form, and hand- written rules were applied to connect such forms to the environment. More recent pipeline-based approaches use learning algorithms to map language to program that can then interface with a planner (Chen & Mooney [201]] Matuszek et al. 2013) Wang et al., 20166), and/or a controller, both of which may have priviledged information about how the world connects the program. It is likely that pretrained language encoders could add robustness to parts of these approaches, much as they do here. Our focus on end-to-end learning. however, is motivated by the intuition that it may eventually scale or adapt more flexibly to arbitrary environments or problems than pipeline approaches.
5 CONCLUSIONS
In this work, we have developed an agent that can follow natural human instructions requiring the identification, manipulation and positioning of visually-realistic assets. Our method relies on zero- shot transfer from template language instructions to those given by human annotators when asked to refer and instruct in natural ways. The results show that, with powerful pretrained language en- coders, this transfer effect is sufficiently strong to permit decoding of complex language-dependes motor behaviours, despite the shift in distribution of the agent's 's input. More generally, we hope that this contribution serves to bring research on text-based and situated language learning closer
8

---

Under review as a conference paper at ICLR 2020
together. To facilitate research in this direction, provide our dataset of natural instructions and referring expressions aligned to ShapeNet models.
We have explored only tiny fraction of the many possible ways to transfer knowledge from text corpus to a situated environment. For instance, our approach involves freezing rather than fine-tuning the BERT encoder weights to our desired behaviour policy, to avoid overfitting. but techniques such as knowledge distillation (Hinton et al.; (0115) could point to more elegant ways to learn jointly from text and environmental experiences. Moreover, we have focused on BERT, but improvements may be possible by applying alternative general-purpose language encoders, such as GPT-2 (Radford al. 2019). Roberta (Liu al. 2019) and Transformer XL (Dai al. 2019).
REFERENCES
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton den Hengel. Vision-and aasion-and-aanguag language navigation: Interpreting visually-grounde navigation instructions in real environments. Proceedings of the IEEE Con- ference Computer Vision and Pattern Recognition, pp. 3674-3683, 2018.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. Learning to understand goal specifications by modelling reward. arXiv preprint arXiv: 01946, 2018.
Harris Chan, Yuhuai Wu, Jamie Kiros, Sanja Fidler, and Jimmy Ba. Actree: Augmenting experience via teacher's advice for multi- goal reinforcement learning. arXiv preprint urXiv:1902.044 04566, 2019.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Ra- jagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
David L Chen and Raymond Mooney. Learning to interpret natural language navigation instruc- tions from observations. In Twenty-Fifth AAAI Conference on Artificial Intelligence, 2011.
Maxime Chevalier-Boisverr, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Babyai: First steps towards grounded language learning with a human in the loop. arXiv preprint arXiv:1810.082/1 2018.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160-167. ACM, 2008
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv: 002860, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2018.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv: 080..0056..1 2018.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded lan- guage learning in simulated 3d world. arXiv preprint arXiv:1706.06555 2017.
Felix Hill, Roi Reichart, and Anna Korhonen. Simlex-999 999: Evaluating semantic models with (gen- uine) similarity estimation. Computational Linguistics, 41(4):665-695, 2015.

---

Under review as a conference paper at ICLR 2020
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv
preprint arXiv:1503.0253: 2015.
Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language as an abstraction for hier- archical deep reinforcement learning. arXiv preprint arXiv: 07343, 2019.
Michal Kempka, Marek Wydmuch, Grzegorz Rune, Jakub Toczek, and Wojciech Jaškowski. Viz- doom: A doom-based ai research platform for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-8, IEEE, 2016.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Jamie Kiros and William Chan. Inferlite: Simple universal sentence representations from natural language inference data. In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 4868-4874, 2018.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907.19907.11999 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task- visiolin- guistic representations for vision-and-language tasks. arXiv preprint 190x:::90.002263 2019.
Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. Learning to parse natural language commands to a robot control system. In Experimental Robotics, pp. 405-415. Springer, 2013.
Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. arXiv preprint 17Xiv:1704.08795, 2017.
Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi Mapping instructions actions in 3d environments with visual goal prediction. arXiv preprint arXiv:1809.00786 2018.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference machine learning, pp. 1928-19999999 1937, 2016.
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Learning-Volume Volume 70, pp. 2661-2670. JMLR. org, 2017.
Danish Pruthi, Bhuwan Dhingra, and Zachary C Lipton. Combating adversarial misspellings with robust word recognition. arXiv preprint arXiv: 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Mike Schuster and Kaisuke Nakajima. Japanese and korean voice search. In 2012 IEEE Interna- tional Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5159-5152. IEEE, 2012.
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. Approaching the symbol grounding problem with probabilistic graphi- cal models. Al magazine, 32(4):64-76, 2011.
Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, and Nicholas Roy. probabilistic approach for enabling robots to acquire information from human partners using language, 2012.
Yao-Hung Hubert Tsai. Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis- Mouis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. arXiv preprint arXiv: 1000:00000...5.. 2019.
10

---

Under review as a conference paper at ICLR 2020
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998 5998-6008, 2017.
Matthew R Walter, Sachithra Hemachandra, Bianca Homberg, Stefanie Tellex, and Seth Teller. A framework for learning semantic maps from grounded natural language descriptions. The Inter- national Journal of Robotics Research, 3379);1167-1190, 2014.
Sida Wang, Perey Liang, and Christopher D Manning. Learning language games through interac- tion. arXiv preprint arXiv: (60666004442 2016.
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imita- tion learning for vision-language navigation. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 6629-6638, 2019.
John Wieting and Douwe Kiela. No training required: Exploring random encoders for sentence classification, arXiv preprint arXiv: 10444, 2019.
Terry Winograd. Understanding natural language. Cognitive psychology, 3(1):1-191, 1972.
Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with realistic and rich 3d environment. arXiv preprint arXiv:1801.022001 2018.
Haonan Yu, Xiaochen Lian, Haichao Zhang, and Wei Xu. Guided feature transformation (gft): A neural language grounding module for embodied agents. arXiv preprint 1xXi::805.00829, 2018a.
Haonan Yu, Haichao Zhang, and Wei Xu. Interactive grounded language acquisition and general- 2018b.
ization in 2d world. arXiv preprint

---

Under review as a conference paper at ICLR 2020
A FULL EXPERIMENTAL RESULTS
The results presented in charts here are those shown in the tables in the paper, with two additional conditions involving typo noise.
X", changed
1,0
Model
01. Randon emboodiing
0.9
Wordkivell translomerr
Word-ivel tranalomer-
04. Wardpiece Innalomee
06. Wordipecc transfomer TN
06. BERT
07. (Sub-IWord cmbs transtomer
08, BERT MP -N
Mean of plct valus, fteed
05, BERT SA
T0, BERT CMSA
0.8
11. BERT SA TN
12. RFRT CMSA + TN
13, Mult-Hask DEAT MP PSA
0.5
14, Multi task BERT BA TN
15. MuTii task BEAT CMBA + TN
0.2
0.1
Figure 3: Lifting task, synonym evaluation. Left: accuracy of different agents over 1,000 episodes when no change is made the environment template instruction. Right: accuracy of different agents when a synonym is introduced. Dotted line indicates score of an agent that carries out the correct behaviour but with random objects from the room.
Litting tasi:: natural reforring expression
1.0
Model
01. Random embedding MP
0.5
Wond-level tranefomme
C8. Word-len level transfomor TN
04. Word piece transfomee
C5 Word piece hanaforme TN
06 BEAT MP
07. (Sus-Ward embe tra transformer
07-
OR BETT MM TN
Mean of piot value, fieed
08 BERT SA
10 BERT CMRSA
0.6
11, BERT SA TN
12. BERT CMBA TN
15. Mull-tooi took BERT MP - BA
0.0
14. Mull-itek BERT SA TN
15. Mull-ttaik BERT CMSA TN
0.8
0.2
Figure 4: Lifting task, natural referring expression evaluation. Performance of agents on 1 1,000 evaluation episodes when part of the environment template lifting instruction is replaced by a natural referring expression from human annotators. Dotted line indicates score of an agent that carries out the correct behaviour but with random objects from the room.
12

---

Under review as a conference paper at ICLR 2020
a combinaions
original nngght
Kodgral Y syronym
Xapnarym, origita
Kynneny,, syronym
00-
Model
Thndoo undaekding
07: Word-ewe trandtrre
Wordlive tonutto
08-
SERT MP
07. (oubo (Wood snct Iurolorrer
MP
Mean plot Valae Ited
SEAT CMSA
0.-
BERT SA TN
GEAT CMAA
Muul BEAT CMSA
0.4
02
00
Figure 5: Putting task, synonym evaluation. Performance of agents 1,000 evaluation episodes when (left) no change is made to the template environment instruction and (rightmost three) different parts of the template environment instruction replaced with synonyms. Dotted line indicates score an agent that carries out the correct behaviour but with random objects from the room.
Putting taok: natural human instruction
Model
0.6
Word lovel sansoorme
Word
0.8
(Sub-WWord
transformor
0.7
0.6
Mull-task
t..
0.6
Mull-task BERT -S
15. Mull-task BERT CMSA
of
0.4
0.1
00
Figure 6: Putting task, natural instruction evaluation. Performance of agents on 1,000 evaluation episodes when the full environment template putting instruction is replaced by natural instruction from human annotators. Dotted line indicates score of an agent that carries out the correct behaviour but with random objects from the room.
13

---

Under review as a conference paper at ICLR 2020
B INSTRUCTIONS TO HUMAN ANNOTATORS
Human annotators keyboard and mouse control player in the environment simulator, as is standard in first-person video games. The annotators were given the following instructions as part of the each annotation lask.
B.1 NATURAL REFERRING EXPRESSIONS
This is task called Name The Object. You will find yourself in room containing single object. Please move around the room get a good view of the object. When you know what the object is:
1. Hit Enter
2. Type the name of the object
3. Hit Enter again
Examples of good responses:
1. A kettle
2. Some trousers
3. A pair of scissors
4. A tennis ball
Please don't describe the object. Just write down what you see, with an article like or 'some' appropriate.
Example of bad responses
1. A brown ball
2. A large piano with long black legs
3. A small thing with lumps the side
You should not need more than 4-5 different words, and most objects will require just or 2 words to name.
Sometimes, you might be unsure what the object is. If that's 's the case, just make your best guess.
B.2 FULL HUMAN INSTRUCTIONS
This is a task called Ask put. You will find yourself in room. Your job is imagine giving an instruction to somebody else so that that person puts the red object on to the white object. Move around the room get a good look at what the objects are. When you ready to give your instruction:
Hit Enter
2. Type your instruction
3. Hit Enter again
Examples of good instructions:
1. Place the cup onto the table
2. Put the ball on the plate
3. Move the pencil onto the box
14

---

Under review as a conference paper at ICLR 2020
Words to avoid Your instruction must not contain words for colours other properties of the object. Please do not use the words red, white, scarlet, dark, large etc. in your instruction. Instead, refer to objects by their name as you recognise them. If you don't recognise what object is, just make your best guess.
Examples of bad instructions:
1. Put the red thing on the table
2. Put the large object on the small object
3. Put the small round thing on the chest
Keep your language varied. Try to use various ways to express your instruction different episodes to keep things interesting.
C AGENT ARCHITECTURE AND TRAINING DETAILS
To process visual input, the agent uses residual convolutional network with 64, 64, 32 channels in the first, second and third layers respectively and 2 residual blocks in each layer.
As the agent learns, actors carry replicas the latest learner network, interact with the environment independently and send trajectories (observations and agent actions) back to a central learner. The learning algorithm (Espeholt et al., 2018) modifies the learner weights optimize an actor-critic objective, with updates importance-weighinn
to correct for differences between the actor policy and the current state of the learner policy. We used a learning rate of 0.0005, a learner batch size of 32, an agent unroll length of 50, a discounting factor of 0.99, epsilon (epsilon-greed) policy) of le and an entropy cost of 0.0003 (Mnih et al. 2016). We use an Adam optimizer (Kingma & Ba 2014) with 31 0.90 and B2 = 0.95.
In order to train the agent on the putting task, it was necessary to combine episodes of the task itself with episodes of simpler tasks, in form of curriculum (although learning in parallel all tasks). In particular, we trained it concurrently on lifting task that involved the same moveable objects the putting task (so that the agent could receive signal about their names without relying on a complete act of putting). We also found it beneficial to add put-near task to the curriculum, where the instructions were of the form Put a cup near the tray rather than Put cup on the tray, and a reward was emitted if the agent moved the cup short distance from the tray and placed on the ground.
In the lifting task, training the agent requires approximately 200 million frames of experience (ap- proximately million episodes), which takes about 24 hours with 250 actors on GPU. the putting task, training in each condition was stopped after 30 million episodes, approximately three days of training.
Body movement actions
Movement and grip actions
Object manipulation
NOOP
GRAB
GRAB SPIN OBJECT_RIGHT
MOVE FORWARD
GRAB + MOVE FORWARD
GRAB SPIN OBJECT LEFT
MOVE BACKWARD
GRAB MOVE BACKWARD
GRAB SPIN OBJECT UP
MOVE.RIGHT
GRAB + MOVE MOVE.RIGHT
GRAB SPIN OBJECT DOWN
MOVE LEFT
GRAB MOVE LEFT
GRAB SPIN OBJECT FORWARD
LOOK RIGHT
GRAB L.OK_RIGHT
GRAB
BOOKLEFFT
GRAB LOOK_LEFT
GRAB PUSWWO
LOOK UP
GRAB LOOK UP
GRAB PULL OBJECT CLOSE
LOOK DOWN
GRAB + LOOK DOWN
D FURTHER ENVIRONMENT DETAILS
For all experiments, the environment is a Unity room of dimension 4m 4m. The walls, floor and ceiling are always the same color, but we add a window and door (positioned randomly per episode) give some sense of absolute location to the agent.
15

---

Under review as a conference paper at ICLR 2020
When importing ShapeNet models, use the scaling provided in the metadata, unless it it is very small (less than 0,000001), in which case we interpret the coordinates in the OBJ file as meters. All objects have rigid bodies with collision meshes generated using Unity's 's built in MeshCollider, with convex set to true. The masses all movable objects are set to kg. so that our avatar has enough strength to pick all of them up (except for beds and trays, which made kinetic).
When selecting ShapeNet models, for performance reasons we discarded all models with a vertex count higher than 8000, and OBJ file size greater than MB. The native ShapeNet category names are often not natural everyday names (for instance, they can be highly specific, like "dual shock analog controller"). To mitigate this, we used ShapeNet's WordNet tags, grouping models into categories according to WordNet synsets and assigning the name of the first synset lemma to the category.
Because depth- perception is challenging without binocular vision, the agent is assisted in manip- ulation by visual guide (bottom-right) that highlights objects within grasping range and drops vertical line from held objects.
Full lists of objects and synonyms are on the following pages.
16

---

Under review as a conference paper at ICLR 2020
manne
synonym
unique modelss
example shapenel model id
wordiel synsel
chest drawers
cupboard
241
Tae
n3018908
table
deck
199
n4386330
chair
scat
167
6bcabd23a8 Bdtaaaddddddd000oeeeeeeeaad
n3005231
158
n42636300
television receiver
aerial
147
n4413042
lamp
light
138
n3641940
desk
bench
98
n3184367
box
tin
81
n2886585
flute
78
4
n4529463
bed
bank
73
n2821967
lamp
lamp
n33719955
cabinet
cuphourd
68
150246336/55 1e1166336635114hh511111 hato:h11011000666:555550300000
n2936496
magazine
c869
103 003
n6422547
pencil
67
839923al
n3914325
laptop
computer
66
n3648 120
monitor
n3787725
coffee table
table
53
n3006991
picture
image
n3937282
bench
chair
52
c6 2 hwlta//55511111107555050044e
n2832068
painting
portrait
n3882197
rug
44
shelf
surface
43
482F3284 1177:3771117777hhhh 2ecn
n4 14197095
switch
flower
n4379457
plant
shrub
42
n1 1117
n4334034
bottle
jug
40
n2879899
loudspeaker
speaker
82691
n3696785
electric refrigerator
fridge
36
n3278824
toilet
urinal
34
n4453665
dining table
table
n3205892
poster
picture
28
n6806283
clock
clock
1c22
collulor telephone
mobile
26
n2995984
person
buman
n5224994
cup
mug
24
5422355448JJ22333333847777777773
n3152175
stapler
clipper
24
n4310633
reflector
n37778568
trilet tissue
toilet paper
24
n1 n1.509700 708
desktop computer
ne
7809
table lamp
desk light
23
85b55253ec
n4387620
flag
banner
22
ac50bdl (0000llleed53/7///00444444l////
n3359749
armchair
seliee
21
806bce I29 wtt.11660500000ccc/////44e4eeee
n2741540
cuphoard
wardmbe
21
3hh-80ad htt.a//0666....11100000000711110
n3152290
pen
pencil
62%s5c83b 0
n3913116
bag
sack
20
70c
n2776843
can
drink
16526d
n4262696
sword
knife
20
n4380981
bookshell
shell'
20
n2874800
fireplace
burner
d' difhalloss5777cccce5522222222223
n3351301
curtain
drape
19
n3155743
baltery
62811606
bookcase
bookshelf
18
n2874241
pizza
pie
07889783
hammer
mallet
a49966
434862255
microwave
oven
6l99661n
cereal box
natmenl
01100010
food
meal
16
n21445
24159912
glass
beaker
16
89c19a775 1est9a/755/3ccc0////
13443167
wine bottle
magnum
14599016
lamppost
streetlight
n3642472
oven
cooker
69c5 69657774KK..NN111777751221211113
n3868196
polaroid
48c266 72o-222e77992221155552 051.5544444555111147777722466660
n2946154
globe
world
14
n3445436
player
fe90d87
n2991759
wardrobe
cupboard
14
n4557470
bible
text
n64344286
pillow
cushion
8bik hrkk:Hlha77SSccccecdeaaaa2a22223
n3944520
mug
cup
13
n3802912
chandelier
light
3748 77778888888264444321000000077733
n3008889
chossboard
checkers
13
n3017971
calculator
computer
n2942270
dishwasher
washing machine
12
fF42 144
n3212662
matiress
mat
n3736655
basket
91b1
n2805104
pencil sharpener
sharpener
n39 14835
candle
lantern
n2951508
bowl
dish
12
n28844335
clock
n3050242
cost banger
hanger
Sea 2aa.17778440011100000022223222
n3061905
Table 6: The 80 ShapeNet category names and their sizes, corresponding WordNet synsets and an example model id for category exemplar used ipythe lifting task. In the extra materials we provide full list of ShapeNet model ids in each category.

---

Under review as a conference paper at ICLR 2020
Environment word
Synonym
Immovable objects
tray
box
bed
mattress
boat
ship
hairdryer
dryer
racket bus
bat
coach
rocket
spaceship
car
automobile
Movable objects
plane
aeroplane
mug
cup
robot
android
train
locomotive
keyboard helicopter
piano
airplane
candle
lamp
Table 7: The immovable and ten movable objects in the putting experiment. To begin each episode, both immovable objects and three randomly-selecte movable objects are randomly posi- tioned the room.
18


Reviews:
[{"review_id": "r1epwfv15r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the task of instruction following where an agent navigates/interacts with a 3D environment conditioned on goals provided in natural language. While several existing approaches use synthetic language for instructions, the authors tackle this problem under the setting of noisy instructions provided by humans in natural language. For this, they use large-scale pre-trained representations (e.g. BERT) as initial parameters for representing the textual instructions. Their main result is the demonstration of transfer from agents trained using synthetic instructions to environments with more variation (e.g. synonyms) or natural instructions provided by humans on two tasks involving object manipulation. \n\nPros:\n1. Nice application of BERT to grounded instruction following tasks\n2. Good empirical results\n\nCons:\n1. Not much technical novelty\n2. Empirical experiments could use a bit more rigor in terms of disentangling the major factors that contribute to performance (e.g typo noise)\n\n\n\nOther comments:\n1. Are the BERT weights frozen or finetuned along with the rest of the model? Does the performance depend on this?\n2. The typo noise (TN) seems to be a key driver of performance. Have you tried adding it to the other baselines like wordPiece Transformer? \n3. What are the scores when training on the test tasks (D.O synonym, natural instructions, etc.) directly? It would be good to establish how well the transfer setup is doing compared to the best RL agent trained directly on the test scenarios.\n\n—————\nPost rebuttal update:\nThanks to the authors for their response and for updating the paper! I especially appreciate the additional experiments, but I’m still confused why the authors do not perform a clear ablation study to support their claims. It seems like the main claimed novelty of the paper is the proposed CMSA method. However, the empirical results are not convincing/rigorous enough to provide the reader information on 1) whether CMSA is a useful method (since it is used only with BERT and does not seem to affect results on its own compared to MP, SA, TN, etc.) and 2) when should one use/not use BERT and CMSA (BERT+CMSA actually does quite poorly acc. to table 5). Further, the other reviewers also pointed out concerns regarding the difficulty of the task and complexity of language used. Hence, I feel the paper still requires some revision to form a coherent story — updating my score accordingly.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HyeF7bRoKH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes applying natural language encoders pre-trained on a large text corpora (e.g. BERT) to training agents to follow natural language instructions in a simulated environment. Overall, I greatly enjoyed reading this paper: clear exposition of the idea, sensible model architecture, reasonable baselines and good experimental performance. I only have minor questions & feedback, see below.\n\nPros\n- Well designed experiments with sensible baselines.\n- Strong transfer learning results.\n- Illuminating analysis where using BERT indeed performs better on capturing phrasal and sentence-level equivalence in natural language instructions.\n\nQuestions\n- In Table 5, BERT doesn't give performance improvement on natural instruction (over Word embedding+Transformers) until BERT+CMSA+TN. Why do you think this is the case? To phrase this in a different way, why do you think BERT+MP doesn't perform well on this?\n- Are the results in Figure 2 computed from a BERT+MP model or a BERT+CMSA model?\n- In the lifting results in Table 4, why doesn't BERT+CMSA+MP outperform BERT+MP?\n\n------ \n\nUpdates:\n\nHaving read other reviewers' comments and also the authors' response, I would also like to call into question the difficulty of the experiments in the paper -- for the lifting task, the model is always presented with a command \"Lift X\", and essentially only needs to identify the correct object out of two at test time. Also, for the putting task, the model only needs to disambiguate between 6 possible combinations (3 movable and 2 fixed objects). Especially with a sophisticated model like BERT, I would have liked to see tasks where human instructions are more complex than this simple task. Hence, I'm changing my score to 6.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeZVasCFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a method of transferring template-based instruction following agents to natural language instructions by using language encoders trained on large text corpora. They explore different ways of combining text-based language encoders with visual representations and compare them. They find that contextual phrase-based representations learned by BERT significantly improve the performance on natural language instructions. \n\nStrengths:\n- The paper is written well, it is easy to understand and follow.\n- The task setup is good, authors collect natural language instruction data from humans.\n- The paper presents several language encoding methods for the task and systematically evaluates them in a scientific manner.  \n- The experimental results indicate that it is possible to transfer an agent trained on template-based instructions to natural language instructions using language models trained on large text corpora. It is not necessary to train the agent on natural language instructions. I find this result important and useful.\n\nWeaknesses:\n- The paper lacks significant technical novelty. It essentially combines known reinforcement learning based instruction following agents with known language models. The different ways of combining language encoding with visual representations are either trivial or adapted from prior work.\n- A major concern is that the natural language instructions considered in the paper do not have much diversity with respect to language. The paper only considers lifting and putting tasks and trains a separate model for both the tasks.  \n-- The lifting task always uses the verb 'lift' and replaces the object word with synonyms or referring expressions. There are 80 objects in the lifting task and I suspect there are very few referring expressions for these objects and they mostly involve a synonym. Furthermore, at test time, the agent only needs to distinguish between 2 objects. The performance with random embedding is around 50% for this task and the best model is around 76% which means the agent is not recognizing the correct object around 50% of the time. \n-- For the putting task, authors consider synonyms for object words and natural instructions which involve changing the verb ‘put’. It seems like humans mostly use only 4 verb words for this task, ‘put’, ‘keep’, ‘move’, ‘place’. This might be an artifact of the examples given to the human annotators. In any case, this word is inconsequential as the agent always lifts one of 3 available objects on one of 2 fixed objects. \n- It seems like the most diversity is coming from synonyms which can probably be handled with a dictionary or wordnet rather than requiring a language model. There is also some prior work on handling synonyms (https://arxiv.org/pdf/1902.04546.pdf). I would have liked to see many more tasks and a multi-task learning model which is also able to distinguish between the task based on natural language instructions in addition to understanding object word synonyms and referring expressions. More objects would also help.\n- The authors claim to tackle “more behavioural and environmental challenges than previous work”. I do not agree with this claim. It is true that this paper handles object interaction and natural language instructions in a partially observable setting, however, previous work has tackled other challenges which this work does not tackle. For example, Oh et al. 2017 generalize to new sequence of instructions, Hermann et al. 2017 and Chaplot et al. 2018 also handle compositionality and generalize to unseen instructions referring to new objects, Hermann et al. 2017 handle negation, Chaplot et al. 2018 handle instructions involving ‘largest’ or ‘smallest’ objects, Misra et al. 2018 handle more diverse natural language and so on.\n- I wouldn’t call moving objects using high-level symbolic actions as ‘manipulation’. This is a whole research area in robotics involving taking low-level actions to move an object. Also, the environment used in the paper is not visually realistic in my opinion. It looks game-like and visual encoders trained in this environment are unlikely to generalize to the real world. This is fine as it is mostly irrelevant to handling natural language instructions, but authors should not claim visual realism and object manipulation in my opinion. \n\n\nComments/Questions\n- I do not understand the meaning and purpose of some actions. Why are there GRAB + actions? Doesn’t the object move with the agent once it is grabbed? What is SPIN_OBJECT? Why is it needed? It seems like there is no ‘place’ action, how does the agent place the object? I am guessing when the agent stops taking GRAB+ actions. If that is the case, then wouldn’t it be easier to just have Grab and Place actions rather than 16 GRAB+ actions?\n- The meaning of ‘(sub)-’ in (sub)-word is not described.\n- What is the probability of typo noise in the experiments?\n- Many experimental details are missing. How long was the model trained for both the tasks? How many training samples/episodes? What were the hyperparameters used for reinforcement learning? Learning rate, optimizer, discount value and so on.\n\nUpdates after author response:\nI have examined the author response and additional experiments carefully. I am maintaining my score due to the following reasons:\n- The authors seem to agree that the paper does not provide any 'substantial algorithmic advance'.\n-  The authors argue that the number of objects in the paper is much more than prior work. However, the focus of prior work (referenced by authors) was not to tackle natural language. Since the focus of this paper is to tackle natural language, I believe the number and diversity of objects and tasks need to be much higher than 80 objects and 2 tasks used in the paper.\n- \"We would language referring to an even wider range of motor-behaviours, e.g. more 'verbs'\" -> I do not understand what the authors are trying to say here, but if \"learning the motor programmes for such concepts in an environment\" is not \"the focus of the present paper\", I believe the focus is only handling synonyms and referring expressions for objects. In my opinion, these are relatively easier to tackle, (for example using a dictionary or wordnet) than grounding 'verbs' into sequence of actions, which limits the scope of this paper further.\n- The authors still claim to tackle object manipulation and visual realism which I do not agree with. I do not believe taking high-level \"GRAB\" action can be called object manipulation. The objects are taken from shape net with relatively realistic shapes, but neither the appearance of objects (textures, shadows) nor the relative arrangement of objects is realistic. In my opinion, a model trained in this environment has no hope of generalizing to the real-world. \n- I do not agree with the authors' argument against chance performance in the lifting task. From my experience, I believe an RL agent trained without any language input would perform at 50% if it receives a reward for lifting one of the two available objects. I do not understand why the authors chose to put only 2 objects in the environment. Why not put 5 or more objects?\n- I appreciate authors' efforts towards multi-task learning results, however, tackling only 2 tasks is not convincing enough.\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper examines whether it is possible to train agents to follow synthetic instructions that perceives and modifies a 3D scene based on a first-person viewpoint, and have the trained agents follow natural language instructions provided by humans.

The paper received two weak rejects and one weak accept.  The main concerns voiced by the reviewers are:
1. Lack of variety in natural language
One of the key claims of the paper is that previous work on instruction following can only handle instructions generated from templates and cannot handle ambiguous expressions used by real people, and that the contribution of this work is that it can handle such expresssions.  However, as pointed out by R1, the language considered in this work is very simplistic in form (close to being template based) with the main variation coming from synonyms.  Even the free-form natural instructions that are collected, are done so with very specific instructions that restrict diversity of language (e.g don't use colors or other properties of the object). R1 also point out that there are prior work that handles much more diverse language.

2. Limited technical novelty and questions about how much the proposed CMSA method actually contribute

3. Overclaims and lack of precision when using terminology
There is concern that the task that is addressed is not actually that complex.  The environments are simple (with just 2 objects) and not that realistic.  Tackling 2 tasks is barely "multi-task", and commonly, "manipulation" refers to low-level grasping/picking up of objects which is not how it is used here.

While the paper has many strong elements and is mostly well written, considerable improvements still need to be made for the paper to have claims it can support.  It is currently below the bar for acceptance. The authors are encouraged to improve their paper and resubmit to an appropriate venue.

Decision: Reject