Robust Instruction-Following in a Situated Agent via Transfer-Learning from Text

1 INTRODUCTION . Developing machines that can follow natural human commands , particularly those pertaining to an environment shared by both machine and human , is a long-standing and elusive goal of AI ( Winograd , 1972 ) . Recent work has applied end-to-end , representation-learning-based methods to this challenge , where a neural-network-based agent is optimized to process language input , perceive its surroundings and execute appropriate movements jointly ( Oh et al. , 2017 ; Hermann et al. , 2017 ; Chaplot et al. , 2018 ) . End-to-end learning promises a way to deal flexibly with the complexity of the physical , visual and linguistic world without relying on ( potentially brittle ) hand-crafted features , rules or policies . Nevertheless , the cost of this flexibility is the large number of environment interactions ( samples ) required for a randomly-initialized network to learn behaviour policies from raw experience . To make the approach feasible , many studies thus employ a synthetic language that is generated on demand from templates by the environment simulator ( Chevalier-Boisvert et al. , 2018 ; Jiang et al. , 2019 ; Yu et al. , 2018b ; a ) . The studies that do combine both end-to-end learning with natural-language data do so in less realistic grid-like environments ( Misra et al. , 2017 ) or grant agents access to privileged global observations to make learning more tractable ( Misra et al. , 2018 ) . Here , we take a different learning-based approach to instruction-following that is robust to human commands . We train agent policies to respond to synthetic , template-based language but also endow them with powerful language encoders that are pretrained on natural language text . The synthetic instructions that our agents are trained to follow require mastery of 26 fine-grained motor actions in order to identify and manipulate visually-realistic models from the ShapeNet dataset ( Chang et al. , 2015 ) in a 3D room . The visual realism of the world makes it possible to elicit diverse natural human ways of instructing and/or referring to things , and to study the agents ’ robustness to this diversity . Unsurprisingly , we find that agents that are trained on template-based commands do not cope well with the diversity and variation of natural keyboard-typed language . In contrast , when powerful pretrained language encoders are integrated into our agent , we find that agents can satisfy human instructions with substantially above-chance accuracy . To better analyze the generalization that supports this robustness , we probe trained agents with specific modifications to their training instructions . We find that methods based on conventional ( context-free ) word embeddings support generalization that is driven by lexical similarity ( executing lift a vehicle when trained to lift a car ) , but capture phrasal equivalence less well ( failing at put a plate on the container when trained to put the dish on the tray ) . In contrast , methods that integrate contextual word representations support both types of generalization . We also find that robustness to synonyms can often be improved when pretrained encoders are combined with an additional ( cross-modal ) self-attention tuned to the agent ’ s environmental objectives . Ablation experiments highlight the role that WordPiece tokenization ( Schuster & Nakajima , 2012 ) plays in robustness to human instructions . This motivates the addition of typo noise to our training pipeline , which further improves the accuracy of the agent ’ s responses . Overall , our principal contributions are the following : • We train an agent that can both interpret human language commands and overcome a similar range of behavioural and environmental challenges to state-of-the-art policy-learning approaches ( Table 1 ) . • We develop the techniques of transfer-learning from a text representation-learning model to an embodied agent . Note that this is very different from transfer between language classification tasks ( Collobert & Weston , 2008 ; Devlin et al. , 2018 ) ; in our evaluations the agent must interpret unfamiliar instructions zero-shot ( i.e . without additional learning steps ) , and must realize complex language-conditional behaviours ( decoding ≈ 50 appropriate actions ) in spite of this uncertainty . 1.1 HIGH-LEVEL APPROACH . We experiment in a simulated room in the Unity1 game engine ( Fig 1 ) . Importantly , the realism of the visual assets in this room enable human observers to recognize the objects ( and hence refer to them in natural ways ) . Our experimental pipeline then involves the following steps : 1 . Define a ( language-dependent ) task , including initial conditions for the simulated room , a template-based process for generating synthetic language instructions and a reward function for checking game states and exposing reward if the agent succeeds . 2 . Train agents with different language encoders to solve the task when conditioned on the synthetic language produced by the environment generator . 3 . Evaluate the trained agent on 1000 episodes of the task with the synthetic instructions replaced with synonyms or free-form commands typed by human annotators . 2 ARCHITECTURES FOR INSTRUCTION-FOLLOWING AGENTS . Since the object of our study is instruction-following , we consider an agent architecture that is as conventional as possible in aspects not related to language processing . It consists of standard components for computing visual representations of pixels , embedding text strings and predicting actions contingent on these and memory of past inputs.2 Visual processing The visual observations received by the agent at each timestep are 96× 72× 3 real-valued tensors , which are processed by a 3-layer residual convnet . Memory core The output of the visual processing is combined with language information according to a particular encoding strategy , as described below . In all conditions , some combination of vision and language input at each timestep passes into a LSTM memory core with hidden dim . 128 . Action and value prediction The state of the memory core at each timestep is passed through a linear layer and softmax to compute a distribution over 26 actions . Independently , the memory state is passed to a linear layer to yield a scalar value prediction . Training algorithm The agent is trained using an importance-weighted actor-critic algorithm with a central learner and distributed actors ( Espeholt et al. , 2018 ) . 2.1 LANGUAGE ENCODING . The agent must process both string observations in the simulated environment , during training , and human instructions , also encoded as strings , during evaluation . Its representations of language must be combined with visual information to make decisions about how to act . We compare various different ways of achieving this encoding . For methods that transfer knowledge from unsupervised text-based learning , we take weights from the well-known BERT model ( Devlin et al. , 2018 ) , specifically the uncased BERTBASE model made available by the authors.3 BERT + mean pool For a given input of w words , BERTBASE returns w context-dependent ( sub ) word representations of 768 units . In this condition , a mean pooling operation is applied over the w dimension to yield a single representation of dimension 768 , which is concatenated with the flattened output of the visual processing module . This multi-modal representation is passed through a single layer MLP with tanh activation and output dimension 128 before entering the memory core of the agent . We apply the standard BERTBASE WordPiece vocabulary ( of size 30,000 ) . Note 1Unity . http : //unity3d.com . 2See Appendix C for details not included here . 3Available at https : //tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1 that WordPiece encodes language in terms of subwords ( a mix of characters , common word chunks , morphemes and words ) rather than the word-level vocabulary applied in more traditional neural language models . BERT + self-attention layer When applying BERT to text classification , performance can sometimes be improved by fine-tuning the weights in the BERT encoder to suit the task . Because of the large number of gradient updates required to learn a complex behaviour policy , fine-tuning the BERT weights in this way would cause substantial overfitting to the synthetic environment language . We therefore keep the BERT weights frozen , but experiment with an additional learned self-attention layer ( Vaswani et al. , 2017 ) to create a language encoder whose bottom layers are pretrained and whose final layer optimizes these representations to the present environment or tasks . This additional layer has 4 attention heads , and uses 64 dimensional key and value embeddings . BERT+ cross-modal self-attention We also consider a cross-modal self-attention layer of the type suggested by e.g . Tsai et al . ( 2019 ) ; Lu et al . ( 2019 ) , which in our application provides an explicit pathway for the agent to bind visual experience to specific ( contextual ) word representations . In this case , we treat each of the output channels of the visual processing network as word-like entities , by passing them through a linear layer of output size 768 to match the BERT output . The embeddings for all words and the visual channels are then processed with a single self-attention layer , whose parameters are again learned with all other agent parameters . Pretrained ( sub ) -word embeddings Language classification experiments with BERT show the value of highly context-dependent ( sub ) -word representations , but transfer in that context is also possible with more conventional ( context-independent ) word embeddings ( Collobert & Weston , 2008 ) . To measure the effect of this distinction , we consider a simpler encoder based on the ( context independent ) input ( sub- ) word embeddings from BERT , which are also of dimension 768.4 Taking a mean of these context-independent vectors would yield a word-order-invariant representation of language . We therefore process them with single-layer transformer with 4 attention heads . As in BERT + mean pool , this output is averaged , reduced by a single-layer MLP ( from 768 to 128 units ) and passed to the agent ’ s core memory . Typo noise Finally , in one condition we introduce typo noise ( each key character was replaced with that of an adjacent character on a standard keyboard , with a probability of 0.01 ) to the synthetic language strings produced by the environment , allowing the agent to learn to compensate during training . No modifications were made to the evaluation stimuli . Much previous work applies typing noise for language classifier robustness ; see e.g . ( Pruthi et al. , 2019 ) for a recent survey . We compare to various baselines designed to isolate specific components of these encoders . Random mean pool As a direct baseline for BERT mean pool , we consider an identical architecture but in which the contextual BERT embeddings are replaced by fixed random ( subword-specific ) vectors ( also of dimension 768 ) . The weights in the rest of the agent network are trained as in other conditions . We note that random sentence vectors are a competitive baseline on many language classification tasks ( Wieting & Kiela , 2019 ) . Word-level and WordPiece transformers In addition to pre-trained weights , a potentially important aspect of encoders based on BERT is WordPiece tokenization , which can afford greater ability to make sense of typos or rare words with familiar morphemes than in a more conventional wordlevel encoder . To isolate the effect of tokenization from that of pretrained weights , we compared two further encoders . In one , we split all input strings by whitespace and hash each word string to a unique index representing an input to a single-layer transformer with 4 attention heads and embedding size 768 ( chosen to match BERTBASE ) , the output of which is processed identically to the Random mean pool condition . We contrast this with an otherwise identical condition in which the WordPiece tokenization from BERT is applied rather than splitting on white space .

Summary: This paper considers the task of instruction following where an agent navigates/interacts with a 3D environment conditioned on goals provided in natural language. While several existing approaches use synthetic language for instructions, the authors tackle this problem under the setting of noisy instructions provided by humans in natural language. For this, they use large-scale pre-trained representations (e.g. BERT) as initial parameters for representing the textual instructions. Their main result is the demonstration of transfer from agents trained using synthetic instructions to environments with more variation (e.g. synonyms) or natural instructions provided by humans on two tasks involving object manipulation.