Generalized Natural Language Grounded Navigation via Environment-agnostic Multitask Learning

Recent research efforts enable study for natural language grounded navigation in photo-realistic environments , e.g. , following natural language instructions or dialog . However , existing methods tend to overfit training data in seen environments and fail to generalize well in previously unseen environments . In order to close the gap between seen and unseen environments , we aim at learning a generalizable navigation model from two novel perspectives : ( 1 ) we introduce a multitask navigation model that can be seamlessly trained on both Vision-Language Navigation ( VLN ) and Navigation from Dialog History ( NDH ) tasks , which benefits from richer natural language guidance and effectively transfers knowledge across tasks ; ( 2 ) we propose to learn environment-agnostic representations for navigation policy that are invariant among environments , thus generalizing better on unseen environments . Extensive experiments show that our environment-agnostic multitask navigation model significantly reduces the performance gap between seen and unseen environments and outperforms the baselines on unseen environments by 16 % ( relative measure on success rate ) on VLN and 120 % ( goal progress ) on NDH , establishing the new state of the art for the NDH task . 1 INTRODUCTION . Navigation in visual environments by following natural language guidance ( Hemachandra et al. , 2015 ) is a fundamental capability of intelligent robots that simulate human behaviors , because humans can easily reason about the language guidance and navigate efficiently by interacting with the visual environments . Recent efforts ( Anderson et al. , 2018b ; Das et al. , 2018 ; Thomason et al. , 2019 ) empower large-scale learning of natural language grounded navigation that is situated in photorealistic simulation environments . Nevertheless , the generalization problem commonly exists for these tasks , especially indoor navigation : the agent usually performs poorly on unknown environments that have never been seen during training . One of the main causes for such behavior is data scarcity as it is expensive and time-consuming to extend either visual environments or natural language guidance . The number of scanned houses for indoor navigation is limited due to high expense and privacy concerns . Besides , unlike vision-only navigation tasks ( Mirowski et al. , 2018 ; 2016 ; Xia et al. , 2018 ; Manolis Savva * et al. , 2019 ; Kolve et al. , 2017 ) where episodes can be exhaustively sampled in simulation , natural language grounded navigation is supported by human demonstrated interaction and communication in natural language . It is impractical to fully collect and cover all the samples for individual tasks . Therefore , it is essential though challenging to efficiently learn a more generalized policy for natural language grounded navigation tasks from existing data ( Wu et al. , 2018a ; b ) . In this paper , we study how to resolve the generalization and data scarcity issues from two different angles . First , previous methods are trained for one task at the time , so each new task requires training a brand new agent instance that can only solve the one task it was trained on . In this work , we propose a generalized multitask model for natural language grounded navigation tasks such as Vision-Language Navigation ( VLN ) and Navigation from Dialog History ( NDH ) , aiming at efficiently transferring knowledge across tasks and effectively solving both tasks with one agent simultaneously . Moreover , although there are thousands of trajectories paired with language guidance , the underlying house scans are restricted . For instance , the popular Matterport3D dataset ( Chang et al. , 2017 ) contains only 61 unique house scans in the training set . The current models perform much better in seen environments by taking advantage of the knowledge of specific houses they have acquired over multiple task completions during training , but fail to generalize to houses not seen during training . Hence we propose an environment-agnostic learning method to learn a visual representation that is invariant to specific environments but still able to support navigation . Endowed with the learned environment-agnostic representations , the agent is further prevented from the overfitting issue and generalizes better on unseen environments . To the best of our knowledge , we are the first to introduce natural language grounded multitask and environment-agnostic training regimes and validate their effectiveness on VLN and NDH tasks . Extensive experiments demonstrate that our environment-agnostic multitask navigation model can not only efficiently execute different language guidance in indoor environments but also outperform the single-task baseline models by a large margin on both tasks . Besides , the performance gap between seen and unseen environments is significantly reduced . We also set a new state of the art on NDH with over 120 % improvement in terms of goal progress . 2 BACKGROUND . Vision-and-Language Navigation . Vision-and-Language Navigation ( Anderson et al. , 2018b ; Chen et al. , 2019 ) task requires an embodied agent to navigate in photo-realistic environments to carry out natural language instructions . The agent is spawned at an initial pose p0 = ( v0 , φ0 , θ0 ) , which includes the spatial location , heading and elevation angles . Given a natural language instruction X = { x1 , x2 , ... , xn } , the agent is expected to perform a sequence of actions { a1 , a2 , ... , aT } and arrive at the target position vtar specified by the language instruction X , which describes stepby-step instructions from the starting position to the target position . In this work , we consider VLN task defined for Room-to-Room ( R2R ) ( Anderson et al. , 2018b ) dataset which contains instructiontrajectory pairs across 90 different indoor environments ( houses ) . Previous VLN methods have studied various aspects to improve the navigation performance , such as planning ( Wang et al. , 2018 ) , data augmentation ( Fried et al. , 2018 ; Tan et al. , 2019 ) , cross-modal alignment ( Wang et al. , 2019 ; Huang et al. , 2019b ) , progress estimation ( Ma et al. , 2019a ) , error correction ( Ma et al. , 2019b ; Ke et al. , 2019 ) , interactive language assistance ( Nguyen et al. , 2019 ; Nguyen & Daumé III , 2019 ) etc . This work tackles VLN via multitask learning and environmentagnostic learning , which is orthogonal to all these prior arts . Navigation from Dialog History . Different from Visual Dialog ( Das et al. , 2017 ) which involves dialog grounded in a single image , the recently introduced Cooperative Vision-and-Dialog Navigation ( CVDN ) dataset ( Thomason et al. , 2019 ) includes interactive language assistance for indoor navigation , which consists of over 2,000 embodied , human-human dialogs situated in photo-realistic home environments . The task of Navigation from Dialog History ( NDH ) is defined as : given a target object t0 and a dialog history between humans cooperating to perform the task , the embodied agent must infer navigation actions towards the goal room that contains the target object . The dialog history is denoted as < t0 , Q1 , A1 , Q2 , A2 , ... , Qi , Ai > , including the target object t0 , the questions Q and answers A till the turn i ( 0 ≤ i ≤ k , where k is the total number of Q-A turns from the beginning to the goal room ) . The agent , located in p0 , is trying to move closer to the goal room by inferring from the dialog history that happened before . Multitask Learning . The basis of Multitask ( MT ) learning is the notion that tasks can serve as mutual sources of inductive bias for each other ( Caruana , 1993 ) . When multiple tasks are trained jointly , MT learning causes the learner to prefer the hypothesis that explains all the tasks simultaneously , hence leading to more generalized solutions . MT learning has been successful in natural language processing ( Collobert & Weston , 2008 ) , speech recognition ( Deng et al. , 2013 ) , computer vision ( Girshick , 2015 ) , drug discovery ( Ramsundar et al. , 2015 ) , and Atari games ( Teh et al. , 2017 ) . The deep reinforcement learning methods that have become very popular for training models on natural language grounded navigation tasks ( Wang et al. , 2019 ; Huang et al. , 2019a ; b ; Tan et al. , 2019 ) are known to be data inefficient . In this work , we introduce multitask reinforcement learning for such tasks to improve data efficiency by positive transfer across related tasks . Environment-agnostic Learning . A few studies on agnostic learning have been proposed recently . For example , Model-Agnostic Meta-Learning ( MAML ) ( Finn et al. , 2017 ) aims to train a model on a variety of learning tasks and solve a new task using only a few training examples . Liu et al . ( 2018 ) proposes a unified feature disentangler that learns domain-invariant representation across multiple domains for image translation . Other domain-agnostic techniques are also proposed for supervised ( Li et al. , 2018 ) and unsupervised domain adaption ( Romijnders et al. , 2019 ; Peng et al. , 2019 ) . In this work , we pair the environment classifier with a gradient reversal layer ( Ganin & Lempitsky , 2015 ) to learn an environment-agnostic representation that can be better generalized on unseen environments in a zero-shot fashion where no adaptation is involved . Distributed Actor-Learner Navigation Learning Framework . To train models for the various language grounded navigation tasks like VLN and NDH , we develop a distributed actor-learner learning infrastructure1 . The framework design is inspired by IMPALA ( Espeholt et al. , 2018 ) and uses its off-policy correction method called V-trace to efficiently scale reinforcement learning methods to thousands of machines . The framework additionally supports a variety of supervision strategies important for navigation tasks such as teacher-forcing ( Anderson et al. , 2018b ) , studentforcing ( Anderson et al. , 2018b ) and mixed supervision ( Thomason et al. , 2019 ) . The framework is built using TensorFlow ( Abadi et al. , 2016 ) and supports ML accelerators ( GPU , TPU ) . 3 ENVIRONMENT-AGNOSTIC MULTITASK LEARNING . 3.1 OVERVIEW . Our environment-agnostic multitask navigation model is illustrated in Figure 1 . First , we adapt the reinforced cross-modal matching ( RCM ) model ( Wang et al. , 2019 ) and make it seamlessly transfer across tasks by sharing all the learnable parameters for both NDH and VLN , including joint word embedding layer , language encoder , trajectory encoder , cross-modal attention module ( CM-ATT ) , and action predictor . Furthermore , to learn the environment-agnostic representation zt , we equip the navigation model with an environment classifier whose objective is to predict which house the agent is . But note that between trajectory encoder and environment classifier , a gradient reversal layer ( Ganin & Lempitsky , 2015 ) is introduced to reverse the gradients backpropagated to the trajectory encoder , making it learn representations that are environment-agnostic and thus more generalizable in unseen environments . During training , the environment classifier is minimizing the environment classification loss Lenv , while the trajectory encoder is maximizing Lenv and minimizing the navigation lossLnav . The other modules are optimized with the navigation loss Lnav simultaneously . Below we introduce multitask reinforcement learning and environmentagnostic representation learning . A more detailed model architecture is presented in Section 4 . 1The identity is not disclosed to respect the anonymity of the submission . 3.2 MULTITASK REINFORCEMENT LEARNING . Interleaved Multitask Data Sampling . To avoid overfitting to individual tasks , we adopt an interleaved multitask data sampling strategy to train the model . Particularly , each data sample within a mini-batch can be from either task , so that the VLN instruction-trajectory pairs and NDH dialogtrajectory pairs are interleaved in a mini-batch though they may have different learning objectives . Reward Shaping . Following prior art ( Wang et al. , 2018 ; 2019 ) , we first implement a discounted cumulative reward function R for the VLN and NDH tasks : R ( st , at ) = T∑ t′=t γt ′−tr ( st′ , at′ ) , where r ( st′ , at′ ) = { d ( st′ , vtar ) − d ( st′+1 , vtar ) if t′ < T 1 [ d ( sT , vtar ) ≤ dth ] if t′ = T ( 1 ) where γ is the discounted factor , d ( st′ , vtar ) is the distance between state st and the target location vtar , and dth is the maximum distance from vtar that the agent is allowed to terminate for success . Different from VLN , NDH is essentially room navigation instead of point navigation because the agent is expected to reach a room that contains the target object . Suppose the goal room is occupied by a set of nodes { vi } N1 , we replace the distance function d ( st , vtar ) in Equation 1 with the minimum distance to the goal room droom ( st , { vi } N1 ) for NDH : droom ( st , { vi } N1 ) = min 1≤i≤N d ( st , vi ) ( 2 ) Navigation Loss . Since human demonstrations are available for both VLN and NDH tasks , we use behavior cloning to constrain the learning algorithm to model state-action spaces that are most relevant to each task . Following previous works ( Wang et al. , 2019 ) , we also use reinforcement learning to aid the agent ’ s ability to recover from erroneous actions in unseen environments . During multitask navigation model training , we adopt a mixed training strategy of reinforcement learning and behavior cloning , so the navigation loss function is : Lnav = −Eat∼π [ R ( st , at ) − b ] − E [ log π ( a∗t |st ) ] ( 3 ) where we use REINFORCE policy gradients ( Williams , 1992 ) and supervised learning gradients to update the policy π. b is the estimated baseline to reduce the variance and a∗t is the human demonstrated action .

Summary: This paper aims to apply the model of Wang 2019 to the new NDH task of Thomason '19.  Both of these datasets are built on the same room-to-room environment and both are for natural language instruction following.  Thomason's work extends the R2R paradigm to include a dialogue history which is collapsed into a single instruction.  The contribution of this paper is to build a single model which alternatingly samples trajectories from each of the two datasets to train a more general actor and the authors also believe that the presence of an environment classifier assists in generalization.