Generalized Natural Language Grounded Navigation via Environment-agnostic Multitask Learning

Recent research efforts enable study for natural language grounded navigation in photo-realistic environments , e.g. , following natural language instructions or dialog . However , existing methods tend to overfit training data in seen environments and fail to generalize well in previously unseen environments . In order to close the gap between seen and unseen environments , we aim at learning a generalizable navigation model from two novel perspectives : ( 1 ) we introduce a multitask navigation model that can be seamlessly trained on both Vision-Language Navigation ( VLN ) and Navigation from Dialog History ( NDH ) tasks , which benefits from richer natural language guidance and effectively transfers knowledge across tasks ; ( 2 ) we propose to learn environment-agnostic representations for navigation policy that are invariant among environments , thus generalizing better on unseen environments . Extensive experiments show that our environment-agnostic multitask navigation model significantly reduces the performance gap between seen and unseen environments and outperforms the baselines on unseen environments by 16 % ( relative measure on success rate ) on VLN and 120 % ( goal progress ) on NDH , establishing the new state of the art for the NDH task . 1 INTRODUCTION . Navigation in visual environments by following natural language guidance ( Hemachandra et al. , 2015 ) is a fundamental capability of intelligent robots that simulate human behaviors , because humans can easily reason about the language guidance and navigate efficiently by interacting with the visual environments . Recent efforts ( Anderson et al. , 2018b ; Das et al. , 2018 ; Thomason et al. , 2019 ) empower large-scale learning of natural language grounded navigation that is situated in photorealistic simulation environments . Nevertheless , the generalization problem commonly exists for these tasks , especially indoor navigation : the agent usually performs poorly on unknown environments that have never been seen during training . One of the main causes for such behavior is data scarcity as it is expensive and time-consuming to extend either visual environments or natural language guidance . The number of scanned houses for indoor navigation is limited due to high expense and privacy concerns . Besides , unlike vision-only navigation tasks ( Mirowski et al. , 2018 ; 2016 ; Xia et al. , 2018 ; Manolis Savva * et al. , 2019 ; Kolve et al. , 2017 ) where episodes can be exhaustively sampled in simulation , natural language grounded navigation is supported by human demonstrated interaction and communication in natural language . It is impractical to fully collect and cover all the samples for individual tasks . Therefore , it is essential though challenging to efficiently learn a more generalized policy for natural language grounded navigation tasks from existing data ( Wu et al. , 2018a ; b ) . In this paper , we study how to resolve the generalization and data scarcity issues from two different angles . First , previous methods are trained for one task at the time , so each new task requires training a brand new agent instance that can only solve the one task it was trained on . In this work , we propose a generalized multitask model for natural language grounded navigation tasks such as Vision-Language Navigation ( VLN ) and Navigation from Dialog History ( NDH ) , aiming at efficiently transferring knowledge across tasks and effectively solving both tasks with one agent simultaneously . Moreover , although there are thousands of trajectories paired with language guidance , the underlying house scans are restricted . For instance , the popular Matterport3D dataset ( Chang et al. , 2017 ) contains only 61 unique house scans in the training set . The current models perform much better in seen environments by taking advantage of the knowledge of specific houses they have acquired over multiple task completions during training , but fail to generalize to houses not seen during training . Hence we propose an environment-agnostic learning method to learn a visual representation that is invariant to specific environments but still able to support navigation . Endowed with the learned environment-agnostic representations , the agent is further prevented from the overfitting issue and generalizes better on unseen environments . To the best of our knowledge , we are the first to introduce natural language grounded multitask and environment-agnostic training regimes and validate their effectiveness on VLN and NDH tasks . Extensive experiments demonstrate that our environment-agnostic multitask navigation model can not only efficiently execute different language guidance in indoor environments but also outperform the single-task baseline models by a large margin on both tasks . Besides , the performance gap between seen and unseen environments is significantly reduced . We also set a new state of the art on NDH with over 120 % improvement in terms of goal progress . 2 BACKGROUND . Vision-and-Language Navigation . Vision-and-Language Navigation ( Anderson et al. , 2018b ; Chen et al. , 2019 ) task requires an embodied agent to navigate in photo-realistic environments to carry out natural language instructions . The agent is spawned at an initial pose p0 = ( v0 , φ0 , θ0 ) , which includes the spatial location , heading and elevation angles . Given a natural language instruction X = { x1 , x2 , ... , xn } , the agent is expected to perform a sequence of actions { a1 , a2 , ... , aT } and arrive at the target position vtar specified by the language instruction X , which describes stepby-step instructions from the starting position to the target position . In this work , we consider VLN task defined for Room-to-Room ( R2R ) ( Anderson et al. , 2018b ) dataset which contains instructiontrajectory pairs across 90 different indoor environments ( houses ) . Previous VLN methods have studied various aspects to improve the navigation performance , such as planning ( Wang et al. , 2018 ) , data augmentation ( Fried et al. , 2018 ; Tan et al. , 2019 ) , cross-modal alignment ( Wang et al. , 2019 ; Huang et al. , 2019b ) , progress estimation ( Ma et al. , 2019a ) , error correction ( Ma et al. , 2019b ; Ke et al. , 2019 ) , interactive language assistance ( Nguyen et al. , 2019 ; Nguyen & Daumé III , 2019 ) etc . This work tackles VLN via multitask learning and environmentagnostic learning , which is orthogonal to all these prior arts . Navigation from Dialog History . Different from Visual Dialog ( Das et al. , 2017 ) which involves dialog grounded in a single image , the recently introduced Cooperative Vision-and-Dialog Navigation ( CVDN ) dataset ( Thomason et al. , 2019 ) includes interactive language assistance for indoor navigation , which consists of over 2,000 embodied , human-human dialogs situated in photo-realistic home environments . The task of Navigation from Dialog History ( NDH ) is defined as : given a target object t0 and a dialog history between humans cooperating to perform the task , the embodied agent must infer navigation actions towards the goal room that contains the target object . The dialog history is denoted as < t0 , Q1 , A1 , Q2 , A2 , ... , Qi , Ai > , including the target object t0 , the questions Q and answers A till the turn i ( 0 ≤ i ≤ k , where k is the total number of Q-A turns from the beginning to the goal room ) . The agent , located in p0 , is trying to move closer to the goal room by inferring from the dialog history that happened before . Multitask Learning . The basis of Multitask ( MT ) learning is the notion that tasks can serve as mutual sources of inductive bias for each other ( Caruana , 1993 ) . When multiple tasks are trained jointly , MT learning causes the learner to prefer the hypothesis that explains all the tasks simultaneously , hence leading to more generalized solutions . MT learning has been successful in natural language processing ( Collobert & Weston , 2008 ) , speech recognition ( Deng et al. , 2013 ) , computer vision ( Girshick , 2015 ) , drug discovery ( Ramsundar et al. , 2015 ) , and Atari games ( Teh et al. , 2017 ) . The deep reinforcement learning methods that have become very popular for training models on natural language grounded navigation tasks ( Wang et al. , 2019 ; Huang et al. , 2019a ; b ; Tan et al. , 2019 ) are known to be data inefficient . In this work , we introduce multitask reinforcement learning for such tasks to improve data efficiency by positive transfer across related tasks . Environment-agnostic Learning . A few studies on agnostic learning have been proposed recently . For example , Model-Agnostic Meta-Learning ( MAML ) ( Finn et al. , 2017 ) aims to train a model on a variety of learning tasks and solve a new task using only a few training examples . Liu et al . ( 2018 ) proposes a unified feature disentangler that learns domain-invariant representation across multiple domains for image translation . Other domain-agnostic techniques are also proposed for supervised ( Li et al. , 2018 ) and unsupervised domain adaption ( Romijnders et al. , 2019 ; Peng et al. , 2019 ) . In this work , we pair the environment classifier with a gradient reversal layer ( Ganin & Lempitsky , 2015 ) to learn an environment-agnostic representation that can be better generalized on unseen environments in a zero-shot fashion where no adaptation is involved . Distributed Actor-Learner Navigation Learning Framework . To train models for the various language grounded navigation tasks like VLN and NDH , we develop a distributed actor-learner learning infrastructure1 . The framework design is inspired by IMPALA ( Espeholt et al. , 2018 ) and uses its off-policy correction method called V-trace to efficiently scale reinforcement learning methods to thousands of machines . The framework additionally supports a variety of supervision strategies important for navigation tasks such as teacher-forcing ( Anderson et al. , 2018b ) , studentforcing ( Anderson et al. , 2018b ) and mixed supervision ( Thomason et al. , 2019 ) . The framework is built using TensorFlow ( Abadi et al. , 2016 ) and supports ML accelerators ( GPU , TPU ) . 3 ENVIRONMENT-AGNOSTIC MULTITASK LEARNING . 3.1 OVERVIEW . Our environment-agnostic multitask navigation model is illustrated in Figure 1 . First , we adapt the reinforced cross-modal matching ( RCM ) model ( Wang et al. , 2019 ) and make it seamlessly transfer across tasks by sharing all the learnable parameters for both NDH and VLN , including joint word embedding layer , language encoder , trajectory encoder , cross-modal attention module ( CM-ATT ) , and action predictor . Furthermore , to learn the environment-agnostic representation zt , we equip the navigation model with an environment classifier whose objective is to predict which house the agent is . But note that between trajectory encoder and environment classifier , a gradient reversal layer ( Ganin & Lempitsky , 2015 ) is introduced to reverse the gradients backpropagated to the trajectory encoder , making it learn representations that are environment-agnostic and thus more generalizable in unseen environments . During training , the environment classifier is minimizing the environment classification loss Lenv , while the trajectory encoder is maximizing Lenv and minimizing the navigation lossLnav . The other modules are optimized with the navigation loss Lnav simultaneously . Below we introduce multitask reinforcement learning and environmentagnostic representation learning . A more detailed model architecture is presented in Section 4 . 1The identity is not disclosed to respect the anonymity of the submission . 3.2 MULTITASK REINFORCEMENT LEARNING . Interleaved Multitask Data Sampling . To avoid overfitting to individual tasks , we adopt an interleaved multitask data sampling strategy to train the model . Particularly , each data sample within a mini-batch can be from either task , so that the VLN instruction-trajectory pairs and NDH dialogtrajectory pairs are interleaved in a mini-batch though they may have different learning objectives . Reward Shaping . Following prior art ( Wang et al. , 2018 ; 2019 ) , we first implement a discounted cumulative reward function R for the VLN and NDH tasks : R ( st , at ) = T∑ t′=t γt ′−tr ( st′ , at′ ) , where r ( st′ , at′ ) = { d ( st′ , vtar ) − d ( st′+1 , vtar ) if t′ < T 1 [ d ( sT , vtar ) ≤ dth ] if t′ = T ( 1 ) where γ is the discounted factor , d ( st′ , vtar ) is the distance between state st and the target location vtar , and dth is the maximum distance from vtar that the agent is allowed to terminate for success . Different from VLN , NDH is essentially room navigation instead of point navigation because the agent is expected to reach a room that contains the target object . Suppose the goal room is occupied by a set of nodes { vi } N1 , we replace the distance function d ( st , vtar ) in Equation 1 with the minimum distance to the goal room droom ( st , { vi } N1 ) for NDH : droom ( st , { vi } N1 ) = min 1≤i≤N d ( st , vi ) ( 2 ) Navigation Loss . Since human demonstrations are available for both VLN and NDH tasks , we use behavior cloning to constrain the learning algorithm to model state-action spaces that are most relevant to each task . Following previous works ( Wang et al. , 2019 ) , we also use reinforcement learning to aid the agent ’ s ability to recover from erroneous actions in unseen environments . During multitask navigation model training , we adopt a mixed training strategy of reinforcement learning and behavior cloning , so the navigation loss function is : Lnav = −Eat∼π [ R ( st , at ) − b ] − E [ log π ( a∗t |st ) ] ( 3 ) where we use REINFORCE policy gradients ( Williams , 1992 ) and supervised learning gradients to update the policy π. b is the estimated baseline to reduce the variance and a∗t is the human demonstrated action .

Summary: This paper addresses some challenges of following natural language instructions for navigating in visual environments. The main challenge in such tasks is the scarcity of available training data, which results in generalization problems where the agent has difficulty navigating in unseen environments. Therefore, the authors propose two key ideas to tackle this issue and incorporate them in the reinforced cross-modal matching (RCM) model (Wang et al, 2019). First, they use a generalized multitask learning model to transfer knowledge across two different tasks: Vision-Language Navigation (VLN) and Navigation from Dialog History (NDH). This results in learning features that explain both tasks simultaneously and hence generalize better. Moreover, by training on both tasks the effective size of training data is increased significantly. Second, they propose an environment-agnostic learning technique in order to learn invariant representations that are still efficient for navigation. This prevents overfitting to specific visual features of the environment and therefore helps improving generalization. The contribution of this paper is combining and incorporating these two key ideas in the RCM framework and verifying it on VLN and NDH tasks. This approach is novel compared to prior results in tackling the VLN task. Their experimental results show that the two proposed techniques improve generalization in a complementary fashion, measured by decreased performance gap between seen and unseen environments.  They demonstrate that their technique outperforms state-of-the-art methods on unseen data on some evaluation metrics.