History Aware Multimodal Transformer for Vision-and-Language Navigation

Vision-and-language navigation ( VLN ) aims to build autonomous visual agents that follow instructions and navigate in real scenes . To remember previously visited locations and actions taken , most approaches to VLN implement memory using recurrent states . Instead , we introduce a History Aware Multimodal Transformer ( HAMT ) to incorporate a long-horizon history into multimodal decision making . HAMT efficiently encodes all the past panoramic observations via a hierarchical vision transformer ( ViT ) , which first encodes individual images with ViT , then models spatial relation between images in a panoramic observation and finally takes into account temporal relation between panoramas in the history . It , then , jointly combines text , history and current observation to predict the next action . We first train HAMT end-to-end using several proxy tasks including single step action prediction and spatial relation prediction , and then use reinforcement learning to further improve the navigation policy . HAMT achieves new state of the art on a broad range of VLN tasks , including VLN with fine-grained instructions ( R2R , RxR ) , high-level instructions ( R2R-Last , REVERIE ) , dialogs ( CVDN ) as well as long-horizon VLN ( R4R , R2R-Back ) . We demonstrate HAMT to be particularly effective for navigation tasks with longer trajectories . 1 Introduction . Vision-and-language navigation ( VLN ) has recently received growing attention [ 1 , 2 , 3 , 4 , 5 ] . VLN requires an agent to understand natural language instructions , perceive the visual world , and perform navigation actions to arrive at a target location . A number of datasets have been proposed to support various VLN tasks such as indoor and outdoor navigation with fine-grained instructions [ 2 , 6 , 7 ] , language-driven remote object finding [ 8 ] and navigation in dialogs [ 9 ] . VLN agents are faced with several challenges . First , as opposed to static vision-text grounding [ 10 ] , the agent continuously receives new visual observations and should align them with instructions . Most of existing works adopt recurrent neural networks ( RNNs ) [ 6 , 11 , 12 , 13 , 14 , 15 , 16 ] to encode historical observations and actions within a fixed-size state vector to predict the next action . Such condensed states might be sub-optimal for capturing essential information in extended trajectories [ 17 ] . For instance , “ bring the spoon to me ” requires the agent to remember its start location after navigating to the “ spoon ” , while early memories are prone to fade in the recurrent state . Few endeavors [ 18 , 19 ] construct external map-like memories for received observations . Nevertheless , these approaches still rely on RNNs to track the navigation state . As the history plays an important role in environment understanding and instruction grounding , we propose to explicitly encode the history as a sequence of previous actions and observations instead of using recurrent states . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Another VLN challenge concerns the generalizations of agents to new environments that have not been observed during training [ 4 ] . One direction is to learn more generic text-image representations . The PRESS model [ 20 ] improves language representation with a pretrained BERT encoder [ 21 ] , and PREVALENT [ 22 ] uses pairs of instruction and single-step observations to pretrain a multimodal transformer . Though achieved promising results , these works do not optimize visual representation for the target navigation task . Moreover , lack of history in training [ 22 ] makes it hard to learn cross-modal alignment and increases the risk of overfitting to training environments . Another direction towards better generalization is to overcome exposure bias [ 23 ] due to discrepancy between training and inference . Different methods have been adopted for VLN including DAgger [ 6 , 24 ] and scheduled sampling [ 20 , 25 ] . Reinforcement Learning ( RL ) [ 12 , 26 ] is one of the most effective approach among them , but it is considered unstable to directly train large-scale transformers via RL [ 27 ] . To address the above challenges , we propose the History Aware Multimodal Transformer ( HAMT ) , a fully transformer-based architecture for multimodal decision making in VLN tasks . As illustrated in Figure 1 , HAMT consists of unimodal transformers for text , history and observation encoding , and a cross-modal transformer to capture long-range dependencies of the history sequence , current observation and instruction . Since our history contains a sequence of all previous observations , its encoding is computationally expensive . To resolve complexity issues , we propose a hierarchical vision transformer as shown in Figure 2 , which progressively learns representations for a single view , spatial relationships among views within a panorama and , finally , the temporal dynamics across panoramas of the history . In order to learn better visual representations , we propose auxiliary proxy tasks for end-to-end training . Such tasks include single-step action prediction based on imitation learning , self-supervised spatial relationship reasoning , masked language and image predictions and instructiontrajectory matching . We empirically show that our training facilitates the subsequent fine-tuning of our model with RL [ 28 ] . We carry out extensive experiments on various VLN tasks , including VLN with fine-grained instructions ( R2R [ 6 ] and RxR [ 7 ] ) , high-level instructions ( REVERIE [ 8 ] and our proposed R2R-Last ) , dialogs [ 9 ] as well as long-horizon VLN ( R4R [ 3 ] and our proposed R2R-Back which requires the agent to return back after arriving at the target location ) . HAMT outperforms state of the art on both seen and unseen environments in all the tasks . We summarize our contributions as follows : ( 1 ) We introduce HAMT to efficiently model longhorizon history of observed panoramas and actions via hierarchical vision transformer ; ( 2 ) We train HAMT with auxiliary proxy tasks in an end-to-end fashion and use RL to improve the navigation policy ; ( 3 ) We validate our method and outperform state of the art in a diverse range of VLN tasks , while demonstrating larger gains for long-horizon navigation . 2 Related work . Vision-and-language navigation . Training instruction-following navigation agents has attracted increasing research attention [ 1 , 2 , 6 , 7 , 8 , 29 ] . Anderson et al . [ 6 ] propose a sequence-to-sequence % = 0 % = 1 view image panorama image embeding panorama embedding history embedding Temporal Transformer $ ! '' $ # '' Transformer ⋯ ⋯ ( a ) Hierarchical vision transformer for history encoding . ( b ) Flattend history encoding . Transformer ( c ) Temporal-only history encoding . ViT Panoramic Transformer ViT ViT ⋯ ⋯ ⋯ ViT Panoramic Transformer ViT ViT ⋯ ⋯ ⋯ ViT Panoramic Transformer ViT ViT ⋯ ⋯ ⋯ % = 2 $ $ '' ( a ) Hierarchical history encoding . It first encodes individual view images with ViT , then models the spatial relation between images in each panorama , and finally captures the temporal relation between panoramas in the history . 𝑡 = 0 𝑡 = 1 view image panorama image embeding panorama embedding history embedding Temporal Transformer 𝑣 ! '' 𝑣 # '' Transformer ⋯ ⋯ ( a ) Hierarchical vision transformer for history encoding . It first encodes individual view images with ViT , then models the spatial relation betwee images in each panorama , and finally capture the temporal relation between panoramas in the history . ( b ) Flattend his ory encoding . It jointly encode spatial and temporal relations of all images in the history . Transformer ( c ) Temporal-only history encoding . It only onsiders temp ral relati n of agent ’ s oriented images in the history . ViT Panoramic Transformer ViT ViT ⋯ ⋯ ⋯ ViT Panoramic Transformer ViT ViT ⋯ ⋯ ⋯ ViT P r ic Transformer ViT ViT ⋯ ⋯ ⋯ 𝑡 = 2 𝑣 $ '' ( b ) Flattened history encoding . It encodes spatial and temporal relations at the same time . 𝑡 = 0 𝑡 = 1 view image p norama i age embeding panorama embedding history embedding Temporal Transformer 𝑣 ! '' 𝑣 # '' Transformer ⋯ ⋯ ( a ) Hierarchical vision transformer for history encoding . It first encodes individual view images with ViT , then models the spatial relation between images in each panorama , and finally capture the temporal relation between panoramas in the history . ( b ) Flattend history encoding . It jointly encode spatial and tempora relations of all images in the history . Transformer ( c ) Temporal-only history encoding . It only considers temporal relation of agent ’ s oriented images in the history . ViT Panoramic Transformer ViT ViT ⋯ ⋯ ⋯ ViT Panoramic Transformer ViT ViT ⋯ ⋯ ⋯ ViT Panoramic Transformer ViT ViT ⋯ ⋯ ⋯ 𝑡 = 2 𝑣 $ '' ( c ) Tempo al-only history encodi g. It only consider temporal relation of oriented views . Figure 2 : A comparison of history encoding methods . Circle nodes in different colors denote view images of panorama at different steps . Darker circle nodes are the oriented view of the agent . LSTM baseline for the VLN task . Fried et al . [ 11 ] extend it with panoramic action space and synthesized instructions . To improve cross-modal alignment , the self-monitoring agent [ 13 ] proposes co-grounding and progress estimation , and RelGraph [ 15 ] uses graphs to model relationships across scene , objects and directions . Reinforcement learning ( RL ) is typically used to improve navigation policy . The EnvDrop model [ 12 ] mixes imitation learning and A3C [ 28 ] . The RCM [ 14 ] utilizes intrinsic reward of cross-modal matching in REINFORCE algorithm . Wang et al . [ 30 ] propose to learn rewards via soft expert distillation . Due to the success of transformer [ 31 ] , recent works explore transformer architectures in VLN . PRESS [ 20 ] replaces LSTM instruction encoder with pretrained BERT [ 21 ] . SIA [ 16 ] uses transformer for single-step multimodal fusion and LSTM for sequential action prediction . PTA [ 32 ] is a transformer VLN model using CNNs to extract visual features [ 33 ] . Here we propose the first full transformer architecture for VLN and train it end-to-end . Memory-based policy for navigation . LSTMs [ 34 ] have been the dominant approach to encode memories for navigation [ 6 , 11 , 12 , 14 ] . Condensing all history into one feature vector , however , is prone to the loss of information . Alternative approaches include topological map memory structures [ 35 , 36 ] . Deng et al . [ 18 ] use graphs to capture environment layout and enable long-term planing . A similar graph is adopted in [ 19 ] with frontier-exploration based decision making . But these works still utilize LSTMs for state tracking . To exploit long-term spatio-temporal dependencies , Fang et al . [ 17 ] store histories in a sequence encoded with transformer . Recurrent VLN-BERT [ 5 ] injects a recurrent unit to encode histories in transformer for VLN . The most similar work to ours is Episodic Transformer ( E.T . ) [ 37 ] . Differently from [ 37 ] , we propose a hierarchical encoding of the panoramic observation history and optimize the whole model in end-to-end training . Multimodal pretraining with transformers . Recent works show significant progress in vision and language tasks using multimodal pretraining . In particular , transformer architectures such as one-stream [ 38 , 39 ] and dual-stream [ 40 , 41 ] achieve state of the art for a number of downstream tasks including visual question answering , image-text retrieval and image captioning . While most previous methods rely on CNN to extract image representations , ViLT [ 42 ] adopts Vision Transformer ( ViT ) [ 43 ] and trains it with associated texts in an end-to-end manner thanks to the efficiency of ViT . A few endeavors [ 22 , 44 ] explore multimodal pretraining for VLN . PREVALENT [ 22 ] pretrains a transformer using instructions and single-step observations without referring to trajectory history . VLN-BERT [ 44 ] measures the compatibility between an instruction and images in a path but does not support action prediction . Our work presents the first end-to-end trainable VLN transformer that jointly encodes text , history and observation , and is able to sequentially predict actions .

Summary: This paper proposes a history-aware multimodal transformer (HAMT) style model for Vision-and-Language Navigation task which explicitly takes the history of observations as the input for the transformer model. HAMT efficiently encodes all past panoramic observations using a hierarchical vision transformer. It consists of unimodal transformers for text, history (of panoramic visual observations), and current visual observation and then fuses these representations together using a cross-modal transformer. They also propose additional auxiliary proxy tasks for end-to-end training of HAMT. These are Single-step Action Prediction & Regression (SAP/SAR) and Spatial Relationship Prediction (SPREL). Through extensive ablations, they show that how each of their contributions makes sense and gets state-of-the-art on the single-run R2R task, R4R, and R2R-Back & R2R-Last (two augmented tasks they described in the paper).