History Aware Multimodal Transformer for Vision-and-Language Navigation

Vision-and-language navigation ( VLN ) aims to build autonomous visual agents that follow instructions and navigate in real scenes . To remember previously visited locations and actions taken , most approaches to VLN implement memory using recurrent states . Instead , we introduce a History Aware Multimodal Transformer ( HAMT ) to incorporate a long-horizon history into multimodal decision making . HAMT efficiently encodes all the past panoramic observations via a hierarchical vision transformer ( ViT ) , which first encodes individual images with ViT , then models spatial relation between images in a panoramic observation and finally takes into account temporal relation between panoramas in the history . It , then , jointly combines text , history and current observation to predict the next action . We first train HAMT end-to-end using several proxy tasks including single step action prediction and spatial relation prediction , and then use reinforcement learning to further improve the navigation policy . HAMT achieves new state of the art on a broad range of VLN tasks , including VLN with fine-grained instructions ( R2R , RxR ) , high-level instructions ( R2R-Last , REVERIE ) , dialogs ( CVDN ) as well as long-horizon VLN ( R4R , R2R-Back ) . We demonstrate HAMT to be particularly effective for navigation tasks with longer trajectories . 1 Introduction . Vision-and-language navigation ( VLN ) has recently received growing attention [ 1 , 2 , 3 , 4 , 5 ] . VLN requires an agent to understand natural language instructions , perceive the visual world , and perform navigation actions to arrive at a target location . A number of datasets have been proposed to support various VLN tasks such as indoor and outdoor navigation with fine-grained instructions [ 2 , 6 , 7 ] , language-driven remote object finding [ 8 ] and navigation in dialogs [ 9 ] . VLN agents are faced with several challenges . First , as opposed to static vision-text grounding [ 10 ] , the agent continuously receives new visual observations and should align them with instructions . Most of existing works adopt recurrent neural networks ( RNNs ) [ 6 , 11 , 12 , 13 , 14 , 15 , 16 ] to encode historical observations and actions within a fixed-size state vector to predict the next action . Such condensed states might be sub-optimal for capturing essential information in extended trajectories [ 17 ] . For instance , “ bring the spoon to me ” requires the agent to remember its start location after navigating to the “ spoon ” , while early memories are prone to fade in the recurrent state . Few endeavors [ 18 , 19 ] construct external map-like memories for received observations . Nevertheless , these approaches still rely on RNNs to track the navigation state . As the history plays an important role in environment understanding and instruction grounding , we propose to explicitly encode the history as a sequence of previous actions and observations instead of using recurrent states . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Another VLN challenge concerns the generalizations of agents to new environments that have not been observed during training [ 4 ] . One direction is to learn more generic text-image representations . The PRESS model [ 20 ] improves language representation with a pretrained BERT encoder [ 21 ] , and PREVALENT [ 22 ] uses pairs of instruction and single-step observations to pretrain a multimodal transformer . Though achieved promising results , these works do not optimize visual representation for the target navigation task . Moreover , lack of history in training [ 22 ] makes it hard to learn cross-modal alignment and increases the risk of overfitting to training environments . Another direction towards better generalization is to overcome exposure bias [ 23 ] due to discrepancy between training and inference . Different methods have been adopted for VLN including DAgger [ 6 , 24 ] and scheduled sampling [ 20 , 25 ] . Reinforcement Learning ( RL ) [ 12 , 26 ] is one of the most effective approach among them , but it is considered unstable to directly train large-scale transformers via RL [ 27 ] . To address the above challenges , we propose the History Aware Multimodal Transformer ( HAMT ) , a fully transformer-based architecture for multimodal decision making in VLN tasks . As illustrated in Figure 1 , HAMT consists of unimodal transformers for text , history and observation encoding , and a cross-modal transformer to capture long-range dependencies of the history sequence , current observation and instruction . Since our history contains a sequence of all previous observations , its encoding is computationally expensive . To resolve complexity issues , we propose a hierarchical vision transformer as shown in Figure 2 , which progressively learns representations for a single view , spatial relationships among views within a panorama and , finally , the temporal dynamics across panoramas of the history . In order to learn better visual representations , we propose auxiliary proxy tasks for end-to-end training . Such tasks include single-step action prediction based on imitation learning , self-supervised spatial relationship reasoning , masked language and image predictions and instructiontrajectory matching . We empirically show that our training facilitates the subsequent fine-tuning of our model with RL [ 28 ] . We carry out extensive experiments on various VLN tasks , including VLN with fine-grained instructions ( R2R [ 6 ] and RxR [ 7 ] ) , high-level instructions ( REVERIE [ 8 ] and our proposed R2R-Last ) , dialogs [ 9 ] as well as long-horizon VLN ( R4R [ 3 ] and our proposed R2R-Back which requires the agent to return back after arriving at the target location ) . HAMT outperforms state of the art on both seen and unseen environments in all the tasks . We summarize our contributions as follows : ( 1 ) We introduce HAMT to efficiently model longhorizon history of observed panoramas and actions via hierarchical vision transformer ; ( 2 ) We train HAMT with auxiliary proxy tasks in an end-to-end fashion and use RL to improve the navigation policy ; ( 3 ) We validate our method and outperform state of the art in a diverse range of VLN tasks , while demonstrating larger gains for long-horizon navigation . 2 Related work . Vision-and-language navigation . Training instruction-following navigation agents has attracted increasing research attention [ 1 , 2 , 6 , 7 , 8 , 29 ] . Anderson et al . [ 6 ] propose a sequence-to-sequence % = 0 % = 1 view image panorama image embeding panorama embedding history embedding Temporal Transformer $ ! '' $ # '' Transformer ⋯ ⋯ ( a ) Hierarchical vision transformer for history encoding . ( b ) Flattend history encoding . Transformer ( c ) Temporal-only history encoding . ViT Panoramic Transformer ViT ViT ⋯ ⋯ ⋯ ViT Panoramic Transformer ViT ViT ⋯ ⋯ ⋯ ViT Panoramic Transformer ViT ViT ⋯ ⋯ ⋯ % = 2 $ $ '' ( a ) Hierarchical history encoding . It first encodes individual view images with ViT , then models the spatial relation between images in each panorama , and finally captures the temporal relation between panoramas in the history . 𝑡 = 0 𝑡 = 1 view image panorama image embeding panorama embedding history embedding Temporal Transformer 𝑣 ! '' 𝑣 # '' Transformer ⋯ ⋯ ( a ) Hierarchical vision transformer for history encoding . It first encodes individual view images with ViT , then models the spatial relation betwee images in each panorama , and finally capture the temporal relation between panoramas in the history . ( b ) Flattend his ory encoding . It jointly encode spatial and temporal relations of all images in the history . Transformer ( c ) Temporal-only history encoding . It only onsiders temp ral relati n of agent ’ s oriented images in the history . ViT Panoramic Transformer ViT ViT ⋯ ⋯ ⋯ ViT Panoramic Transformer ViT ViT ⋯ ⋯ ⋯ ViT P r ic Transformer ViT ViT ⋯ ⋯ ⋯ 𝑡 = 2 𝑣 $ '' ( b ) Flattened history encoding . It encodes spatial and temporal relations at the same time . 𝑡 = 0 𝑡 = 1 view image p norama i age embeding panorama embedding history embedding Temporal Transformer 𝑣 ! '' 𝑣 # '' Transformer ⋯ ⋯ ( a ) Hierarchical vision transformer for history encoding . It first encodes individual view images with ViT , then models the spatial relation between images in each panorama , and finally capture the temporal relation between panoramas in the history . ( b ) Flattend history encoding . It jointly encode spatial and tempora relations of all images in the history . Transformer ( c ) Temporal-only history encoding . It only considers temporal relation of agent ’ s oriented images in the history . ViT Panoramic Transformer ViT ViT ⋯ ⋯ ⋯ ViT Panoramic Transformer ViT ViT ⋯ ⋯ ⋯ ViT Panoramic Transformer ViT ViT ⋯ ⋯ ⋯ 𝑡 = 2 𝑣 $ '' ( c ) Tempo al-only history encodi g. It only consider temporal relation of oriented views . Figure 2 : A comparison of history encoding methods . Circle nodes in different colors denote view images of panorama at different steps . Darker circle nodes are the oriented view of the agent . LSTM baseline for the VLN task . Fried et al . [ 11 ] extend it with panoramic action space and synthesized instructions . To improve cross-modal alignment , the self-monitoring agent [ 13 ] proposes co-grounding and progress estimation , and RelGraph [ 15 ] uses graphs to model relationships across scene , objects and directions . Reinforcement learning ( RL ) is typically used to improve navigation policy . The EnvDrop model [ 12 ] mixes imitation learning and A3C [ 28 ] . The RCM [ 14 ] utilizes intrinsic reward of cross-modal matching in REINFORCE algorithm . Wang et al . [ 30 ] propose to learn rewards via soft expert distillation . Due to the success of transformer [ 31 ] , recent works explore transformer architectures in VLN . PRESS [ 20 ] replaces LSTM instruction encoder with pretrained BERT [ 21 ] . SIA [ 16 ] uses transformer for single-step multimodal fusion and LSTM for sequential action prediction . PTA [ 32 ] is a transformer VLN model using CNNs to extract visual features [ 33 ] . Here we propose the first full transformer architecture for VLN and train it end-to-end . Memory-based policy for navigation . LSTMs [ 34 ] have been the dominant approach to encode memories for navigation [ 6 , 11 , 12 , 14 ] . Condensing all history into one feature vector , however , is prone to the loss of information . Alternative approaches include topological map memory structures [ 35 , 36 ] . Deng et al . [ 18 ] use graphs to capture environment layout and enable long-term planing . A similar graph is adopted in [ 19 ] with frontier-exploration based decision making . But these works still utilize LSTMs for state tracking . To exploit long-term spatio-temporal dependencies , Fang et al . [ 17 ] store histories in a sequence encoded with transformer . Recurrent VLN-BERT [ 5 ] injects a recurrent unit to encode histories in transformer for VLN . The most similar work to ours is Episodic Transformer ( E.T . ) [ 37 ] . Differently from [ 37 ] , we propose a hierarchical encoding of the panoramic observation history and optimize the whole model in end-to-end training . Multimodal pretraining with transformers . Recent works show significant progress in vision and language tasks using multimodal pretraining . In particular , transformer architectures such as one-stream [ 38 , 39 ] and dual-stream [ 40 , 41 ] achieve state of the art for a number of downstream tasks including visual question answering , image-text retrieval and image captioning . While most previous methods rely on CNN to extract image representations , ViLT [ 42 ] adopts Vision Transformer ( ViT ) [ 43 ] and trains it with associated texts in an end-to-end manner thanks to the efficiency of ViT . A few endeavors [ 22 , 44 ] explore multimodal pretraining for VLN . PREVALENT [ 22 ] pretrains a transformer using instructions and single-step observations without referring to trajectory history . VLN-BERT [ 44 ] measures the compatibility between an instruction and images in a path but does not support action prediction . Our work presents the first end-to-end trainable VLN transformer that jointly encodes text , history and observation , and is able to sequentially predict actions .

Summary: This paper presents a transformer-based model for VLN, which makes use of history trajectory information. Panoramic visual observations of all passed viewpoints are embedded by two transformer modules, one handling panorama at each step and the other one handling panorama representations of all steps. The well-known pre-training technology is employed. Experiments on R4R, R2R, and R2R variants are conducted.