History Aware Multimodal Transformer for Vision-and-Language Navigation
Year: 2022

Abstract:
Vision-and-language navigation (VLN) aims to build autonomous visual agents that follow instructions and navigate in real scenes. To remember previously visited locations and actions taken, most approaches to VLN implement memory using recurrent states. Instead, we introduce a History Aware Multimodal Transformer (HAMT) to incorporate a long-horizon history into multimodal decision making. HAMT efficiently encodes all the past panoramic observations via a hierarchical vision transformer (ViT), which first encodes individual images with ViT, then models spatial relation between images in a panoramic observation and finally takes into account temporal relation between panoramas in the history. It, then, jointly combines text, history and current observation to predict the next action. We first train HAMT end-to-end using several proxy tasks including single step action prediction and spatial relation prediction, and then use reinforcement learning to further improve the navigation policy. HAMT achieves new state of the art on a broad range of VLN tasks, including VLN with fine-grained instructions (R2R, RxR), high-level instructions (R2R-Last, REVERIE), dialogs (CVDN) as well as long-horizon VLN (R4R, R2R-Back). We demonstrate HAMT to be particularly effective for navigation tasks with longer trajectories.

text: Vision-and-language navigation (VLN) aims to build autonomous visual agents that follow instructions and navigate in real scenes. To remember previously visited locations and actions taken, most approaches to VLN implement memory using recurrent states. Instead, we introduce a History Aware Multimodal Transformer (HAMT) to incorporate a long-horizon history into multimodal decision making. HAMT efficiently encodes all the past panoramic observations via a hierarchical vision transformer (ViT), which first encodes individual images with ViT, then models spatial relation between images in a panoramic observation and finally takes into account temporal relation between panoramas in the history. It, then, jointly combines text, history and current observation to predict the next action. We first train HAMT end-to-end using several proxy tasks including single step action prediction and spatial relation prediction, and then use reinforcement learning to further improve the navigation policy. HAMT achieves new state of the art on a broad range of VLN tasks, including VLN with fine-grained instructions (R2R, RxR), high-level instructions (R2R-Last, REVERIE), dialogs (CVDN) as well as long-horizon VLN (R4R, R2R-Back). We demonstrate HAMT to be particularly effective for navigation tasks with longer trajectories.

heading: 1 Introduction
text: Vision-and-language navigation (VLN) has recently received growing attention [1, 2, 3, 4, 5]. VLN requires an agent to understand natural language instructions, perceive the visual world, and perform navigation actions to arrive at a target location. A number of datasets have been proposed to support various VLN tasks such as indoor and outdoor navigation with fine-grained instructions [2, 6, 7], language-driven remote object finding [8] and navigation in dialogs [9].
VLN agents are faced with several challenges. First, as opposed to static vision-text grounding [10], the agent continuously receives new visual observations and should align them with instructions. Most of existing works adopt recurrent neural networks (RNNs) [6, 11, 12, 13, 14, 15, 16] to encode historical observations and actions within a fixed-size state vector to predict the next action. Such condensed states might be sub-optimal for capturing essential information in extended trajectories [17]. For instance, “bring the spoon to me” requires the agent to remember its start location after navigating to the “spoon”, while early memories are prone to fade in the recurrent state. Few endeavors [18, 19] construct external map-like memories for received observations. Nevertheless, these approaches still rely on RNNs to track the navigation state. As the history plays an important role in environment understanding and instruction grounding, we propose to explicitly encode the history as a sequence of previous actions and observations instead of using recurrent states.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Another VLN challenge concerns the generalizations of agents to new environments that have not been observed during training [4]. One direction is to learn more generic text-image representations. The PRESS model [20] improves language representation with a pretrained BERT encoder [21], and PREVALENT [22] uses pairs of instruction and single-step observations to pretrain a multimodal transformer. Though achieved promising results, these works do not optimize visual representation for the target navigation task. Moreover, lack of history in training [22] makes it hard to learn cross-modal alignment and increases the risk of overfitting to training environments. Another direction towards better generalization is to overcome exposure bias [23] due to discrepancy between training and inference. Different methods have been adopted for VLN including DAgger [6, 24] and scheduled sampling [20, 25]. Reinforcement Learning (RL) [12, 26] is one of the most effective approach among them, but it is considered unstable to directly train large-scale transformers via RL [27].
To address the above challenges, we propose the History Aware Multimodal Transformer (HAMT), a fully transformer-based architecture for multimodal decision making in VLN tasks. As illustrated in Figure 1, HAMT consists of unimodal transformers for text, history and observation encoding, and a cross-modal transformer to capture long-range dependencies of the history sequence, current observation and instruction. Since our history contains a sequence of all previous observations, its encoding is computationally expensive. To resolve complexity issues, we propose a hierarchical vision transformer as shown in Figure 2, which progressively learns representations for a single view, spatial relationships among views within a panorama and, finally, the temporal dynamics across panoramas of the history. In order to learn better visual representations, we propose auxiliary proxy tasks for end-to-end training. Such tasks include single-step action prediction based on imitation learning, self-supervised spatial relationship reasoning, masked language and image predictions and instructiontrajectory matching. We empirically show that our training facilitates the subsequent fine-tuning of our model with RL [28]. We carry out extensive experiments on various VLN tasks, including VLN with fine-grained instructions (R2R [6] and RxR [7]), high-level instructions (REVERIE [8] and our proposed R2R-Last), dialogs [9] as well as long-horizon VLN (R4R [3] and our proposed R2R-Back which requires the agent to return back after arriving at the target location). HAMT outperforms state of the art on both seen and unseen environments in all the tasks.
We summarize our contributions as follows: (1) We introduce HAMT to efficiently model longhorizon history of observed panoramas and actions via hierarchical vision transformer; (2) We train HAMT with auxiliary proxy tasks in an end-to-end fashion and use RL to improve the navigation policy; (3) We validate our method and outperform state of the art in a diverse range of VLN tasks, while demonstrating larger gains for long-horizon navigation.

heading: 2 Related work
text: Vision-and-language navigation. Training instruction-following navigation agents has attracted increasing research attention [1, 2, 6, 7, 8, 29]. Anderson et al. [6] propose a sequence-to-sequence
% = 0 % = 1
view image
panorama
image embeding
panorama embedding
history embedding
Temporal Transformer
$!" $#"
Transformer
⋯
⋯
(a) Hierarchical vision transformer for history encoding.
(b) Flattend history encoding.
Transformer
(c) Temporal-only history encoding.
ViT
Panoramic Transformer
ViT ViT
⋯
⋯ ⋯
ViT
Panoramic Transformer
ViT ViT
⋯
⋯ ⋯
ViT
Panoramic Transformer
ViT ViT
⋯
⋯ ⋯
% = 2
$$"
(a) Hierarchical history encoding. It first encodes individual view images with ViT, then models the spatial relation between images in each panorama, and finally captures the temporal relation between panoramas in the history.
𝑡 = 0 𝑡 = 1
view image
panorama
image embeding
panorama embedding
history embedding
Temporal Transformer
𝑣!" 𝑣#"
Transformer
⋯
⋯
(a) Hierarchical vision transformer for history encoding. It first encodes individual view images with ViT, then models the spatial relation betwee images in each panorama, and finally capture the temporal relation between panoramas in the history.
(b) Flattend his ory encoding. It jointly encode spatial and temporal relations of all images in the history.
Transformer
(c) Temporal-only history encoding. It only onsiders temp ral relati n of agent’s oriented images in the history.
ViT
Panoramic Transformer
ViT ViT
⋯
⋯
⋯
ViT
Panoramic Transformer
ViT ViT
⋯
⋯
⋯
ViT
P r ic Transformer
ViT ViT
⋯
⋯
⋯
𝑡 = 2
𝑣$"
(b) Flattened history encoding. It encodes spatial and temporal relations at the same time.
𝑡 = 0 𝑡 = 1
view image
p norama
i age embeding
panorama embedding
history embedding
Temporal Transformer
𝑣!" 𝑣#"
Transformer
⋯
⋯
(a) Hierarchical vision transformer for history encoding. It first encodes individual view images with ViT, then models the spatial relation between images in each panorama, and finally capture the temporal relation between panoramas in the history.
(b) Flattend history encoding. It jointly encode spatial and tempora relations of all images in the history.
Transformer
(c) Temporal-only history encoding. It only considers temporal relation of agent’s oriented images in the history.
ViT
Panoramic Transformer
ViT ViT
⋯
⋯
⋯
ViT
Panoramic Transformer
ViT ViT
⋯
⋯
⋯
ViT
Panoramic Transformer
ViT ViT
⋯
⋯
⋯
𝑡 = 2
𝑣$"
(c) Tempo al-only history encodi g. It only consider temporal relation of oriented views.
Figure 2: A comparison of history encoding methods. Circle nodes in different colors denote view images of panorama at different steps. Darker circle nodes are the oriented view of the agent.
LSTM baseline for the VLN task. Fried et al. [11] extend it with panoramic action space and synthesized instructions. To improve cross-modal alignment, the self-monitoring agent [13] proposes co-grounding and progress estimation, and RelGraph [15] uses graphs to model relationships across scene, objects and directions. Reinforcement learning (RL) is typically used to improve navigation policy. The EnvDrop model [12] mixes imitation learning and A3C [28]. The RCM [14] utilizes intrinsic reward of cross-modal matching in REINFORCE algorithm. Wang et al. [30] propose to learn rewards via soft expert distillation. Due to the success of transformer [31], recent works explore transformer architectures in VLN. PRESS [20] replaces LSTM instruction encoder with pretrained BERT [21]. SIA [16] uses transformer for single-step multimodal fusion and LSTM for sequential action prediction. PTA [32] is a transformer VLN model using CNNs to extract visual features [33]. Here we propose the first full transformer architecture for VLN and train it end-to-end.
Memory-based policy for navigation. LSTMs [34] have been the dominant approach to encode memories for navigation [6, 11, 12, 14]. Condensing all history into one feature vector, however, is prone to the loss of information. Alternative approaches include topological map memory structures [35, 36]. Deng et al. [18] use graphs to capture environment layout and enable long-term planing. A similar graph is adopted in [19] with frontier-exploration based decision making. But these works still utilize LSTMs for state tracking. To exploit long-term spatio-temporal dependencies, Fang et al. [17] store histories in a sequence encoded with transformer. Recurrent VLN-BERT [5] injects a recurrent unit to encode histories in transformer for VLN. The most similar work to ours is Episodic Transformer (E.T.) [37]. Differently from [37], we propose a hierarchical encoding of the panoramic observation history and optimize the whole model in end-to-end training.
Multimodal pretraining with transformers. Recent works show significant progress in vision and language tasks using multimodal pretraining. In particular, transformer architectures such as one-stream [38, 39] and dual-stream [40, 41] achieve state of the art for a number of downstream tasks including visual question answering, image-text retrieval and image captioning. While most previous methods rely on CNN to extract image representations, ViLT [42] adopts Vision Transformer (ViT) [43] and trains it with associated texts in an end-to-end manner thanks to the efficiency of ViT. A few endeavors [22, 44] explore multimodal pretraining for VLN. PREVALENT [22] pretrains a transformer using instructions and single-step observations without referring to trajectory history. VLN-BERT [44] measures the compatibility between an instruction and images in a path but does not support action prediction. Our work presents the first end-to-end trainable VLN transformer that jointly encodes text, history and observation, and is able to sequentially predict actions.

heading: 3 Method
text: Problem definition The VLN problem [6] is formulated as a partially observable Markov decision process, where future observations are independent of the past conditioning on current state st. Given
an instructionW containing a sequence of L words (w1, w2, · · · , wL), an agent should follow the instruction to move in a connectivity graph to reach the goal location. At each step t, the agent receives an observation Ot, a panorama of its surrounding environment. The Ot consists of K single view images split from the panoramaOt , ([vo1; ao1], · · · , [voK ; aoK ]), where voi is the visual feature of the i-th view and aoi denotes the relative angle to face the view (subscript t is omitted for simplicity). There are n navigable viewpoints among all the K views1, denoted as Oct , ([vc1; ac1], · · · , [vcn; acn]). We follow the setup in [11] and use Oct as the decision space, so the agent only needs to select a candidate in Oct at each step. All observations Oi and performed actions ahi before step t form the historyHt , ([O1; ah1 ], · · · , [Ot−1; aht−1]), where ahi denotes the turned angles at step i. The goal is to learn a policy π parametrized by Θ to predict the next action based on the instruction, history and the current observation, which is π(at|W,Ht,Ot,Oct ; Θ). Unlike dominant recurrent approaches to condense Ht into a fixed-size vector, in this section, we present the History Aware Multimodal Transformer (HAMT) that jointly encodes text, long-horizon history, and observation for sequential action prediction. The model architecture is described in Section 3.1. We propose end-to-end training for HAMT in Section 3.2 to learn unimodal and multimodal representations, and then use RL to fine-tune the navigation policy in Section 3.3.

heading: 3.1 HAMT: History Aware Multimodal Transformer
text: Figure 1 illustrates the model architecture of HAMT. The inputs textW , historyHt and observation Ot are first encoded via the corresponding unimodal transformers respectively, and then fed into the cross-modal transformer encoder to capture multimodal relationships.
Text Encoding. For each token i in the instruction W , we embed it as the summation of its word embedding wi, position embedding EPi and type embedding of text E T 0 . Then we employ a transformer with NL layers to obtain contextual representation xi following the standard BERT [21].
Observation Encoding. For each view [voi ; aoi ] in the panoramic observation Ot, we first represent the relative angle aoi as E A aoi
= (sin θi, cos θi, sinφi, cosφi) where θi and φi are the relative heading and elevation angle to the agent’s orientation. Then the observation embedding oi is as follows:
oi = LN(W o v v o i ) + LN(W o aE A aoi ) + ENoi + E T 1 (1)
where W ov ,W o a are learnable weights. The E N oi denotes the navigable embedding to differentiate types of views, with EN0 for non-navigable view, E N 1 for navigable view and E N 2 for stop view (we append a stop token in observation to support stop action). The ET1 is the type embedding of observation. We omit bias terms for simplicity. The LN denotes layer normalization [45]. Because aoi has much lower feature dimensions than v o i , we apply LN to balance the encoded a o i and v o i .
Hierarchical History Encoding. As Ht consists of all the past panoramic observations Oi and performed actions ahi before step t, it is important to encodeHt efficiently as context. Figures 2b-2c depict the flattened and temporal-only history encoding approaches used in VLN-BERT [44] and E.T. [37] respectively. The flattened approach treats each view image in Oi as a token, so the history sequence contains tK tokens. Though it enables to learn relationships among all image views, the computation cost quadratically increases with the sequence length, making it inefficient for long-horizon tasks. In the temporal-only approach, only the oriented view of the agent in each Oi is taken as inputs instead of the whole panorama, so only t temporal tokens are encoded. However, this approach can lose critical information in past observations. For example, in the instruction “with the windows on your left, walk through the large room past the sitting areas”, the object “window” does not appear in the oriented view of the agent. Therefore, the encoded history is insufficient to tell whether the agent passed the window or not, making the model confused to take the next action.
In order to balance computational efficiency and information integrity, we propose a hierarchical history encoding approach as illustrated in Figure 2a. It hierarchically encodes view images within each panorama and then temporal relationships across panoramas, similar to the factorized spatialtemporal video transformer [46]. For each Oi, its constituent view images are first embeded via ViT and Eq (1), and then encoded via a panoramic transformer withNh layers to learn spatial relationships within the panorama. We apply average pooling to obtain panorama embedding, and add it with the
1A navigable view can lead to one or multiple viewpoints. We follow [5, 12] to use different features for these viewpoints. The viewpoints share the same visual features but differ in angle features.
oriented view image feature in residual connection. The parameters in ViT and panoramic transformer are shared for different steps. In this way, each historical observation Oi is represented as vhi , and the final temporal token hi is computed as:
hi = LN(W h v v h i ) + LN(W h a E A ahi ) + ESi + E T 2 (2)
where ESi denotes the i-th step embedding, E T 2 is the type embedding of history. The computational cost isO(tK2+t2), which significantly reduces fromO(t2K2) in the flattened approach. To be noted, we add a special token [cls] to the start of the history sequence to obtain a global representation. The embedding of [cls] is a parameter to learn, which is initialized from a zero vector.
Cross-modal Encoding. We concatenate history and observation as the vision modality, and use cross-modal transformer with Nx layers to fuse features from text, history and observation as shown in the right of Figure 1. The reason of using such dual-stream architecture rather than onestream is that the length of different modalities can be highly imbalanced, and the dual-stream architecture can balance the importance of intra- and inter-modal relationships by model design [47]. In each cross-modal layer, a vision-text cross-attention is firstly performed for vision modality to attend relevant text information and vice versa for text modality. Then each modality uses selfattention to learn intra-modal relationship such as interaction between observation and history, followed by a fully-connected neural network. Finally, the HAMT model outputs embeddings X ′
= (x′cls, x ′ 1, · · · , x′L), H
′ t = (h ′ cls, h ′ 1, · · · , h′t−1), O ′ t = (o ′ 1, · · · , o′K , o′stop) for tokens in text,
history and observation respectively.

heading: 3.2 End-to-end training with proxy tasks
text: As it is difficult to train large-scale transformers with RL due to sparse supervision [27], we propose to first end-to-end train HAMT via several proxy tasks to learn unimodal and multimodal representation.
Table 1 compares our HAMT with previous VLN transformers PREVALENT [22] and VLNBERT [44] in inputs and proxy tasks. As neither PREVALENT nor VLN-BERT jointly encodes text, history and observation, a limited choice of proxy tasks can be applied in training. Our model instead can take advantage of various proxy tasks to learn cross-modal alignment, spatial and temporal reasoning, and history-aware action prediction. Given the input pair (W,HT ) where T is the length of full trajectory, we can apply common proxy tasks as in vision-and-language pretraining [40, 44], including Masked Language Modeling (MLM), Masked Region Modeling (MRM) and Instruction Trajectory Matching (ITM). Details of the three proxy tasks are presented in the supplementary material. In the following, we introduce new proxy tasks given the triplet input (W,Ht,Ot) specifically for VLN tasks.
Single-step Action Prediction/Regression (SAP/SAR). The task deploys imitation learning to predict the next action based on instruction, history from expert demonstration and the current observation. We formulate it as a classification and a regression task respectively. In the SAP classification task, we predict action probability for each navigable view in Oct which is pt(o′i) = exp(fSAP(o ′ i x ′ cls))∑
j exp(fSAP(o ′ j x′cls))
,
where fSAP is a two-layer fully-connected network, is element-wise multiplication and x′cls is output embedding of special text token [cls]. The objective is to minimize negative log probability of the target view action o′∗: LSAP = −log pt(o′∗). In SAR regression task, we directly predict the action heading and elevation angles based on the text token [cls] which is θ̂t, φ̂t = fSAR(x′cls). The loss function is LSAR = (θ̂t− θt)2 + (φ̂t− φt)2. The two proxy tasks enable the model to learn how to make action decision conditioning on instruction and contextual history.
Spatial Relationship Prediction (SPREL). Expressions of egocentric and allocentric spatial relations are frequent in navigational instructions, such as “walk into the room on your left” and “enter
the bedroom next to the stairs”. In order to learn spatial relation aware representations, we propose the SPREL self-supervised task to predict relative spatial position of two views in a panorama based on only visual feature, angle feature or both. Assume [voi ; a o i ] and [v o j ; a o j ] are two views in Ot, we randomly zero out vo∗ or a o ∗ with probability of 0.3. Their encoded representations are o ′ i and o ′ j , and their relative heading and elevation angles are θij , φij . We then predict θ̂ij , φ̂ij = fSPREL([o′i; o ′ j ]) where [; ] denotes vector concatenation and optimize LSPREL = (θ̂ij − θij)2 + (φ̂ij − φij)2. The task helps for spatial relationship reasoning in the observation.
Training Strategy. Instead of directly training the whole HAMT model at once, we propose to progressively train HAMT in two stages. In the first stage, we freeze ViT pretrained on ImageNet [48] and train the rest of the modules which are randomly initialized. This aims to avoid catastrophic forgetting of the pretrained weights in ViT. Then we unfreeze ViT and train the whole model end-toend. The learning rate for ViT is set to be higher than for others modules to avoid vanishing gradients and to speedup convergence. We empirically show that the proposed two-stage training outperforms one-stage training in the supplementary material.

heading: 3.3 Fine-tuning for sequential action prediction
text: Structure Variants. We present two variants of HAMT for action prediction in the following. 1) MLP action head: we directly reuse the action prediction network fSAP in the SAP task to predict navigable views. We use it as default for VLN tasks. 2) MLP action head based on encoder-decoder structure: the original HAMT model applies cross-modal attention for both vision-to-text and text-tovision, which is computationally expensive when instructions are long. Therefore, we remove the cross-modal attention from text to vision. In this way, we separate the cross-modal transformer into an encoder which only takes instruction as input, and a decoder that inputs history and observation as query and attends over encoded text tokens. Please see supplementary material for details.
RL+IL Objective. We combine Reinforcement Learning (RL) and Imitation Learning (IL) to finetune HAMT for sequential action prediction. The IL relies on the SAP loss defined in Section 3.2 and follows the expert action at each step while RL samples actions according to the policy π. Specifically, we use the Asynchronous Advantage Actor-Critic (A3C) RL algorithm [28]. At each step t, the agent samples an action based on policy π: âht ∼ π(at|W,Ht,Ot,Oct ) and receives an immediate reward rt. For non-stop actions, we set rt as the reduced distance of taking the action to the target and the increased alignment score [3] compared to expert demonstration as defined in [5]; for the stop action, rt = 2 if the agent successfully arrives at the target otherwise -2. A critic network is trained to estimate the value of each state st, which is Rt = ∑T−t k=0 γ
krt+k where γ is discount factor. We implement it as Vt = fcritic(x′cls h′cls). As the reward signal favors shortest distance, we empirically find it benefits to combine A3C RL with IL weighted by λ, which is:
Θ← Θ + µ 1 T T∑ t=1
∇Θlog π(âht ; Θ)(Rt − Vt)︸ ︷︷ ︸ Reinforcement Learning (RL)
+λµ 1
T ∗ T∗∑ t=1
∇Θlog π(a∗t ; Θ)︸ ︷︷ ︸ Imitation Learning (IL)
(3)
where µ is the learning rate, a∗t is the expert action at step t of the expert trajectory of length T ∗.

heading: 4 Experiments

heading: 4.1 Experimental setup
text: Datasets. We evaluate our method on four VLN tasks (seven datasets): VLN with fine-grained instructions (R2R [6], RxR [7]); VLN with high-level instructions (REVERIE [8], R2R-Last); visionand-dialogue navigation (CVDN [9]); and long-horizon VLN (R4R [3], R2R-Back).
• R2R [1] builds upon Matterport3D [49] and includes 90 photo-realistic houses with 10,567 panoramas. It contains 7,189 shortest-path trajectories, each associated with 3 instructions. The dataset is split into train, val seen, val unseen and test unseen sets with 61, 56, 11 and 18 houses respectively. Houses in val seen split are the same as training, while houses in val unseen and test splits are different from training.
• RxR [7] is a large multilingual VLN dataset based on Matterport 3D. The instructions are in three different languages (English, Hindi and Telugu). The dataset emphasizes the role of language in VLN by addressing biases in paths and describing more visible entities than R2R.
• R4R [3] extends R2R dataset by concatenating two adjacent tail-to-head trajectories in R2R. Therefore, it has longer instructions and trajectories. The trajectories are also less biased as they are not necessarily the shortest-path from start to end location.
• R2R-Back is a new VLN setup proposed in this work. The agent is required to return to its start location after arriving at the destination. The agent needs to remember its navigation histories to solve the task. We add a return command at the end of each instruction in R2R and a reverse path from the end to start locations as expert demonstration.
• CVDN [9] defines a navigation from dialog history task, which requires an agent to arrive at goal regions based on multi-turn question-answering dialogs. Such types of instructions are often ambiguous and under-specified. The lengths of instructions and paths are also long.
• REVERIE [8] replaces step-by-step instructions in R2R with high-level instructions, which mainly describe the target location and object. The agent, hence, is required to navigate to the goal without detailed guidance and depends on its past experiences.
• R2R-Last is our proposed VLN setup similar to REVERIE. It only uses the last sentence from the original R2R instructions describing the final destination.
Evaluation metrics. We adopt standard metrics [1], including (1) Trajectory Length (TL): the agent’s navigated path in meters; (2) Navigation Error (NE): the average distance in meters between the agent’s final position and the target; (3) Success Rate (SR): the ratio of trajectories reaching the destination with a maximum error of 3 meters to the target; and (4) Success Rate normalized by the ratio between the length of the shortest path and the predicted path (SPL). SPL is more relevant than SR as it balances the navigation accuracy and efficiency. For long-horizon VLN task (R4R and R2R-Back), we further employ three metrics to measure the path fidelity between the predicted path and target path, including (5) Coverage weighted by Length Score (CLS) [3]; (6) the normalized Dynamic Time Warping (nDTW) [50]; and (7) the Success weighted by nDTW (SDTW). Implementation details. For the HAMT model, we set NL = 9 for language transformer, Nh = 2 for panoramic transformer in hierarchical history encoding, and Nx = 4 for cross-modal transformer. There are K = 36 view images in each panoramic observation. We use ViT-B/16 [43] for image encoding if not otherwise specified. In training with proxy tasks, we randomly select proxy tasks for each mini-batch with predefined ratio. We train HAMT for 200k iterations with fixed ViT using learning rate of 5e-5 and batch size of 64 on 4 NVIDIA Tesla P100 GPUs (∼1 day). The whole HAMT model is trained end-to-end for 20k iterations on 20 NVIDIA V100 GPUs with learning rate of 5e-5 for ViT and 1e-5 for the others (∼20 hours). We use R2R training set and augmented pairs from [22] for training unless otherwise noted. In fine-tuning with RL+IL, we set λ = 0.2 in Eq (3) and γ = 0.9. The model is fine-tuned for 100k iterations with learning rate of 1e-5 and batch size of 8 on a single GPU. Unimodal encoders are fixed by default. The best model is selected according to performance on val unseen split. We use the same augmented data as [5] for R2R for fair comparison, while no augmented data is used for other datasets. Greedy search is applied in inference following the single-run setting. Please see supplementary material for more details.

heading: 4.2 Ablation studies
text: In this section, we evaluate each component in the HAMT model, including: hierarchical history encoding, end-to-end training with proxy tasks, and fine-tuning objectives.
How important is the history encoding for VLN? For fair comparison with the state-of-the-art recurrent architecture RecBERT [5], we use the same Resnet152 visual features and train all the models from scratch with RL+IL objectives to avoid the influence of different weight initialization. The models are optimized for 300k iterations end-to-end except for the visual feature. Table 2 compares different history encoding approaches on
R2R dataset. Our recurrent model slightly differs from RecBERT (no init. OSCAR) [5] in trans-
Table 3: Ablations for end-to-end HAMT training on R2R dataset using proposed proxy tasks.
(a) Comparison of visual features and end-to-end training. The “PT” stands for proxy tasks in training; “e2e” for optimizing the visual representation.
feature PT e2e Val Seen Val UnseenSR↑ SPL↑ SR↑ SPL↑ Resnet
152 × × 65.5±1.2 61.3±1.4 54.4±0.4 48.7±0.4 X × 69.3±1.0 64.8±1.2 63.5±0.5 57.5±0.5
ViT X × 75.7±1.0 72.5±1.0 64.4±0.3 58.8±0.0 X X 75.0±0.9 71.7±0.7 65.7±0.7 60.9±0.7
(b) Comparison of different proxy tasks. The “SAP(R)” denotes the single step action prediction and regression task, and “SPREL” is the spatial relationship prediction task.
SAP (R) SP REL Val Seen Val Unseen SR↑ SPL↑ SR↑ SPL↑
× × 71.2±2.3 67.2±2.0 62.8±1.3 57.7±1.0 X × 74.7±0.6 71.1±0.9 63.6±0.1 58.1±0.4 X X 75.7±1.0 72.5±1.0 64.4±0.3 58.8±0.0
former architecture as shown in Figure 1. It achieves slightly better performance on val unseen split. The temporal-only model uses transformer to encode agent’s oriented visual observations in history sequence, and outperforms the recurrent method by relative gains of 1.9% on SR and 2.1% on SPL for val unseen split. Adding panoramic observations in a hierarchical way results in 4.2% (SR) and 3.6% (SLP) relative improvements on the val unseen split compared to the recurrent method. Even larger improvements are achieved on val seen split as the hierarchical model has a larger capacity to fit the seen environments. This evaluation demonstrates the advantage of our hierarchical history representation compared to the recurrent and temporal-only history representation.
How much does training with proxy tasks help? We next evaluate the advantage of training HAMT end-to-end with proxy tasks. In Table 3a, the first row uses RL+IL objectives to train HAMT from scratch, while the second row uses proxy tasks for training prior to RL+IL fine-tuning. We can see that it significantly boosts the performance to first train with proxy tasks. It improves on val unseen split with 16.7% and 18.0% relative gains on SR and SPL respectively, indicating that training with auxiliary proxy tasks enables better generalization. In the third row, we replace the visual feature from Resnet152 to ViT. The ViT feature improves the performance on both val seen and val unseen splits, showing that more powerful visual representations matter. Finally, training ViT end-to-end obtains 2.1% gains on SPL on val unseen split. This is the first time to show that optimizing visual representations end-to-end is beneficial for VLN tasks. In Table 3b, we evaluate the benefit of the two new proxy tasks for frozen ViT features using the other proxy tasks by default. The SAP(R) uses imitation learning to predict actions, which directly influences the navigation policy and improves the performance by a large margin. The SPREL is a self-supervised proxy task that forces the model to learn spatial relationships in panorama and helps generalization in unseen environments. More experiments to ablate contributions from history encoding and proxy tasks, contributions of proxy tasks in end-to-end training etc. are presented in supplementary material.
What is the impact of the fine-tuning objectives? Table 4 presents results using different objectives in fine-tuning. The first row directly applies HAMT trained by proxy tasks, which achieves lower performance than that after IL finetuning, because we mainly use augmented data in proxy task training to increase visual diversity, but such noisy data deteriorates action prediction performance. Previous work [12] has shown that RL alone performs poorly. However, training with
proxy tasks stabilizes the followup RL fine-tuning. HAMT optimized by RL achieves much better performance than that when fine-tuning with IL on the SR metric. It indicates that RL is able to learn better exploration strategy on unseen environments. However, as the reward for RL focuses more on shortest paths rather than path fidelity with instructions, the improvement on SPL metric is relatively small compared to SR metric. Moreover, the fluctuation of the pure RL objective is larger than IL. Therefore, mixing the RL and IL achieves the best performance.

heading: 4.3 Comparison to state of the art
text: VLN with fine-grained instructions: R2R and RxR. Table 5 compares HAMT with previous VLN methods on the R2R benchmark. Our model outperforms state-of-the-art results of RecBERT [5]
by relative 5.9% and 7.0% improvements in SPL on val seen and unseen splits respectively. We achieve state-of-the-art performance under the single-run setting on the unseen testing split of the leaderboard2. It demonstrates the effectiveness and generalization of our model. We further provide computation time in inference for HAMT and RecBERT in the supplementary material to show the efficiency of our HAMT model. We also achieve large improvements on RxR dataset. The full results are presented in supplementary material.
Long-horizon VLN: R4R and R2R-Back. Table 6 shows navigation results on R4R dataset. As R4R contains longer instructions and trajectories compared to R2R, we use the encoder-decoder variant of HAMT for better efficiency. Our method outperforms previous approaches in all metrics and shows particularly large improvements for the path fidelity related metrics. Compared to RecBERT, HAMT achives 8.2% and 9.5% relative improvement in CLS and nDTW respectively. The large improvements on these path fidelity related metrics indicate that HAMT is better to follow the designated path of the fine-grained instruction. Figure 3 evaluates the performance of HAMT and RecBERT with respect to instruction length measured by words. Though the nDTW decreases for longer instructions, the relative improvement of HAMT increases with the instruction length.
The navigation performance on R2R-Back dataset is presented in Table 7. We compare with two state-of-the-art recurrent models EnvDrop [12] and RecBERT [5] based on LSTM and transformer respectively (both models are trained on R2R-Back for fair comparison). The improvements are more significant on this task as it requires the agent to remember the way it came to the target to successfully return back. The recurrent state is insufficient to capture such history and leads to inferior performance compared to the HAMT model.
2We report the published results on the testing unseen split as shown in https://eval.ai/web/ challenges/challenge-page/97/leaderboard/270 (25/10/2021).
Vision-and-Dialog Navigation: CVDN.
player’s navigation trajectory. We mix the two types of demonstrations as supervision in training which has shown to be the most effective in previous works [22, 52, 53]. As navigation paths in CVDN dataset are much longer than R2R dataset, we adopt the encoder-decoder variant of HAMT. As shown in Table 8, HAMT outperforms existing recurrent approaches on both seen and unseen environments, and achieves the top position in the leaderboard3. It demonstrates that our HAMT model is generalizable to different types of instructions in new VLN tasks.
VLN with high-level instructions: R2R-Last and REVERIE. Table 9 shows results on the R2R-Last dataset that specifies the goal location and contains no step-by-step instructions. The HAMT model with the hierarchical history encoding is able to better accumulate the knowledge of the environment and achieves 9.8% and 10.5% relative gains on SPL metric on seen and unseen splits respectively compared to RecBERT [5]. The REVERIE dataset also
contains high-level instructions but requires object grounding at the target location besides navigation. We provide results on REVERIE dataset in supplementary material. Our HAMT achieves SPL 30.20 and 26.67 on val unseen and test splits respectively, outperforming the state of the art navigation performance [5] by 5.3% and 2.7%.

heading: 5 Conclusion
text: This paper presents the first end-to-end transformer for vision-and-language navigation, denoted as History Aware Multimodal Transformer (HAMT). Our method efficiently encodes long-horizon history and combines it with instructions and observations to derive multimodal action prediction. The HAMT is first trained with proxy tasks in an end-to-end manner, and is then fine-tuned with RL to improve the navigation policy. We achieve state-of-the-art navigation performance on a diverse range of challenging VLN tasks, demonstrating improved accuracy and generalization of our approach compared to the dominant recurrent methods. Future work could extend our history-aware transformer to VLN with continuous actions [54] and could benefit from pretraining on larger navigation datasets. This paper has minimal ethical, privacy and safety concerns.
Acknowledgments and Disclosure of Funding This work was granted access to the HPC resources of IDRIS under the allocation 101002 made by GENCI. It was funded in part by the French government under management of Agence Nationale de la Recherche as part of the “Investissements d’avenir” program, reference ANR19-P3IA-0001 (PRAIRIE 3IA Institute) and by Louis Vuitton ENS Chair on Artificial Intelligence.

Review summaries:
This paper presents a transformer-based model for VLN, which makes use of history trajectory information. Panoramic visual observations of all passed viewpoints are embedded by two transformer modules, one handling panorama at each step and the other one handling panorama representations of all steps. The well-known pre-training technology is employed. Experiments on R4R, R2R, and R2R variants are conducted.

This paper proposed history-aware multi-modal transformers for vision and language navigation tasks. Different from prior work (RecBert, PREVALENT), the proposed method takes long-horizon history with vision transformers and spatial relationship prediction proxy tasks. The proposed method achieves the state of the art performance on VLN R2R tasks. 

This paper proposes a history-aware multimodal transformer (HAMT) style model for Vision-and-Language Navigation task which explicitly takes the history of observations as the input for the transformer model. HAMT efficiently encodes all past panoramic observations using a hierarchical vision transformer. It consists of unimodal transformers for text, history (of panoramic visual observations), and current visual observation and then fuses these representations together using a cross-modal transformer. They also propose additional auxiliary proxy tasks for end-to-end training of HAMT. These are Single-step Action Prediction & Regression (SAP/SAR) and Spatial Relationship Prediction (SPREL). Through extensive ablations, they show that how each of their contributions makes sense and gets state-of-the-art on the single-run R2R task, R4R, and R2R-Back & R2R-Last (two augmented tasks they described in the paper).