History Aware Multimodal Transformer for Vision-and-Language Navigation
Authors: Shizhe Chen
Pierre-Louis Guhur
Cordelia Schmid
Ivan Laptev
Venue: NeurIPS 2021 Poster (2021)
Keywords: vision-and-language navigation
transformer

Abstract: Vision-and-language navigation (VLN) aims to build autonomous visual agents that follow instructions and navigate in real scenes. To remember previously visited locations and actions taken, most approaches to VLN implement memory using recurrent states. Instead, we introduce a History Aware Multimodal Transformer (HAMT) to incorporate a long-horizon history into multimodal decision making. HAMT efficiently encodes all the past panoramic observations via a hierarchical vision transformer (ViT), which first encodes individual images with ViT, then models spatial relation between images in a panoramic observation and finally takes into account temporal relation between panoramas in the history. It, then, jointly combines text, history and current observation to predict the next action. We first train HAMT end-to-end using several proxy tasks including single step action prediction and spatial relation prediction, and then use reinforcement learning to further improve the navigation policy. HAMT achieves new state of the art on a broad range of VLN tasks, including VLN with fine-grained instructions (R2R, RxR), high-level instructions (R2R-Last, REVERIE), dialogs (CVDN) as well as long-horizon VLN (R4R, R2R-Back). We demonstrate HAMT to be particularly effective for navigation tasks with longer trajectories. 

History Aware Multimodal Transformer for Vision-and-Language Navigation
Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, Ivan Laptev
Inria, École normale supérieure, CNRS, PSL Research University
(shizhe chen, pierre-louis guhur, cordelia.schmig. ivan www.laaaattriia....
https://cshizhe github. gittuu..io/preee lo/projects/vln. hamt. html
Abstract
lision-and-language navigation (VLN) aims to build autonomous visual agents that follow instructions and navigate in real scenes. To remember previously visited locations and actions taken, most approaches VLN implement memory using recurrent states. Instead, we introduce a History Aware Multimodal Transformer (HAMT) to incorporate long-horizon history into multimodal decision making. HAMT efficiently encodes all the past panoramic observations via a hierarchical vision transformer (ViT), which first encodes individual images with ViT, then models spatial relation between images in a panoramic observation and finally takes into account temporal relation between panoramas in the history. It, then, jointly combines text, history and current observation to predict the next action. We first train HAMT end-to-end using several proxy tasks including single step action prediction and spatial relation prediction, and then use reinforcement learning to further improve the navigation policy. HAMT achieves new state of the on a broad range of VLN tasks, including VLN with fine-grained instructions (R2R, RxR), high-level instructions (R2R-Last, REVERIE), dialogs (CVDN) well long-horizon VLN (R4R, #2R-Back) We demonstrate HAMT to be particularly effective for navigation tasks with longer trajectories.
Introduction
Vision-and- ann-uuundageneeeeeaa navigation (VLN) has recently received growing attention 2,,2,,,4, 5]. VLN requires agent to understand natural language instructions, perceive the visual world, and perform navigation actions to arrive at a target location. number datasets have been proposed to support various VLN tasks such indoor and outdoor navigation with fine-grained instructions [2, 6, 7}, language-driven remote object finding [8] and navigation in dialogs [9].
VLN agents are faced with several challenges. First, opposed static vision-text grounding [10], agent continuously receives new visual observations and should align them with instructions. Most of existing works adopt recurrent neural networks (RNNs) [6. 11, 12, 13, 14, 15, 16] to encode historical observations and actions within fixed-size state vector to predict the next action. Such condensed states might be sub-optimal for capturing essential information in extended trajectories [17]. For instance, "bring the spoon to me" requires the agent to remember start location after navigating to the "spoon", ", while early memories are prone to fade in the recurrent state. Few endeavors [18, 19] construct external map-like memories for received observations. Nevertheless, these approaches still rely on RNNs to track the navigation state. As the history plays important role in environment understanding and instruction grounding, we propose to explicitly encode the history as a sequence of previous actions and observations instead of using recurrent states.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Action Prediction
Next Action vwww!
X' Hi
Wy (N' M4 m(-1)
(o) Og. Sampp
Multimodal Encading
Cross-modal Transformer Encoder
X'
[M]00] 011
X = (1,,))
Og = (0), Pig Petop)
Unimodal Encoding
FFN
FFN
Multi-ayer Transformer
Embedding
Embesiiing
Fyl
Navigatle Embuuding
Self-attn
Solt attn
Eimbeddiigg
Embbedding
Q Kt
X/%
Emoedding
Posit
Arp
E
Embedding
Action
Action Embedding
Egg
stop
Cross attn
Crass attn
tmboedding
Word
Embodoing
Emberiding
View
stop
at* v
* v
Look up Table
Hiararchical Vision Transformer
Visinn Transformer
[
Inputs
(cha) go strraghhht bahind the bus onuch furn left when first of brick aod turn left and wse (tve)
stop
Cross-modal Transformer Layer
diech 2-3
Vilnar
Text
History
Observation
Figure 1: The architecture of History Aware Multimodal Tranformer (HAMT). HAMT jointly encodes textual instruction, full history of previous observations and actions, and current observation to predict next action.
Another VLN challenge concerns the generalizations of agents to new environments that have not been observed during training [4]. One direction is learn more generic text-image representations. The PRESS model [20] improves language representation with a pretrained BERT encoder [21], and PREVALENT [22] uses pairs of instruction and single-step observations pretrain multimodal transformer. Though achieved promising results, these works do not optimize visual representation for the target navigation task. Moreover, lack history in training [22] makes it hard to learn cross-modal alignment and increases the risk of overfitting to training environments. Another direction towards better generalization to overcome exposure bias [23] due discrepancy between training and inference. Different methods have been adopted for VLN including DAgger [6, 24] and scheduled sampling [20, 25]. Reinforcement Learning (RL) [12, 26] is one of the most effective approach among them, but it is considered unstable to directly train large-scale transformers via RL [27].
To address the above challenges, propose the History Aware Multimodal Transformer (HAMT), fully ransformer-basel architecture for multimodal decision making in VLN tasks. As illustrated in Figure 1, HAMT consists of unimodal transformers for text, history and observation encoding, and cross-mes moddll transformer to capture long-range dependencies of the history sequence, current observation and instruction. Since our history contains sequence of all previous observations, its encoding is computationally expensive. To resolve complexity issues, we propose a hierarchical vision transformer as shown Figure which progressively learns representations for a single view, spatial relationships among views within a panorama and, finally, the temporal dynamics across panoramas of the history. In order to learn better visual representations, we propose auxiliary proxy tasks for end-to- to-end training. Such tasks include single-step action prediction based on imitation learning, self-supervised spatial relationship reasoning, masked language and image predictions and instruction- trajectory matching. We empirically show that our training facilitates the subsequent fine-tuning of our model with RI. [28]. We carry out extensive experiments on various VIN tasks, including VI.N with -ra-ggd instructions (R2R [6] and RxR [7]), high-level instructions (REVERIE [8] and our proposed R2R-Last), dialogs [9] as well as
VLN (R4R [3] and our proposed R2R-Back which requires the agent to return back after arriving at the target location). HAMT outperforms state of the art on both seen and unseen environments in all the tasks.
We summarize our contributions as follows: (1) We introduce HAMT to efficiently model long- horizon history of observed panoramas and actions via hierarchical vision transformer, (2) We train HAMT with auxiliary proxy tasks in an end-to-end fashion and use RI. to improve the navigation policy; (3) We validate our method and outperform state of the art in a diverse range of VLN tasks, while demonstrating larger gains for long-horizon navigation.
2 Related work
-adinnaand-aanguage navigation. Training instruction-follovii navigation agents has attracted increasing research attention [1, 2,2,66 6, 7, 29]. Anderson et al. [6] propose sequence-to-seqqenee

---

history
embedding
Temporal Transformer
Transformer
panorama
embedding
Panoramic Transformer
Panoramic Transformer
Panoramic Transformer
image embeding
(b) Flattened history encod- ing. It encodes spatial and tem- poral relations the same time.
Image
Transformer
2-2
(a) Hierarchical history encoding. It first encodes individual view images with ViT, then models the spatial relation between images in each panorama, and finally captures the temporal relation between panoramas in the history.
(c) Temporal-only history
coding. only considers tem- poral relation oriented views.
Figure 2: comparison of history encoding methods. Circle nodes in different colors denote view images of panorama at different steps. Darker circle nodes are the oriented view of the agent.
LSTM baseline for the VLN task. Fried al. [11] extend it with panoramic action space and synthesized instructions. To improve cross-modee alignment, the self-monitoring agent [13] proposes co-grounding and progress estimation, and ReIGraph [ [15] uses graphs to model relationships across scene, objects and directions. Reinforcement learning (RL) is typically used to improve navigation policy. The EnvDrop model [12] mixes imitation learning and A3C [28]. The RCM [14] utilizes intrinsic reward of cross-modal matching in REINFORCE algorithm. Wang et al. [30] propose to
learn rewards via soft expert distillation. Due the success of transformer [31], recent works explore transformer architectures in VLN. PRESS [20] replaces LSTM instruction encoder with pretrained transformer for single-step multimodal fusion and LSTM for sequential action prediction. PTA [32] is transformer VLN model using CNNs to extract visual features [33]. Here we propose the first full transformer architecture for VLN and train end-to-end.
BERT [21]. SIA [16]
Memory-based policy for navigation. LSTMs [34] have been the dominant approach to encode memories for navigation [6, 11, 12, 14]. Condensing all history into one feature vector, however, is prone to the loss of information. Alternative approaches include topological map memory struc- tures [35, 36]. Deng et al. [18] use graphs capture environment layout and enable long-term planing. A similar graph is adopted in [19] with frontier-exploration based decision making. But these works still utilize LSTMs for state tracking. To exploit long-term spatio-temporal dependencies, Fang el al. [17] store histories in a sequence encoded with transformer. Recurrent VLN-BERT [5] injects recurrent unit encode histories in transformer for VLN. The most similar work to is Episodic Transformer (E.T.) [37]. Differently from [37], we propose hicrarchical encoding of the panoramic observation history and optimize the whole model in end-to-end training.
Multimodal pretraining with transformers. Recent works show significant progress in vision and language tasks using multimodal pretraining. In particular, transformer architectures such as one-stream [38, 39] and dual-stream [40, 41] achieve state the art for a number of downstream tasks including visual question answering, image-text retrieval and image captioning. While most previous methods rely on CNN to extract image representations, ViLT [42] adopts Vision Transformer (ViT) [43] and trains it with associated texts in end-to-end manner thanks to the efficiency of ViT. few endeavors [22, 44] explore multimodal pretraining for VI... PREVALENT [22] pretrains transformer using instructions and single-step observations without referring trajectory history. VLN-BERT [44] measures the compatibility between an instruction and images in path but does support action prediction. Our work presents the first end-to-end trainable VLN transformer that jointly encodes text, history and observation, and is able to sequentially predict actions.
Method
Problem definition The VIN problem [6] is formulated as a partially observable Markov decision process, where future observations are independent of the past conditioning current state s;. Given

---

an instruction W containing a sequence of L words (w1. ,..,,,,,,,, an agent should follow the instruction to move connectivity graph to reach the goal location. At each step t, the agent receives an observation O, a panorama of its surrounding environment. The O2 consists of K single view images split from the panorama O _ [[uf:a7),... the i-th view and af denotes the relative angle to face the view (subscript t is omitted for simplicity).
ag],-.. [ek: ax)) D). where og is the visual feature of
There are n navigable viewpoints among all the K views', denoted as Of ê (lVf: af). ... [0% a.]). We follow the setup in [11] and use Of as the decision space, the agent only needs to select candidate in Of at each step. All observations O2 and performed actions af before step t form the history He _ ([O1; at), [O,- i; (e)_))), where at denotes the turned angles at step i. The goal is to learn policy π parametrized by θ to predict the next action based on the instruction, history and the current observation, which is a(a [W, He. O,, Of; θ)
Unlike dominant recurrent approaches to condense H. into a fixed-size vector, in this section, we present the History Aware Multimodal Transformer (HAMT) that jointly encodes text, long-horizon history, and observation for sequential action prediction. The model architecture is described in Section 3.1. We propose end-to-end training for HAMT in Section 3.2 to learn unimodal and multimodal representations, and then use RL to fine-tune the navigation policy in Section 3.3.
3.1 HAMT: History Aware Multimodal Transformer
Figure illustrates the model architecture of HAMT. The inputs text W. history Hi and observation O are first encoded via the corresponding unimodal transformers respectively, and then fed into the cross-modal transformer encoder to capture multimodal relationships.
Text Encoding. For each token i in the instruction W, we embed it the summation of its word embedding W/s position embedding E," and type embedding of text ET Then we employ a transformer with N2 layers to obtain contextual representation v; following the standard BERT [21].
Observation Encoding. For each view [e%: af] in the panoramic observation O, we first represent the relative angle of as Edg = (sin 02, cos 0,, sin Φi, cos Φ.) where 0; and di are the relative heading and elevation angle to the agent's orientation. Then the observation embedding og is as follows:
= L.(WW"() + LN(W F. + E FT
where W Wa are learnable weights The EV denotes the navigable embedding to differentiate types of views, with EN for nav-navigable view, EN for navigable view and E for stop view (we append a stop token in observation to support stop action). The ET is the type embedding of observation. We omit bias terms for simplicity. The LN denotes layer normalization [45]. Because af has much lower feature dimensions than uf, we apply LN to balance the encoded af and v?.
Hierarchical History Encoding. As Hr consists of all the past panoramic observations O and performed actions at before step I, it is important to encode H. efficiently as context. Figures 2b-2c depict the flattened and temporal-only history encoding approaches used in VLN-BERT [44] and E.T. [37] respectively. The flattened approach treats each view image in O₂ as a token, so the history sequence contains tk tokens. Though it enables to learn relationships among all image views, the computation cost quadratically increases with the sequence length, making it inefficient for long-horizon tasks. In the temporal-oniy approach, only the oriented view of the agent in each O is taken as inputs instead of the whole panorama, so only t temporal tokens are encoded. However, this approach can lose critical information in past observations. For example, in the instruction "with the windows on your left, walk through the large room past the sitting areas", ", the object "window" " does not appear in the oriented view of the agent. Therefore, the encoded history is insufficient to tell whether the agent passed the window or not, making the model confused to take the next action.
In order to balance computational efficiency and information integrity, propose hierarchical history encoding approach as illustrated in Figure 2a. It hierarchically encodes view images within each panorama and then temporal relationships across panoramas, similar the factorized spatial- temporal video transformer [46]. For each O, its constituent view images are first embeded via ViT and Eq (I), and then encoded via a panoramic transformer with N,, layers to learn spatial relationships within the panorama. We apply average pooling to obtain panorama embedding, and add it with the
A navigable view can lead to one or multiple viewpoints. We follow [5, 12] to use different features for these viewpoints. The viewpoints share the same visual features but differ in angle features.

---

Table 1: Comparison of HAMT and previous VLN transformers.
Inputs
Proxy Tasks ITM
Models
Text
History
Observation
MLM
MRM
SAPISAR
SPREL
PREVALENT [22]
VIN-BERT [44] HAMT (Ours)
oriented view image feature in residual connection. The parameters in ViT and panoramic transformer are shared for different steps. In this way, each historical observation O; is represented as v and the final temporal token hy is computed as:
h4 = LN((W IN FE + F. F
(2)
where E denotes the i-th step embedding, E is the type embedding of history. The computational
cost O(LK2 + +12), which significantly reduces from O K2) in the flattened approach. To be noted, we add special token [cls] to the start of the history sequence to obtain a global representation. The embedding of [cls] is parameter to learn, which is initialized from zero vector.
Encoding. We concatenate history and observation as the vision modality, and use cross-modal transformer with N₂ layers fuse features from text, history and observation as shown in the right of Figure 1. The reason of using such dual-stream architecture rather than one- stream that the length of different modalities can highly imbalanced, and the dual-stream architecture can balance the importance of intra- and inter-modal relationships by model design [47]. In each cross-modal layer, a vision-text cross-attention firstly performed for vision modality to attend relevant text information and vice versa for text modality. Then each modality uses self- attention to learn intra-modal relationship such as interaction between observation and history, followed by fully-connected neural network. Finally, the HAMT model outputs embeddings x" (I'as: "'.... =²r,), HI (Kels hi,,. h1-1), (di, ,k dstop) for tokens in text, history and observation respectively.
3.2 End-to-end training with proxy tasks
As it is difficult to train large-scale transformers with RL due to sparse supervision [27], we propose to first end-to-end train HAMT via several proxy tasks to learn unimodal and multimodal representation.
Table compares our HAMT with previous VLN transformers PREVALENT [22] and VLN- BERT [44] in inputs and proxy tasks. As neither PREVALENT nor VLN-BERT jointly encodes text, history and observation, a limited choice of proxy tasks can be applied training. Our model instead take advantage of various proxy tasks to learn cross modal alignment, spatial and temporal reasoning, and history-aware action prediction. Given the input pair (W, Hr) where T is the length of full trajectory, we can apply common proxy tasks as in iision-and-language pretraining [40, 44], including Masked Language Modeling (MLM), Masked Region Modeling (MRM) and Instruction Trajectory Matching (ITM). Details of the three proxy tasks are presented the supplementary mate- rial. In the following, we introduce new proxy tasks given the triplet input (W, He, O₂) specifically for VLN tasks.
Single-step Action Prediction/Regression (SAP/SAR). The task deploys imitation learning to pre- dict the next action based on instruction, history from expert demonstration and the current observation. We formulate it as a classification and a regression task respectively. In the SAP classification task,
exp(fsaroo ( Ga predict action probability for each navigable view in Of which is (o() Σ, exp(Ssael (of (o) Gala))
where fSAP is two-layer fully-connected network, O is element-wise multiplication and x' is output embedding of special text token [cls]. The objective is to minimize negative log probability of the target view action d,: LSAP log P: (o))). In SAR regression task, we directly predict the action heading and elevation angles based on the text token [cls] which is 0,, &r = fSAR (2'(a) The loss function LSAR = (6. 2 + (6; - 2.) The two proxy tasks enable the model to learn how make action decision conditioning instruction and contextual history.
Spatial Relationship Prediction (SPREL.. Expressions of egocentric and allocentric spatial rela- tions are frequent in navigational instructions, such "walk into the room your left" and "enter
5

---

the bedroom next to the stairs". In order to learn spatial relation aware representations, we propose the SPREL self-supervised task to predict relative spatial position of two views in a panorama based on only visual feature, angle feature or both. Assume [uf: ag] and lef:a]] are two views in O₁ we randomly zero out e. or a with probability of 0.3. Their encoded representations are 0f and 0j. and their relative heading and elevation angles are 0ij. Øij. We then predict oij. Voj = fSPREL (lo): ofD) where [:] denotes vector concatenation and optimize LSPREL (0., 02,)² + (Ōis 0 The
task helps for spatial relationship reasoning in the observation.
Training Strategy. Instead of directly training the whole HAMT model at once, propose to progressively train HAMT in two stages. In the first stage, we freeze ViT pretrained on ImageNet [48] and train the rest of the modules which are randomly initialized. This aims avoid catastrophic forgetting of the pretrained weights in VIT. Then we unfreeze ViT and train the whole model end-to- end. The learning rate for ViT is set be higher than for others modules to avoid vanishing gradients and to speedup convergence. We empirically show that the proposed two-stage training outperforms one-stage training in the supplementary material.
3.3 Fine-tuming for sequential action prediction
Structure Variants. We present two variants of HAMT for action prediction in the following. MLP action head: we directly reuse the action prediction network fSAP in the SAP task predict navigable views. We use it as default for VLN tasks. 2) MLP action head based on encoder-decoder structure: the original HAMT model applies cross-modal attention for both vision-to-text and text-to- vision, which is computationally expensive when instructions are long. Therefore, we remove the cross-modal attention from text to vision. In this way, we separate the cross-modal transformer into an encoder which only takes instruction as input, and a decoder that imputs history and observation as query and attends over encoded text tokens. Please see supplementary material for details.
RL+II Objective. We combine Reinforcement Learning (RI.) and Imitation Learning (II.) to fine- tune HAMT for sequential action prediction. The II. relies the SAP loss defined in Section 3.2 and follows the expert action at each step while RL samples actions according to the policy т. Specifically, we use the Asynchronous Advantage Actor-Critic (A3C) RL algorithm [28]. At each step f, the agent samples an action based on policy π: at ~ <(, W, He. Or Oi) and receives an immediate reward rr- For non-stop actions, we set r2 as the reduced distance of taking the action to the target and the increased alignment score [3] compared to expert demonstration as defined in [5]; for the stop action, r, if the agent successfully arrives at the target otherwise -2, A critic network trained to estimate the value of each state 8t, which is Rg = Σx- "rrii""reee rhre+* where Y is discount factor. We implement it as Vn = feritie (x'ds O Hels). As the reward signal favors shortest distance, we empirically find it benefits to combine A3C RL with IL weighted by X, which is:
T
T T Σ Volog (a;;; Q)
θ θ
1 T Σ
Velog x (a;; 0)( - Vi) Ap ATT
(3)
Reinforvement Learning (RL)
Imitation Learning (IL.
where μ is the learning rate, at is the expert action at step t of the expert trajectory of length T*.
4 Experiments
4.1 Experimental setup
Datasets. We evaluate our method on four VLN tasks (seven datasets): VLN with fine-grained instructions (R2R [6], RxR [7]): VLN with high-level instructions (REVERIE [8], R2R-Last); vision-
and-dialogue navigation (CVDN [9]); and long-horizon VLN (R4R [3], R2R-Back).
R2R [1] builds upon Matterport3D [49] and includes 90 photo-realistic houses with 10,567 panoramas. contains 7,189 shortest-path trajectories, each associated with 3 instructions. The dataset is split into train, val seen, val unseen and test unseen sets with 61, 56, 11 and 18 houses respectively. Houses in val seen split are the same as training, while houses in val unseen and test splits are different from training.

---

RxR [7] is a large multilingual VLN dataset based on Matterport 3D. The instructions are in three different languages (English, Hindi and Telugu). The dataset emphasizes the role of language in VIN by addressing biases in paths and describing more visible entities than R2R. R4R [3] extends R2R dataset by concatenating two adjacent tail-to-head trajectories in R2R. Therefore, it has longer instructions and trajectories. The trajectories are also less biased as they are not necessarily the shortest-path from start to end location.
R2R-Back is a new VLN setup proposed in this work. The agent is required to return to its start location after arriving at the destination. The agent needs to remember its navigation histories to solve the task. We add a return command at the end of each instruction in R2R and a reverse path from the end to start locations as expert demonstration.
CVDN [9] defines navigation from dialog history task, which requires an agent arrive at goal regions based on multi-turn question-answeriin
dialogs. Such types of instructions are often ambiguous and under-speciffed. The lengths of instructions and paths also long.
REVERIE [8] replaces step-by-step instructions in R2R with high-level instructions, which mainly describe the target location and object. The agent, hence, required to navigate to the goal without detailed guidance and depends on its past experiences.
R2R- is our proposed VLN setup similar to REVERIE. It only uses the last sentence from the original R2R instructions describing the final destination.
Evaluation metries. We adopt standard metrics [1], including (1) Trajectory Length (TL)): the agent's navigated path in meters; (2) Navigation Error (NE): the average distance in meters between the agent's final position and the target: (3) Success Rate (SR): the ratio of trajectories reaching the destination with a maximum error of 3 meters to the target; and (4) Success Rate normalized by the ratio between the length of the shortest path and the predicted path (SPL). SPL is more relevant than SR as it balances the navigation accuracy and efficiency. For long-horizon VLN task (R4R and we further employ three metrics to measure the path fidelity between the predicted path and target path, including (5) Coverage weighted by Length Score (CLS) [3]; (6) the normalized Dynamic Time Warping (nDTW) [50]; and (7) the Success weighted by nDTW (SDTW).
Implementation details. For the HAMT model, we set Nı = 9 for language transformer, N,, for panoramic transformer in hicrarchical history encoding, and N₂ = 4 for cross-modal transformer. There are K- 36 view images in each panoramic observation. We use ViT-B/16 [43] for image encoding if not otherwise specified. In training with proxy tasks, we randomly select proxy tasks for each mini-batod with predefined ratio. We train HAMT for 200k iterations with fixed ViT using learning rate of 5e-5 and batch size of 64 on 4 NVIDIA Tesla P100 GPUs (~1 day). The whole HAMT model trained end-to-end for 20k iterations on 20 NVIDIA V100 GPUs with learning rate of 5e-5 for VIT and le-5 for the others (~20 hours). We use R2R training set and augmented pairs from [22] for training unless otherwise noted. In fine-tuning with RL+II. we set X = 0.2 in Eq (3) and 7 = 0.9. The model is fine-tuned for 100k iterations with learning rate of le-5 and batch size of on a single GPU. Unimodal encoders are fixed by default. The best model is selected according to performance on val unseen split. We use the same augmented data as [5] for R2R for fair comparison, while no augmented data is used for other datasets. Greedy search is applied in inference following the single-run setting. Please see supplementary material for more details.
4.2 Ablation studies
In this section, we evaluate each component in the HAMT model, including: hierarchical history encoding, end-to-end training with proxy tasks, and fine-tuning objectives.
How important is the history en- coding for VIN? For fair compari- son with the state-of-the-art recurrent architecture RecBERT [5], we use the same Resnet152 visual features and train all the models from scratch with RL+IL objectives to avoid the influ- ence of different weight initialization. The models are optimized for 300k iterations end-to-end except for the vi-
Table 2: R2R navigation results for alternative methods of history encoding. All methods use Resnet152 visual features and are trained from scratch on R2R dataset.
History Encoding
Val Seen
Val Unseen
SR
SPL
SR1
SPL
RecBERT [S]
62
59
50
46
Recurrent
60.941.0
56.641.1
52.2+0.7
47.0
Temporal-only
61.5
57.7
53.2
48.0+a.4
sual feature. Table 2 compares differ- ent history encoding approaches on R2R datasct. Our recurrent model slightly differs from RecBERT (no init. OSCAR) [5] in trans-
Hierarchical
65.5
61.3
54.4.0.4
48.7
7

---

Table 3:
Ablations for end-to-end HAMT training on R2R dataset using proposed proxy tasks.
(a) Comparison of visual features and end-to-end train- ing. The "PT" stands for proxy tasks in training: "c2e" for optimizing the visual representation.
(b) Comparison of different proxy tasks. The "SAP(R)" denotes the single step action prediction and regression task, and "SPREL" is the spatial relationship prediction task.
Val Seen
Val Unseen
feature PT e2e
SR1
SPLt
SR
SPL
SAP SP Val Seen (R)
Val Unseen SPL
REL
SR1
SPLt
SR
Resnet
x
x
65.5 61.3
54.4 48.7+0..4
152
V
X 69. 64.8
63.5
57.5
71.2+23
67.2+2.0 62.8 57.7
VIT V
x 75.7=1.0 1..
64.4
58.8
x
74.7
71.1 63.610.1 18.4 $8.8
/ 75.0=0.9 71.7
65.7.107 60.9..0.7
75.7 72.5 64.4
former architecture as shown in Figure 1. It achieves slightly better performance on val unseen split. The temporal-only model uses transformer encode agent's 's oriented visual observations in history sequence, and outperforms the recurrent method by relative gains of 1.9% on SR and 2. 1% on SPL for val unseen split. Adding panoramic observations in a hierarchical way results in 4.2% (SR) and 3.6% (SLP) relative improvements on the val unseen split compared to the recurrent method. Even larger improvements are achieved on val seen split as the hierarchical model has larger capacity to fit the seen environments. This evaluation demonstrates the ad advantage of our hierarchical history representation compared to the recurrent and temporal-only history representation.
How much does training with proxy tasks help? We next evaluate the advantage of training HAMT end-to-end with proxy tasks. In Table 3a, the first row uses RL+II objectives train HAMT from scratch, while the second row uses proxy tasks for training prior to RL+II fine-tuning. We can see that it significantly boosts the performance first train with proxy tasks. It improves on val unseen split with 16.7% and 18.0% relative gains on SR and SPL respectively, indicating that training with auxiliary proxy tasks enables better generalization. In the third row, we replace the visual feature from Resnet152 152 to VIT. The ViT feature improves the performance on both val seen and val unseen splits, showing that more powerful visual representations matter. Finally, training ViT end-to-end obtains 2.1 % gains on SPL on val umseen split. This is the first time to show that optimizing visual representations end-to-end is beneficial for VLN tasks. In Table 3b, we evaluate the benefit of the two new proxy tasks for frozen ViT features using the other proxy tasks by default. The SAP(R) uses imitation learning to predict actions, which directly influences the navigation policy and improves the performance by a large margin. The SPREL is self-supervised proxy task that forces the model to learn spatial relationships in panorama and helps generalization in unseen environments. More experiments to ablate contributions from history encoding and proxy tasks, contributions of proxy tasks in end-to-end training etc. are presented in supplementary material.
What is the impact of the fine-tuning objec- tives? Table presents results using different objectives in fine-tuning, The first row directly applies HAMT trained by proxy tasks, which achieves lower performance than that after IL fine- tuning, because we mainly use augmented data in proxy task training to increase visual diversity, but such noisy data deteriorates action prediction performance. Previous work [12] has shown that RL alone performs poorly. However, training with proxy tasks stabilizes the followup RL fine-tuning, HAMT optimized by RL achieves much better performance than that when fine-tuning with IL on the SR metric. It indicates that RL is able to learn better exploration strategy on unseen environments. However, the reward for RL focuses more on shortest paths rather than path fidelity with instructions, the improvement on SPL metric is relatively small compared to SR metric. Moreover, the fluctuation of the pure RL objective is larger than IL. Therefore, mixing the RL and IL achieves the best performance.
Table 4: Ablations for fine-tuning objectives of sequential action prediction on R2R dataset.
Val Seen
Val Unseen
IL RL
SR1
SPL.1
SR
SPL.1
x
x
57.9
54.8
51.8
48,9
V
63.7=2.1
61.7=2.2
57.2
54.7
70.5
65.6a2.
63.5a14
57.5
75.0
71.7
65.7...7
60.9
4.3 Comparison to state of the art
VLN with fine-grained instructions: R2R and RxR. Table compares HAMT with previous VLN methods on the R2R benchmark. Our model outperforms state-of-the-a art results of RecBERT [5]
8

---

Table 5: Comparison with state-of-the- art methods on R2R dataset.
Validation Seen
Validation Unseen
Test Unseen
Methods
TL.
NEJ
SRT
SPL
TL
NEL
SR1
SPLt
TL.
NE.
SR1
SPL1
Seq2Seq [6]
11.33
6.01
39
8.39
7.81
22
8.13
7.85
20
18
SF [11]
3.36
66
6.62
35
14.82
6.62
35
28
PRESS [20]
10.57
4.39
55
10.36
5.28
49
10.77
5.49
49
EnyDrop [12]
11.00
3.99
62
59
10.70
5.22
52
48
11.66
5.23
51
47
AuxRN [51]
3.33
70
67
5.28
55
50
5.15
55
51
PREVALENT [22]
10.32
3.67
69
65
10.19
4.71
58
53
10.51
5.30
54
51
ReIGraph [15]
10.13
3.47
65
9.99
4.73
57
10.29
4.75
55
RecBERT [5]
11.13
2.90
68
12.01
3.93
63
57
12.35
4.09
63
57
HAMT (Ours)
11.15
2.51
72
11.46
2.29
66
12.27
3.93
65
60
by relative 5.9% and 7.0% improvements in SPL on val seen and unseen splits respectively, We achieve state-of-the-art performance under the single-run setting on the unseen testing split of the leaderboard It demonstrates the effectiveness and generalization of our model. We further provide computation time in inference for HAMT and RecBERT in the supplementary material to show the efficiency of our HAMT model. We also achieve large improvements on RxR dataset. The full results are presented supplementary material.
Long-horizon VLN: R4R and R2R-Back. Table 6 shows navigation results on R4R dataset. As R4R contains longer instructions and trajectories compared to R2R,
the encoder-decoder variant of HAMT for better efficiency. Our method outperforms previous approaches in all metrics and shows particularly large improvements for the path fidelity related metrics. Compared to RecBERT, HAMT achives 8.2% and 9.5% relative improvement in CLS and nDTW respectively. The large improvements these path fidelity related metries indicate that HAMT is better to follow the designated path of the fine mee-griind instruction. Figure 3 evaluates the performance of HAMT and RecBERT with respect instruction length measured by words. Though the nDTW decreases for longer instructions, the relative improvement of HAMT increases with the instruction length.
Table 6: Comparison on R4R val unseen split.
0.60
RecBERT
Methods
NEL
SR
CLSt
nDTW>
SDTW1
056
SF [11]
8.47
65e
RCM [14]
29
35
30
13
PTA [32]
8.25
24
37
32
10
0/5
EGP [18]
8.0
30.2
44.4
37.4
17.5
ReIGraph [15]
7.43
36
47
34
23
30
RecBERTi [5]
6.67
43.6
51.4
45.1
29.9
instruction lengths
Figure 3: nDTW with respect instruc- tion length on R4R val unseen split.
HAMT (Ours)
6.09
44.6
57.7
50.3
31.8
The navigation performance on R2R-Back dataset is presented in Table 7, We compare with two state-of-the-arr recurrent models EnvDrop [12] and RecBERT [5] based on LSTM and transformer respectively (both models trained on R2R-Back for fair comparison). The improvements more significant on this task as it requires the agent to remember the way it came to the target to successfully return back. The recurrent state insufficient capture such history and leads to inferior performance compared to the HAMT model.
Table Comparison of methods on the R2R-Back dataset.
Val Seen
Val Unseen
Methods
TL
SR1
SPI.1
nDTW>
SDTW
TL.
SR1
SPI.
nDTW
SDTW
Enypr [12]
23.83
44.1
42.0
61.3
39.4
24.57
32.4
30.2
51.1
28.0
RecBERT [5]
22.33
51.4
48.4
67.3
45.7
23.35
41.1
37.7
58.2
35.6
HAMT (Ours)
22.76
64.8
61.8
73.7
58.9
23.78
57.2
53.1
65.1
49.5
We report the published results on the testing unseen split as shown in challenges/ www.llenge/challenneee httpa//ss//aalllggggggg////////e wttp://///eeeeebrrrre//////////e hat.l/gg///lllllleeee/////////// (25/10/2021).

---

Table 8: Navigation performance on CVDN dataset.
Vision-and-Dialon Navigation: CVDN.
The CVDN dataset contains dialogs as in- structions and use Goal Progress (GP) in meters as the primary evaluation metric. GP measures the difference between com- pleted distance and left distance to the goal, the higher the better. There are two types of demonstrations in the dataset. One is shortest-paie trajectory and the other is player's 's navigation trajectory. We mix the two types of demonstrations as supervision in training which has shown to be the most effective in previous works [22, 52, 53]. As navigation paths in CVDN dataset are much longer than R2R dataset, we adopt the encoder-decoder variant of HAMT. As shown in Table 8, HAMT outperforms existing recurrent approaches aatirraaaa
Val Seen
Val Unseen
Test Unseen
PREVALENT [22]
3.15
2.44
VISITRON [52]
5.11
3.25
3.11
MT-RCM-EnVAg [53]
5.07
4.65
3.91
HAMT (Ours)
6.91
5.13
5.58
both seen and unseen environments, and achieves the top position the leaderboard It demonstrates that our HAMT model is generalizable different types of instructions in new VLN tasks.
VLN with high-level instructions: R2R-Last and REVERIE. Table 9 shows results on the R2R-Last dataset that specifies the goal loca- tion and contains no step-by-step instructions. The HAMT model with the hierarchical his- tory encoding is able to better accumulate the knowledge of the environment and achieves 9.8% and 10.5% relative gains on SPL metric on seen and unseen splits respectively compared to RecBERT [5]. The REVERIE dataset also contains high-level instructions but requires object grounding at the target location besides navigation. We provide results on REVERIE dataset in supplementary material. Our HAMT achieves SPI. 30.20 and 26.67 on val unseen and test splits respectively, outperforming the state of the art navigation performance [5] by 5.3% and 2.7%.
Table 9: Comparison on the R2R-Last dataset.
Val Seen SR1
Val Unseen SR
Methods
SPL1
SPL1
EnvDrop [12]
42.8
38.4
34.3
28.3
RecBERT" [5]
50.2
45,8
41.6
37.3
HAMT (Ours)
53.3
50.3
45.2
41.2
Conclusion
This paper presents the first end-to-end transformer for Dision-and-languaae navigation, denoted as History Aware Multimodal Transformer (HAMT). Our method efficiently encodes long-horizon history and combines it with instructions and observations to derive multimodal action prediction. The HAMT is first trained with proxy tasks in an end-to-end manner, and is then fine-tuned with RL to improve the navigation policy. We achieve state-of-th th--te-rrtt navigation performance on a diverse range of challenging VLN tasks, demonstrating improved accuracy and generalization of our approach compared to the dominant recurrent methods. Future work could extend our history-aware transformer VIN with continuous actions [54] and could benefit from pretraining larger navigation datasets. This paper has minimal ethical, privacy and safety concerns.
Acknowledgment and Disclosure of Funding
This work was granted access to the HPC resources of IDRIS under the allocation 101002 made by GENCI. It was funded in part by the French government under management of Agence Nationale de la Recherche as part of the "Investissements d' d'avenir" program, reference ANR 1NR19-P31 (PRAIRIE 31A Institute) and by Louis Vuitton ENS Chair on Artificial Intelligence.
References
[1] Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi. Manolis Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv:1807.06755 2018. 1,2,
[2] Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In Proceedings of the
Phttpa://eval wwt./eeb/caalleeggs/chhaaaa hhhhllaaallleeaaeeee 2eoge/443/100 /eadeedrbrbbb/1122 (25/10/2021)
We run the original implementation of methods released by the authors.
10

---

IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12338-1257,,.1 2019.
1,2
[3] Vihan Jain, Gabriel Magalhaes, Alexander Ku, Ashish Vaswani, Eugene Ie, and Jason Baldridge. Stay the path: Instruction fidelity in vision-and- aisoo-aad-langguae navigation. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 1862-1872, 2019.
1, 1,2,6 6,
[4] Yubo Zhang, Hao Tan, and Mohit Bansal. Diagnosing the environment bias in vision-and- language navigation. International Joint Conferences on Artificial Intelligence, 2020. 1,2
[5] Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould. VIn bert: A recurrent aaiio--and-angguae bert for navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1643-1653, 2021. 1,3, 4, 7, 8, 10
[6] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan Reid, Stephen Gould, and Anton Van Den Hengel. Assion-and-lnnuree navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3674-3683, 2018. 1, 6,
[7] Alexander Ku, Peter Anderson, Roma Patel, Eugene le, and Jason Baldridge. Room-across- room: Multilingual vision-and-languaae navigation with dense spatiotemporal grounding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1392-4412 2020. 1, 2, 6,
[8] Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, and Anton den Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9982-9991, 2020. 1,2,66 6,
[9] Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-andddialog navigation. In Conference on Robot Learning, pages 394 406. PMLR, 2020. 1, 2, 6,
[10] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang. Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1307-1315, 2018.
[11] Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Barg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker- follower models for vision-and- lan-uage navigation. Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 3318 3-332,, 2018. 1,3, 1,9
[12] Hao Tan, Licheng Yu, and Mohit Bansal. Learning to navigate unseen environments: Back translation with environmental dropout. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 2610-2621, 2019. 1, 4, 8, 9, 10
[13] Chih- Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, and Caim- ing Xiong. self-mooitoring navigation agent via auxiliary progress estimation. Proceedings of the International Conference on Learning Representations, 2019. 1,3
[14] Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6629-6638, 2019. 1, 9
[15] Yicong Hong, Cristian Rodriguez, Yuankai Qi, Qi Wu, and Stephen Gould, Language and visual entity relationship graph for agent navigation. Advances in Neural Information Processing Systems, 33:7685-7696, 2020. 1,
[16] Xiangru Lin, Guanbin Li, and Yizhou Yu. Scene-intuitive agent for remote embodied visual grounding. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 7036-7045, 2021. 1,3

---

[17] Kuan Fang, Alexander Toshev, Li Fei-Fei, and Silvio Savarese. Scene memory transformer for embodied agents long-horizon tasks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2019. 1,3
[18] Zhiwei Deng, Karthik Narasimhan, and Olga Russakovsky. Evolving graphical planner: Con- textual global planning for vision-an vision-and-languee and-language navigation. In Advances in Neural Information Processing Systems, volume 33, 2020. 1,3,9
[19] Hanqing Wang, Wenguan Wang, Wei Liang, Caiming Xiong, and Jianbing Shen. Structured scene memory for vision-language navigation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8455-8464 8464, 2021. 1,3
[20] Xiujun Li, Chunyuan Li, Qiaolin Xia, Yonatan Bisk, Asli Celikyilmaz, Jianfeng Gao, Noah A Smith, and Yejin Choi. Robust navigation with language pretraining and stochastic sampling. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNL.--JCNNPP) pages 1494-1499 2019. 3,
[21] Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171 14866, 2019. 2,
[22] Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. Towards learning generic agent for usioo-andd-lanuue navigation via pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13137-13146, 2020. 5, 7,9, 10
[23] Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level training with recurrent neural networks. International Conference on Learning Representations, 2016.
[24] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[25] Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, volume 28, 2015. 2
[26] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018. 2
[27] Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayaku- mar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International Conference on Machine Learning, pages 7487-7498. PMLR, 2020. 2,
[28] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference machine learning, pages 1928-1937, PMLR, 2016. 2, 6
[29] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10740-10749, 2020. 2
[30] Hu Wang, Qi Wu, and Chunhua Shen. Soft expert reward learning for vision-and-language navigation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pages 126-141. Springer, 2020. 3
12

---

[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 9998,600, 2017.
[32] Federico Landi, Lorenzo Baraldi, Marcella Cornia, Massimiliano Corsini, and Rita Cucchiara. Perceive, transform, and act: Multi-modal attention networks for vision-and-language naviga- tion. arXiv preprini arXiv: 191 /2377, 2019. 3,
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 770-778, 2016.
[34] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation,
173: 1997. 3
[35] Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cogni- tive mapping and planning for visual navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2616-2625, 2017. 3
[36] Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. International Conference on Learning Representations, 2018.
[37] Alexander Pashevich, Cordelia Schmid, and Chen Sun. Episodic transformer for vision-and- language navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15942-15952, 2021. 3,
[38] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision, pages 104-120. Springer, 2020. 3
[39] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for lision-language tasks. In European Conference Computer Vision, pages 121-137. Springer, 2020.
[40] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining task-agnostic visiolinguistic representations for sssion-and-lagguaae tasks. In Advances in Neural Information Processing Systems, volume 32, 2019. 5,5
[41] Hao Tan and Mohit Bansal. LXMERT: Learning coos-modaaiity encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language P Processing (EMNLP-IJCNLP), pages 5103-51 14,,,114777 2019.
[42] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: ssiionandd-anguuee transformer without convolution or region supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 3583-55944 2021. 3
[43] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2020. 3.7
[44] Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruy Batra. Improving aidoonaadd-aanguage navigation with image-text pairs from the web. In European Conference Computer Vision, pages 259-274. Springer, 2020. 3, 4.5
[45] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint VrXiv:1607.6644 2016. 4
[46] Anurag Amab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: video vision transformer. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 6836-6846, 2021.
13

---

[47] Jize Cao, Zhe Gan, Yu Cheng, Licheng Yu, Yen-Chun Chen, and Jingjing Liu. Behind the scene: Revealing the secrets of pre-trained sasion-and-languaae models. In European Conference on Computer Vision, pages 565-580. Springer, 2020.
[48] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-24182 255. IEEE, 2009.
[49] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In 2017 International Conference 3D Vision (3DV), pages 667-676. IEEE, 2017. 6
[50] Gabriel Ilharco, Vihan Jain, Alexander Ku, Eugene Ie, and Jason Baldridge. General evaluation for instruction conditioned navigation using dynamic time warping. In Visually Grounded Interaction and Language (ViGIL), NeurIPS 2019 Workshop, 2019.
[51] Fengda Zhu, Yi Zhu, Xiaojun Chang, and Xiaodan Liang. Vision-lange languaee navigation with self-supervised auxiliary reasoning tasks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10012-10022, 2020. 9
[52] Ayush Shrivastava, Karthik Gopalakrishnan, Yang Liu, Robinson Piramuthu, Gokhan Tür, Devi Parikh, and Dilek Hakkani-Tür, Visitron: Visual semantics-aligned interactively trained object-navigator. arXiv preprint 2 105.///89, 2021. 10
[53] Xin Eric Wang, Vihan Jain, Eugene Ie, William Yang Wang. Zornitsa Kozareva, and Sujith Ravi. Environment-agnostie multitask learning for natural language grounded navigation. In Computer Vision-ECCT 2020: 16th European Conference, Glasgow; UK. August 23-28, 2020, Proceedings, Part XXIV 16, pages 413 430. Springer, 2020. 10
[54] Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav- graph: Vision-and-language navigation in continuous environments. In European Conference on Computer Vision, pages 104-120. Springer, 2020. 10
14


Reviews:
[{"review_id": "zMSXnWpEWww", "reviewer": "Reviewer_xpbr", "summary": "This paper proposed history-aware multi-modal transformers for vision and language navigation tasks. Different from prior work (RecBert, PREVALENT), the proposed method takes long-horizon history with vision transformers and spatial relationship prediction proxy tasks. The proposed method achieves the state of the art performance on VLN R2R tasks. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strength]\n\n- Very good performance on VLN R2R datasets. \n\n- The idea of encoding the whole history as input is intuitive, and the paper is well written and the authors perform extensive ablation study to verify the effectiveness of each module. \n\n- The author also try with different visual backbone and imitation learning and A3C for the tasks. \n\n[Weakness]\n\n- In the contribution section, the author claims their model is the first fully transformer-based model for VLN tasks. This is not super clear to me, it will be better to illustrate more precisely. \n\n- Since the model encodes the whole history image as input, I wonder what is the computation time during inference time. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the author addresses the limitation and potential negative social impact of the work. ", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "gytHFAGtmn4", "reviewer": "Reviewer_E9jz", "summary": "This paper proposes a history-aware multimodal transformer (HAMT) style model for Vision-and-Language Navigation task which explicitly takes the history of observations as the input for the transformer model. HAMT efficiently encodes all past panoramic observations using a hierarchical vision transformer. It consists of unimodal transformers for text, history (of panoramic visual observations), and current visual observation and then fuses these representations together using a cross-modal transformer. They also propose additional auxiliary proxy tasks for end-to-end training of HAMT. These are Single-step Action Prediction & Regression (SAP/SAR) and Spatial Relationship Prediction (SPREL). Through extensive ablations, they show that how each of their contributions makes sense and gets state-of-the-art on the single-run R2R task, R4R, and R2R-Back & R2R-Last (two augmented tasks they described in the paper).", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality and Significance:\n-\tThis paper proposes a novel modelling approach where they explicitly model the history of panoramic visual observations in addition to instruction and current visual observation. Their “history-aware” transformer style model first encodes individual images with Vision Transformer and then combine it with unimodal transformer output of instruction and current visual observation using a cross-modal transformer.\n-\tThey also propose 2 new auxiliary tasks for pretraining their model: Single-step Action Prediction/Regression and Spatial Relationship Prediction.\n-\tThey show results on 2 VLN tasks: R2R and R4R. \n    -\tUsing ablations, they show that their history encoding for VLN improves Success Rate by 4.2% and 2.1% SPL over temporal-only baseline.\n    -\tThey show that using all their proxy tasks and visual representation finetuning provides gains of 12.2% SPL (48.7 -> 60.9) and 11.3% SR (54.4 -> 65.7).\n    -\tThey achieve state-of-the-art on R2R and beats recurrent-VLN-BERT by 3% on SPL and 2% on SR.\n    -\tThey get state-of-the-art on R4R and two additional VLN setup tasks (introduced in the paper), R2R-Back where the agent needs to remember the trajectory and come back to starting position, R2R-Last where only the last instruction is provided.\n\n### Quality and Clarity:\nThe paper is well-written, is clear about their claims and explanations and is easy to follow.\n\n### Typos:\nL98: and-to-end -> end-to-end\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "FCxA0z7Jh8X", "reviewer": "Reviewer_7aBY", "summary": "This paper presents a transformer-based model for VLN, which makes use of history trajectory information. Panoramic visual observations of all passed viewpoints are embedded by two transformer modules, one handling panorama at each step and the other one handling panorama representations of all steps. The well-known pre-training technology is employed. Experiments on R4R, R2R, and R2R variants are conducted.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contribution of this work are twofold: (1) introducing navigated trajectory into transformers in the form of hierarchical fashion; (2) employing the pre-training technology. Experiments are conducted on two commonly used datasets R2R and R4R. The authors also created two additional datasets, R2R-Back and R2R-Last, to further demonstrate the effectiveness of the proposed history-aware model.\n\nThe writing is not very clear as some crucial details are missing. For example, (1) Do ViT and Panoramic Transformer are shared between time steps as shown in Figure 2a? (2) Is the panorama at t=0 fed into the model for steps after t=0? If so, is the repeated calculation necessary? (3) Regarding observation encoding, 36 view embeddings are added with navigable embeddings as shown in Figure 1. However, one view may have zero or multiple navigable viewpoints. How are such cases handled by the proposed model? (4) How do you guarantee and verify the information is not lost regarding the claim in line 156 \"In this way, xxx without information loss\".\n\nRegarding the performance, the main contribution hierarchical history module only brings 1%~2% SPL improvement on unseen split (which is the main reference) as shown in Table 2 (row Recurrent vs row Hierarchical), but the pre-triaing technology brings about 10% improvement. Performance is further improved if ViT is used. However, the pre-training technology is not the contribution of this work as they are taken from existing work. Thus, the claimed contribution is not matched with the experimental results. Furthermore, some ablation studies are missing, such as for the ViT setting in Table 3a, how about remove both PT and e2e? What are the contributions of SAP and SAR  respectively to the final performance?  In addition, it is weird not using the final model (ViT+PT+e2e) to do the ablation study in Table 3b.\n\nFor comparisons on R2R-Back and R2R-Last, it is not very fair for existing methods as they are not designed for such tasks. In addition, do you retrain them on R2R-Back and R2R-Last datasets?\n\nOverall, I think this work is not ready for publication based on the novelty, writing, technique contribution, and experiment issues listed above and below.\n\nOther issues:\n1. Why zero out v_*^o or a_*^o (line 198) for spatial relationship prediction? How does it affect the performance if not zero out them?\n2. Is there any explanation for the performance drop when optimizing visual representation in ViT setting of Table 3a?\n3. The explanation for marginal SPL improvement in line 328 is not reasonable, because for the R2R task the SPL metric is a good measurement for path fidelity as the ground-truth path is exactly the shortest path. \n4. Since pre-training technology is utilised, the proposed model should be tested on more downstream tasks, such as Touchdown, REVERIE, CVDN, etc.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Thanks for the response from the authors. Most of my issues have been addressed, such as more results on downstream tasks and clarifications about model details. Overall, I think this is a good work but not good enough for NeurIPS. This is because most of the performance comes from the application of existing pre-training instead of the proposed history-aware scheme (10% vs 2% improvement), and there is no novelty in neither pretraining model design nor loss functions. \n\nI have read initial comments from other reviewers, which tend to be positive. However, if this paper does not get lucky due to my review, I suggest rewriting the paper in line with the contribution importance of each proposed module, and paying extra attention to the newly designed xxx-back VLN task which usually can be a separate article.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "EVwmxFHSoX", "reviewer": "Reviewer_euGp", "summary": "This paper introduces a History Aware Multimodal Transformer to incorporate a long-horizon history into multimodal decision making. The authors perform experiments on three VLN datasets (R2R, R4R, and R2R-last), and demonstrate that HAMT is particularly effective for navigation tasks with longer trajectories.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper introduces a History Aware Multimodal Transformer to incorporate a long-horizon history into multimodal decision making. The authors perform experiments on three VLN datasets (R2R, R4R, and R2R-last), and demonstrate that HAMT is particularly effective for navigation tasks with longer trajectories.\n\nStrengths:\n- A new multimodal transformer based method for VLN is introduced particularly for solving the longer paths.\n- The proposed HAMT outperforms the state of the art and achieves excellent results on three VLN datasets. \n- Training HAMT with auxiliary proxy tasks is interesting.\n\nWeaknesses:\n- More clarification and ablation study is needed about the training strategy. Have the authors tried to train HAMT in a single stage end-to-end rather than the current two-stage strategy?\n- Experimental details were not made clear and many important hyper-parameters are missing, making this paper hard to reproduce. Are all the Panoramic Transformer layers randomly initialized, or are they initialized from certain checkpoints? How do you initialize the discount factor? Besides, the authors introduce a new VLN framework, thus it is important to report some model details and training details, for example, the numbers of model parameters, memory cost and inference time (the model seems to be rather complex), and so on.\n- Line 160, how do you get the special token? From my guess you are treating it as a totally free parameter during training. If so, how do you initialize it?\n- The presentation of the paper can be further improved. For example, the authors could give a brief formulation of Temporal Transformer, Panoramic Transformer, and Multimodal Transformer first, and then give a unified and specific description.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an effective history-aware multi-modal transformer for VLN tasks, which captures well the long-horizon history and spatial relationships.  This method efficiently encodes all past panoramic observations using a hierarchical vision transformer, with promising performances on a variety of datasets.  It achieves SOTA on R2R.  The main concerns raised by reviewers are the unclear writing and the relatively smaller gap from using a new history-aware scheme compared to pretraining models. The authors later provide extensive experiments to demonstrate the generalization capability and large improvements by their modules in more downstream tasks and more diverse datasets.  The experiments indeed validating well the contribution of the proposed transformers. Considering the positive comments by three reviewers and the responses solve the most issues by reviewer 7aBY, the AC thus recommends accepting this paper.
Decision: Accept (Poster)